Outdated Lockfile Please Run yarn Install and Try Again

Coding is fun, no doubt about information technology!

Other fun things: Testing! Code maintenance! Keeping dependencies in sync!

No? But for me? Well, it might not be fun for many of y'all, but to keep your library/application working properly, you lot'll need to adapt and at least effort some of it.

If yous take written a JavaScript library and published it on NPM (Node Parcel Managing director), at some point in its development lifecycle, yous have experienced inconsistent behavior, and you lot didn't know the root crusade. It was working fine for you and some of your coworkers/contributors but crashing for everybody else. So what is going on hither?

There is a high chance unhandled dependencies acquired it in your project. This article addresses that problem and focuses on a particular task: lockfile maintenance. I'll show you:

• What yarn.lock (lockfile) is
• Why exercise you need to do lockfile maintenance
• What is the possible solution
• What mistakes we've made and how y'all can avoid them

Disclaimer: Everything mentioned below is based on true events. No developers were harmed during implementation and/or for the purposes of this article.

Trouble

For the last few months, I've been working at Stoplight, equally a part of 11Sigma, on an open up-source library called Elements.

An issue emerged in our library integration that made us claiming our conventionalities that our dependencies are under control. In short, our app crashed considering of a issues in a dependency's dependency. Despite fixing the nested dependency, the issue remained to be unsolved.

Fortunately, it turned out to exist a non-issue on a fresh install of Elements.

Unfortunately, that meant that we weren't testing what users were using at a given moment.

To sympathise this issue comprehensively, we start need to learn almost the lockfile and how dependencies are installed in a projection.

What is a lockfile?

If you are familiar with how a lockfile works, feel gratuitous to skip to "Problem – 2nd await"

Information technology is, first and foremost, necessary to know what a lockfile is and how it works. Although it tin can have different names depending on whether you utilize npm or yarn, the premise is similar. I'm using yarn so I'll use yarn.lock as an example in this article.

When you run yarn in your project, two things can happen:

1. A lockfile (yarn.lock) is generated (if in that location isn't any) according to the contents of packet.json
2. Packages are installed according to contents of an existing yarn.lock

Important: When you install dependencies in your application or library, only the top-level yarn.lock file is respected. Lockfiles within your dependencies will be ignored.

In brusk:

When present in the projection, yarn.lock is the principal source of information about the current versions of dependencies in a project. Yarn uses that information to cheque if it needs to update annihilation – it compares dependency versions currently installed in a project (listed in yarn.lock) to version restrictions in package.json and updates packages if needed. Data from the lockfile tin can exist further used past other users to create a repeatable environment elsewhere.

Generating yarn.lock

Whenever you run yarn (which is the equivalent of running yarn install) upon a fresh install, a yarn.lock file is generated. It lists the versions of dependencies that are used at the time of the installation process. That means information technology looks into your package.json, and depending on the versioning syntax, it will install your projection dependencies, then their dependencies, and so their dependencies, and and so on…

For more than info well-nigh dependency versioning cheque this link

Allow's say your project uses two dependencies: chicken and farm. Both of these are external packages, over which we don't have any control:

          // bundle.json (your project)   dependencies: {    "chicken": "^1.2.0",    "farm": "2.iii.0"  }        

and farm package uses a pinned (specific) version of chicken:

          // package.json (`farm` bundle)   dependencies: {    "chicken": "1.0.0",    (...)   }        

This will result in your project requiring two versions of chicken:

• ane.0.0 for the farm dependency
• ^i.ii.0 as divers in your projection'southward package.json. This will vary upon a fresh install depending on whatever the latest version afterward 1.2.0 is – the ^ symbol allows changes that practise not modify the left-nearly non-zero chemical element in the version number. For this particular version range, it means 1.2.0 <= installed version < 2.0.0. To give an example – if v1.ii.4 is available at the moment of (fresh) installation of your project, it will be installed. Same for v1.5.8, v1.7.2, v1.9.nine, but non for v2.0.0.

Both of these versions will exist present in the yarn.lock

The binder structure will wait similar this:

At this point, you lot have probably noticed that this isn't a perfect situation – if a sub-dependency (craven) has a bug introduced in one of its versions, and dependency that uses it ('farm') doesn't pin the version – it could introduce a issues to your project.

There is a bright side, though – if your projection requires craven and uses that aforementioned/matching version range, eastward.thou.

                      dependencies: {    "chicken": "^1.2.0",    (...)   }        

y'all won't install ii versions of the chicken package. This decreases the size of your project and prevents potential conflicts betwixt different versions of the same packet.

Updating the lockfile

Updating the lockfile is a fleck easier to explicate. An update can happen in iii situations: When the dependency is added, removed, or modified.

This can happen in two ways: Automagically or manually. You can trigger an update via yarn CLI (which updates both packet.json and yarn.lock) using the following commands:

          # adding dependencies  yarn add Parcel-Name  # removing dependencies  yarn remove Package-NAME  # upgrading all dependencies  yarn upgrade  # upgrading a specific packet  yarn upgrade Parcel-Name  # Adding the `--latest` flag at the end of 'upgrade' commands makes yarn ignore the specified version range and install the latest version(s).        

If you want to go the manual update road, y'all modify the contents of bundle.json and then run yarn install. If yarn doesn't detect whatever differences betwixt versions in bundle.json and yarn.lock, information technology won't install anything new and/or update yarn.lock

Question: You lot installed project dependencies at some point. Time has passed, maybe a few of your project dependencies released a new version. Yous take likewise added some additional dependencies. What will happen if you run yarn?

Answer: Well, yarn will, of course, install dependencies freshly added to package.json that are non yet installed and/or not nowadays in yarn.lock. But if y'all have " dependecy_A": "v1.3.5" already in your yarn.lock, latest version available on NPM is " dependency_A": "v.1.4.0" and version range in packet.json is ^1.2.0 – will yarn upgrade to the latest version? No, information technology won't. v1.3.5 falls into the requirement of being ^ane.2.0, then yarn doesn't see the need of upgrading the package.

Problem – A 2nd Look

Permit's come up back to the original issue for a moment:

1. An integration of Elements in Storybook (a tool for building an testing UI components) was broken because of a issues present in the Elements dependency, JSV (JSON Schema Viewer)
2. JSV was using a dependency called JST (JSON Schema Tree), which is effectively a sub-dependency of Elements, with a non-pinned (not-specified) version (^1.1.0) listed in its parcel.json.
3. A new version of JST (ane.1.2), which falls into the requirement of being ^1.1.0, included a set that would solve our problem
4. Even though the JST version could be 1.1.0 or above, Elements itself would still hang for u.s.. Installing dependencies (by running yarn inside the Elements directory) didn't assist either. Why?

The reply at this bespeak is really pretty simple – even though both of the JSV and JST versions are not pinned and should update upon a fresh install, our local yarn.lock file was blocking these updates, having v1.1.0 of JST in itself.

Mayhap we should only deploy the lockfile alongside other files?

As explained in the paragraphs to a higher place – when present, yarn.lock serves equally the primary source of information about which versions of packages should be installed. If that'due south the case, can we just deploy it with the rest of the parcel when releasing a new version?

tl;dr(too long, didn't read) – no you lot don't (sometimes)

That depends on what your project is:

• Is your project an awarding? Then: Yes
• Is your project a library? If and then: No

Why should you care nearly lockfile maintenance for libraries?

In that location seems to be an agreement as to whether the lockfile should be committed. There'southward an splendid mail on yarnpkg covering this topic (both for applications and libraries) if yous want to understand the reasoning backside it.

We'll focus on libraries, such as Elements. Plus, committing the lockfile alongside the awarding pretty much solves the effect of unwanted updates.

Handling lockfile in libraries

Because just the top-level lockfile is respected (the 1 course users project root directory), yarn will look into the used library'south package.json and install the packages with versions described there. Unless you pivot each dependency in your library to an exact version, users' projects might end upwardly having different sub-dependencies depending on the time of installation.

So are nosotros doomed? Kind of. Users volition always be the first people to discover a breaking modify in a dependency (and hopefully file a bug written report). To give you lot some perspective:

• Let'southward presume your library has 20 external, sub-dependencies
• Each of these sub-dependencies can get a new release someday
• Thousands (potentially) of users install your library each twenty-four hours
• Each such installation volition fetch the latest sub-dependencies
• If any of those sub-dependencies introduce a bug, your users might exist affected
• The above will happen, unless your development squad has a way to regularly test that sub-dependency upgrades don't break your library

How to proceed dependencies up to engagement?

By now, we've established that yarn.lock left alone without any maintenance can innovate confusion most the electric current state of the projection/library, equally developers might end upwardly having dissimilar versions of dependencies installed locally on their machines.

Permit'due south take a await at possible solutions for keeping the lockfile upwards to engagement.

Dependabot

The first arroyo we looked at was Dependabot – a well-known tool for bumping dependencies. Information technology checks for possible updates, opens Pull Requests with them, and allow users to review and merge (if you're confident enough with your test suite, you tin even ready auto-merge)

Nosotros'd already been using Dependabot for security updates and it served the purpose really well!

Why did nosotros determine non to become with information technology?

Unfortunately, it misses (at least at the time of writing this article) the ability to have duplicate updates for different allow types. That means you can't take east.g. daily updates for dependencies and weekly updates for devDependencies in the same project. In our case, it was about not being able to daily update to versions that include security-related changes and all of the other changes (features, fixes, major updates) on a weekly basis using the same tool.

Besides, as it turned out, afterward on, having new PR for each dependency update is a scrap of a hurting.

Renovate

After figuring out that Dependabot does not allow us to do the higher up, nosotros've decided to look for alternatives. One of the most promising ones (and open up-source!) was Renovate.

Even though the basic principle of bumping dependencies is the same, the tool itself seems very powerful and customizable. It has 3 applications (Github, Gitlab, and self-hosted), highly granular settings (you tin even set custom rules for auto-merging of PR), and allows opening a PR for a batch of dependencies, instead of for each i.

As we are using GitHub for version control, the supported application for it was an obvious selection. Because our usage was a fleck unorthodox – updating merely yarn.lock and not parcel.json to have a representation of current users environments and at the same controlling the version ranges – we wanted to test it on the self-hosted version first, to avoid unnecessary PRs created by Renovate, or even worse – unwanted merges.

This is where we hit a wall with Renovate – fifty-fifty though it has a great range of options, we didn't manage to configure information technology the way we wanted – update But yarn.lock once a calendar week and create a single PR.

Because of that, we decided not to spend more fourth dimension on publicly available solutions, and handle the lockfile maintenance ourselves.

Your own CI job

You lot may ask:

"Why did yous even bother with setting those dependency direction systems? Isn't it easier to merely run yarn upgrade on everything and call information technology a day?"

And y'all would be partially right. The thing is that these systems probably do the exact aforementioned thing under the hood but put more attention to the possible failures and corner cases. And just because they are already battle-tested, nosotros decided to check them first. Custom solutions built from scratch, in full general, tend to exist more frail than the commercially bachelor ones.

Since neither Dependabot nor Renovate met our needs at a time though, our way out was writing a custom CI job that:

1. Would bump dependencies for us
2. Run some basic tests against those changes
3. Create a PR

Our toolchain was:

• CircleCI for CI/CD
• git and GitHub for VCS
• Yarn as a bundle manager
• Jest for testing
• Coffee® for energy

Custom command

          ### bash   $ git checkout main  $ export BRANCH_NAME=feat/lockfile-maintenance-ci-chore-$(appointment +"%m-%d-%Y") && git checkout -b $BRANCH_NAME  $ yarn upgrade  $ git add yarn.lock  $ git commit -grand "job: weekly lockfile maintenance"  $ git push --set-upstream origin $BRANCH_NAME  $ BODY='{"head":''"'${BRANCH_NAME}'"'',"base of operations":"principal","title":"Weekly lockfile maintenance"}'      && curl -X POST      -H "Take:application/vnd.github.v3+json"      -u $GIT_AUTHOR_NAME:$GH_TOKEN https://api.github.com/repos/stoplightio/elements/pulls      -d "$BODY"        

The premise of this is:

1. Become the latest changes from main (no demand to git fetch as this is being run in a fresh CI job each time) and create a feature branch with a name corresponding to the lockfile maintenance

                 $ git checkout main  $ export BRANCH_NAME=feat/lockfile-maintenance-ci-task-$(appointment +"%m-%d-%Y") && git checkout -b $BRANCH_NAME            

2. Upgrade all of the dependencies in yarn.lock according to packet.json – this mimics what happens for users upon a fresh install

                 $ yarn upgrade            

3. Push button changes to remote

                 $ git add yarn.lock $ git commit -m "chore: weekly lockfile maintenance" $ git push --set-upstream origin $BRANCH_NAME            

4. Create a PR using GitHub API (more than details in GitHub API Documentation)

                 $ BODY='{"head":''"'${BRANCH_NAME}'"'',"base":"primary","title":"Weekly lockfile maintenance"}'  && gyre -Ten Post    -H "Accept:application/vnd.github.v3+json"    -u $GIT_AUTHOR_NAME:$GH_TOKEN https://api.github.com/repos/stoplightio/elements/pulls    -d "$Trunk"            

Both $GIT_AUTHOR_NAME and $GH_TOKEN are secrets from CircleCI – brand sure you don't hardcode your credentials in the CI config file and/or the command itself.

CI configuration

          workflows:  version: 2  examination-and-release:    ...  perform-lockfile-maintenance:    triggers:        - schedule:            cron: "0 3 * * ane"            filters:              branches:                only:                  - main    jobs:      - lockfile-maintenance        

Brand certain yous define the chore as well:

          jobs: lockfile-maintenance:    docker:      - image: circleci/node:12    steps:      - checkout      - run:          command: |            ### THIS IS A PLACE FOR THE COMMAND FROM PREVIOUS PARAGRAPH        

By default, CircleCI runs workflows against all commits from all branches. This is definitely not the behavior we want to take for lockfile maintenance. The desired outcome is that it will run once a calendar week against the main co-operative. We also don't run any tests at this stage, as the PR created against the main branch volition trigger the test-and-release workflow that is existence run for each branch and contains a examination suite, checks linting, and builds a projection to see if there are no crashes.

That's where cron jobs come in handy. We start define that our perform-lockfile-maintenance workflow will be triggered past one (exam yours using this online tool) past putting cron job description in the triggers/schedule department. Then we apply an additional filter to it, so it only targets main at any given moment.

As for scheduling, we decided to go with Mon before work (Central European Time), so it is the first affair nosotros look into at the beginning of the calendar week. A contributor opens a PR containing changes made to yarn.lock, approves if information technology looks correct, and merges the change to main.

And that's it! You've just prepare your first lockfile maintenance menstruum!

Possible improvements / aftermath

There are few more things y'all tin can do to ameliorate your confidence even more:

• If you include examples of usage for your library like us (an integration for GatsbyJS, Angular, CRA) yous can bump their dependencies every bit well. This will assure that your library not only is properly tested internally but doesn't crash when applied to a existent-life scenario
• Serve an environment containing these integrations for each PR eastward.g. using Netlify. That volition make the whole testing process much quicker as you won't demand to check out the changes and run them locally on your own
• Strengthen your CI pipeline in full general: the more that is covered past your testing suite, the less y'all will have to check

Summary

So there yous go, nosotros have but gone to a dependency hell and came back live!

I believe that what I have described above will help you encounter fewer bug when developing your library, especially if you don't have a full team dedicated to testing bugs.

But fifty-fifty if I didn't convince you to do a weekly/monthly/any dependency crash-land I promise that this commodity gave you a strong understanding of the lockfile itself, why it is important when talking about compatibility across different machines, and seeing that lockfile maintenance does not take to be a terrible chore that takes an unreasonable amount of time.

If you feel similar this article added some value to your electric current skill set, though, please consider resharing it on your social media.

If you'd similar to acquire more near Stoplight'due south open-source tool, Elements, which was a ground for this article, visit Stoplight website.

Photo by Jude Al-Safadi on Unsplash

ridgwayraceiziendas.blogspot.com

Source: https://11sigma.com/blog/2021/09/03/yarn-lock-how-it-works-and-what-you-risk-without-maintaining-yarn-dependencies-deep-dive/

Share this post