Dependencies of our dependencies

This story began on November 30 in the morning. When a completely normal Test environment build suddenly fell. Probably, some linter fell off, without waking up, I thought and was wrong.

To whom it is interesting what this story ended and what thoughts it suggested - I ask for cat.

When I opened the build log, I saw that npm install had crashed. I thought strangely. Last night everything worked fine. After studying the logs for some time, a suspicious line was found:

node --eval 'if (require("./package.json").name === "coffee-script") { var red, yellow, cyan, reset; red = yellow = cyan = reset = ""; if (!process.env.NODE_DISABLE_COLORS) { red = "\x1b[31m"; yellow = "\x1b[33m"; cyan = "\x1b[36m"; reset = "\x1b[0m"; } console.warn(red + "CoffeeScript has moved!" + reset + " Please update references to " + yellow + "\"coffee-script\"" + reset + " to use " + yellow + "\"coffeescript\"" + reset + " (no hyphen) instead."); console.warn("Also, a new major version has been released under the " + yellow + "coffeescript" + reset + " name on NPM. This new release targets modern JavaScript, with minimal breaking changes. Learn more at " + cyan + "http://coffeescript.org" + reset + "."); console.warn(""); } 

Here I was again surprised. Again, yesterday we did not use coffee-script on the project and it’s unlikely that much has changed overnight. A quick review of package.json confirmed that no adversary had added anything to it. So, probably, we have updated some kind of dependency that uses coffee-script. But it was against this idea that for quite some time I had set strict versions for all dependencies of the project and, as it seemed to me, this could not happen. Having no results looking for a similar problem on the Internet, I again returned to the thought of a renewed dependency. Therefore, a script was written on the knee which bypassed all package.json s in node_modules in search of coffee-script. There were about 5-6 such dependencies. This further strengthened my suspicions, and not much time thinking, I demolished the entire node_modules, and at the same time all dependencies except one in the local repository and launched npm install again. The process was successful. Then, step by step, the dependency was found, which brought down install.

It turned out to be karma-typescript, which had a pad in transitive dependency, which in turn depended on coffee-script. And here I am depressed again. There were few options. Or temporarily disable the tests, or wait for the fix, or fork and fix it yourself (and it’s not very clear what exactly needs to be fixed). Without much hope, I went to Github to create an issue. What was my surprise when I was answered literally after 20 minutes. It turned out that some comrade decided to update the coffee-script package to npm and instead of declaring the old package obsolete, he simply made him unpublish.

One of the comments from the community:

Fortunately, @ondrejbase has already suggested crutch temporary solution that helped me. And the whole story ended relatively cheaply. But it could not be so.

It was a saying. And now I propose to talk about the dependencies of our projects and the problems that they bring to us.

Let's start with the simple.

Global dependencies

Recently, I once again came across an article for beginners that began with the line

npm install -g typescript

And could not resist. In my opinion, this is one of the worst tips you can give a beginner developer. I'm serious. Here are the problems this advice leads to:

• Local repository conflict. Collecting all the code using global dependencies (tsc, for example) or validating it with global linters, we can no longer easily update these dependencies. After all, updating something global, we endanger all repositories in which it is used. As a result, you have to either spend time updating these dependencies in all repositories, or not updating them at all.
• Conflict with team members. The use of global dependencies in teamwork leads to their desynchronization. And, as a result, to the fact that my code does not gather or does not pass the test of my colleague.
• Conflict with build machine. What I gather is not guaranteed to build on the build machine. Or, which is much worse and more cunning - it will not be collected in the same way as mine. And this is a guaranteed debug hell for the next hour or more.

And all this happens because most manuals start with npm install abc -g . Although you can put everything locally and connect to package.json as ./node_modules/.bin/tsc

If you are thinking of writing your guide, please remember this article and save my nerves, as well as the nerves of all those who will use it in practice.

NB Globally installing code generators (create-react-app, create-angular-app) is fine. They will work once and that's it. In addition, you do not need to install them again when you decide to create the next repository.

Soft dependencies

Let's go further. Install the create-react-app, and create the base application. We go in package.json and what we see there?

"react": "^16.2.0"

Everyone (or almost everyone) knows what the ^ symbol means. It means that npm can set any version older or equal to that specified, within the major release. And what's wrong with that? I do not want to be categorical, but, as for me, this approach is also “not very”, and this is why:

• Loss of control. As soon as you left ^ or ~ in your package.json you lost control of your code. Of course I exaggerate, but think. You are loading into the project a version of a third-party library about which you don’t even know its version. Only that it lies in the repository from some publisher. And suddenly, what will happen, we can only hope that this package has a fairly organized community to notice.
• Changes breaking backward compatibility. Of course, publishing a major version, the developers promise that there will not be breaking changes within it. But, gentlemen, let's be realistic. This is the Internet, and this is open source: no one owes you anything. What worked an hour ago can fall off with a comment like: “This is a new release targets modern JavaScript, with minimal breaking changes”.
• Invisible updates. Suppose I have a library of the form abc: ^ 2.1.1. And, it turns out, I do not know what version of the library I actually have. Maybe 2.1.1, and maybe 2.9.9. And it seems that this is not a problem, on the contrary. I do not need to look for documentation for a particular version, it is enough to always look the latest. And here I look at it, and I see a new chip, and it does not work. But it does not work because I just forgot to update the library. I update it, comedy code, and after 30 minutes my colleague runs to me because his application does not work! And three more come for him and make me sad. Who needs it?
• And finally: dependency dependencies. This is the cherry on the cake and this is what I would like you to think about.

Dependencies of our dependencies or ZNZ

Let's start with why we have a broken build. All dependencies were set rigidly and nevertheless we fell. Despite the fact that the versions of our main dependencies remained the same (remember, no ^, ~ in package.json) their dependencies were not so strict. We did not control the dependencies of our dependencies, although we tried. And, that the most unpleasant such behavior is encouraged by default. I don't know who did it and why, but he put a big pig on all of us, and especially on those who practice continuous-integration.

Of course, this particular problem is easy to fix. It is enough to create a lock-file (for example, using the npm shrinkwrap command) or use yarn, a package manager that by default captures all dependencies of your project.

However, this solves only part of the problem. There remains one more, much more dangerous part of it, and its name is unpublish . If you have not encountered this problem before, then here is an excellent article that shows the entire vulnerability of modern web development. At any time, deliberately or through carelessness, your project may stop being assembled just because someone has deleted his package from npm. And it's not difficult to do that. Simply enter the unpublish command. This misfortune can be fought . But let's be honest with ourselves? Which one of us has its own local npm? And who even thought about it? I'm afraid not so much. And I only hope that you are now warned.

By the way, if you log into my repository (do not do this, please), you will see that almost all my projects violate everything that was said above. And this is another proof of how widespread the problem is.

findings

• Use the global installation flag wisely. You should not use it for those dependencies on which your project will constantly depend.
• If you are not working on one or several physical machines, consider setting the dependencies strictly. This will allow you to have the same basic project dependencies in all repositories and with colleagues. In addition, you will increase control over your own code and the process of updating it.
• Use lock file. Especially if you are going to use continues-integration.
• Think about private register packages. This not only saves you from sudden removal of packages, but also speeds up the installation of dependencies on the build machine. And this will save your time and the amount of coffee that is drunk while the build is underway during pull requests.

I have everything on it, I hope it was interesting and useful. According to a strange tradition, there should be some kind of advertisement here, but I don’t have it, so I’ll just pass on thanks to my colleagues who put out this fire with me.

And I apologize for the English words, but in the Russian equivalent they severely hurt the eyes.