Diarrhea for your backend on Node.JS - reduce build weight

For certain you often noticed how much garbage is inside node modules. These are tests, benchmarks, readme files, licenses, typing script, and even an insane amount of garbage that can be more or less safely removed. What we actually do in this post.
I have already mentioned the last few publications about the weight of the node module, so here's another one, which generally reflects the current situation. Little Big, “Life in da trash” is recommended as a soundtrack for the post.

Start

So, once again, by accidentally going inside the node modules, I mourn, and I began by saying that for the experiment I wrote several scripts with my hands, which I drove through my project. They looked like this:

find ./ -iname "tests" -exec rm -rf {} \; find ./node_modules -iname "test" -exec rm -rf {} \; find ./node_modules -iname "*.gif" -exec rm -rf {} \; find ./node_modules -iname "*.jpg" -exec rm -rf {} \; find ./node_modules -iname "*.jpeg" -exec rm -rf {} \; find ./node_modules -iname "*.png" -exec rm -rf {} \; find ./ -iname "*.md" -exec rm -rf {} \; find ./ -iname "*.log" -exec rm -rf {} \; find ./ -iname "LICENSE*" -exec rm -rf {} \; find ./ -iname "README*" -exec rm -rf {} \; find ./ -iname "LICENCE*" -exec rm -rf {} \; find ./ -iname "AUTHORS*" -exec rm -rf {} \; find ./ -iname "NOTICE*" -exec rm -rf {} \; find ./ -iname "changelog*" -exec rm -rf {} \; find ./ -iname ".travis.yml" -exec rm -rf {} \; find ./ -iname ".coveralls.yml" -exec rm -rf {} \; find ./ -iname ".npmignore" -exec rm -rf {} \; find ./ -iname ".gitignore" -exec rm -rf {} \; find ./ -iname ".jshintrc" -exec rm -rf {} \; find ./ -iname ".eslintrc" -exec rm -rf {} \; find ./ -iname ".jscs.json" -exec rm -rf {} \; find ./ -iname ".editorconfig" -exec rm -rf {} \; find ./ -iname ".vs" -exec rm -rf {} \; find ./ -iname ".babelrc" -exec rm -rf {} \; 

In general, it was already quite good, and it gave some savings, but I wanted some more elegant and general solution. And quickly enough, I found a module that does exactly what I wanted - ModClean .

Modclean

The module has a lot of settings, and also lists of “extra files” in the form of plug-ins, which are connected as npm packages. By default, this is the list.

There are three levels of deletion security files in it - I highly recommend that you look at the list and see what may cause you problems. For example, I did not understand why on the list they included “semver” ...

The only drawback is that for some reason the author of the module did not think that you might also want to remove all the garbage before building in your own project - for example, you absolutely do not need production test files ...
True, it turned out that it was easy to get around by specifying the project directory as the module directory:

 modclean --modules-dir . 

results

It is clear that the results can be very different in different projects, but I have saved from 10% to 20% of the savings. In advanced cases, there may be more. This may not seem like the biggest figure, but this reduction in the storage space required by 20% or 20% faster rolling out projects from virtually nothing - so why not?
For the curious on the module site there are also benchmarks for removal, but it is better to just try it at home.

Diarrhea

On this positive note, I would like to say that I have solved my task, and I recommend everyone to use this excellent module. But you saw the word “diarrhea” in the title, not “modclean”, right? This means that one more thing remains. The nuance is quite fat in the literal sense of the word - it turned out that the module has 123 transitive dependencies and it weighs 4MB. What is generally considered to be more or less the standard for the modules of the node, but can not deliver the heartache.

At the same time, 1.2 megabytes is occupied by the wildly popular update-notifier module , all it does is that it draws such a frame if updates are available:

1 megabyte for this, Karl! I, as a Node.JS developer, want to fail with shame.

And most of the rest of the space is occupied by the clui module , which contains all sorts of decorations for working with the terminal - only a couple of functions are used.

In general, I could not bear it, and did my build - without blackjack and everything else. I completely cut out the notification of updates from there, and I assembled the necessary functions from thick modules with a large number of dependencies into a minified bundle using a webpack. In general, it was possible to play a little more and still reduce the weight of the module - but I reached the weight of 700 kilobytes and so far calmed down. I also added the option --root, which allows you to more clearly not clean the dependencies, but the root of your project.

Issues for all the above reasons I threw in the modclean, and I also made a pull request there, and if everything is approved, then I will deny this fork in favor of the main module - but for now there is a feeling that the modclean is abandoned - so if there are good ideas, throw them to me.

You can install my version, as always, from npm .

A warning

Double-check that you have everything you need in the repository - diarrhea can delete unnecessary files, files, teeth, and kill kittens. I take off for this all responsibility.

CI Integration

It is best to use diarrhea like this:

1. First, we remove all the garbage from node modules;
2. Then we run the unit tests to make sure that we haven't taken anything extra;
3. Then we remove the garbage from the main project, including the actual tests;
4. We do npm prune --production;
5. We collect artifact.

ToDo

The module was made in haste for personal use, so in the future I would like to add the following:

• Tests Oddly enough, modClean doesn't have them at all;
• Compatibility with node up to version 8 Since there is already a webpack, it is not difficult;
• Cut a few more dependencies in which there is no special meaning;
• Minimal version without terminal decorations for CI

findings

• Once again, I get scared of what terrible things create a low threshold for entering my favorite ecosystem. With this, you definitely need to do something, and I hope that this will somehow be decided at the npm level. But right now I see that this beautiful update-notifier is in dependencies of the npm itself, which is very distressing. This unfortunate notifier has 2 million downloads per week! 2 million megabytes (almost 2 terabytes) spent on the fucking frame with the message about the update!
• Always check the weight of your dependencies, even on the backend;
• Clean your dependencies;
• Strangely enough, a little bit I come to the idea that backend dependencies also need to be assembled into a common bundle. What is being done quite easily and naturally with any tool of your choice, and does not add pain when debugging - after all, you can generate a search tool. The only drawback is that not every package can be included in a bundle, I regularly encounter some problems in this, mostly due to the fact that everyone loves to connect dynamic dependencies. For example, the same update-notifier cannot be shoved into a bundle.

Well, that's all.

If you were interested, you might also like my other articles on this topic:

• Check for duplication of dependencies using wtfwith;
• Check security dependencies using snyk;
• A story about how I live via meltdown on your phones using npm.

I can also tell you about various instances of the use of Node.JS on the backend, participate in the survey if you are interested.