Reduce the size of published npm modules
By default, npm publishes the entire module to the registry. Except for those explicitly specified in .gitignore files. This drops the dependencies, but still allows a bunch of not-so-needed files to seep into the published one. After that, grateful users wait until all this is downloaded. For grunt , by the way, will have to wait about 6 megabytes. But he is usually not alone.I decided to figure out how to measure the size of my modules after publication and, if possible, reduce this size. As an example I will use the check-more-types module, which contains only a few files. Plus unit tests and documentation that is compiled into the readme markdown file.
First of all, we must calculate the current size of the module. NPM stores all files as tar archives, so it will be enough to create such an archive and see its size. Moreover, npm has for this a special npm pack command that creates an archive from the contents of the specified directory. Mathias Bynens offers the following script to determine the size of the module:
tarball="$(npm pack .)"; wc -c "${tarball}"; tar tvf "${tarball}"; rm "${tarball}"; ')
I measured the archive size for a commit with hash 3ada360 :
little shell magic
$ tarball="$(npm pack .)"; wc -c "${tarball}"; tar tvf "${tarball}"; rm "${tarball}"; 25184 check-more-types-2.1.2.tgz -rw-r--r-- 0 501 20 1977 Nov 19 13:55 package/package.json -rw-r--r-- 0 501 20 64 Nov 19 13:18 package/.npmignore -rw-r--r-- 0 501 20 19703 Nov 19 13:49 package/README.md -rw-r--r-- 0 501 20 1073 Nov 19 13:18 package/LICENSE -rw-r--r-- 0 501 20 2534 Nov 19 13:18 package/Gruntfile.js -rw-r--r-- 0 501 20 18204 Nov 19 13:18 package/check-more-types.js -rw-r--r-- 0 501 20 6723 Nov 19 13:49 package/check-more-types.min.js -rw-r--r-- 0 501 20 600 Nov 19 13:49 package/bower.json -rw-r--r-- 0 501 20 162 Nov 19 13:18 package/.travis.yml -rw-r--r-- 0 501 20 1756 Nov 19 13:18 package/.jshintrc -rw-r--r-- 0 501 20 655 Nov 19 13:18 package/docs/README.tmpl.md -rw-r--r-- 0 501 20 1936 Nov 19 13:18 package/docs/badges.md -rw-r--r-- 0 501 20 255 Nov 19 13:18 package/docs/footer.md -rw-r--r-- 0 501 20 240 Nov 19 13:18 package/docs/install.md -rw-r--r-- 0 501 20 13707 Nov 19 13:49 package/docs/use.md -rw-r--r-- 0 501 20 127 Nov 19 13:18 package/test/check-more-types-minified-spec.js -rw-r--r-- 0 501 20 78 Nov 19 13:18 package/test/check-more-types-spec.js -rw-r--r-- 0 501 20 467 Nov 19 13:18 package/test/load-under-node-test.js -rw-r--r-- 0 501 20 738 Nov 19 13:18 package/test/synthetic-browser-spec.js -rw-r--r-- 0 501 20 37754 Nov 19 13:18 package/test/unit-tests.js What do we see? A bunch of files and the size of the compressed data in 251184 bytes. First, we automate this process a little. I found a good utility for getting size without using shell commands: pkgfiles by Tim Oxley .
I installed the utility as a dependency maiden and added it to the “prepublish” script for package.json :
{ "devDependencies": { "pkgfiles": "2.3.0" }, "scripts": { "prepublish": "pkgfiles" } } The “Prepublish” script will be executed each time with a local installation and with the command npm publish . Let's look at the result of npm run publish after our changes:
npm run prepublish result
$ npm run prepublish > check-more-types@2.1.2 prepublish /Users/kensho/git/check-more-types > pkgfiles PATH SIZE % .npmignore 0 B 0% test/check-more-types-spec.js 78 B 0% test/check-more-types-minified-spec.js 127 B 0% .travis.yml 162 B 0% docs/install.md 240 B 0% docs/footer.md 255 B 0% test/load-under-node-test.js 467 B 0% bower.json 600 B 1% docs/README.tmpl.md 655 B 1% test/synthetic-browser-spec.js 738 B 1% LICENSE 1.07 kB 1% .jshintrc 1.76 kB 2% docs/badges.md 1.94 kB 2% package.json 2.05 kB 2% Gruntfile.js 2.53 kB 2% check-more-types.min.js 6.72 kB 6% docs/use.md 13.71 kB 13% check-more-types.js 18.2 kB 17% README.md 19.7 kB 18% test/unit-tests.js 37.75 kB 35% DIR SIZE % docs/ 16.79 kB 15% test/ 39.16 kB 36% . 108.77 kB 100% PKGFILES SUMMARY Size on Disk with Dependencies ~126.72 MB Size with Dependencies ~88.58 MB Publishable Size ~108.77 kB Number of Directories 3 Number of Files 20 Very detailed information. And the most "heavy" files are listed last, which makes it possible to conveniently analyze the results in the terminal. Most of the space occupied by the directory with the documentation and tests - and this despite the fact that we are not going to publish them!
There are three ways to specify which files will not be published to npm. We use the default method: the files specified in .gitignore are automatically blacklisted. And we can create another file, .npmignore , in which we specify a set of files independent of git that we don’t want to publish. Alternative way: add files to the “white list” using package.json . Personally, I prefer this way. Please note that a number of files, such as package.json or README , are automatically whitelisted.
{ "files": [ "bower.json", "check-more-types.js", "check-more-types.min.js" ] } And to exclude files from already added ones, you can use an exclamation mark. For example, if in your src directory that you want to publish, there is a test subdirectory that you don’t want to publish at all, then:
{ "files": [ "src", "!src/test" ] } Well, if in one src directory you have files for both production and test / staging, then you can exclude files one by one or in groups:
{ "files": [ "src/*.js", "!src/*-spec.js" ] } The commit hash with all these changes starts with bc3e2a1 . Let's see what happened with the size of the published module:
another npm run prepublish result
$ npm run prepublish > check-more-types@2.1.2 prepublish /Users/kensho/git/check-more-types > pkgfiles PATH SIZE % bower.json 600 B 1% LICENSE 1.07 kB 2% package.json 2.15 kB 4% check-more-types.min.js 6.72 kB 14% check-more-types.js 18.2 kB 38% README.md 19.7 kB 41% DIR SIZE % . 48.45 kB 100% PKGFILES SUMMARY Size on Disk with Dependencies ~126.72 MB Size with Dependencies ~88.58 MB Publishable Size ~48.45 kB Number of Directories 1 Number of Files 6 Everything turned out quite well: the size of the published module was reduced by 55% from 107 kilobytes to 48. This is a general reduction in size, but we can still see what has changed in the tar archive. Unfortunately, the npm pack causes the prepublish script and cannot correctly process its output. Therefore, I will temporarily rename prepublish and add a “tarball = ...” under the name reuse :
npm run size result
$ npm run size > check-more-types@2.1.2 size /Users/kensho/git/check-more-types > tarball="$(npm pack .)"; wc -c "${tarball}"; tar tvf "${tarball}"; rm "${tarball}"; 13179 check-more-types-2.1.2.tgz -rw-r--r-- 0 501 20 2256 Nov 19 14:09 package/package.json -rw-r--r-- 0 501 20 19703 Nov 19 13:58 package/README.md -rw-r--r-- 0 501 20 1073 Nov 19 13:18 package/LICENSE -rw-r--r-- 0 501 20 18204 Nov 19 13:58 package/check-more-types.js -rw-r--r-- 0 501 20 6723 Nov 19 13:58 package/check-more-types.min.js -rw-r--r-- 0 501 20 600 Nov 19 13:58 package/bower.json Now the client needs to say only 13 kilobytes instead of 28, and this is a 50% reduction in size!
I also had the idea to show the size of the published module with each push from the local repository in the “remote master”. To do this, it is enough to add both size and pkgfiles commands to the pre-push step and use the “pre-git” module:
npm install -D pre-git package.json
To check that everything worked, I increased the version of the package from 2.1.2 to 2.2.0 and used to install the clean directories and the npm version “3.4.0”:
Almost 3 seconds. Erase the node_modules directory and try the new version of the package:
We successfully bite off 1 second from the installation - and this is 30% of the total installation time for our little module!
The full version of package.json that I used to “clean up” (you can see in my repository ):
By the way, this version can be slightly reduced by using “t” instead of “tarball”:
{ "scripts": { "pkgfiles": "pkgfiles", "size": "tarball=\"$(npm pack .)\"; wc -c \"${tarball}\"; tar tvf \"${tarball}\"; rm \"${tarball}\";" }, "config": { "pre-git": { "pre-push": [ "npm run size", "npm run pkgfiles" ] } } } </spoiler> To check that everything worked, I increased the version of the package from 2.1.2 to 2.2.0 and used to install the clean directories and the npm version “3.4.0”:
$ time npm i check-more-types@2.1.2 /private/tmp/test-small └── check-more-types@2.1.2 real 0m2.706s user 0m1.419s sys 0m0.323s Almost 3 seconds. Erase the node_modules directory and try the new version of the package:
$ rm -rf node_modules/ $ time npm i check-more-types@2.2.0 /private/tmp/test-small └── check-more-types@2.2.0 real 0m1.716s user 0m1.244s sys 0m0.198s We successfully bite off 1 second from the installation - and this is 30% of the total installation time for our little module!
The full version of package.json that I used to “clean up” (you can see in my repository ):
npm install -D pkgfiles pre-git package.json entirely
{ "devDependencies": { "pkgfiles": "2.3.0", "pre-git": "1.3.0" }, "scripts": { "pkgfiles": "pkgfiles", "size": "tarball=\"$(npm pack .)\"; wc -c \"${tarball}\"; tar tvf \"${tarball}\"; rm \"${tarball}\";" }, "config": { "pre-git": { "pre-push": [ "npm run size", "npm run pkgfiles" ] } } } By the way, this version can be slightly reduced by using “t” instead of “tarball”:
"scripts": { "pkgfiles": "pkgfiles", "size": "t=\"$(npm pack .)\"; wc -c \"${t}\"; tar tvf \"${t}\"; rm \"${t}\";" } Source: https://habr.com/ru/post/283552/
All Articles