Many JS packages in one repository

Hello! Not so long ago, we began to develop a comprehensive project that has or plans several types of front-end, many back-end services, command line interface, demons, and a lot more. All this, in turn, has a sharp code, and completely new applications should be possible to assemble from existing bricks in a simple and understandable way.

If you don’t get bored with the terminology, we make a platform. Platform for visual programming for DIY-electronics.

Despite the fact that the project is at an early stage, the code base has already threatened to turn into a mush. To do this, we transferred the project to the so-called monorepo-approach. On Habré there were no materials on this topic, so I will try to fill the gap.

What was at the beginning

It all started pretty traditional. Our repository looked like this:

dist/ node_modules/ src/ assets/ components/ containers/ reducer/ actions.js actionTypes.js constants.js test/ package.json 

Those who deal with the React + Redux stack will instantly recognize the pattern. The src/ front-end source code is located; by a command, they are collected by the Webpack into dist/ wherefrom the front-end can be served as a simple static.

Expanding

Such a structure works well if the application is not very large. But we quickly got a lot of React-components and containers, Redux-reducers and-actions, which began to crowd in their directories.

The semantic division into groups suggested itself: something was responsible for rendering the graph of the user’s program, something for editing tools of this graph, something for navigating through files and tabs, something for displaying informational messages, etc.

It's time to share. At this point, there was a choice in front of two generally accepted approaches of separation.

Rails style:

 src/ components/ project/ projectBrowser/ editor/ messages/ containers/ project/ projectBrowser/ editor/ messages/ reducers/ project/ projectBrowser/ editor/ messages/ actions/ project.js projectBrowser.js editor.js messages.js ... 

Or pod style:

 src/ project/ components/ containers/ reducers/ actions.js actionTypes.js constants.js projectBrowser/ components/ containers/ reducers/ actions.js actionTypes.js constants.js editor/ components/ containers/ reducers/ actions.js actionTypes.js constants.js messages/ components/ containers/ reducers/ actions.js actionTypes.js constants.js 

The Rails approach is good because the layers are well defined. The structure of “packages” is regulated and does not provoke inventions.

But therein lies the problem. We want, let us assume, now the CLI interface. React for command line utilities makes little sense: the components and containers layers are not needed. But you need somewhere to put modules for a nice output in the terminal, for parsing arguments, etc. There are no layers for this, you only have to add for CLI.

Then we invent something else and see that it does not climb into the structure again. We'll have to inflate again. Inevitably there will be a garbage dump with the name utils , helpers , tools , shared or whatever it is usually disguised by something unclear. Bad option.

And most importantly: there is no simple way to tear out some kind of “package” from the code base, to say that now it is something independent, throw it onto a diskette and send it by mail.

Therefore, we stopped at the pod-concept.

Ways up

Now, if one package wants to use the benefits of another package, it must import it in a relative path:

 // src/editor/containers/Editor.jsx import { validateProject } from '../../core/project/selectors' 

There is something controversial about this. Although packages are separated by directories, there is a strict assumption about their placement. The number of "dots" varies depending on the nesting of the module that imports. Among other things, it also makes refactoring difficult.

It would be desirable as with libraries: to begin importing from the name of the library, and so that someone can figure out for us where to get this library:

 // src/xod-editor/containers/Editor.jsx import { validateProject } from 'xod-core/project/selectors' 

Core evolved xod-core to eliminate the possibility of conflicts with third-party libraries in the case of using simple names. XOD is the name of the project we are doing.

So, how to come to this? That's right, make real JS packages that run through NPM and node_modules , exactly as it does with libraries.

The classic approach is to have a repository with its package.json , versioning, etc. for each JS package.

However, with dynamic development, juggling with a dozen repositories with npm install, build, publish, npm link, git pull, git push even feels like hell. You need to somehow leave everything in one repository.

As long as we refactor the structure, clearly highlighting the packages:

 node_modules/ xod-cli/ bin/ src/ test/ xod-client/ dist/ src/ test/ xod-client-browser/ ... xod-client-electron/ xod-core/ xod-espruino/ xod-fs/ xod-server/ package.json 

Linking

In theory, for such scenarios, there is an npm link , but it is just elementary to correctly link all the links on a new machine; to reproduce the structure of the project is no longer easy: npm link is not stateless. And everything is done for the sake of simplicity. So no, thanks.

There is a trick that is based on how Node looks for modules. Namely: the node runs up the file tree in search of node_modules/ , starting from the directory where the importing module is located.

Thus, we can make the symlinks we need with our hands inside src/ . On the one hand, we will make our own packages visible for import and do not conflict with the usual dependencies from package.json .

 node_modules/ react/ redux/ webpack/ xod-client/ dist/ src/ node_modules/ xod-core -> ../../../xod-core/src this-package-js-files.js xod-core/ src/ project/ selectors.js package.json 

Hurray, regardless of the position of the importing module, we can do:

 import { validateProject } from 'xod-core/project/selectors' 

Simlinks can be safely stored in a Git repository. And while Windows is not found among developers, everything will work fine: the code is immediately ready for assembly after the clone.

Turning to package

What turned out is not yet full JS-packages. In order for the package to be filled with NPM and on equal terms with all rights used in third-party projects, you need to provide each with its own package.json and prescribe its sole dependencies. Now we have the only description of the mega-package is at the root. All dependencies of all packages have been dumped there. We fix:

 xod-client/ dist/ node_modules/ babel/ ramda/ react/ redux/ webpack/ src/ node_modules/ package.json xod-core/ node_modules/ babel/ ramda/ webpack/ src/ package.json package.json Makefile ←  

The symlink story continues to work as it did, but each package received its own meta-information, its own dependencies, which are necessary only for him. The structure has become more manageable.

Only now, having 10 packages, in order to make the same npm install , it is necessary to go into each directory and run the script for each package. Not cool.

To return the ability to build, test, lint or run in one command, we added a Makefile . With content type:

 install: npm install cd xod-cli && npm install cd xod-client && npm install cd xod-client-browser && npm install cd xod-client-electron && npm install cd xod-core && npm install cd xod-espruino && npm install 

And so for every action. A little awkward, but it works.

Stupid builds

The problem of such a structure surfaced rather quickly. The fact that each package began to have its own dependencies from an academic point of view is good, but from a practical point of view it led to the fact that the same dependencies began to be established several times. On one only make install took under 10 minutes.

The lion's share of time was eaten off by the installation of the Webpack, Babel and their friends.

Additionally, during the build, the same source files were transported / packaged several times: once per package. Not productive.

Solution: let each package build itself into its dist/ once, and dependent packages use ready-made artifacts. The build tools themselves can be put once in the root node_modules/ .

With this approach, symlinks between packages are sufficient to migrate from src/ to dist/ and slightly tweak the Webpack configs so that it does not process "foreign" sources.

You should also separately make sure that the order of the build is not broken: the packages on which depend must be built before the dependent packages.

All tools from dev-dependencies moved to the root: Webpack, Babel, Mocha, ESLint.

This pair of measures returned the full build and check on the CI server in three minutes. Correspondingly, on localhost, things went brighter.

Lerna

While we moved the package directories back and forth, I came across Lerna . This is a tool that was extracted from Babel at one time and just helps keep many packages in one repository. This is done , of course, in Babel himself.

Among the utilities, Lerna allows you to run the npm command inside each package, bump the version of each package, and most importantly it allows you to do the so-called bootstraping.

Bootstrapping is the creation of symlinks to local packages, as we did, only automatically (based on the package.json package) and in its regular node_modules/ , and not in src/ . The final step of bootstrapping is to install third dependencies for each of the packages. And all this is cross-platform.

Everything is good, only Lerna is incompatible with the current structure in two articles:

• Packages must be in the packages/ subdirectory
• Simlinks are created directly on the package directory, and not on its subdirectory like dist/ or src/

The first problem is solved trivially. With the second all harder.

The fact is that we will not be able to write:

 import { validateProject } from 'xod-core/project/selectors' 

It is necessary to write everywhere:

 import { validateProject } from 'xod-core/dist/project/selectors' 

There is something unnatural about it. But what if any package wants to build for several types of targets, and the appropriate subdirectories appear in its dist/ ? It is necessary to rewrite absolutely all import paths. Bad bad.

The package.json file allows you to specify a so-called main file, for example, dist/index.js , but does not allow you to specify a "main directory". Based on what I read, this is the official position of the node and will not change. To not dabbled.

How to be? Exhale, look at the experience of others. And the experience is such that almost anywhere you will not find imports with paths. Those. if there is a foo library, you simply import directly from it: import { blabla } from 'foo' . No import {blabla } from 'foo/bla/bla' .

And this is pretty damn good, we thought. The package acquires a clear, clear framework: it has an API from a number of functions, constants, classes that neighbors can use. This API can be described in its own README.md , cut from this repository, placed in a separate, published independently, etc.

Inside, conditionally, even though the grass does not grow, but I will show you kindly: a good and beautiful API.

As a result, all of our numerous imports of the form:

 import { validateProject } from 'xod-core/project/selectors' 

turned into elegant:

 import { validateProject } from 'xod-core' validateProject(...) //  import core from 'xod-core' core.validateProject(...) 

The packages themselves, in their root index.js simply re-export the necessary characters for everything.

Conclusion

As a result, we got a pretty nice structure, which is comfortable to work with and one feels that it will withstand more than one thousand commits.

 node_modules/ webpack/ babel/ mocha/ eslint/ packages/ xod-cli/ xod-client/ dist/ node_modules/ src/ api/ editor/ messages/ processes/ projectBrowser/ user/ index.js test/ package.json weback.config.js xod-client-browser/ xod-client-electron/ xod-core/ xod-espruino/ xod-fs/ .babelrc dist/ node_modules/ src/ backup.js index.js load.js save.js package.json xod-server/ package.json lerna.json 

The most attentive ones might have noticed that I started the examples with the separation of front-end components, and continued with some larger packages. And there is. The whole front with us and now lies inside one xod-client . There he is organized in the style of pods. It turned out that while he is not so tight. And when it starts to reap, we know what to do: take it to the upper level, in separate packages.

TODO:

• Lerna is not yet friends with Yarn . We are waiting for the developers to agree and npm install will become a rocket for mono-repositories as well.
• Lerna is able to execute the npm script in each package, but cannot do this, given the cross dependencies. Therefore, you have to manually register the build order in the root package.json . It is worth trying to simplify this through Gulp.

I do not pretend that the presented approach is “correct”. So it happened with us and it was formed on the basis of the evolutionary changes that the project went through. If someone will be useful to the thoughts, I will be glad;)

PS In the project we are looking for the full-stack of the JS developer. If you can recommend someone for this vacancy , I will be immensely grateful.