Broccoli is a new automated build system. It can be compared with the Rails asset pipeline, however there are some differences: it runs on Node.JS and is independent of the server side of the application.
After a long string of 0.0.x alpha releases, I just released the first beta version, Broccoli 0.1.0.
Table of contents:
- Quick example
- Motivation / Features
- Architecture
- Behind the Scenes / General View
- Comparison with other build systems
- What's next?
1. Quick example
Below is an example of the configuration file for the build (Brocfile.js). Comments are intentionally omitted; an example is given only to illustrate the syntax:
')
module.exports = function (broccoli) { var filterCoffeeScript = require('broccoli-coffee'); var compileES6 = require('broccoli-es6-concatenator'); var sourceTree = broccoli.makeTree('lib'); sourceTree = filterCoffeeScript(sourceTree); var appJs = compileES6(sourceTree, { ... outputFile: '/assets/app.js' }); var publicFiles = broccoli.makeTree('public'); return [appJs, publicFiles]; };
Run
broccoli serve
to start tracking changes to the source files. Each time any file in the list of monitored changes is changed, broccoli automatically rebuilds it in the target directory. Broccoli is optimized to serve serve as fast as possible, so you should not pause between successive assemblies.
Run
broccoli build dist
to do a one-time build and put the result in the dist folder.
For a more detailed example, take a look at the
broccoli-sample-app2. Motivation / Features
2.1. Fast rebuild
The main challenge in designing Broccoli was the implementation of fast incremental builds. And that's why:
For example, you use Grunt to build an application written in CoffeeScript, SASS, and several other preprocessors. When you are developing something, you want to edit the files and immediately see the result in the browser, without constantly running the build of the system. So, for this purpose you are using
grunt watch
, but as your application grows, the build is going slower and slower. After several months of working on the project, your “edited-updated” cycle turns into “edited-waited-10-seconds-updated”.
Accordingly, in order to speed up your build, you are trying to reassemble only those files that have been changed. It's pretty hard, because it happens that one output file depends on several input files. You have to manually configure the rules to rebuild the correct files, based on the modified and their dependencies. But Grunt is not designed to cope with this task easily, and therefore, even having written your own sets of rules, you cannot be sure that the necessary files will be rebuilt. Sometimes, it will re-compile files when it is not necessary (and thus slow down the build), but worse, sometimes it will not re-compile files when it has to do it (which makes your build unreliable).
With Broccoli, you can just run
broccoli serve
, and he himself will understand which files need to be monitored, and will reassemble only those that need it.
As a result, this means that a rebuild, as a rule, should have O (1) constant execution time, regardless of how many files are used in a project, since there is always only one going. I strive for a 200ms result for each build with a typical set of tasks, but because such a delay time seems almost instantaneous for the human brain, for me the results are acceptable up to half a second.
2.2. Plugin Chains
Another important task is the possibility of linking call plugins. Let us consider an example to show how simple, using Broccoli, you can compile CoffeeScript with the following minifacting:
var tree = broccoli.makeTree('lib') tree = compileCoffeeScript(tree) tree = uglifyJS(tree) return tree
Using Grunt, you would have to create a temporary directory (besides the final one) to save the CoffeeScript output there. As a result of all these manipulations, the Gruntfile usually grows to rather large sizes. With the help of Broccoli, all such actions are resolved automatically.
3. Architecture
For the curious, let me tell you about the Broccoli architecture.
3.1. Trees, not files
As a level of abstraction for describing the source and output data, not files are used, but trees are directories with files and sub-directories. It turns out that we have not “file-to-input-file-to-exit”, but “tree-to-input-tree-to-exit”.
If Broccoli worked with separate files, we would still be able to compile CoffeeScript without problems (because files are compiled according to 1 to 1), but this would cause problems when interacting with API preprocessors like SASS (which allows
@import
, which allows you to compile n files in 1).
However, Broccoli is designed in such a way that solving tasks for preprocessors like SASS (n: 1) does not cause problems, and tasks for preprocessors like CoffeeScript (1: 1) are easily solved as a special case n: 1. In general, for such (1: 1) transformations, we have the
Filter
class, which makes it as easy as possible to use them in our solutions.
3.2. Plugins just return new trees
First, I designed Broccoli with two primitives:
tree
(hereafter referred to as "tree"), which represent file directories and
transform
(hereinafter
transform
to as "transformation"), which takes the tree as input and returns a new, compiled tree at the output, after transformations.
This implies that we are converting trees 1: 1. Surprisingly, this is not always a good abstraction. For example, in SASS there are “download paths” that are used to find files when using the
@import
directive. On a similar principle, concatenators like r.js work: it has the “paths” option, which is responsible for searching for imported modules. The best way to represent such paths is a set (data structure) consisting of trees.
As you can see, in the real world, many compilers / preprocessors assemble n “trees” into one. The simplest way to follow this approach is to allow plugins to deal with their input “trees” themselves, thereby allowing them to take 0, 1, or n “trees” as input.
But now that we have decided that the plugins themselves will process their input trees, we no longer need to know about compilers as first-order objects. Plugins simply export a function that takes from zero to n source trees (and possibly some settings), and returns an object representing the new tree. For example:
broccoli.makeTree('lib')
3.3. The file system is our API.
Remember that due to the fact that Grunt does not support the use of "chains" of plug-ins, we have to bother with temporary directories for intermediate results of assemblies, which makes our configs too large and heavily supported.
To avoid this, the first thing that comes to mind is to bring the file system view into memory, where our trees will be presented as a collection of threads. Gulp does that. I also tried to implement this approach in earlier versions of Broccoli, but it turned out that the code became quite complicated: with threads, plugins had to keep track of queues and deadlocks. Also, speaking about streams and paths: we may need attributes like the last modification time or file size. Or, for example, if we need the ability to read the file again or find something, display the file in memory, or, eventually, if we need to transfer the input tree to another process through the terminal - here our API for working with streams will not be able to help, and we have to first write the whole tree to the file system. Too hard!
But wait, if we are going to copy every feature of the file system, and sometimes pull out our trees from memory and distill them into a physical representation, then put them into memory again, and again ... Why not just use the file system instead of streams?
The
fs
Node.JS module already provides the required file system API for us - everything we can only wish for.
The only inconvenience is that we will have to work with temporary directories "behind the scenes", and then we will have to clean up. But, in fact, it is not as difficult as it seems.
People sometimes worry that writing to disk is slower than into memory. But even if you take a real hard disk, the throughput of modern SSDs becomes so high that it can be compared with the speed of the CPU, which means that we get only minor overheads.
3.4. Caching instead of partial rebuild
When I tried to solve the problem of incremental builds, I tried to develop a way to check which files to recompile to allow Broccoli to trigger this event only for a subset of the source files. With an incremental build, we need to know which source files the result depends on, because often we are confronted with n: 1 relations. “Partial build” is a classic Make approach, just like the Rails asset pipeline, Rake :: Pipeline or Brunch, but I personally decided for myself that these are unnecessary difficulties.
The Broccoli approach is much simpler: we ask all plugins to cache the output. When we rebuild the project completely or when we restart individual plugins, most of the information will be taken from the cache of the plugins themselves, which will take meager time.
At first Broccoli provided some caching primitives, but later it was decided to exclude them from the core API. Now we just limit ourselves to providing an architecture that does not interfere with the implementation of caching.
For plugins that are 1: 1 mapped, such as CoffeeScript, we can use a common caching mechanism (represented in the broccoli-filter package), leaving the plug-in code very simple. Plugins that are built n: 1, such as SASS, require more careful care about caching, so they need to implement special logic to work with the cache. I believe that in the future we will still be able to isolate some common part of caching logic.
3.5. "No" parallelism
If we all suffer from slow builds, maybe we should try performing tasks in parallel?
My answer is
“no” : parallel launch of tasks makes it possible to have problems with the order inside the plug-ins, which you can overlook until the deployment. These are the worst of the bugs, so moving away from a parallel launch, we protect ourselves from the whole category of bugs.
On the other hand, the Amadal law stops us from gaining great performance when using parallel launches. Let's take a simple example: our assembly takes 16 seconds. Imagine that 50% we can run in parallel, and the rest should be run in the order of the queue (a la coffee-> concate-> uglify). If we run such an assembly on a quad-core machine, the assembly will take 10 seconds: 8 seconds for the synchronous part, and 8/4 = 2 seconds for the parallel part. As a result, the build time is still as much as 10 seconds, and this is only + 40% of the performance.
For incremental builds that interest us most of all, caching minimizes most of the advantages of running parallel tasks, so we barely lose in performance.
That is why I believe that the implementation of the parallelization of the assembly is not worth its advantages. In principle, you are not prohibited from writing a plugin for Broccoli, which would provide the opportunity to parallelize some of the tasks of your build process. However, Broccoli primitives are also designed in such a way as to allow the most convenient to design plug-ins that will be run in a deterministic sequence.
4. Behind the scenes / General view
At the core of my decision to write a good build system are two reasons.
The first reason is improved performance, which is achieved by incremental builds.
In general, I believe that the developer’s productivity is determined by the quality of the libraries and tools he uses. The cycle “edited, reloaded page” we repeat every day several hundred times, and this is probably the main way to get feedback. And if we want to improve the performance of our tools, we need to make this cycle as fast as possible.
The second reason is in supporting the ecosystem of front-end packages.
I believe that Bower and the modular ES6 system will help us build a great ecosystem, but Bower itself is useless unless you build an assembly system over it. That is why Bower is an absolutely unattached tool that allows you to load all the dependencies of your project (recursively, along with their dependencies) into the file system - this is all that can be done with it. Broccoli, on the other hand, aims to become just that missing link - a superstructure in the form of an assembly system under which it will work.
By the way, Broccoli itself is in no way connected with Bower or ES6 modules - you can use it with whatever you want. (I know that there are other bundles, for example npm + browserify, or npm + r.js.) I’ll touch on them in one of my next posts.
5. Comparison with other build systems
If I’ve almost convinced you, but you’re still interested in how other build systems behave in comparison to Broccoli, then let me explain why I wrote Broccoli instead of using one of the following:
Grunt is a tool for running tasks, and it has never been positioned as a system for building. If you try to use it as an assembly system, you will quickly be disappointed, because it does not provide the ability to use chain calls (composition), and you will quickly get tired of dealing with temporary directories, and in the meantime the configuration file will grow and grow. Also, it cannot provide reliable incremental builds, so your re-assemblies will be slow and / or unreliable, see “Fast Rebuild” above.
Grunt was created as a tool for running tasks that allows you to get the functionality of shell scripts on any platform, such as scaffolding or deploying your application. In the future, Broccoli will be available as a plugin for Grunt, so you can call it directly from your Gruntfile.
Gulp is trying to solve the problem with the sequential invocation of plug-ins, but, as for me, it suffers from certain errors in architecture: despite the fact that all his work revolves around the "trees", they are implemented by a sequence of files. This works fine when one source file is converted to one destination file. But when the plugin is required to follow
import
directives, and this requires accessing files out of turn, the work becomes more complicated. Now, plugins that use
import
directives suspend execution of the build and read the necessary files directly from the file system. In the future, I heard that libraries will be used that will allow all threads on the virtual file system to run and pass them to the compiler. I believe that all these complications are a symptom of a complete discrepancy between the build system and the compiler. You can once again read the section "Trees, not files", where I stopped in more detail on this issue. I'm also not at all sure that by abstracting from files to streams or a buffer, we will get a more convenient API; see "The file system is our API."
Brunch , like Gulp, uses an API based on files (rather than trees). Just like in Gulp, plugins, after all, bypass the build system when they need to get more than one file. Brunch also tries to perform partial rebuild instead of caching, see “Caching instead of partial rebuild” above.
Rake :: Pipeline is written in Ruby, which is less ubiquitous than Node in the world of front-end development, and it also tries to do partial builds. Yehuda Katz said that the system is not very actively supported, and he puts on Broccoli.
The Rails asset pipeline also uses partial assemblies, and, moreover, uses very different approaches for development and production, which can lead to very unexpected errors at the time of deployment. But more importantly, it requires a ROR on the backend.
6. What's next
The list of plugins is still small. But if this is enough for your needs, I would highly recommend giving Broccoli a chance:
https://github.com/joliss/broccoli#installationI would like to see how other people are involved in the development of plug-ins. Compiling compilers is quite simple, but the most important and difficult thing is to achieve proper caching and minimal loss in speed. We want to highlight more caching patterns to reduce duplicate code in plugins.
In the next couple of weeks, my plans include improving the documentation and cleaning up the Broccoli kernel code and plugins. We also want to add tests for the Broccoli core and provide an elegant solution for integration tests for plug-ins. Also, in our existing plugins there is no support for source maps. This is very expensive in terms of performance, because plugins, when making a sequential call, have to take the Source Maps of the previous plug-in and correctly interpolate them, but I haven’t yet found the time to do it.
Soon you will be able to see the active use of Broccoli in the ecosystem of the Ember framework, which will provide the default ember-cli stack (it will appear soon, its functionality is similar to rails command line). We also hope to replace Rake :: Pipeline and Grunt during the Ember core build process.
I would also very much like to see Broccoli adapted for projects outside the Ember community. JS MVC applications written using frameworks such as Angular or Backbone, various JS and CSS libraries that require assemblies are the main candidates to be compiled using Broccoli. Using Broccoli for real build scripts, we need to make sure its API is reliable, and I hope that in the next few months we will be able to release the first stable (1.0.0) version.
This post is the first post detailing the Broccoli architecture, so there is still little background information / documentation. I will be happy to help you get started, and fix any bugs you encounter. You can find me on #broccolijs on Freenode, or by writing me a mail / calling Google Talk: joliss42@gmail.com.
Github.
Jonas Nicklas, Josef Brandl, Paul Miller, Erik Bryn, Yehuda Katz, Jeff Felchner, Chris Willard, Joe Fiorini, Luke Melia, Andrew Davey, and Alex Matchneer .
From translator
z6Dabrata .