Build projects using Gulp.js. Yandex Workshop

Hi, my name is Boris. I work in Yandex in the testing department and create tools that allow us to make the life of our testers easier and happier. Our team is partly research, so we can afford to use rather unusual tools and experiments. Recently, I told my colleagues about one of these experiments: Gulp.js. Today I would like to share this experience with you.







First, a little background, about how web technologies developed. In the beginning there was no frontend as a separate concept, most of the logic was performed on the server. Therefore, a variety of tasks for the assembly of scripts and styles, as well as the preparation of images, fonts and other resources were performed by the backend, and their collectors, for example, Apache Ant or Maven. The frontend was at a disadvantage, the tools provided by these collectors were not very suitable for him. This problem began to be solved only recently, when Grunt appeared. This is the first collector written in JS. Every fronder knows JavaScript, so he can easily write tasks under Grunt and understand already written ones. This led to the success of this collector. Grunt has a lot of advantages, but there are also disadvantages.



For example, this is what the simplest Grunt file looks like.

')



We have a task, for its performance plugins are used. If we need more action, we connect more plugins. As a result, we get a huge sheet of code in which nothing can be found. And since the Grunt-file is large, the build becomes prohibitively long. And how to accelerate it is completely incomprehensible, because in the Grunt architecture there are no ways to do this.





Therefore, the only way out is to try to start everything from the beginning and enter from the other side. Well, for starters, you can see what is already useful in the outside world. For example, there is a UNIX shell. It has a useful concept - pipeline - the direction of exhaust of one process to the input of another process, and it can send its next and so on along the chain.

 $ cat *.coffee \ | coffee \ | concat \ | uglify \ > build/app.min.js 

Thus, we can build a real conveyor that will perform our assembly. This is pretty damn logical to do assembly on the pipeline. This also applies to frontend tasks. However, if you do this on a pure shell, there may be some problems. Firstly, not every operating system has a shell, and secondly, we have no commands that, for example, make the conversion of coffee to JS.



But this can make Gulp. This utility is written in JavaScript. It uses the same principle as the Shell script, but the pipe() function is used instead of the vertical bar here.

 gulp.src('*.coffee') .pipe(coffee()) .pipe(concat()) .pipe(uglify()) .pipe(gulp.dest('build/')) 

Those. we can do exactly the same thing, while it is clear that what we do, if necessary, we can change blocks of places, delete and generally configure our conveyor as you like.



Gulp is already quite stable, has developed to the third version and has found its fans. It is installed in our favorite way:

 npm install -g gulp 

I decided to test it on one of my projects and was surprised to find that the build with it runs a little faster than with Grunt. And now I will try to explain why.







The thing is that the most expensive operation at the time of assembly is access to the file system: the assembly takes place in the processor, the file system is somewhere far away, you need to go to it, and it takes some time. The diagram shows the red arrows as these operations. It can be seen that there are only two of them in Gulp (read at the entrance, recorded at the output), and at Grunt - four: each plugin reads and writes. Well, since everything works faster, why not switch to Gulp. But first I decided to check everything carefully. I prepared a test case in which coffee files and styles are assembled and packaged, described this task for Grunt and Gulp, ran them one by one and saw that there was indeed a gain, gulp was about a quarter faster: 640 ms versus 850. I also prepared another test, a little more complicated. In it, we need to still slightly style the styles. Most styles, of course, in bootstrap. Let's try to build it from the original less-files, and then, to reduce its size, walk through CSSO. In Gulp, this is done quite easily: there is a plugin for both less and csso .

 var gulp = require('gulp'); var csso = require('gulp-csso'); var less = require('gulp-less'); gulp.task('default', function() { return gulp.src('bower_components/bootstrap/less/bootstrap.less') .pipe(less()) .pipe(csso()) .pipe(gulp.dest('dest/')); }); 

Grunt-file turns out more.

 module.exports = function(grunt) { require('time-grunt')(grunt); grunt.loadNpmTasks('grunt-contrib-less'); grunt.loadNpmTasks('grunt-csso'); grunt.initConfig({ less: { compile: { src: 'bower_components/bootstrap/less/bootstrap.less', dest: 'dest/bootstrap.css' } }, csso: { compile: { src: 'dest/bootstrap.css', dest: 'dest/bootstrap.min.css' } } }); grunt.registerTask('default', ['less', 'csso']); }; 

As a result, Gulp won again: 2 seconds against 2.3. Grunt spent 300 milliseconds reading and writing unnecessary files.



There are not as many plugins for Gulp as for Grunt, but the 400 that are are enough for typical tasks. Well, if you still lack something, you can always write your own. The main idea of ​​Gulp is streams. They already exist in the core node.js, for this you do not need to connect anything. Consider a small example: a plugin that will greet everyone. We give him a word, and he greet us:







This is how it will look in JavaScript:

 var stream = require('stream'), greeterStream = new stream.Transform({objectMode: true}); greeterStream._transform = function(str) { this.push('Hello, '+str+'!'); }; greeterStream.pipe(process.stdout) greeterStream.write('world'); // Hello, world! greeterStream.write('uncle Ben'); // Hello, uncle Ben! 

We have a ready-made native object in which we need to define the _transform method. He is given a string at the entrance so that we process it and return it. We write to him, and he converts. You do not need to connect anything, this is the node.js native API. To see how all this is embedded in the Gulp, remove the cover from it and look inside. There we will see two modules: Orchestrator and Vinyl fs. Orchestrator conducts streams, builds them in a queue, tries to perform them with maximum parallelism, and generally ensures that everything works like an orchestra. With Vinyl everything is a little more interesting. A stream is a data set, and we collect files. It is more than just data; it is also a name, an extension, and other attributes. I would like to somehow separate the continuous stream into separate files. Vinyl does all that. In essence, this is a wrapper over files: we receive not just data, but objects. Vinyl puts all the required fields there. We can modify and manage them.

 var coffeeFile = new File({ cwd: "/", base: "/test/", path: "/test/file.coffee" contents: new Buffer("test = 123") }); 

Every plugin does this, for example, gulp-freeze , written by me specifically to show how easy it is. It is designed to freeze static. In Gulp, this is all done very simply: we have the content, we calculate the md5-hash from it and say that it is the file name. Then we write the file further to the stream. Gulp will do the rest of the operations for us: read the files, give them to our plugin, then pass them on to the rest of the plugins and eventually write them to the file system. And we write only the most interesting, our plugin.

 var through = require('through2'); module.exports = function() { return through.obj(function(/**Vinyl*/file, enc, callback) { var content = file.contents.toString('utf-8'), checksum = createMD5(content), file.path = checksum; this.push(file); callback(); }); }; 

And since we have nothing superfluous, the test is pretty simple. Let's create a test stream in which we put the fake data, and we can not even use the file system. If we write a large plugin, and CI will be configured for it, for example, Travis, we will be pleasantly surprised at the speed of the build. For all test cases, you can generate virtual files, write them to the stream and listen to the output. If the output is correct, everything is fine, the test is passed, if not - we have a mistake, we go to correct it.

 var freeze = require('./index.js') var testStream = freeze() testStream.on('data', function(file) { //assert here }); testStream.write(fakeFile); 

Sometimes it is not even necessary to write a plugin. Some functions can be inserted directly into the stream. For example, no one has yet written a Gulp plugin for the Yate template engine. But we can call it directly:

 var map = require('vinyl-map'); var yate = require('yate'); gulp.src('*.yate') .pipe(map(function(code, filename) { //        return yate.compile(code.toString('utf-8')).js; })) .pipe(gulp.dest('dist/')) 

There are more exotic applications of such a system. For example, this picker can be replaced by Jekyll. Suppose we have articles in markdown, and we collect web pages from them in HTML. This scheme ideally fits into the ideology of Gulp, with its help you can collect Jekyll-templates. To do this, you just need to read our posts, process them, you may have to write a couple of small plug-ins, and as a result get a full-fledged Jekyll port on node.js. Very comfortably. It seems to me that in Grunt it is impossible to do this in principle.

 gulp.task('jekyll', function() { return gulp.src('_posts/**') .pipe(liquid.collectMeta()) //    .pipe(post()) //    .pipe(gulp.src('!_*/?*')) //   .pipe(markdown()) //  html,   .pipe(liquid.template()) // .pipe(gulp.dest('_site/')); //  }); 

PS The report was told in the spring of 2014, over the past six months, tools have evolved, something could have changed, but the basic idea remains the same.