node-seq in a new way (again about asynchrony)

Hello, Habr! I am writing to you as a slop. Indeed, while spacecraft with vertical takeoff and landing travel through the oceans, and the most impatient are using features of ES6 in their projects, I brought you another library to facilitate the life of the asynchronist.

Of course, there are a million promises realized for a long time, and async for those who want it. There is also, well-known in narrow circles, the Seq library from the notorious fellow Substack. I started using it practically from the first days of my javascript and used it wherever I could. The approach proposed by this library seems to me more understandable and logical for curbing asynchronous noodles than the approach used, for example, in async. See for yourself:

var fs = require('fs'); var Hash = require('hashish'); var Seq = require('seq'); Seq() .seq(function () { fs.readdir(__dirname, this); }) .flatten() .parEach(function (file) { fs.stat(__dirname + '/' + file, this.into(file)); }) .seq(function () { var sizes = Hash.map(this.vars, function (s) { return s.size }) console.dir(sizes); }); 

Everything is so simple and clear that even explaining laziness. Unfortunately, the library has not been developed or supported for a long time. In addition, during the time I used it, I got a list of bugs that somehow stumbled upon, a list of features I wanted, a list of complaints - because not everything worked as well as in my imagination. Once, I once again stumbled upon the incoherence of the world and decided that this moment had come. It's time to fork and fix. I often do this, but what makes this library seem to me a real magic.

So happy and determined to do git clone, cd, gvim, and try to understand what's going on here. I do not understand. The author uses a couple of his libraries and for enlightenment it is necessary to deal with them first. After a couple of hours I get bored and I discover the Fatal Deficit. No sooner said than done. Sit down and write from scratch Library of Dreams. Oddly enough, there was no magic in all this. The prototype was ready for the evening. Then, for some time, I finished using it in a real project, replacing it with Seq completely. And so it turned out what happened. Let's get acquainted.
')
YAFF - Yet Another Flow Framework.

Overall, I tried to make it compatible with Seq. And most of the code was migrated simply by replacing the import (it was var Seq = require ('seq'); it became var Seq = require ('yaff');). Of course, I had to replace something else. Seq uses the .catch () method to catch fleas. For example, the above piece of code can be changed like this:

 var fs = require('fs'); var Hash = require('hashish'); var Seq = require('seq'); Seq() .seq(function () { fs.readdir(__dirname, this); }) .flatten() .parEach(function (file) { fs.stat(__dirname + '/' + file, this.into(file)); }) .catch(function (err)( console.error(err); )) .seq(function () { var sizes = Hash.map(this.vars, function (s) { return s.size }) console.dir(sizes); }); 

This construction is terrible because after we “caught” the error, we can continue. You can not do it this way. First, it is not clear what to do if parEach (or other similar methods) throw a few errors. Catch only the first? Catch all? And if we have already gone far down and in some parEach the error popped up on the timer? And if below we already have no catch? And if those catch that below are not prepared for error handling which will come out on the timer from forEach? There are many unanswered questions. Therefore, I decided that in each YAFF there should be only one construction for error handling and it should be at the end. And in order not to violate the nodejs traditions, let it process the result as well. It turns out beauty. Make sure:

 var fs = require('fs'); YAFF(['./', '../']) .par(function (path) { fs.readdir(path, this); }) .par(function (path) { fs.readdir(path, this); }) .flatten() .parMap(function (file) { fs.stat(__dirname + '/' + file, this); }) .map(function (stat) { return stat.size; }) .unflatten() .finally(function (e, sizes) { if (e) throw e; log(sizes); }); 

If we assume that this is all inside the asynchronous function, then in finally we can simply throw the callback provided to us, a la:

 var listDirs = function (dirs, cb) { YAFF(dirs) [ ] .finally(cb); }; 

Conveniently? I like it too. In general, this library is built on the concept of a stack-array of arguments (it was worth writing about it from the very beginning). And all the methods that are here anyway, this stack is changed by applying to it or to its individual elements. For example, a flock of functions wrapped in par will take one element from the stack, in the order in which these par are written and only after all par shoot callbacks (and the callback inside all this this methods) does YAFF go to what he has to do next the queue. Suppose then that we have several functions wrapped in seq. YAFF will apply the entire stack to them and replace it with something that returns a wrapped function to the callback. Here is the code to make it clear:

 YAFF(['one', 'two', 'three']) .par(function (one) { this(null, one); }) .par(function (two) { this(null, two); }) .par(function (three) { this(null, three); }) .seq(function (one, two, three) { this(null, one, three); }) .seq(function (one, three) { this(null, null); }) .seq(function () { this(null, 'and so on and so forth'); }) 

Of course, if someone calls his callback with a non-zero first argument (error), then YAFF will immediately spit on all the functions that still remain there, and go to do what is written in finally. If you forgot to write finally or thought that there could be no error in your error code, then YAFF, in case of an error, will unceremoniously throw an exception. So it is better to finally be.

Still, there are all sorts of synchronous functions for working with the stack of arguments as with an array: push, pull, filter, set, reduce, flatten, extend, unflatten, empty, shift, unshift, splice, reverse, save (name), load (fromName ). Fuf, like everything. The names speak for themselves, but if that - do not hesitate to ask and look at the source. There is one file ( main.js ) and everything is very simple.

And, of course, for the sake of which everything was started - asynchronous functions for processing data arrays: forEach (YAFF will not wait for processing of this block to end, the results of this block will not affect the stack. YAFF immediately goes to the next handler in the chain), seqEach , parEach (YAFF will wait until all functions are fired, but the results will not affect the stack). seqMap, parMap, seqFilter, parFilter - do what you expect; YAFF waits for them to work and the results of the work of these blocks replace those values ​​that were on the stack before. In addition, all methods with the par prefix after the function, you can specify a number. This number is the limit of simultaneously running asynchronous functions. Something like this:

 var resizePhotos(photos, cb) { YAFF(photos) .parMap(function (photo) { asyncReisze(photo.image, photo.params, this); }, 10) .unaflatten() .finally(cb); } 

In this example, we resynim a stack of photos asynchronously. In order not to overload the server, we resize no more than 10 pictures simultaneously for each client. unflatten is needed in order to collect photos spread over the stack into an array which will be one argument for callback.

YAFF also has mseq and mpar methods - a nod to async users. These methods accept an array of functions that will be executed sequentially or in parallel. With the same success, you could write a bunch of seq () and par (), but sometimes you want to generate functions dynamically. We still have a functional language, right?

In order to completely confuse you, I came up with the following example and drew a picture (in desperate hope that it will clarify everything):

 YAFF(['./']) .mseq([ function (path1) { fs.readdir(path1, this); }, function (arg) { this(null, arg); } ]) .flatten() .parMap(function (file) { fs.stat(__dirname + '/' + file, this); }) .map(function (stat) { return stat.size; }) .unflatten() .finally(function (e, sizes) { log(sizes); }); 

I hope that I was able to clearly explain what I wanted; you are imbued with the idea and I'm not the only one who does not understand why you need async.

All specific requests and suggestions are better arranged in the form of issues (in English, do not hesitate - I also do not know it well, we will train literature together) on a githaba or even in the form of pull requests .

Oh yeah, the library is on npmjs.org .

PS Just now, in a fit of passion, added the synchronous method apply — now all other synchronous methods can be thrown away. But I will leave for convenience and compatibility.