Asynchronous JavaScript Programming - Staying Alive

Programmers take certain features for granted - sequential programming, for example, when writing an algorithm that takes one step only after the other.

However, if you are writing JavaScript code that uses blocking I / O or other lengthy operations, sequential coding is out of the question, since blocking a single execution thread in the system is a very bad idea. The solution is to implement algorithms using asynchronous callbacks, that is, to split the sequential code into several callbacks.

This solves the problem, but means that we lose the ability to write a sequential algorithm, and a nontrivial sequential code is converted into a graph of callback functions.
')
This is becoming even more critical for large scale applications that make extensive use of asynchrony. Using the transfer of callback functions for asynchronous actions is not very convenient and can create complex processes for transferring callbacks due to the need to handle return values.

The JavaScript community is aware of this, especially the Node.JS community, since Node.JS focuses on asynchronous code.

The CommonJS team responded to this call in the form of Promises , which are aimed at providing an interface for interacting with an object that is the result of an action that is performed asynchronously and that may or may not be completed at any given time.
Thus, various components can return promises for asynchronous actions, and consumers can use promises in a predictable way. Promises can also provide a fundamental entity to be used for a more convenient syntactically level of language extensions that help to work with asynchrony.

Stratified JavaScript is another approach that suggests simplifying programming with a superset of the JavaScript language. But if you cannot change programming languages, you can use a flexible API that allows you to emulate sequential code. If this API allows the use of short notation, it is often called an embedded DSL.

InfoQ decided to take a look at the list of these APIs and DSLs, and talk with their creators about how they approached the problem, the design principles, the paradigms they follow, and much more. And, of course, about the limitations of these solutions.

In particular, InfoQ has contacted:

• Tim Caswell of Step ,
• Will Conant from Flow-js ,
• Kris Zyp from node-promise ,
• Caolan McMahon from Async ,
• Fabian Jakobs from Async.js ,
• AJ O'Neal from FuturesJS and
• Isaac Z. Schlueter from slide-flow-control

InfoQ: What problems does your library focus on? That is, it focuses mainly on removing template code, avoiding manual processing of asynchronous I / O, or it also provides conducting or other functionality (for example, to help with the simultaneous processing of several I / O calls with the expectation of the results of their execution, and so on).

Tim (Step) : Step’s goals are to remove both the template code and the more readable asynchronous code. Our library is very minimalistic, and does not do anything that you can not repeat manually with the abundant use of try..catch blocks and variable counters. A feature of our library is the simple formation of chains of sequential calls with optional groups of parallel calls at each step.

Will (Flow-js) : Flow-JS provides a JavaScript construct that is similar to a continuation or fiber that exists in other programming languages. Practically, it can be used to destroy the so-called "pyramids" from your multi-step asynchronous logic. Instead of directly using the literals of nested callback functions, you use the special value “this” as the callback function for the next function specified in the definition of the execution thread.

Kris Zyp (node-promise) : The problem is that the typical style of the flow of callbacks (transmission of continuity) combines the complexity of the interface and the mixing of functional parameters with the result handlers. Promises encapsulate the event completion of the calculation, allowing functions / methods to be executed with clean input parameters, while the return value, like a promise, stores the result.

I explained these principles a little more here and here .

Encapsulating event-based computations, promises work fine for conducting parallel and sequential actions, even with complex conditional logic. The node-promise library includes functions to do this simply (all () and step () functions at promised-io)

By the way, just for the information, promised-io is actually the heir of node-promise. It has the same core, but promised-io also includes promise-style NodeJS I / O functions, and a platform normalization layer is available that provides access to similar functions in the browser.

Caolan (Async) : Yes, the main task is to remove the template code. The code for calling functions in series or in parallel, followed by waiting for callbacks in JavaScript, is quite detailed, but obvious. Shortly after node.js moved from promises to callbacks as the underlying asynchronous processing mechanism, I discovered that I was using the same templates over and over again. It seemed obvious that they needed to be extracted into a separate library.

Since then, it has grown to cover more sophisticated features that allow you to conduct callbacks based on their dependencies. For the most part, however, this is a fairly low-level library that leaves the overall structure to developers. Nevertheless, I found that JavaScript is often suitable for a more functional programming style, and immediately added asynchronous versions of map, reduce, filter, and other common functions. The library really shows its strength when used in this style, and allows you to stick with ordinary callbacks without using sequels or promise objects.

Fabian (Async.js) : Async.js tries to simplify standard asynchronous templates in javascript. Its main purpose is to apply a series of asynchronous functions to a set of homogeneous objects. It grew from a forEach asynchronous function into a set of generic concepts. This is especially useful for asynchronous work with the file system in node.js, although the library is not tied to node.js and can be used for any other similar cases. This piece of code shows the use of async.js:

async.readdir(__dirname) .stat() .filter(function(file) { return file.stat.isFile() }) .readFile("utf8") .each(function(file) { console.log(file.data); }) .end(function(err) { if (err) console.log("ERROR: ", err); else console.log("DONE"); }); 

This code operates with all elements of the current directory. This is a uniform set of objects. For each item in the catalog, a sequence of asynchronous operations is performed. First, all elements that are not files are filtered out, then the contents of the files are read from the disk in utf-8 encoding and printed to the console. After all the operations have been completed, the final callback function is called with the error indicator parameter.

AJ (FuturesJS) : It’s pretty hard to talk about asynchronous and event-driven programming.

I created Futures mainly for:

• providing one library of asynchronous control flow for both browser and server (Node.JS);
• publishing a quality template for handling callbacks and error handler calls;
• control the flow of application execution, in which events depend on each other;
• handling callbacks for a variety of resources, such as mash-ups;
• encouraging the use of advanced programming practices, such as model use and error handling.

Futures.future and Futures.sequence simply reduce the number of boilerplate code and provide some flexibility.

Futures.join can connect (in the same manner as join works for operating system execution threads) or synchronize (for events that occur periodically) several future objects.

Futures.chainify makes it easy to create asynchronous models, similar to the Twitter Anywhere API.

Isaac (slide-flow-control) : The task to which the slide is dedicated is that I needed something I could tell about at the OakJS meeting, and did not want to come up with any new idea, because I am very lazy. Basically, I wanted to do almost no work, just show what was done, drink some beer, enjoy Chinese cuisine, chat with interesting people, swim a little in positive attention, and then return home. The ratio of the amount of work to the win received is very important for me, both in software and in life. So, I just put together terribly simple asynchronous helper methods that I use at npm so that they fit into a set of slides, called it “slide” because of this feature, and introduced this library.

Another task this library is dedicated to is to show how easy it is to write your own library of flow control. Everyone thinks that his own library is the best, so it’s important to just give people a few basic patterns and let them create.

InfoQ: Does the library implement ideas from computer science scientists?

Tim (Step) : Directly - nothing.

Will (Flow-js) : Not that I know. It was just my first blow to the creation of business logic, which simplifies the execution of many synchronous calls to external services managed from Node.js.

Kris Zyp (node-promise) : Yes, of course. Most computer-aided research on asynchronous design points to Promises (promises) in various forms, as the most appropriate mechanism for functional flows and proper division of interests. The term “promise” was originally proposed by Daniel P. Friedman and David Wise, beginning in 1976. You can learn more about the rich history of computer science regarding promises from a Wikipedia article .

Caolan (Async) : I have no experience in the field of computer science, and I implemented the Async library on a purely pragmatic basis. When I needed a higher order function to clean up a bit of asynchrony in JavaScript, and I used it repeatedly, it went into the library.

Fabian (Async.js) : The implementation of async.js is vaguely reminiscent of Haskell Monads , but this is rather random.

AJ (FuturesJS) : Yes. The following materials had the greatest impact:

• Google's How to think about OO
• Yahoo! S Crockford on JavaScript Act III
• Google's Lazy Programmers Guide to Secure Computing

The best thing about asynchronous programming is that it naturally forces you to write more modular code, and if you have asynchronous models of any kind, then you have to follow the principle of always passing parameters inside the model and never transmit data that relates to the model. , beyond this model.

Isaac (slide-flow-control) : It uses one type of continuation pattern. A lot of computer science research misses this topic, I think. This is my finger pointing to the moon. To get there, you need a rocket. Longer fingers will not help. When you realize this, deeper mysteries will reveal themselves.

InfoQ: Does the library offer any error handling strategies? How does it interact with throwing exceptions?

Tim (Step) : If an exception is thrown at any step, it is caught and passed to the next step as an error parameter. Also, any non-undefined return values ​​are passed to the next step as a callback parameter. Thus, the steps can be synchronous or asynchronous using the same syntax.

Will (Flow-js) : Flow-JS does not have any built-in exception handling, which is definitely its weak point. Tim Caswell wrote a flow-based module called Step, which frames the calls to each of the provided functions in try / catch blocks and passes caught exceptions to the next function in sequence.

Kris Zyp (node-promise) : Yes, promises are designed to provide the asynchronous equivalent of a synchronous execution thread. As a JavaScript function can throw an exception or return a value successfully, a promise can be resolved to a successful value or to an error state. Promises returned to callers can propagate errors until the error handler “catches” them, just as the thrown exception propagates until it is caught. The node-promise library supports this concept shortly, allowing you to easily register error handlers or propagate errors until they are caught (to eliminate the situation of silently swallowing errors). Having direct synchronous equivalents of promises, the flow of code execution using promises is very easy to read.

Caolan (Async) : Exception handling with asynchronous code can be a bit more complicated, especially if you are not familiar with highly asynchronous environments such as node.js. For me, in fact, error handling is more important than the style that you adapt to this processing. In the browser, this is especially important because it is easy to accidentally raise the exception to the top level, thereby killing all the JavaScript on the page.

There is much that can be said about this. Exception handling should be straightforward and preferably familiar, so that it is easy to implement and it becomes obvious if you forget about it. Unfortunately, JavaScript in the browser does not help us in this sufficiently yet, but node.js provided a simple convention that can be easily used in both environments.

The Async library adopted this convention using the first callback argument to pass the error to the next step in your program. If the first argument is null (or another false value), then it can be ignored, otherwise it is considered as an exception. Where possible, the implementation will be carried out by the Async library using an abbreviated scheme to speed up the process. If one function from the executed collection is executed with an error, the subsequent functions in the collection will not be executed.

Fabian (Async.js) : Async.js is built using the node.js error handling convention. The first argument to any callback function is reserved for the error object. If the evaluation ends in an error, or an exception is thrown, the error / exception is passed as the first argument to the callback function. Async.js supports two error handling strategies that can be configured through its API. In the event of an error, either the whole set operation is stopped, and the error handler is called, or the error element is skipped.

AJ (FuturesJS) : Since exceptions cannot be thrown asynchronously, the user is instead prompted to pass any exceptions as the first parameter to the callback function.

The basic idea is to execute try {} catch (e) {} for an error, and pass the error instead of stopping the application at some indefinite point in time. Futures.asyncify () does this in order to use synchronous functions in the dominant asynchronous environment.

Here is an example:

 (function () { "use strict"; var Futures = require('futures'), doStuffSync, doStuff; doStuffSync = function () { if (2 % Math.floor(Math.random()*11)) { throw new Error("Some Error"); } return "Some Data"; }; doStuff = Futures.asyncify(doStuffSync); doStuff.whenever(function (err, data) { if (err) { console.log(err); return; } console.log(data); }); doStuff(); doStuff(); doStuff(); doStuff(); }()); 

Isaac (slide-flow-control) : Never throw exceptions! Never! Throwing an exception is evil. Do not do this. When the callback function is called, the first argument is either an error or null. If this is an error, process it, or pass callback functions to handle it. Send the error to the first parameter of your callback function to signal the appearance of an error.

InfoQ: Was there a mastermind or under what influence your library was created (for example, F # Workflows, Rx (Javascript version), or other projects)?

Tim (Step) : Yes, the style was borrowed directly from the project flow-js .

Will (Flow-js) : Not really. It was just the first decision that occurred to me.

Kris Zyp (node-promise) : The Mark Miller E programming language and the use of promises in it, the Tyler Close ref_send library, the Kris Kowal Q library, the Neil Mix NarrativeJS library, in the Twisted and Dojo, and many other libraries.

Caolan (Async) : I’m afraid that since I didn’t use F # or Rx, I can’t express my attitude to these projects. However, I drew inspiration from Underscore.js, an excellent functional library for programming in JavaScript. Most of the functions that use iterators from Underscore were modified to start working asynchronously with callbacks, and implemented in the Async library.

Fabian (Async.js) : The characteristic call chains in the API were inspired by jQuery. One of my goals was to provide a jQuery-like API for the node.js file system module. Python-style generators also had a great influence. Each element in the chain generates values ​​that can be used by subsequent elements of the chain. The whole operation is called the last element in the chain, which “pulls” values ​​through the chain. From this point of view, async.js differs from jQuery and Rx, where the values ​​are pushed by the source. This "pulling" system makes it possible to calculate all values ​​lazily, and also allows you to create generators that return an infinite number of values ​​(for example, all even integers).

AJ (FuturesJS) : Not immediately, no.

I built a mashup site using Facebook and Amazon, and my first attempt was a jumble because I simply did not understand how to work with a model created from two resources (I was not really familiar with JavaScript at the time; I was going by trial and error in the “WTFJS” style and used a bit of jQuery to simplify the painful work with the DOM).

So I found it easier to always assume that it would take some time to get any data than ever to assume that the data would already exist if necessary, and then I realized that I had to reorganize the whole chain from bottom to top to make any particular set data could be processed asynchronously.

I tried, failed, and half-done using several different methods, and then, to my luck, someone in the mailing list of my local JavaScript user group mentioned Crawford's lecture series on JS . After watching the entire series (I watched the third section at least 3 times), I finally began to better understand how to manage the “problems” (or, rather, the possibilities) of asynchronous programming. Next, I found Crawford's slides and started with an example of promises, which he cited, as from his starting point.

Later, I started playing with Node.JS, and as a result I changed my error handling strategy (but I updated the documentation only a few days ago). In Futures 2.0, which I will release this coming Sunday, I also added EventEmitter from Node.JS for use in the browser.

Isaac (slide-flow-control) : No It was, I believe, inspired by the templates that we came to NodeJS for using callbacks. I'm just a supporter of completeness, because I'm not smart enough to remember more than one kind of things (or maybe two, on a good day, with lots of coffee) without getting confused and going to the toilet, thinking that this is a bathroom. and then getting all the clothes smelling, like ... However, you understood the idea.

Use things of the same type, everywhere. That's all. Functions that take a callback as the last argument. Callback functions that receive an error message as the first argument, or null / undefined if everything went well. Slide is just a few helper functions that make it easier to do a bunch of things using this scheme.

InfoQ: Are there any new features or changes in the JavaScript language that can make the libraries better, for example, they can be more concise, etc.?

Tim (Step) : Perhaps, but not without major changes in the semantics of the language. Maybe a coffeescript preprocessor can help with syntax, but I think it’s better to stick with vanilla javascript most of the time.

Will (Flow-js) : In my opinion, Javascript desperately needs something like fibers from Ruby 1.9. I spent a lot of time working with Node.js for my future projects, but at some point asynchronous programming always makes the brain melt. There are many tools to make it more manageable, but I can’t prove it, but I feel that having so many libraries like Flow-js is only evidence that Javascript is really unsuccessful for parallel programming.

I know that one of the goals of Node was to avoid modifications in the core of the V8 JavaScript engine, but as far as I know, guys from Asana added fibers to the core without any special problems .

Kris Zyp (node-promise) : Yes, there has recently been a discussion around single-staff or small sequels similar to generators, which can help avoid the need for a callback, which complicate branching and thread loops. This can be used in conjunction with promises to create extremely simple and easy-to-read asynchronous code.

By the way, one more note - the node-promise library also implements the http://wiki.commonjs.org/wiki/Promises/A specification, that is, it can interact with Dojo and, probably, with future JQuery promises.

Caolan (Async) : The Async library was designed to make the most of the language in its current form. To work as is, without trying to create a new language over JavaScript.

However, adding yield to JavaScript 1.7 may have some interesting applications for future projects. Using yield, it would be possible to port some Twisted-like functions to JavaScript for a more synchronous-like coding style. This is what my colleague is studying with Whorl , although this project seems to have stopped developing.

Fabian (Async.js) : Standardizing generators and iterators supported by Mozilla can make the async.js code more concise and simplify asynchronous problems.

AJ (FuturesJS) : The most frequently repeated code in the library is the one that makes the code work the same in the browser and in Node.JS. I know that some libraries (such as teleport) were created to try to resolve this issue, but I have not played with any of them yet. This, of course, would be nice if asynchronous require was embedded in the language.

From my point of view, a language with such natural asynchrony, like JavaScript, should have something similar to futures built into the core of the language.

Although CommonJS has several proposals for standardizing server promises, while their focus is more on data privacy, futures are more focused on flow control, ease of use by end-developers, and browser compatibility.

Isaac (slide-flow-control) : No My flow control library is the best. It cannot be improved in any other way, because this best is directly related to my self, so any external influence would make it less mine, and thus less good. If you want to gain experience, I suggest you write your own library. You will see that it is the best, as soon as you write it. If it seems to any other library that something could be better, then you can rush back to the editor, hiding your shame, and quickly reinvent all your ideas, but a little differently than before, and then you will know in your heart that your new is now the best.