Delayed array processing

Once, when developing a complex application in JavaScript, an insurmountable problem arose - a large array in Internet Explorer (hereafter IE). Since I initially checked the work in Chrome, I did not notice any problems when using arrays. I will say more - the nested arrays did not cause fear. Chrome easily coped with difficult tasks. However, IE, in a very rigid form, pointed out all my flaws and shortcomings of the code, especially the processing of large arrays.

In a nutshell

The developed application had to, in real time, respond to events coming from the server. For these purposes, they used the Socket.IO library, which made it easy to implement two-way communication with the server. At the same time, the application was also tied to a third-party service, to which I had access only through the external REST API.

The weak link turned out to be a large cycle that was supposed to process an array of data received just from a third-party service. In this case, it was necessary to simultaneously change the DOM model in a loop. In these cases, IE went into a state of deep thought and naturally tore the connection. Superficially, it was unacceptable - locking the application and constantly spinning the download icon could not satisfy the customer. Yes, many can rightly point out the high costs of changing the DOM model. And I must assure you that this part of the problem was solved by moving all changes out of the cycle. However, the main problem is that the connection was broken during array processing.

Full refactoring was scary because thousands of lines of code had already been written. In addition, time is running out. It was in such conditions that the idea of ​​a “delayed”, partial processing of an array was born.
')

Algorithm

The array is logically divided into several ranges, the processing of each of which is done with a delay. The essence of the work is presented in the figure:



Thus, having a single global array, the Array function processes it in parts, with a predetermined delay. For asynchronous language, which is JavaScript, this technique allows you to unload the main stream (just one) when processing very large arrays.

The main body of the handler:

//Lazy array processing /** * Input parameters * params = { * outerFuntion : function(array, start, stop), * mainArray: [] * startIndex: Int, * stepRange Int, * timeOut: Int, * callback: function() *} */ var lazyProcessing = function(params) { var innerFunction = (function(_) { return function() { var arrayLength = _.mainArray.length, stopIndex = _.startIndex + _.stepRange; if(arrayLength < stopIndex) { _.outerFunction( _.mainArray, _.startIndex, arrayLength - _.startIndex); if(_.callback != undefined) { _.callback(); } return; } else { _.outerFunction( _.mainArray, _.startIndex, stopIndex); _.startIndex += _.stepRange; lazyProcessing(_); } } })(params); setTimeout(innerFunction, params.timeOut); }; //Outer function works with mainArray in given range var func = function(mainArray, start, stop) { //TODO: this should be everything you want to do with elements of for (var i = start; i < stop; ++i) { // do something } }; 

In the body of the lazyProcessing function, a local function called innerFunction is created, which closes in itself the obtained params parameters. This allows each time, when recursively calling the lazyProcessing function from itself, to save unique parameters.

The innerFunction function returns a nameless function that performs the following fairly simple operations:

1. Checks for reaching the end of a global array
2. Calls the outermost outerfuntion function with different stop values.
3. In case of reaching the end of the array, it calls the callback function.
4. Otherwise, it will recursively call lazyProcessing .

The outer function outerFuntion is passed to the array itself (due to its global nature , this could not be done, but visibility will suffer as well), as well as the indices of the beginning and end of the cycle. In this case, the processing results can be either stored in an array or in other global variables. It all depends on the current task.

When the end of the array is reached, a callback function may be called, but this is optional.

Pros and cons

Naturally, this solution has its pitfalls and disadvantages:

1. If you set the stepRange to be minimal, with a sufficiently large mainArray array, you can tritely "fall" over the stack overflow
2. The thread will still be blocked when calling the outer outerFunction. Those. performance will directly depend on the algorithm for processing the elements of the array
3. “Noodles” of nested and returned functions does not look very friendly

At the same time, partial processing of an array at regular intervals does not block the program execution flow. That allows you to handle other callback functions.



Ps. The user zimorodok took off a wonderful example - the same in spirit and in essence. I can not add it.
Passing through an array with a callback for each element with the ability to set timeOut:

 /* example of use var arr = ['masha','nadya', 'elena']; iterate_async(arr, function (el, index, arr) { console.log(el + ' is #' + (index + 1)); }, 99); */ function iterate_async (arr, callback, timeout) { var item_to_proceed; item_to_proceed = 0; (function proceed_next () { if (item_to_proceed < arr.length) { setTimeout(function () { callback.call(arr, arr[item_to_proceed], item_to_proceed, arr); item_to_proceed += 1; proceed_next(); }, timeout || 50); } }()); }