Multi-threaded file uploader on JS (jQuery)
Good day, colleagues. In this article I will describe the experience of creating a multi-threaded file loader (with limited server load) on JS (jQuery).
Most recently, I had a task (the solution to which I want to share with you): to do, in the admin area, the ability to select and download more than one file at a time. The task seems to be trivial and not difficult, but in the end my solution seemed rather interesting to me, since no analogues were found.
Since the task was simple, and, naturally, there are plenty of solutions already written, I turned to Google for help. But, the search for positive results did not work - most of the proposed solutions use Flash (which, due to some specifics, did not allow me to use such solutions) or the written libraries on JS were very huge and, most regrettably, non-working. I had to collect the bike.
The task was urgent, so an urgent solution was needed. Not for a long time, I think I screwed the multiple attribute to the field (available with HTML5).
')
This was followed by small changes in the handler to receive not one file, but an array of files - and the problem is solved! (naivety (inexperience) of mine was not chapel).
How many people, with a laugh, have already thought that five hundred thirds answered the first, normal, batch of nginx files. We had to think further.
Since the past decision was made beautifully and conveniently for admins, it was decided to build on it. It was necessary to solve the problem of error number 503, which was returned by nginx due to lengthy file processing.
Half a minute of thinking and a new solution appears: we will send ajax not all the files at once, but one at a time.
The solution, approximately, was as follows:
It's simple: we iterate over the array of files (which is in the right input), create an instance of the class for working with files (more on that later) and send the request to the server as ajax. It is worth paying attention to the " async: false " parameter - here we set the synchronous execution of the ajax request, since the asynchronous will create us a lot of requests to the server, which we can easily set it with.
The solution works, there is no error, but here’s one problem — it works slowly. And then I came up with the idea of ​​another solution to the problem.
To speed up the upload of files to the server, you can increase the number of requests in which files will be transferred. Such a decision to rest on the solution of two problems:
1). With a large number of requests I quickly go to hell with my server.
2) A large amount of simultaneously uploaded files will score us the whole channel.
Judging by the problems, our bootloader should count the number of running requests and the amount of data transferred and, under certain conditions, expect the end of some requests to start others.
Proceed to the code:
Legend:
dataArr - an array of data to send.
fStek - write timeout identifiers here to further stop recursion and clear memory of incomplete functions.
vStek - the number of threads invoked .
deley - recursion delay of the function that checks threads and volumes.
debug - debaby mode. It is necessary for debugging, but in this example I deleted all its signs.
maxFilesSize - the maximum amount of downloaded files
maxThreads - the maximum number of threads.
Full clarity in the variables (especially fStek and deley ) will be introduced by the second considered function FilesUploader.controller () . For now, let's move on to class initialization:
This is the function that handles the event of a click of a button in the form. The work of the function is simple: we run through the files ( jQuery.each ) entered in the input and add ( FilesUploader.dataArr.push (f) ) record about each to the array. Next, call the controller, which is the most important and most complex part of the system:
In the first line of the function, we asynchronously call (after a certain period of time) the same function (ie, create recursion), and put the identifier of the called function into a variable to be able to interrupt its execution.
Next comes the condition for checking threads.
After getting the file from the array ( FilesUploader.dataArr.pop () ), we check it for the presence.
1. If there is no file, then we “kill” the called functions by their identifier ( clearTimeout (FilesUploader.fStek.pop ()) );
2. If the file exists, we do a check for the size of the loaded files, and if it is exceeded, we return the file back to the stack and exit the function, otherwise, if it is not exceeded: we take the volume, increase the counter of the started threads and call the following function ( FilesUploader.worker ( item) ).
To send a file to the server using ajax, you need to put data about the file (file.append ()) into an instance of the FormData class.
Next, we call the $ .ajax function, which will transfer our file to the loader on the server. Upon completion of each request (the complete () function), you need to increase the allowable volume and reduce the number of executable threads (which is done in the “ FilesUploader.maxFilesSize + = this.fileData.size ” and “ FilesUploader.vStek— ” lines ).
And the final touch is the console output function and closing bracket:
That's all - the class for multi-threaded file upload to the server is ready. Next, you should set, depending on the server configuration, the permissible number of threads and the amount of simultaneously uploaded files - and you can work.
Most recently, I had a task (the solution to which I want to share with you): to do, in the admin area, the ability to select and download more than one file at a time. The task seems to be trivial and not difficult, but in the end my solution seemed rather interesting to me, since no analogues were found.
Decision number 1
Since the task was simple, and, naturally, there are plenty of solutions already written, I turned to Google for help. But, the search for positive results did not work - most of the proposed solutions use Flash (which, due to some specifics, did not allow me to use such solutions) or the written libraries on JS were very huge and, most regrettably, non-working. I had to collect the bike.
Decision number 2
The task was urgent, so an urgent solution was needed. Not for a long time, I think I screwed the multiple attribute to the field (available with HTML5).
')
<form id='FilesupLoadForm'> <input type='file' id='fileinput' name='files' multiple="multiple" > <input type="submit" value='upload'> </form> This was followed by small changes in the handler to receive not one file, but an array of files - and the problem is solved! (naivety (inexperience) of mine was not chapel).
How many people, with a laugh, have already thought that five hundred thirds answered the first, normal, batch of nginx files. We had to think further.
Decision number 3
Since the past decision was made beautifully and conveniently for admins, it was decided to build on it. It was necessary to solve the problem of error number 503, which was returned by nginx due to lengthy file processing.
Half a minute of thinking and a new solution appears: we will send ajax not all the files at once, but one at a time.
The solution, approximately, was as follows:
jQuery.each($('#fileinput')[0].files, function(i, f) { var file = new FormData(); file.append('file', f); $.ajax({ url: 'uploader.php', data: file, async : false, contentType: false, processData: false, dataType: "JSON", type: 'POST', beforeSend: function() {}, complete: function(event, request, settings) {}, success: function(data){ } }); }); It's simple: we iterate over the array of files (which is in the right input), create an instance of the class for working with files (more on that later) and send the request to the server as ajax. It is worth paying attention to the " async: false " parameter - here we set the synchronous execution of the ajax request, since the asynchronous will create us a lot of requests to the server, which we can easily set it with.
The solution works, there is no error, but here’s one problem — it works slowly. And then I came up with the idea of ​​another solution to the problem.
Decision number 4
To speed up the upload of files to the server, you can increase the number of requests in which files will be transferred. Such a decision to rest on the solution of two problems:
1). With a large number of requests I quickly go to hell with my server.
2) A large amount of simultaneously uploaded files will score us the whole channel.
Judging by the problems, our bootloader should count the number of running requests and the amount of data transferred and, under certain conditions, expect the end of some requests to start others.
Proceed to the code:
FilesUploader = { dataArr : new Array(), fStek : new Array(), vStek : 0, deley : 100, debug : true, maxFilesSize : 1024*1024*10, maxThreads : 10, Legend:
dataArr - an array of data to send.
fStek - write timeout identifiers here to further stop recursion and clear memory of incomplete functions.
vStek - the number of threads invoked .
deley - recursion delay of the function that checks threads and volumes.
debug - debaby mode. It is necessary for debugging, but in this example I deleted all its signs.
maxFilesSize - the maximum amount of downloaded files
maxThreads - the maximum number of threads.
Full clarity in the variables (especially fStek and deley ) will be introduced by the second considered function FilesUploader.controller () . For now, let's move on to class initialization:
run : function() { jQuery.each($('#fileinput')[0].files, function(i, f) { FilesUploader.dataArr.push(f); }); FilesUploader.controller(); }, This is the function that handles the event of a click of a button in the form. The work of the function is simple: we run through the files ( jQuery.each ) entered in the input and add ( FilesUploader.dataArr.push (f) ) record about each to the array. Next, call the controller, which is the most important and most complex part of the system:
controller : function() { FilesUploader.fStek.push(setTimeout(FilesUploader.controller, FilesUploader.deley)); if(FilesUploader.vStek>=this.maxThreads) { return; } item = FilesUploader.dataArr.pop(); if(item) { if(FilesUploader.maxFilesSize-item.size < 0) { FilesUploader.dataArr.push(item); return; } FilesUploader.maxFilesSize-=item.size; FilesUploader.vStek++; FilesUploader.worker(item); } else clearTimeout(FilesUploader.fStek.pop()); }, In the first line of the function, we asynchronously call (after a certain period of time) the same function (ie, create recursion), and put the identifier of the called function into a variable to be able to interrupt its execution.
Next comes the condition for checking threads.
After getting the file from the array ( FilesUploader.dataArr.pop () ), we check it for the presence.
1. If there is no file, then we “kill” the called functions by their identifier ( clearTimeout (FilesUploader.fStek.pop ()) );
2. If the file exists, we do a check for the size of the loaded files, and if it is exceeded, we return the file back to the stack and exit the function, otherwise, if it is not exceeded: we take the volume, increase the counter of the started threads and call the following function ( FilesUploader.worker ( item) ).
worker : function(item) { var file = new FormData(); file.append('file', item); $.ajax({ url: 'uploader.php', data: file, contentType: false, processData: false, dataType: "JSON", type: 'POST', beforeSend: function() {}, complete: function(event, request, settings) { FilesUploader.maxFilesSize+=this.fileData.size; FilesUploader.vStek--; }, success: function(data){ } }); }, To send a file to the server using ajax, you need to put data about the file (file.append ()) into an instance of the FormData class.
Next, we call the $ .ajax function, which will transfer our file to the loader on the server. Upon completion of each request (the complete () function), you need to increase the allowable volume and reduce the number of executable threads (which is done in the “ FilesUploader.maxFilesSize + = this.fileData.size ” and “ FilesUploader.vStek— ” lines ).
And the final touch is the console output function and closing bracket:
out : function(message) { if(console.log && this.debug) console.log(message); } } That's all - the class for multi-threaded file upload to the server is ready. Next, you should set, depending on the server configuration, the permissible number of threads and the amount of simultaneously uploaded files - and you can work.
Source: https://habr.com/ru/post/149701/
All Articles