Uploading files with the html5 File API, with preference and dancers

Foreword

File uploading has always been a special place in web development.
A lot has already been said about the difficulty of styling with <input type = file />, you can read about it, for example, from the links one , two , three , four , five , six .
But the file uploading process itself is non-trivial, there are many different ways - and not one perfect one.

I already wrote about the implementation on our project Files@Mail.Ru silverlight-loader six months ago . At that time we had iframe, flash, silverlight and regular file uploads. But progress does not stand still, and now the latest beta versions of all beloved browsers fully support html5 FileAPI (in fairness, it is worth noting that, as usual, some support peculiar, but more on that - below).

While the article was being written, Chrome 9 was declared stable and was forcibly updated already on 75% of installations of version 8 . So, we celebrate File API support with the first stable browser, hurray!
')
We thought that not using such technology would be a crime against users of users.
We thought - and implemented html5 download in addition to the existing options.
As a result, our users received a lot of buns:
- transparent reloading after the connection is broken (and even the browser restart!);
- download queue;
- progress bar (MacOS and Safari users can finally see progress without any foreign plug-ins), the ability to remove files from the queue, if he changed his mind.


Using File API we can programmatically from javascript code:
1. get a list of files selected in the dialog, their sizes and mime types (which, by the way, should not be counted on, because some popular file types are not defined by browsers by extension).
2. get the required range of bytes from the file, without loading the entire contents of the file into memory (unlike Flash and Firefox 3 - see note 1).
3. Upload the server as a whole file, and its piece.
4. upload files to one drag-n-drop.
5. upload several files simultaneously (in parallel).
Those. we don't need any plugins to manipulate files, and this is definitely very cool!

Plot

Actually downloading files is implemented in the File API in just a few lines, but we added some nice features (download queue, reloading when the connection was broken) and the code became a bit more complicated.
The loader code on the Files@Mail.Ru project is available and not obfuscated and can be studied , but it is tied to the project and its features, so we will consider this download mechanism in its pure form using the example of the lightweight uploader project.

So let's go ...

We hang on input onchange handler.

oself.file_elm.onchange = function() { oself.onSelect(this); // 'this' is a DOM object here } 

The input object supports multiple html5 attributes for allowing multiple files to be selected at a time in the dialog and accept (see note 2), which filters the files in the dialog according to the specified mime types.

In the onSelect method, we go over the files array (which contains the list of selected files generated by the browser), set default properties and generate an onSelect event for each file.
After that we re-create the button, i.e. we delete input and we create it again. This is done in order to prevent re-loading of selected files when sending a form to the server when the button is inside the form.
The initiator of the start of loading in this case is the listener of the onSelect event, calling the enqueueUpload method of the loader object.

 /* n -    ,    file -    File idx -     cnt -       */ function onSelect(n, file, idx, cnt) { if(file.size > 1 * 1024 * 1024) { alert("File is too big!\nMaximum size is 1 MB."); return; } var d = document.createElement('div'); d.id = 'file_' + file.id + '_' + n; document.getElementById('file_list_' + n).appendChild(d); d.innerHTML = '<a href="#" id="file_' + file.id + '_cancel_' + n + '">X</a>' + file.name + ' (' + file.size + ') <span id="file_' + file.id + '_status_' + n + '">...</span>' document.getElementById('file_' + file.id + '_cancel_' + n).onclick = function() { window['up' + n].cancelUpload(file.id); return false; }; window['up' + n].enqueueUpload(file, 'http://lwu.no-ip.org/upload', "arg1=val1&arg2=val2"); } 

The enqueueUpload method adds the file to the internal loader queue, adds the file to the frontend queue (the frontend is an entity that interacts with the user and allows him to select files, that is, either input or a Flash or Silverlight plugin) and calls the startNextUpload method, which immediately starts downloading this file, or postpones it, if the number of files specified during initialization is already loading at the same time.

When adding a file to the frontend queue, html5 frontend starts the mechanism for calculating the unique hash of the file, with the help of which [hash] is implemented reloading. Details can be found in the article about silverlight-loader .
Yes, the hash is again calculated using the Adler32 algorithm.

 oself.addFile = function(fo) { upFE_html5.superclass.addFile.apply(oself, [fo]); oself.calcChunkSize(fo); oself.calcFileHash(); // run calculation for next file }; 

After the hash is calculated, the local storage is accessed to check if there is any information about the previous unsuccessful download of this file. If the information is found, the attributes of the file url, sessionID and uploadedRange are overwritten with information from the local storage.
Local storage (it’s also WebStorage ) is another html5 element that allows you to store arbitrary data in the key-value format on the user’s side, either for the duration of the session ( SessionStorage ) or permanently ( LocalStorage ).
When the queue reaches the file download, the startUpload method of the loader is called, which generates the onStart event and starts the download.

 oself.startUpload = function(id, url, data) { var fo = oself.getFile(id); fo.url = url; // at this moment url already fetched from localStorage if info presents fo.data = data; fo.full_url = fo.url + (fo.url.match(/\?/) ? '&' : '?') + fo.data; fo.retry = oself.opts.maxChunkRetries; oself.broadcast('onStart', fo); oself.uploadFile(fo); }; 

The uploadFile method directly uploads a file to the server.

 oself.uploadFile = function(fo) { oself.calcNextChunkRange(fo); var blob, simple_upload = 0; try { blob = fo.slice(fo.currentChunkStartPos, fo.currentChunkEndPos - fo.currentChunkStartPos + 1); } catch(e) { // Safari doesn't support Blob.slice method blob = new FormData(); blob.append('Filedata', fo); simple_upload = 1; }; fo.xhr = new XMLHttpRequest(); fo.xhr.onreadystatechange = function() { if(this.readyState == 4) { try { if(this.status == 201) { // chunk was uploaded succesfully var range = this.responseText; try { // getResponseHeader throws exception during cross-domain upload, but this is most reliable variant range = this.getResponseHeader('Range'); } catch(e) {}; if(!range) { throw new Error('No range in 201 answer'); } fo.uploadedRange = range; // store range for case of later retry fo.retry = oself.opts.maxChunkRetries; // restore retry counter userStorage.set(fo); // add or update file info in localStorage oself.uploadFile(fo); } else if(this.status == 200) { fo.responseText = this.responseText; fo.loaded = fo.size; // all bytes were uploaded userStorage.del(fo); // delete file info from localStorage oself.broadcast('onDone', fo, fo.responseText); } else if(this.status == 0 && fo.cancel == 1) { //t('Aborted uploading for id=' + fo.id); } else { throw new Error('Bad http answer code'); } } catch(e) { // any exception means that we need to retry upload oself.retryUpload(fo); }; } }; fo.xhr.open("POST", fo.full_url, true); fo.xhr.upload.onprogress = function(evt) { fo.loaded = (simple_upload ? 0 : fo._loaded) + evt.loaded; oself.broadcast('onProgress', fo); }; if(!simple_upload) { fo.xhr.setRequestHeader('Session-ID', fo.sessionID); fo.xhr.setRequestHeader('Content-Disposition', 'attachment; filename="' + encodeURI(fo.name) + '\"'); fo.xhr.setRequestHeader('Content-Range', 'bytes ' + fo.currentChunkStartPos + '-' + fo.currentChunkEndPos + '/' + fo.size); fo.xhr.setRequestHeader('Content-Type', 'application/octet-stream'); } fo.xhr.withCredentials = true; // allow cookies to be sent fo.xhr.send(blob); }; 

Comments in the code clearly show incomplete support for the html5 File API in the Safari browser (at least in the Windows OS), see approx. 3
When errors occur, the retryUpload method is run, which repeatedly tries to load the file specified during the bootloader initialization, increasing the interval between attempts at each failure.
In case of exhaustion of the number of attempts, the onError event is generated.

 oself.retryUpload = function(fo) { fo.retry--; if(fo.retry > 0) { var timeout = oself.opts.retryTimeoutBase * (oself.opts.maxChunkRetries - fo.retry); setTimeout(function(){oself.uploadFile(fo)}, timeout); } else { oself.broadcast('onError', fo. lwu.ERROR_CODES.OTHER_ERROR); } }; 

For this miracle to work, nginx with an upload module must be installed on the server. A little more about this was written in the previous article .

Instead of an epilog ...

I would like to express a few thoughts:
1. At the moment, FileAPI support Chrome 8 and higher, Firefox 4 beta and partially Safari 5 . I know nothing about the implementation of support in InternetExplorer and Opera.
However, we have disabled Chrome 8 because of the annoying bug , because of which it is impossible to select many files in the dialog.
Firefox 3 supports FileAPI in its own way, there is no support for the urgently needed FormData object, so large files cannot be uploaded because requires reading the entire contents of the file in the computer's memory.
2. The accept attribute works very clumsily, browsers simply do not understand many mime types. Therefore, it remains a mystery to me why filtering is done this way, and not according to the list of extensions, as is done in Flash and Silverlight.
3. Safari browser does not implement the FileReader object and the Blob.slice method, so the download does not work with html5. Since reloading is a very useful “bun”, we changed the order of the boot loaders in Safari, making Silverlight more preferable.
4. Not quite obvious, but when using bit operations, Javascript converts operands to the signed int32 type. And since to calculate the checksum Adler32 need unsigned numbers, had to abandon the bit shift to the left and use the multiplication by 65536.
5. You need to do URI encoding of the file name on the client and decoding on the server, because the name falls into the Content-Disposition header, and the headers should not contain non-ASCII characters according to the standard.
6. It is necessary to warn users about the need to disable the Firebug plugin or the like, and here's why: Firebug on the Network tab logs all network activity and completely saves all requests, and since our requests are small in size, the plugin’s built-in limiter does not work, and on large files we can get a lot of memory consumption by the browser.

Dmitry Dedyukhin, lead developer Files@Mail.Ru