New attachments in Yandex.Mail
We strive to ensure that all parts of Yandex.Mail work equally well for all users. Today we will tell you about how and why the block for adding attachments was completely rewritten. In this article - about the rejection of flash, support for the capabilities of modern browsers and, as a result, an increase in the speed and reliability of downloading files.
Previously, we divided the whole Yandex audience. Mail users into users with a flash and without.
With the first, everything was simple: users with a flash installed attached files to the letter via a flash loader. He allowed to download several files at once, determined their size and controlled the download process.
')
But with users without a flash (8-10% of the daily audience) was more difficult. We offered them to upload files through the usual form with
And if small files (up to 25 MB) did not cause any particular difficulties, then large ones created a new problem: if the file size exceeded the allowed limit, you had to use the Yandex.Narod service, and then Yandex.Disk (with the new attachments we changed the file storage) *.
* The limit on the size of sent files is explained not so much by technological limitations in Yandex. Mail, as by problems with third-party mail servers. Not all of them are ready to accept and store letters of large sizes. To ensure that such letters reach the addressee, we save attachments of more than 25 MB in size on Yandex.Disk and add links to the letter.
To determine the size of files for users without a flash, we raised an internal service that worked like this: the client sent a POST file with a request to a special url, the server read the request Content-Length header and closed the connection.
The implementation of downloading files in all browsers is designed so that it does not wait for a response from the server until it has completely sent the file. Therefore, the server cannot immediately report the file size. To solve this problem, we made a second GET request, in which the server passed to the client a Content-Length header value equal to the size of the downloaded file.
Problems with attaching files to the letter could arise for both categories of users. For example, the flash loader allows you to select several files and is able to determine their size, but:
And the usual
Of course, we were not satisfied with this state of affairs, and we did not stop searching for an effective solution to these problems.
Here are the new features that appeared during the development of HTML5:
- multiple attribute in the input tag (starting with Chrome 4, Firefox 3.6, IE 10, Opera 11, Safari 5);
- Drag and Drop API (Chrome 4, Firefox 3.5, IE 5.5, Opera 12, Safari 3);
- FormData (Chrome 7, Firefox 4, IE 10, Opera 12, Safari 5);
- XMLHttpRequest level 2 + CORS + progress events (Chrome 7, Firefox 4, IE 10, Opera 12, Safari 5).
Theoretically, we could have introduced them a year and a half ago, but the changes would affect only Chrome and Firefox. These browsers had a good share in total, but were not monopolists. Opera and IE by that time did not yet support these features. So, half the audience would still have to leave on a flash.
Therefore, we waited. And before the June release of Opera 12, in which the introduction of the necessary technologies became possible, began its development.
As for IE10, its release is expected soon.
Usually files are uploaded through the standard form:
Suppose we send a form to a hidden iframe. In this case, the browser will read all the selected files from the input (even if there are many) and send a POST request to / upload. But here the files are loaded all together, but it does not suit us.
Let's see how AJAX will help us. To send files via AJAX, we need FormData support. Without it, you cannot read the files in input and add them to the query. Let's try this:
But in this case, all the files still go from input. It turns out that you need to take each file separately and determine where to download it (to Disk or to a letter), that is, process it independently.
Downloading files individually is also convenient because the error in loading one file does not prevent others.
We support all popular browsers, but not all of them support modern technologies. According to the feature detection policy, we have added four checks to enable new features:
The third and fourth checks resulted in tests for Modernizr :
In Safari 5.1 for Windows, a bug was immediately found: when selecting several files, they all turned out to be zero in size and sent to the server empty. In this browser, all new features had to be disabled.
In addition to AJAX transport, we started using Progress events to draw a beautiful progress bar.
We use it like this:
Note that when loading data to the server, an event handler must be hung on the xhr.upload property, and when loading data from the server, on xhr itself.
In browsers that support the File API, the size of the files can be obtained from the File object. In older specifications, the property was called fileSize, and now just size.
In browsers without File API support (and there are less and less of them), we are degrading before using the internal file sizing service.
By the way, with the transition to new technologies, we were able to implement our long-standing idea: drag-and-drop upload attachments. The drag-and-drop API is very common. It concerns not only files, but any drag and drop of objects on the page. Accordingly, absolutely everything can be moved to the file area.
We had to solve this problem: how to leave in the mail only the ability to download files?
Much does the browser itself, but not all. In the drop event in the event.dataTransfer.files property, of course, there will be only objects from the file system. But these objects can be both folders and files. To prevent downloading folders (not all browsers can load files from folders - Chrome 21 was the first, and Firefox refused to do it with the principle) we use FileReader. This API allows you to read a file from disk and work with it in JavaScript. And if the object is read, then this is a file. A small function that implements this method can be found on GitHub .
However, this check is not needed for all browsers - Chrome for Mac and IE10 for Windows 8 themselves filter out folders.
You should be very careful with FileReader, especially in Chrome, which is not stable: up to the 21st version there were tab drops when reading a file of several hundred megabytes, and in the 21st it began to fall on small files. We even had to stop using FileReader for this browser.
Among other things, we have slightly modified the logic of the appearance of the area for dragging files. Here again, the problem arose: the user can drag the label onto the letter or, for example, accidentally start dragging a picture from the interface.
To solve this problem in the dragover and dragenter handlers, we did the following check:
The “Files” type means that there are real files in the dragged objects, and “return false” - the beginning of the drag and drop process. This check does not work in all browsers, but slightly improves the interface.
It also turned out that dragenter, dragover and dragleave events, if you hang them on a document, are subject to the same problems as mouseover, mouseout: they are thrown every time you move between DOM nodes.
The problem was solved by a timeout for handling these events.
The definition of transport policy remains the same.
For cross-domain queries, it is necessary to do the correct processing of the “preflight” OPTIONS requests. In these requests, the browser asks the remote server if it can be accessed from the current domain. They look like this:
To this, the server must respond with permissive headers, for example, like this:
Such requests do not always occur, but they must be remembered and verified that they are processed correctly.
If the browser does not receive permission for a cross-domain request, the request will end with status = 0 (this can be processed in onreadystatechange). It may also mean that the request was interrupted by the user or server. In any case, it is worth making a fallback to the iframe download.
The process of uploading files to Yandex.Disk itself looks like this: first, a request is made in which the Disk backend returns us the url for which the file should be uploaded to the repository, as well as the oid (operation id) for which you can request the status of the operation. Downloading is not a synchronous operation, and the end of sending a file from the client does not mean that the file is ready on the server, it must be saved in the right place, checked by antivirus, written to the database.
If there is support for progress events, then the status of the operation is not requested until the file download is completed, and the progress bar is drawn by the browser. This allows you to significantly reduce the load on the server and draw a smoother progress.
If progress events are not supported, we request the download status every one or two seconds until the server says that the file is ready.
Problem
Previously, we divided the whole Yandex audience. Mail users into users with a flash and without.With the first, everything was simple: users with a flash installed attached files to the letter via a flash loader. He allowed to download several files at once, determined their size and controlled the download process.
')
But with users without a flash (8-10% of the daily audience) was more difficult. We offered them to upload files through the usual form with
/> . Files from it were sent via the iframe along with the contents of the letter itself, and it took a lot of time. Pressing the "Send" button, the user waited a long time until the files are loaded.And if small files (up to 25 MB) did not cause any particular difficulties, then large ones created a new problem: if the file size exceeded the allowed limit, you had to use the Yandex.Narod service, and then Yandex.Disk (with the new attachments we changed the file storage) *.
* The limit on the size of sent files is explained not so much by technological limitations in Yandex. Mail, as by problems with third-party mail servers. Not all of them are ready to accept and store letters of large sizes. To ensure that such letters reach the addressee, we save attachments of more than 25 MB in size on Yandex.Disk and add links to the letter.
To determine the size of files for users without a flash, we raised an internal service that worked like this: the client sent a POST file with a request to a special url, the server read the request Content-Length header and closed the connection.
The implementation of downloading files in all browsers is designed so that it does not wait for a response from the server until it has completely sent the file. Therefore, the server cannot immediately report the file size. To solve this problem, we made a second GET request, in which the server passed to the client a Content-Length header value equal to the size of the downloaded file.
Problems with attaching files to the letter could arise for both categories of users. For example, the flash loader allows you to select several files and is able to determine their size, but:
- This is a third-party plugin that must be installed on the user's computer, and it can be blocked by other plug-ins or extensions;
- There are problems with SSL connections and security;
- Difficult to solve problems and errors when downloading files.
And the usual
/> at least does not have multiattaching.Of course, we were not satisfied with this state of affairs, and we did not stop searching for an effective solution to these problems.
Opportunity
Over the past year, all browsers have learned independently (without connecting third-party plug-ins) to organize work with files. Take a closer look at all their modern capabilities in the article on the Mozilla Developer Network .Here are the new features that appeared during the development of HTML5:
- multiple attribute in the input tag (starting with Chrome 4, Firefox 3.6, IE 10, Opera 11, Safari 5);
- Drag and Drop API (Chrome 4, Firefox 3.5, IE 5.5, Opera 12, Safari 3);
- FormData (Chrome 7, Firefox 4, IE 10, Opera 12, Safari 5);
- XMLHttpRequest level 2 + CORS + progress events (Chrome 7, Firefox 4, IE 10, Opera 12, Safari 5).
Theoretically, we could have introduced them a year and a half ago, but the changes would affect only Chrome and Firefox. These browsers had a good share in total, but were not monopolists. Opera and IE by that time did not yet support these features. So, half the audience would still have to leave on a flash.
Therefore, we waited. And before the June release of Opera 12, in which the introduction of the necessary technologies became possible, began its development.
As for IE10, its release is expected soon.
Implementation
As mentioned above, we must separate files into large and small ones. For example, a user tries to attach ten files to a letter, nine of which in total fit into the allowed limit, and the tenth is twice as large as the rest. Without the ability to download files individually, all ten files would go to Yandex.Disk. However, this does not seem reasonable - it is better to send only one file to the Disk, the last one, and upload all the rest to the letter. So we decided to download each file separately.Usually files are uploaded through the standard form:
<form action="/upload" method="post"> <input type="file" multiple="true"/> <input type="submit"/> </form> Suppose we send a form to a hidden iframe. In this case, the browser will read all the selected files from the input (even if there are many) and send a POST request to / upload. But here the files are loaded all together, but it does not suit us.
Let's see how AJAX will help us. To send files via AJAX, we need FormData support. Without it, you cannot read the files in input and add them to the query. Let's try this:
var formElement = document.getElementById("myFormElement"); var xhr = new XMLHttpRequest(); xhr.open("POST", "/upload", true); xhr.send(new FormData(formElement)); But in this case, all the files still go from input. It turns out that you need to take each file separately and determine where to download it (to Disk or to a letter), that is, process it independently.
for (var i = 0, j = input.files.length; i < j; i++) { upload(input.files[i]); } function upload(file) { var url = ""; if (file.size > MESSAGE_LIMIT) { url = "uploader.disk.yandex.ru"; } else { url = "uploader.mail.yandex.ru"; } var data = new FormData(); data.append("attachment", file); var xhr = new XMLHttpRequest(); xhr.open("POST", url, true); xhr.send(data); } Downloading files individually is also convenient because the error in loading one file does not prevent others.
We support all popular browsers, but not all of them support modern technologies. According to the feature detection policy, we have added four checks to enable new features:
- No support FormData → use iframe.
- There is support for FormData → use AJAX.
- There is support for Drag-n-Drop and FormData → enable the ability to drag and drop files from the file manager. For example, in IE there is the first, but there is no second, therefore we cannot send the dragged files in any way.
- There is support for multiple input and FormData → we enable the ability to select many files. For example, in Opera 11.6 there are multiple inputs, but there is no FormData, respectively, we cannot send files one by one.
The third and fourth checks resulted in tests for Modernizr :
Modernizr .addTest('draganddrop-files', function() { return !!(Modernizr['draganddrop'] && window['FormData'] && window['FileReader']); }) .addTest('input-multiple', function() { return !!(Modernizr['input']['multiple'] && window['FormData'] && window['FileReader']); }); In Safari 5.1 for Windows, a bug was immediately found: when selecting several files, they all turned out to be zero in size and sent to the server empty. In this browser, all new features had to be disabled.
In addition to AJAX transport, we started using Progress events to draw a beautiful progress bar.
We use it like this:
var xhr = new XMLHttpRequest(); xhr.open('POST', '/upload', true); if (xhr.upload) { xhr.upload.addEventListener('progress', processProgressEvent, false); } else { drawCommonProgressbar() } Note that when loading data to the server, an event handler must be hung on the xhr.upload property, and when loading data from the server, on xhr itself.
In browsers that support the File API, the size of the files can be obtained from the File object. In older specifications, the property was called fileSize, and now just size.
In browsers without File API support (and there are less and less of them), we are degrading before using the internal file sizing service.
By the way, with the transition to new technologies, we were able to implement our long-standing idea: drag-and-drop upload attachments. The drag-and-drop API is very common. It concerns not only files, but any drag and drop of objects on the page. Accordingly, absolutely everything can be moved to the file area.
We had to solve this problem: how to leave in the mail only the ability to download files?
Much does the browser itself, but not all. In the drop event in the event.dataTransfer.files property, of course, there will be only objects from the file system. But these objects can be both folders and files. To prevent downloading folders (not all browsers can load files from folders - Chrome 21 was the first, and Firefox refused to do it with the principle) we use FileReader. This API allows you to read a file from disk and work with it in JavaScript. And if the object is read, then this is a file. A small function that implements this method can be found on GitHub .
function isRegularFile(file, callback) { // , 4, if (file.size > 4096) { callback(true); return; } if (!window['FileReader']) { // callback(null); } else { try { var reader = new FileReader(); reader.onerror = function() { reader.onloadend = reader.onprogress = reader.onerror = null; // Chrome (Linux/Win), Firefox (Linux/Mac), Opera 12.01 (Linux/Mac/Win) callback(false); }; reader.onloadend = reader.onprogress = function() { reader.onloadend = reader.onprogress = reader.onerror = null; // abort if (e.type != 'loadend') { // reader.abort(); } callback(true); }; reader.readAsDataURL(file); } catch(e) { // Firefox/Win callback(false); } } } However, this check is not needed for all browsers - Chrome for Mac and IE10 for Windows 8 themselves filter out folders.
You should be very careful with FileReader, especially in Chrome, which is not stable: up to the 21st version there were tab drops when reading a file of several hundred megabytes, and in the 21st it began to fall on small files. We even had to stop using FileReader for this browser.
Among other things, we have slightly modified the logic of the appearance of the area for dragging files. Here again, the problem arose: the user can drag the label onto the letter or, for example, accidentally start dragging a picture from the interface.
To solve this problem in the dragover and dragenter handlers, we did the following check:
var types = event.dataTransfer.types; if (types) { for (var i = 0, j = types.length; i < j; i++) { if (types[i] == 'Files') { showDragArea(); return false; } } } The “Files” type means that there are real files in the dragged objects, and “return false” - the beginning of the drag and drop process. This check does not work in all browsers, but slightly improves the interface.
It also turned out that dragenter, dragover and dragleave events, if you hang them on a document, are subject to the same problems as mouseover, mouseout: they are thrown every time you move between DOM nodes.
The problem was solved by a timeout for handling these events.
var processTimer = null; $(document).on({ 'dragover dragenter': function() { window.clearTimeout(processTimer); showDragArea(); }. 'dragleave': function() { processTimer = window.setTimeout(function() { hideDragArea(); }, 50); } }); Cross Domain Queries
To upload to the Disk, support for cross-domain queries was needed, which can be checked as follows: window['XMLHttpRequest'] && 'withCredentials' in new XMLHttpRequest() The definition of transport policy remains the same.
For cross-domain queries, it is necessary to do the correct processing of the “preflight” OPTIONS requests. In these requests, the browser asks the remote server if it can be accessed from the current domain. They look like this:
OPTIONS /upload HTTP/1.1 Host: disk-storage42.mail.yandex.net Origin: https://mail.yandex.ru Access-Control-Request-Method: POST Access-Control-Request-Headers: origin, content-type To this, the server must respond with permissive headers, for example, like this:
Access-Control-Allow-Origin: https://mail.yandex.ru Allow: POST, PUT, TRACE, OPTIONS Such requests do not always occur, but they must be remembered and verified that they are processed correctly.
If the browser does not receive permission for a cross-domain request, the request will end with status = 0 (this can be processed in onreadystatechange). It may also mean that the request was interrupted by the user or server. In any case, it is worth making a fallback to the iframe download.
The process of uploading files to Yandex.Disk itself looks like this: first, a request is made in which the Disk backend returns us the url for which the file should be uploaded to the repository, as well as the oid (operation id) for which you can request the status of the operation. Downloading is not a synchronous operation, and the end of sending a file from the client does not mean that the file is ready on the server, it must be saved in the right place, checked by antivirus, written to the database.
If there is support for progress events, then the status of the operation is not requested until the file download is completed, and the progress bar is drawn by the browser. This allows you to significantly reduce the load on the server and draw a smoother progress.
If progress events are not supported, we request the download status every one or two seconds until the server says that the file is ready.
Success
In our opinion, the game was worth the candle. We are completely satisfied with the current solution, including because we solved a number of problems without losing the advantages that the flash has:- eliminated "underwater knocks" - problems of users with sending emails and downloading attachments;
- refused to use the flash;
- subjectively - reduced the time that users spent on sending letters with attachments;
- twice increased the number of file downloads on the disk compared with the people.
Source: https://habr.com/ru/post/154151/
All Articles