Silverlight + nginx = renewable download of files in the browser
This article describes the experience of implementing Silverlight-client for the organization of renewable file uploads on Files.Ru project.
Why do you need it? I think it is not necessary to tell that uploading files to the server and storing them now provides a very large number of web projects, from small to very large. Moreover, the download is usually implemented in the form of the usual
The problem is that the HTTP protocol is initially text-based and is not very adaptable for transferring large amounts of binary data. This implies that when the user breaks down, the computer restarts, etc., the half-transmitted file has to start downloading again, and in the case of a slow channel, this turns into a real mockery.
')
What to do?
The appearance of the first rumors about the new version of Adobe Flash 10 with the support of the FileReference.load () method for reading the contents of the file inspired us. But it was not the case: Adobe “outwitted everyone”. The FileReference.load () method completely loads the entire contents of the file into the computer's memory, thereby “hanging up” the machine when trying to read a large file (in the experiments “large” already turned out to be about 500 MB on a computer with 2 GB of RAM). In addition, Flash does not support files larger than 2GB.
We were sad and disappointed. In addition, support for partial loading from the server side was urgently needed, and it was too lazy to do it yourself.
And once we thought: “Let's look at Silverlight, can it give us something more than Flash?” - and we were not mistaken.
In Silverlight, work with files is implemented more competently and accessible than in Flash - we can read the file selected by the user in the dialog box on an arbitrary offset with buffers of arbitrary size. At the same time, the size of the file in Silverlight is limited to a 64-bit number, i.e. we can upload files of almost infinite size (theoretically up to 16 384 PB).
In addition, Valery Kholodkov’s repository (if someone doesn’t know, then this is the author of the excellent nginx_upload_module module for uploading files) and a branch called partial-upload appeared, and one of its names led us to reverent awe.
Having enlisted Valery’s support, we started writing a Silverlight client and “docking” it with the server module ...
After many hours of rewriting the client code and testing it, we finally got the first working version.
With bated breath, we began to download the first file - oh my god, what a bliss, when you pull out the network cable in the boot process, then you stick it back and, oh, the download resumes almost from the place of the cliff. But bliss quickly passed, because during a cursory test, bugs were discovered in both the client code and the server module code.
Many thanks to Valery for quite a prompt fixing of bugs in his module, and us for fighting Silverlight and C #.
One fine August day, we finally finally tested and fixed all the found bugs and did not fail to take advantage of this in order to make the Files.Ru users happy - that is, we let it go into production.
And finally - a solution to the studio!
Download is as follows.
The client generates a unique session identifier for each uploaded file.
Also for each file is considered a hash, the purpose of which is to uniquely identify a unique file within the user's computer.
After selecting a file in the dialog and calculating its hash, we check the availability of information about this file in the local Silverlight repository, and if there is information, we start the download from the first “hole” in the loaded byte ranges.
The client then sends a piece of the file, specifying the range of bytes sent in the X-Content-Range header (due to Silverlight limitations, this header is used instead of the standard HTTP Content-Range header, although the server module supports both headers) and the session ID in the Session-ID header . In this case, pure binary data is sent in the request body, i.e. the contents of the piece.
In the header of the Range-response from the server comes a list of byte ranges of this file that are already uploaded to the server. Also, this list is duplicated in the response body (for which it is duplicated - see below).
After each load is successfully loaded, the information about the loaded ranges is saved / updated in the local storage of Silverlight, the key being the hash of the file. This allows you to reload the file even after closing the browser. After loading each chunk, the server module returns us the http code 201, and the download request is not proxied to the backend.
When the module determines that the file is fully loaded, it proxies the request for the backend with a link to the temporary file (as well as the standard upload module). In fact, for the backend, the transition from using the standard upload module to using the partial-upload module is completely transparent, i.e. the backend code is not required at all.
1. You cannot set the Content-Range header, so we use the X-Content-Range header.
2. It is impossible to reliably determine the server response code, we see only 200 or 404 codes (when using the Browser HTTP Stack in Silverlight)
3. When using the Client HTTP Stack in Silverlight, we lose the proxy authorization and have to manually set cookies, but we can accurately determine the server response code - so we use the Browser HTTP Stack with some tricks to determine the 201 response code:
4. Calculation of the “correct” file hash (for example, md5) on large files takes a lot of time - tens of seconds - which is unacceptable, so we take 50 parts of a file of 100K, for each part we calculate the amount using the Adler32 algorithm (this algorithm was chosen from - because of its high speed of work, on the advice of a familiar hacker) and then concatenate individual sums - this is the “unique” hash of the file
5. Silverlight with the presence of certain Russian letters in the file name (the letter “z” definitely fell out of favor with Microsoft) produced an exception in the line ...
... so I had to make a modification - encode the file name when downloading and decode it on the server
6. Even though buffers are reset after loading a certain number of bytes, Silverlight caches the POST request and sends it completely. This makes it impossible to download entire files (without chunks), because on large client memory files, there is not enough to buffer the request. This feature also makes it impossible to adequately display download progress.
Therefore, we are trying to divide the file into 100 chunks to display progress from 0% to 100%, but at the same time we limit the size of the chunk from the top and bottom for cases of very large and very small files, respectively, which can lead to more than 100 chunks, so to the lesser.
7. In Opera, there is an unpleasant bug (you can already get lost, which account is): if the response from the server has a body of zero length, then the handler for reading the response body is not called in Silverlight. That is why we asked Valery to duplicate the range of loaded bytes in the response body of the server.
We have attacked a lot of unpleasant rakes, and we want other developers to be less thorny. Therefore, we decided to open the client code. Meet MrUploader . Together with the nginx-upload module by Valery Kholodkov, it is especially tasty.
Why do you need it? I think it is not necessary to tell that uploading files to the server and storing them now provides a very large number of web projects, from small to very large. Moreover, the download is usually implemented in the form of the usual
<input type=file/> , less often - with the help of Flash, even less often - by other means (we do not consider downloading via FTP in this article).
The problem is that the HTTP protocol is initially text-based and is not very adaptable for transferring large amounts of binary data. This implies that when the user breaks down, the computer restarts, etc., the half-transmitted file has to start downloading again, and in the case of a slow channel, this turns into a real mockery.
')
What to do?
How did we get to such a life
The appearance of the first rumors about the new version of Adobe Flash 10 with the support of the FileReference.load () method for reading the contents of the file inspired us. But it was not the case: Adobe “outwitted everyone”. The FileReference.load () method completely loads the entire contents of the file into the computer's memory, thereby “hanging up” the machine when trying to read a large file (in the experiments “large” already turned out to be about 500 MB on a computer with 2 GB of RAM). In addition, Flash does not support files larger than 2GB.
We were sad and disappointed. In addition, support for partial loading from the server side was urgently needed, and it was too lazy to do it yourself.
And once we thought: “Let's look at Silverlight, can it give us something more than Flash?” - and we were not mistaken.
In Silverlight, work with files is implemented more competently and accessible than in Flash - we can read the file selected by the user in the dialog box on an arbitrary offset with buffers of arbitrary size. At the same time, the size of the file in Silverlight is limited to a 64-bit number, i.e. we can upload files of almost infinite size (theoretically up to 16 384 PB).
In addition, Valery Kholodkov’s repository (if someone doesn’t know, then this is the author of the excellent nginx_upload_module module for uploading files) and a branch called partial-upload appeared, and one of its names led us to reverent awe.
Having enlisted Valery’s support, we started writing a Silverlight client and “docking” it with the server module ...
Happy end
After many hours of rewriting the client code and testing it, we finally got the first working version.
With bated breath, we began to download the first file - oh my god, what a bliss, when you pull out the network cable in the boot process, then you stick it back and, oh, the download resumes almost from the place of the cliff. But bliss quickly passed, because during a cursory test, bugs were discovered in both the client code and the server module code.
Many thanks to Valery for quite a prompt fixing of bugs in his module, and us for fighting Silverlight and C #.
One fine August day, we finally finally tested and fixed all the found bugs and did not fail to take advantage of this in order to make the Files.Ru users happy - that is, we let it go into production.
And finally - a solution to the studio!
It is a little about interaction of the client and server
Download is as follows.
The client generates a unique session identifier for each uploaded file.
SessionId = (1100000000 + new Random ().Next(10000000, 99999999)).ToString();
* This source code was highlighted with Source Code Highlighter .Also for each file is considered a hash, the purpose of which is to uniquely identify a unique file within the user's computer.
UniqueKey = "" ;
try
{
if (FileLength < Constants.MinFilesizeToAdd)
{
throw new Exception();
}
// Adler32 version to compute "unique" file hash
// UniqueKey will be Constants.NumPoints * sizeof(uint) length
int part_size = ( int )((file.Length / Constants.NumPoints) < Constants.MaxPartSize ? file.Length / Constants.NumPoints : Constants.MaxPartSize);
byte [] buffer = new Byte [part_size];
byte [] adler_sum = new Byte [Constants.NumPoints * sizeof ( uint ) / sizeof ( byte )];
int current_point = 0;
int bytesRead = 0;
Stream fs = file.OpenRead();
AdlerChecksum a32 = new AdlerChecksum();
while (current_point < Constants.NumPoints && (bytesRead = fs.Read(buffer, 0, part_size)) != 0)
{
a32.MakeForBuff(buffer, bytesRead);
int mask = 0xFF;
for ( int i = 0; i < sizeof ( uint ) / sizeof ( byte ); i++)
{
UniqueKey += ( char )((mask << (i * sizeof ( byte )) & a32.ChecksumValue) >> (i * sizeof ( byte )));
}
fs.Position = ++current_point * file.Length / Constants.NumPoints;
}
}
catch (Exception) { }
* This source code was highlighted with Source Code Highlighter .After selecting a file in the dialog and calculating its hash, we check the availability of information about this file in the local Silverlight repository, and if there is information, we start the download from the first “hole” in the loaded byte ranges.
The client then sends a piece of the file, specifying the range of bytes sent in the X-Content-Range header (due to Silverlight limitations, this header is used instead of the standard HTTP Content-Range header, although the server module supports both headers) and the session ID in the Session-ID header . In this case, pure binary data is sent in the request body, i.e. the contents of the piece.
UriBuilder ub = new UriBuilder(UploadUrl);
HttpWebRequest webrequest = (HttpWebRequest)WebRequest.Create(ub. Uri );
webrequest.Method = "POST" ;
webrequest.ContentType = "application/octet-stream" ;
// Some russian letters in filename lead to exception, so we do uri encode on client side
// and uri decode on server side
webrequest.Headers[ "Content-Disposition" ] = "attachment; filename=\"" + HttpUtility.UrlEncode( File .Name) + "\"" ;
webrequest.Headers[ "X-Content-Range" ] = "bytes " + currentChunkStartPos + "-" + currentChunkEndPos + "/" + FileLength;
webrequest.Headers[ "Session-ID" ] = SessionId;
webrequest.BeginGetRequestStream( new AsyncCallback(WriteCallback), webrequest);
* This source code was highlighted with Source Code Highlighter .In the header of the Range-response from the server comes a list of byte ranges of this file that are already uploaded to the server. Also, this list is duplicated in the response body (for which it is duplicated - see below).
After each load is successfully loaded, the information about the loaded ranges is saved / updated in the local storage of Silverlight, the key being the hash of the file. This allows you to reload the file even after closing the browser. After loading each chunk, the server module returns us the http code 201, and the download request is not proxied to the backend.
When the module determines that the file is fully loaded, it proxies the request for the backend with a link to the temporary file (as well as the standard upload module). In fact, for the backend, the transition from using the standard upload module to using the partial-upload module is completely transparent, i.e. the backend code is not required at all.
Silverlight limitations that we had to circumvent:
1. You cannot set the Content-Range header, so we use the X-Content-Range header.
2. It is impossible to reliably determine the server response code, we see only 200 or 404 codes (when using the Browser HTTP Stack in Silverlight)
3. When using the Client HTTP Stack in Silverlight, we lose the proxy authorization and have to manually set cookies, but we can accurately determine the server response code - so we use the Browser HTTP Stack with some tricks to determine the 201 response code:
if (ResponseText != null && ResponseText.Length != 0)
{
// We cannot check response.StatusCode, see comments in constructor of FileUploadControl
if (Regex.IsMatch(ResponseText, @"^\d+-\d+/\d+" )) // we got 201 response
{
...
}
else // we got 200 response
{
BytesUploaded = FileLength;
}
}
* This source code was highlighted with Source Code Highlighter .4. Calculation of the “correct” file hash (for example, md5) on large files takes a lot of time - tens of seconds - which is unacceptable, so we take 50 parts of a file of 100K, for each part we calculate the amount using the Adler32 algorithm (this algorithm was chosen from - because of its high speed of work, on the advice of a familiar hacker) and then concatenate individual sums - this is the “unique” hash of the file
5. Silverlight with the presence of certain Russian letters in the file name (the letter “z” definitely fell out of favor with Microsoft) produced an exception in the line ...
webrequest.Headers[ "Content-Disposition" ] = "attachment; filename=\"" + File .Name + "\"" ;
* This source code was highlighted with Source Code Highlighter .... so I had to make a modification - encode the file name when downloading and decode it on the server
webrequest.Headers[ "Content-Disposition" ] = "attachment; filename=\"" + HttpUtility.UrlEncode( File .Name) + "\"" ;
* This source code was highlighted with Source Code Highlighter .6. Even though buffers are reset after loading a certain number of bytes, Silverlight caches the POST request and sends it completely. This makes it impossible to download entire files (without chunks), because on large client memory files, there is not enough to buffer the request. This feature also makes it impossible to adequately display download progress.
Therefore, we are trying to divide the file into 100 chunks to display progress from 0% to 100%, but at the same time we limit the size of the chunk from the top and bottom for cases of very large and very small files, respectively, which can lead to more than 100 chunks, so to the lesser.
public long FileLength
{
get { return fileLength; }
set
{
fileLength = value ;
ChunkSize = ( long )(fileLength / (100 / Constants.PercentPrecision));
if (ChunkSize < Constants.MinChunkSize)
ChunkSize = Constants.MinChunkSize;
if (ChunkSize > Constants.MaxChunkSize)
ChunkSize = Constants.MaxChunkSize;
}
}
* This source code was highlighted with Source Code Highlighter .7. In Opera, there is an unpleasant bug (you can already get lost, which account is): if the response from the server has a body of zero length, then the handler for reading the response body is not called in Silverlight. That is why we asked Valery to duplicate the range of loaded bytes in the response body of the server.
We have attacked a lot of unpleasant rakes, and we want other developers to be less thorny. Therefore, we decided to open the client code. Meet MrUploader . Together with the nginx-upload module by Valery Kholodkov, it is especially tasty.
Source: https://habr.com/ru/post/102551/
All Articles