The story of a single application or the struggle for performance
If you are a professional developer, then you should be familiar with the feeling when you want to do something not for money, but for the soul. On one of these evenings, I wanted to get a little distracted and write just such an application.
We are located in Ukraine, where there are not so many local applications for Windows Phone, and there are even fewer applications on national topics. Being a music lover, I decided to make an application with the lyrics of songs of Ukrainian performers. To my surprise, I found more than 18,000 Ukrainian songs on the OUR site that are performed by about 800 artists.
“Not bad” - I thought and sat down to write a simple parser, which folded all my texts locally. I have been writing parsers and other similar applications for many years, so this process did not take much time. For writing a crawler and parsing HTML I used the Data Extracting SDK library I wrote and, undoubtedly, the best library in the .NET world for this purpose - HtmlAgilityPack .
')
After all the information was packaged into a single XML file, the question of how to best unpack this information in the application, so that the user does not feel the brakes, became a question. And at that moment, the “for fun” task turned into a completely applied task of finding the optimal approach for working with large (by the standards of a mobile device) data volumes.
That's what came of it.
What the developer needs to pay attention to in order for the application to be high-performance:
Application start time
The application has only a few seconds to start. If the start time exceeds 8 seconds, the application will be unloaded from memory, and the author of the application will most likely receive an additional unit in the market.
In order to reduce the start time, it is necessary for the application to contain as few resource files as possible (Resources), so multimedia files and other “heavy” files should be included in the project as Content.
But there is a downside - access to resources after launching the application takes less time than reading the content. Therefore, developers need to take into account these differences.
You also need to remember that splash screen does not support animation, because This is a regular jpeg file.
Data load time
After the application starts, data starts to load. In our case there is a lot of data and the reading time is also big.
If data is loaded for a few seconds (from local storage or via a web service), then an additional fake page is often made that emulates a splash screen, but adds animation (a progress bar with the text "Please wait ..."). After loading the data, the user is redirected to the page where the loaded data is already displayed, being amazed that Microsoft finally added the ability to display animated splash screenshots. For more information on how to make an animated splash screen, read here .
If you load data in the main stream, it usually looks like this:
then the user will surely see the interface hangs (this is especially true for the Panorama control).
You can fix it this way - wait until all UI and resources are fully loaded and only then display the data:
The second option is to load everything in a separate thread or create a BackgroundWorker:
Wrote a little more code, but improved performance.
Well, it is also a good idea to show the progress bar while performing heavy operations (you can use Microsoft.Phone.Shell.ProgressIndicator, a component from the Silverlight Toolkit or from Telerik RadControls for Windows Phone).
Screen switching speed
Usually transitions between pages do not take much time, but you can also improve this process by adding beautiful animations (you can also use Telerik RadControls for Windows Phone).
It was a “moment of theory.” Smoothly proceed to the main task: loading and displaying data.
There is a Windows Console program that prepares data, and a Windows Phone application reads this data.
The easiest way: data serialization, namely in XML:
The data model looks like this:
As a result, we received an XML file that occupied more than 24 megabytes.
I also tried to serialize to JSON format, as well as reduce the names of properties, which resulted in saving of half a megabyte, which can be called “micro” optimization.
It’s not a good idea to load 24 megabytes into the application, so it was decided to use the SQL CE database and read the data from it.
A code has been added to the Windows Phone application that parses the XML file and writes all the data into the local database.
Findings:
In order to copy a locally generated database, the Isolated Storage Explorer tool was used:
In order for the Isolated Storage Explorer to work for you, you need to add the appropriate library to the project and register the code:
Since the variant with the base flew by, began to look for other optimization options. It is clear that where there is a large text file, there you can compress it using the archiver. The compression zip file archiver file began to weigh about 5 megabytes.
Thus, when starting the application, you need to 1) copy the file to the local storage 2) unpack it 3) load the data from the file into memory.
Zip archive in the project is added as Content, at the start we do all of the above steps. For this, various combinations of methods were used:
Also in the course went the library SharpZipLib:
I tried many options:
This stage ended with one conclusion: the file should be one and it should be in the archive, because The zip file reading itself is fairly fast.
Since the file size cannot be further reduced, it is possible to reduce the time it is read and loaded into memory. And then I took the path of custom serialization.
Earlier, I knew that embedded serialization is slow and if you need speed, you need to use binary serialization.
SilverlightSerializer by Mike Talbot was chosen as a serializer.
In the Windows Console Application, everything worked as it should (serialization / deserialization), but with the reading of the created file in the Windows Phone project there were difficulties associated with different versions of mscorlib.
On the project description page there is a paragraph about compatibility issues between .NET and Silverlight projects:
Unfortunately, to overcome this problem did not work.
Then I got to the protobuf-net project.
Mark the necessary classes and properties and get a binary file at the output:
There was no problem reading the file from the Windows Phone project.
The result was this solution:
1. Zip file, which contains the artists.bin binary, and the zip file is connected as Content;
2. With the help of BackgroundWorker, after loading the UI, we begin to load data (read the bin file directly from the zip file and deserialize it into the local data model):
Total: application launch and data parsing on Lumia 800: ~ 5-6 seconds, after which you can easily view any content. The result could be better and I will continue research and work on productivity, but this result, in my opinion, is not bad enough.
There is another small bottleneck - during the transitions between texts within one artist or search results, the text parser can slow down a bit (I have already described the solution - you need to add animations that hide this artifact, maybe I will add it in the update if users like the application) .
In general, everything went for everything:
Total ~ 12 hours.
The result of the work can be seen and evaluated here .
Thanks for attention! I hope, Windows Phone Store and Habr became a little bit better after this article!
We are located in Ukraine, where there are not so many local applications for Windows Phone, and there are even fewer applications on national topics. Being a music lover, I decided to make an application with the lyrics of songs of Ukrainian performers. To my surprise, I found more than 18,000 Ukrainian songs on the OUR site that are performed by about 800 artists.
“Not bad” - I thought and sat down to write a simple parser, which folded all my texts locally. I have been writing parsers and other similar applications for many years, so this process did not take much time. For writing a crawler and parsing HTML I used the Data Extracting SDK library I wrote and, undoubtedly, the best library in the .NET world for this purpose - HtmlAgilityPack .
')
After all the information was packaged into a single XML file, the question of how to best unpack this information in the application, so that the user does not feel the brakes, became a question. And at that moment, the “for fun” task turned into a completely applied task of finding the optimal approach for working with large (by the standards of a mobile device) data volumes.
That's what came of it.
Key Performance Issues
What the developer needs to pay attention to in order for the application to be high-performance:
Application start time
The application has only a few seconds to start. If the start time exceeds 8 seconds, the application will be unloaded from memory, and the author of the application will most likely receive an additional unit in the market.
In order to reduce the start time, it is necessary for the application to contain as few resource files as possible (Resources), so multimedia files and other “heavy” files should be included in the project as Content.
But there is a downside - access to resources after launching the application takes less time than reading the content. Therefore, developers need to take into account these differences.
You also need to remember that splash screen does not support animation, because This is a regular jpeg file.
Data load time
After the application starts, data starts to load. In our case there is a lot of data and the reading time is also big.
If data is loaded for a few seconds (from local storage or via a web service), then an additional fake page is often made that emulates a splash screen, but adds animation (a progress bar with the text "Please wait ..."). After loading the data, the user is redirected to the page where the loaded data is already displayed, being amazed that Microsoft finally added the ability to display animated splash screenshots. For more information on how to make an animated splash screen, read here .
If you load data in the main stream, it usually looks like this:
public MainPage() { InitializeComponent(); DataContext = new SomeDataObject(); } then the user will surely see the interface hangs (this is especially true for the Panorama control).
You can fix it this way - wait until all UI and resources are fully loaded and only then display the data:
public MainPage() { InitializeComponent(); this.Loaded += MainPage_Loaded; } void MainPage_Loaded(object sender, RoutedEventArgs e) { DataContext = new SomeDataObject(); }
The second option is to load everything in a separate thread or create a BackgroundWorker:
public MainPage() { InitializeComponent(); StartLoadingData(); } private void StartLoadingData() { this.Dispatcher.BeginInvoke(() => { var backroungWorker = new BackgroundWorker(); backroungWorker.DoWork += backroungWorker_DoWork; backroungWorker.RunWorkerCompleted += backroungWorker_RunWorkerCompleted; backroungWorker.RunWorkerAsync(); }); } void backroungWorker_DoWork(object sender, DoWorkEventArgs e) { // heavy operation } void backroungWorker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e) { this.Dispatcher.BeginInvoke(() => { // update layout with new data }); }
Wrote a little more code, but improved performance.
Well, it is also a good idea to show the progress bar while performing heavy operations (you can use Microsoft.Phone.Shell.ProgressIndicator, a component from the Silverlight Toolkit or from Telerik RadControls for Windows Phone).
Screen switching speed
Usually transitions between pages do not take much time, but you can also improve this process by adding beautiful animations (you can also use Telerik RadControls for Windows Phone).
It was a “moment of theory.” Smoothly proceed to the main task: loading and displaying data.
Download and display data
There is a Windows Console program that prepares data, and a Windows Phone application reads this data.
The easiest way: data serialization, namely in XML:
private static void Serialize(object obj, string name) { var ser = new XmlSerializer(obj.GetType()); var sb = new StringBuilder(); var writer = new StringWriter(sb); ser.Serialize(writer, obj); File.WriteAllText(name + ".xml", sb.ToString().Replace("encoding=\"utf-16\"", null)); } private static void Deserialize(Type type) { //Assuming doc is an XML document containing a serialized object and objType is a System.Type set to the type of the object. XmlNodeReader reader = new XmlNodeReader(doc.DocumentElement); XmlSerializer ser = new XmlSerializer(objType); object obj = ser.Deserialize(reader); // Then you just need to cast obj into whatever type it is eg: var myObj = (typeof(obj))obj; }
The data model looks like this:
public class Artist { public Artist() { Songs = new List<Song>(); } public int Id { get; set; } public string Title { get; set; } public List<Song> Songs { get; set; } } public class Song { public int Id { get; set; } public string Title { get; set; } public string Description { get; set; } public int ArtistId { get; set; } }
As a result, we received an XML file that occupied more than 24 megabytes.
I also tried to serialize to JSON format, as well as reduce the names of properties, which resulted in saving of half a megabyte, which can be called “micro” optimization.
It’s not a good idea to load 24 megabytes into the application, so it was decided to use the SQL CE database and read the data from it.
A code has been added to the Windows Phone application that parses the XML file and writes all the data into the local database.
Findings:
- data verification and insertion was very slow;
- due to the length of some texts, I had to use the DbType.NText type;
- The final size of the database was very large and even despite this it was necessary to rewrite it at the start of the application to the local storage, which took a lot of time.
In order to copy a locally generated database, the Isolated Storage Explorer tool was used:
In order for the Isolated Storage Explorer to work for you, you need to add the appropriate library to the project and register the code:
Since the variant with the base flew by, began to look for other optimization options. It is clear that where there is a large text file, there you can compress it using the archiver. The compression zip file archiver file began to weigh about 5 megabytes.
Thus, when starting the application, you need to 1) copy the file to the local storage 2) unpack it 3) load the data from the file into memory.
Zip archive in the project is added as Content, at the start we do all of the above steps. For this, various combinations of methods were used:
private void CopyFromContentToStorage(IsolatedStorageFile store, string dbName) { var src = Application.GetResourceStream(new Uri(dbName, UriKind.Relative)).Stream; var dest = new IsolatedStorageFileStream(dbName, FileMode.OpenOrCreate, FileAccess.Write, store); src.Position = 0; CopyStream(src, dest); dest.Flush(); dest.Close(); src.Close(); dest.Dispose(); } private static void CopyStream(Stream input, IsolatedStorageFileStream output) { var buffer = new byte[32768]; long tempPos = input.Position; int readCount; do { readCount = input.Read(buffer, 0, buffer.Length); if (readCount > 0) { output.Write(buffer, 0, readCount); } } while (readCount > 0); input.Position = tempPos; } // load items from "fileName" file that exists in "zipName" file private static List<Artist> Load(string zipName, string fileName) { var info = Application.GetResourceStream(new Uri(zipName, UriKind.Relative)); var zipInfo = new StreamResourceInfo(info.Stream, null); var s = Application.GetResourceStream(zipInfo, new Uri(fileName, UriKind.Relative)); var serializer = new XmlSerializer(typeof (List<Artist>)); return serializer.Deserialize(s.Stream) as List<Artist>; }
Also in the course went the library SharpZipLib:
using (ZipInputStream s = new ZipInputStream(src)) { s.Password = "123456";//if archive is encrypted ZipEntry theEntry; try { while ((theEntry = s.GetNextEntry()) != null) { string directoryName = Path.GetDirectoryName(theEntry.Name); string fileName = Path.GetFileName(theEntry.Name); // create directory if (directoryName.Length > 0) { Directory.CreateDirectory(directoryName); } if (fileName != String.Empty) { // save file to isolated storage using (BinaryWriter streamWriter = new BinaryWriter(new IsolatedStorageFileStream(theEntry.Name, FileMode.OpenOrCreate, FileAccess.Write, FileShare.Write, iso))) { int size = 2048; byte[] data = new byte[2048]; while (true) { size = s.Read(data, 0, data.Length); if (size > 0) { streamWriter.Write(data, 0, size); } else { break; } } } } } } catch (ZipException ze) { Debug.WriteLine(ze.Message); }
I tried many options:
- A zip file with a file with meta information (a list of artists and links to their songs), while the lyrics were in separate files - a big minus in performance, because 18,000 files were read and copied very slowly;
- zip with two files: one with a list of artists, the other with songs is bad, because the artists read quickly, and the songs, even while using the application, were also long;
- the boot option at the start of the artist list only, and in the background imperceptibly for the user — the loading of songs — a situation arose that the user clicked on the song before the file had time to load.
This stage ended with one conclusion: the file should be one and it should be in the archive, because The zip file reading itself is fairly fast.
Since the file size cannot be further reduced, it is possible to reduce the time it is read and loaded into memory. And then I took the path of custom serialization.
Binary serialization
Earlier, I knew that embedded serialization is slow and if you need speed, you need to use binary serialization.
SilverlightSerializer by Mike Talbot was chosen as a serializer.
In the Windows Console Application, everything worked as it should (serialization / deserialization), but with the reading of the created file in the Windows Phone project there were difficulties associated with different versions of mscorlib.
On the project description page there is a paragraph about compatibility issues between .NET and Silverlight projects:
This is a reference to the system. System, System.Core and mscorlib.
Unfortunately, to overcome this problem did not work.
Then I got to the protobuf-net project.
Mark the necessary classes and properties and get a binary file at the output:
[ProtoContract] public class Person { [ProtoMember(1)] public int Id {get;set;} [ProtoMember(2)] public string Name {get;set:} [ProtoMember(3)] public Address Address {get;set;} }
There was no problem reading the file from the Windows Phone project.
What is the result
The result was this solution:
1. Zip file, which contains the artists.bin binary, and the zip file is connected as Content;
2. With the help of BackgroundWorker, after loading the UI, we begin to load data (read the bin file directly from the zip file and deserialize it into the local data model):
public List<Artist> LoadData() { var info = Application.GetResourceStream(new Uri("artists.zip", UriKind.Relative)); var zipInfo = new StreamResourceInfo(info.Stream, null); using (var file = Application.GetResourceStream(zipInfo, new Uri("artists.bin", UriKind.Relative)).Stream) { return Serializer.Deserialize<List<Artist>>(file); } }
Total: application launch and data parsing on Lumia 800: ~ 5-6 seconds, after which you can easily view any content. The result could be better and I will continue research and work on productivity, but this result, in my opinion, is not bad enough.
There is another small bottleneck - during the transitions between texts within one artist or search results, the text parser can slow down a bit (I have already described the solution - you need to add animations that hide this artifact, maybe I will add it in the update if users like the application) .
In general, everything went for everything:
- 1 hour to write a parser of songs;
- 6 hours to research various options for improving performance;
- 2 hours to write, in fact, Windows Phone applications;
- 1 hour for drawing icons and creating graphic files;
- 0.25 hours to send for certification;
- 1.5 hours to write this article.
Total ~ 12 hours.
The result of the work can be seen and evaluated here .
Thanks for attention! I hope, Windows Phone Store and Habr became a little bit better after this article!
Source: https://habr.com/ru/post/167001/
All Articles