Performance Analysis of Safari 3 for Windows
The other day bbsod published a note about independent testing of the Safari browser . I think we need to elaborate on this, for example, Google services such as Gmail and Google Maps in Safari 3 do not work as fast as in FireFox and IE, well, more on that below. So in this article we will focus on techniques for testing the speed of web browsers on the example of Safari .
Measuring Apple's real browser performance
On June 11, Apple released its OSX browser version of Safari 3.0 for Windows, stating that it is “the fastest Windows browser.” This statement was made on the basis of tests conducted by Apple, based on the iBench benchmark done by Zif Davis, who shares the results of the HTML, JavaScript performance tests and the launch time of the application. There are countless benchmarks to check performance, and we decided to check if Apple’s statement is true.
Test preparation
Other tests run Safari 3.0 Windows beta through benchmarks and check how they work in strictly defined test conditions. We were more interested in checking how this browser works directly when working with ordinary users. That is, we measured how Safari works while working with web pages, making up for this a few tests that differ from the usual browser benchmarks.
')
What are web pages?
When testing Safari 3.0 only on Gmail and Google Maps, it turned out that Safari 3.0 is slower than Firefox 2 and IE7 under Windows, but testing on the basis of only these two sites cannot show the full capabilities of the browser for all users. Of course, what should be checked on a larger number of websites, but on which ones? The problem was solved by selecting for testing sixteen English-language sites, which are on the first lines of the Alexa ranking on 08/29/07. This allowed us to test sites with completely different designs, ranging from the simplest pages on DHTML. This approach allowed us to take into account the maximum possible options for web page designs. A larger number of tested web pages would certainly increase accuracy, but sixteen is a good number for accurate and simple testing, plus all the most popular sites are covered at the same time.
Network Features
The main problem with testing the browser is the bandwidth. It can vary widely, and it is impossible to determine exactly when the speed is low due to server workload, and when there are problems in the browser itself. This problem can be somehow solved if measurements are taken at different times during the day, but it also does not provide perfect guarantees of accuracy. A server may be lightly loaded while testing one browser and then suddenly the server load will increase when another is tested.
The problem was solved by separate sites on the Internet, and "canned" sites. Testing for sites on the Internet was done several times and then the average value was derived.
The “preserved” statistics were made as follows: the site was completely saved to the local server, and then the statistics were measured for the local network. To make the channel bandwidth close to real, all browsers worked on the channel at a speed of 5 Mbps. In addition, we had two separate measurement techniques, which allows us to avoid errors. In other words, the difference in the browser should be the same in both groups of tests.
Note that we do not claim that this technique is better than any other. It has its drawbacks, and it should be perceived only as one of the possible points of view in assessing browser performance.
Measurement Technique
Performance measurement when working with web pages was made by our own software product Web Performance Analyzer Pro , which measures HTTP traffic. Some programs for testing are faced with a problem when using javascript to calculate the page load time, but this trouble was eliminated with our approach. The reason why our measurement technique is noticeably more accurate is that in order to increase the speed of displaying pages, the browser pre-displays the page as soon as possible, and at the same time pipelines HTTP requests for images. By the time the last image is loaded, the web page will already be fully displayed except for the last image, the download time of which can be neglected.
This technique has drawbacks. For example, pages with AJAX may not display correctly. Fortunately, the program groups HTTP traffic and therefore it is quite simple to determine the statistics associated with the page.
Caching
When determining the speed of loading a web page, two scenarios are possible. In the first case, when the user visits the page for the first time, in the second, when, when opening the page, most of the information is already cached. Both cases were tested separately in order to determine the difference in performance.
Realistic use
In order for the performance measurements to be real, the sites opened as if you were already a user of the resource. That is, the page with the registration of a new user has not been tested. For example, when testing Wikipedia, only the information pages were tested, not the default page, which allows the user to select a region.
Installations
The operating system used was Windows XP Professional SP 2, IIS 5.0 server with HTTP connection and compression enabled. The following browsers were tested: Firefox 2.0.0.6, Safari 3.0.3 and Internet Explorer 7.0.5730
Restrictions
Restrictions for this test were AJAX sites like GMail, as this has already been said, and the participation of such sites in the test can increase the error of the test. So this test rather measures the average performance of browsers when working with HTML, but cannot serve as a good indicator for sites of the GMail class.
Another limitation in the test was that the sites whose hosting was located in the USA were tested. This is due to the fact that if the site is hosted in Asia, then there is a high probability of random changes in the performance of servers, which leads to inaccurate data from online tests.
And finally, the technique when the site is completely transferred to a local server works fine only for simple sites. Sites with a more complex structure are quite difficult to correctly transfer to a local server. However, this does not contradict other features of the test.
Test data
Testing sites on a local server
The above sites were hosted on the local IIS server. Measurements were made for both the first visit, and for pages with cached content. It immediately became clear that no browser would be the best in all tests. One will be faster on one page, one on another.
Load times were averaged, which allowed us to understand the overall picture of the performance of all sites. The graph above shows that Safari 3 was the fastest, then Internet Explorer 7, then Firefox 2. Browser performance was higher on average when loading pages with cached content, but the places were distributed the same - Safari beta was the fastest, then IE7, then slow - Firefox 2.
Online Measurements
The selected web pages were downloaded again, but this time directly from the original servers. This time, the relationship between browser performance remained unchanged, but the overall performance increased due to the wider bandwidth of the channel.
If the tests with the local network were accurate, then online testing will be compared with previous results, and thus all conclusions will be made. The chart below shows the difference between download speeds on the local network and the Internet. Here you can see the relationship between browser performance based on the first page load. Safari 3.0 won, then IE7 comes in and Firefox 2 comes in third. When checking performance for cached pages, the places were distributed differently: a draw between Firefox and IE, but Safari was a little ahead of them.
Analysis
Which browser is faster?
In all cases, the Safari 3.0 beta browser for Windows proved to be the fastest. The difference between the average page load time for all tests is shown in this graph:
The download speed of a remote web page varies from 0.2 seconds for a cached page to 1.4 when the page is loaded for the first time. The difference of 0.2 seconds between Safari beta and Firefox 2 with IE7 is insignificant, and shows that for frequently visited pages we will not see serious gains in page loading speed.
A very different result for pages uploaded for the first time. Safari shows results 1.1 and 1.4 seconds better when loading a page on the Internet. The importance of such a difference in download speed is very individual, but for high-speed Internet these seconds will most likely be very noticeable.
For pages loaded from the local hosting, Safari 3.0 also showed better results, but the difference was less noticeable. So the performance of all browsers when working with LAN was about the same, which is quite expected.
Test logic
The point of repeating tests with a local network was to eliminate the possibility that the online test results were affected by a network disruption or server congestion. The table below shows that the difference in Safari download times over its competitors has remained unchanged both when working with a local network and during tests on the Internet.
Evaluating Apple's performance statements
Apple has literally stated the following: "Safari loads pages twice as fast as Internet Explorer 7 and 1.6 times faster than Firefox 2." Taking into account that Apple talked about the maximum possible numbers, we believe that they only spoke about the first page load, lowering the work with cached sites.
Apple said that Safari is 1.6 times faster than Firefox and showed graphics where the load time for Safari was 2.14 seconds, and for Firefox 2 it was 3.67 seconds. The difference in these figures is 1.7 seconds, which almost exactly corresponds to the results of our testing.
Regarding the comparison with IE7, Apple said that their browser is 2 times faster, and showed a graph where the coefficient was about 2.2. This is somewhat higher than the 1.4 coefficient, which we calculated, but still, Safari really is noticeably faster.
In any case, the reality of Apple’s statements will depend on whether you want to operate with specific numbers, or the basic concept that Safari is faster than Firefox 2 and IE7. We really confirmed the veracity of these statements, if the page is loaded for the first time, and questioned them if the page is already in the cache. However, it should be understood that the performance of the browser in each particular case will depend on what page you visit.
Want to keep abreast of all the news of the browser Safrari? Subscribe to our RSS
Measuring Apple's real browser performance
On June 11, Apple released its OSX browser version of Safari 3.0 for Windows, stating that it is “the fastest Windows browser.” This statement was made on the basis of tests conducted by Apple, based on the iBench benchmark done by Zif Davis, who shares the results of the HTML, JavaScript performance tests and the launch time of the application. There are countless benchmarks to check performance, and we decided to check if Apple’s statement is true.
Test preparation
Other tests run Safari 3.0 Windows beta through benchmarks and check how they work in strictly defined test conditions. We were more interested in checking how this browser works directly when working with ordinary users. That is, we measured how Safari works while working with web pages, making up for this a few tests that differ from the usual browser benchmarks.
')
What are web pages?
When testing Safari 3.0 only on Gmail and Google Maps, it turned out that Safari 3.0 is slower than Firefox 2 and IE7 under Windows, but testing on the basis of only these two sites cannot show the full capabilities of the browser for all users. Of course, what should be checked on a larger number of websites, but on which ones? The problem was solved by selecting for testing sixteen English-language sites, which are on the first lines of the Alexa ranking on 08/29/07. This allowed us to test sites with completely different designs, ranging from the simplest pages on DHTML. This approach allowed us to take into account the maximum possible options for web page designs. A larger number of tested web pages would certainly increase accuracy, but sixteen is a good number for accurate and simple testing, plus all the most popular sites are covered at the same time.
yahoo.com
msn.com
google.com
youtube.com
live.com
myspace.com
orkut.com
wikipedia.org
facebook.com
hi5.com
rapidshare.com
blogger.com
medaupload.com
microsoft.com
friendster.com
fotolog.net
Network Features
The main problem with testing the browser is the bandwidth. It can vary widely, and it is impossible to determine exactly when the speed is low due to server workload, and when there are problems in the browser itself. This problem can be somehow solved if measurements are taken at different times during the day, but it also does not provide perfect guarantees of accuracy. A server may be lightly loaded while testing one browser and then suddenly the server load will increase when another is tested.
The problem was solved by separate sites on the Internet, and "canned" sites. Testing for sites on the Internet was done several times and then the average value was derived.
The “preserved” statistics were made as follows: the site was completely saved to the local server, and then the statistics were measured for the local network. To make the channel bandwidth close to real, all browsers worked on the channel at a speed of 5 Mbps. In addition, we had two separate measurement techniques, which allows us to avoid errors. In other words, the difference in the browser should be the same in both groups of tests.
Note that we do not claim that this technique is better than any other. It has its drawbacks, and it should be perceived only as one of the possible points of view in assessing browser performance.
Measurement Technique
Performance measurement when working with web pages was made by our own software product Web Performance Analyzer Pro , which measures HTTP traffic. Some programs for testing are faced with a problem when using javascript to calculate the page load time, but this trouble was eliminated with our approach. The reason why our measurement technique is noticeably more accurate is that in order to increase the speed of displaying pages, the browser pre-displays the page as soon as possible, and at the same time pipelines HTTP requests for images. By the time the last image is loaded, the web page will already be fully displayed except for the last image, the download time of which can be neglected.
This technique has drawbacks. For example, pages with AJAX may not display correctly. Fortunately, the program groups HTTP traffic and therefore it is quite simple to determine the statistics associated with the page.
Caching
When determining the speed of loading a web page, two scenarios are possible. In the first case, when the user visits the page for the first time, in the second, when, when opening the page, most of the information is already cached. Both cases were tested separately in order to determine the difference in performance.
Realistic use
In order for the performance measurements to be real, the sites opened as if you were already a user of the resource. That is, the page with the registration of a new user has not been tested. For example, when testing Wikipedia, only the information pages were tested, not the default page, which allows the user to select a region.
Installations
The operating system used was Windows XP Professional SP 2, IIS 5.0 server with HTTP connection and compression enabled. The following browsers were tested: Firefox 2.0.0.6, Safari 3.0.3 and Internet Explorer 7.0.5730
Restrictions
Restrictions for this test were AJAX sites like GMail, as this has already been said, and the participation of such sites in the test can increase the error of the test. So this test rather measures the average performance of browsers when working with HTML, but cannot serve as a good indicator for sites of the GMail class.
Another limitation in the test was that the sites whose hosting was located in the USA were tested. This is due to the fact that if the site is hosted in Asia, then there is a high probability of random changes in the performance of servers, which leads to inaccurate data from online tests.
And finally, the technique when the site is completely transferred to a local server works fine only for simple sites. Sites with a more complex structure are quite difficult to correctly transfer to a local server. However, this does not contradict other features of the test.
Test data
Testing sites on a local server
The above sites were hosted on the local IIS server. Measurements were made for both the first visit, and for pages with cached content. It immediately became clear that no browser would be the best in all tests. One will be faster on one page, one on another.
Load times were averaged, which allowed us to understand the overall picture of the performance of all sites. The graph above shows that Safari 3 was the fastest, then Internet Explorer 7, then Firefox 2. Browser performance was higher on average when loading pages with cached content, but the places were distributed the same - Safari beta was the fastest, then IE7, then slow - Firefox 2.
Online Measurements
The selected web pages were downloaded again, but this time directly from the original servers. This time, the relationship between browser performance remained unchanged, but the overall performance increased due to the wider bandwidth of the channel.
If the tests with the local network were accurate, then online testing will be compared with previous results, and thus all conclusions will be made. The chart below shows the difference between download speeds on the local network and the Internet. Here you can see the relationship between browser performance based on the first page load. Safari 3.0 won, then IE7 comes in and Firefox 2 comes in third. When checking performance for cached pages, the places were distributed differently: a draw between Firefox and IE, but Safari was a little ahead of them.
Analysis
Which browser is faster?
In all cases, the Safari 3.0 beta browser for Windows proved to be the fastest. The difference between the average page load time for all tests is shown in this graph:
The download speed of a remote web page varies from 0.2 seconds for a cached page to 1.4 when the page is loaded for the first time. The difference of 0.2 seconds between Safari beta and Firefox 2 with IE7 is insignificant, and shows that for frequently visited pages we will not see serious gains in page loading speed.
A very different result for pages uploaded for the first time. Safari shows results 1.1 and 1.4 seconds better when loading a page on the Internet. The importance of such a difference in download speed is very individual, but for high-speed Internet these seconds will most likely be very noticeable.
For pages loaded from the local hosting, Safari 3.0 also showed better results, but the difference was less noticeable. So the performance of all browsers when working with LAN was about the same, which is quite expected.
Test logic
The point of repeating tests with a local network was to eliminate the possibility that the online test results were affected by a network disruption or server congestion. The table below shows that the difference in Safari download times over its competitors has remained unchanged both when working with a local network and during tests on the Internet.
Evaluating Apple's performance statements
Apple has literally stated the following: "Safari loads pages twice as fast as Internet Explorer 7 and 1.6 times faster than Firefox 2." Taking into account that Apple talked about the maximum possible numbers, we believe that they only spoke about the first page load, lowering the work with cached sites.
Apple said that Safari is 1.6 times faster than Firefox and showed graphics where the load time for Safari was 2.14 seconds, and for Firefox 2 it was 3.67 seconds. The difference in these figures is 1.7 seconds, which almost exactly corresponds to the results of our testing.
Regarding the comparison with IE7, Apple said that their browser is 2 times faster, and showed a graph where the coefficient was about 2.2. This is somewhat higher than the 1.4 coefficient, which we calculated, but still, Safari really is noticeably faster.
In any case, the reality of Apple’s statements will depend on whether you want to operate with specific numbers, or the basic concept that Safari is faster than Firefox 2 and IE7. We really confirmed the veracity of these statements, if the page is loaded for the first time, and questioned them if the page is already in the cache. However, it should be understood that the performance of the browser in each particular case will depend on what page you visit.
Want to keep abreast of all the news of the browser Safrari? Subscribe to our RSS
Source: https://habr.com/ru/post/31287/
All Articles