Apple fixed a bug and accuses Consumer Reports of incorrect testing methodology.

That same setting Safari, because of which the dispute broke out

Two weeks ago, fans of Apple products were surprised by the news: for the first time, the new MacBooks did not receive a recommendation from Consumer Reports for purchase.

The highest quality of Apple notebooks is well known. This technique is used by the best hackers, even if you have to reinstall the operating system. There has not been a single case where laptops have not received a recommendation to purchase Consumer Reports. But this happened with the latest line of MacBook Pro due to inconsistent results on battery life when opening sites in the Safari browser.
')
Immediately the thought crept in that there was some kind of misunderstanding. And so it happened - Apple found and fixed a very strange bug . But now she blames the “too smart” experts of Consumer Reports, who managed to manifest this bug. Like, they have the wrong methodology. Who is right?

Those results really turned out to be extremely strange. Judge for yourself. A 13-inch MacBook Pro with a touchbar worked 16 hours in the first test, 12.75 hours in the second test and 3.75 hours in the third.

The 13-inch MacBook Pro without a touchbar worked 19.5 hours in the first test and 4.5 hours in the second.

For the 15-inch model, the figures were 18.5 and 8 hours.

The test consists of an infinite number of cycles to open 10 pages in the browser, which are transmitted from the local server via WiFi. The cycle starts when the battery is fully charged, and ends when the laptop is turned off. The test was conducted on a standard browser (in this case, Safari) on the OS with the latest patches. The test began a few weeks ago, then continued after the release of macOS Sierra 10.12.2, but there was no difference.

Representatives of Consumer Reports explained that usually the difference between the results of the same tests does not exceed 5%, but here the difference is too great.

Now, in a press comment, Apple explained what was happening : "During testing of the batteries, Consumer Reports used hidden settings for developers instead of the normal settings that people use on a daily basis."

This is a setting that disables the cache in the browser. Experts from a reputable magazine specifically turned off the cache to make the test results more realistic, because during testing the script downloaded the same set of sites over WiFi. If they left the cache enabled, the battery life would not show the true state of affairs, because when loading the site from the cache, data transfer via WiFi is not activated.

Loading sites with a disabled cache in the browser is quite a valid and logical testing methodology.

But the unexpected happened here. As a joint investigation by Apple and Consumer Reports showed, activating this particular setting for developers simultaneously triggered an “unclear and intermittent” bug that reloaded icons (for a more detailed description of the bug, see the foundout user comment ). It is this reason that explains such unusual test results for battery life.

When running tests with normal settings, the bug does not appear.

In the end, now Apple has still fixed this unfortunate bug, so the Consumer Reports editors can once again test and assign a high rating and recommendation to buy to MacBooks.

But Apple and Consumer Reports look at the problem with tests differently. Apple believes that the problem is in their methodology, and Consumer Reports says that laptops flooded the test because of a bug .



In defense of its methodology, Consumer Reports writes that they test all laptops in a uniform way. And since it is impossible to emulate user behavior - everyone uses a laptop in different ways, their tests for battery life are not intended to be a direct emulation of user behavior. Instead, they are designed to take into account the maximum number of variables and produce a result as close as possible to reality with an average load of processors, memory, laptop transceivers and display operation. “This test served as a good indicator of battery life on hundreds of laptops in our ratings,” writes Consumer Reports. But not for the MacBook.

The Safari browser fix is ​​published on the Apple Beta Software Program website and is available to members of this program. After beta testing, in a few weeks the update will be sent to the computers of all users through an automatic update program. Perhaps fixing a strange bug will help those users who complain on the forums that the laptop has too little battery life, which only occasionally shows up.

Some bugs in modern software are so strange that it is almost impossible to identify their true cause and determine when they will manifest. It seems that this bug in Safari is just one of those.

But it is worth paying attention once again to how professional the Apple PR machine works . Experts present the situation in such a way that the point is not at all in the Safari bug. In their opinion, Consumer Reports has a problem in the methodology, and Apple helped them to identify this problem in order to open the eyes of unworthy and unreasonable testers.