Fallout 76 beta self-destruct

“PC players # Fallout76 BETA: We are aware of the problems with the client and deal with it. Please do not click any buttons in the client-launcher yet. ”
- Bethesda support (@BethesdaSupport) October 30, 2018

The instructive story, which caused a lot of noise, happened this week with the company Bethesda. Bethesda games have always been famous for a lot of bugs ; Today, the company continues to follow the precepts of Buggerfall (TES II Daggerfall) and, it seems, this time it was able to reach a new level.

This is the start of the beta test of PC-version of Fallout 76 - a multiplayer spin-off of the well-known role-playing series, more reminiscent in its spirit of Rust - which took place this week, on October 30th. In fact, beta was a stress test of four short sessions lasting 4-5 hours - and the stress test (for CDN and users) can be called not only literally, but also figuratively.

The developers managed to view the bug in the game launcher, because of which its client arbitrarily deleted all game files from the user's computer when launching the launcher and / or clicking on any button in it. As a result, when the long-awaited beta (for which the players pre-ordered) finally started, tens of thousands of users who downloaded the game in advance were forced to download the entire installation files of the game again (almost 50 GB), and the download speed dropped to minimum values ​​(about 1Mb / s). When the beta ended 4 hours later, most of them were never able to get into Fallout 76.
')
If we consider that just a week ago, the company sent a letter to the subscribers that warned that they would be confronted with bugs unseen in the beta, then we can say that Bethesda did not let them down.

The essence of the problem

When Bethesda announced its new game, Fallout 76, at E3, it immediately announced that it would not use Steam to spread the game - instead, the company decided to use its own Bethesda.net launcher.

The reason for this decision is obvious - with this approach, Bethesda does not need to pay Valve deductions - for large companies that publish AAA games to Steam, which is about 20-25% (it all depends on the contract with a particular company - but less than for most developers). However, company director Pete Hines stated that this decision was made in order to “simplify communication with the player community” - since players now had to send feedback directly to Bethesda forums, bypassing Steam forums (and reviews).

For a while, the public was greatly outraged by the fact that now players would have to keep another launcher on their computer, and due to this, they predicted a game of sales failure. Looking ahead, we can say that the most pessimistic predictions did not come true - judging by the very lively forum, the game has already managed to cause some excitement; besides, Fallout 4 assets and game mechanics are actively used in Fallout 76, which significantly reduces the cost of producing such an experiment. The latter is not surprising, because there is a perception that this project Bethesda probes the ground for using the game-as-service model in its subsequent projects - it’s hard to figure out where else Skyrim could be ported after Amazon Alexa .

It would seem that the launcher and infrastructure could have been run a year ago - during the beta testing of Quake Champions, whose members complained about various problems then - but, apparently, it was decided to completely ignore them.

And the appointed day X for PC-players has finally arrived. Those who watched the countdown timer in the game launcher were already waiting for the first disappointment when it turned out that the timer worked two hours earlier - the Play button was unlocked, however, the game could not be started.

On the problem of the timer, Bethesda responded quickly enough and explained: they say, the programmer who was responsible for the countdown timer lives on Mountain Standard Time (MST), mountain time (this time, corresponding to the time zone of the Mountain States, one of the seven time zones of the USA; two hours after the Eastern Standard Time) - adding that this programmer had already been reset to the “factory settings”, and this should not happen again.

However, from this point on, the players' launchers began to massively erase the already installed game and download the installation files again. Faced with thousands of disgruntled players, the company responded by tweeting itself, which is quoted at the beginning of the post. "Please do not press any buttons in the launcher." He, alas, is now removed - but quickly scattered across the network, and its original can be seen, for example, in the article Kotaku .

However, according to the testimony of the players, it was not necessary to press anything - it was enough just to launch the launcher, as the game files disappeared or were erased. Those who learned about the problem earlier and managed to make a backup of the files tried to slip them to the launcher, but they were also disappointed - after spending half an hour checking these files, the launcher still started downloading installation files from scratch.

Here, another player was waiting for the upset players. Download speed suddenly dropped to 0.5-1 MB / s - and none of them had almost no chance to download the client for the 4 hours allocated to the beta.



After a couple of hours, the company's technical support published a tweet stating that the problem was fixed - however, irritated players continued to complain about the difficulties with downloading the game, and they quickly removed it.

And even at this the torment of individual users did not end. At the end of the download of files, the launcher could give the error "Not enough rights to install the game in the selected directory", after which ... I think you already guessed it. In the end, the individual "lucky ones" were forced to deflate the client completely 3 times.

It all ended with Bethesda extending the next day’s beta test period by four hours.

Possible causes and cost of error

Unfortunately, no official information on the cause of the error was received from the company, however there are some assumptions.

In fact, it can be quite trivial - most likely, it worked out the mechanism for updating the game incorrectly. For example, in the update it was necessary to replace the subdirectory, and for this it was necessary to delete it recursively by unpacking the new folder here. And there were more than enough ways to “mess up” here: transferring files could work with an error; somewhere a relative path to the folder could be incorrectly spelled out; the environment variable was set incorrectly or the path was specified; for the beta, special temporary flags in the code could be set that they forgot to reset ...

Let's try to roughly calculate how much this error could cost Bethesda. Bethesda uses AWS CloudFront as a CDN. It can be assumed that such a strong drop in download speed was due to throttling and incorrectly configured configuration parameters. Suppose a company pays $ 0.10 / Gb; each person who loads the game costs them about 50 cents. If, say, a million people had to download the game again, then this mistake cost the company $ 500,000. The question here is how many users this has affected - and it is difficult to answer it. For comparison, Fallout 4 for the first 24 hours after the release on Steam has sold 1.2 million copies, the peak number of users was 470,000 people.

And this is not even counting the pre-orders of digital and physical copies of the game canceled by disgruntled buyers, who took time off from work for the sake of being able to play beta.

P.S

In fairness, it is worth noting that Bethesda is not at all pioneers in this “genre” - more serious mistakes are known to history.

Habr's old-timers could have heard of the EVE Online update problem, which removed the C: \ boot.ini file from the players. For her, the developers from CCP apologized, at the same time detailing the reason for the error .

At one time, the first Half-Life (in the release version 1.0, which was recorded on the disc) included Sierra Utilities, which, when uninstalling the game, deleted not only the folder into which the game was installed, but also its parent directory. It is not difficult to imagine what kind of "joy" waited for those who were installed in c: \ games \ HL , c: \ program files \ hl or c: \ hl ...