From tanks to nuclear power plants: looking back at Heisenbug 2017 Moscow

While after the Heisenbug conference, we collected and analyzed the audience feedback, a detailed post appeared on Habré IvanPonomarev with his impressions of the viewer. And in order not to repeat it, but to supplement it, we decided to build our text about the conference differently.

Not to describe all two days with a party and not to talk about the reports that Ivan has already described, but to see according to the audience estimates which of the rest the audience liked the most (all such marks turned out to be above 4.2), and tell a little about them. The result was a list of “six things that attracted Heisenbug viewers” ​​- it can be used to evaluate specific topics and their spread.
')

Tanks

In the summer at Petersburg's Heisenbug, there was a report on stress testing from Alexey Lavrenyuk , and Yandex.Tank was mentioned there, to which Alexei is directly related. This time, too, the conference didn’t do without tanks, but completely different: Alexander Shukov from Wargaming talked about what it was to do auto-tests in the case of World of Tanks . While microservices are raging somewhere in full swing, the games remain “monolithic” - so the game of such proportions is also a kind of “tank”, with which we must still be able to manage, and for this, Wargaming has made its own framework.

For whom was the report - for those who work directly in game devs (“learn from experience”), or, on the contrary, for everyone else (“while you are indulging in buns, this is what we are doing”)? Perhaps, for those and for others. You did not need to know the specifics of the game in advance - on the contrary, the report just helped to understand the main points. For example, it is easy not to think from the outside that the testing between games and MMOs greatly affects testing: it’s one thing to test a big project that “they have been preparing for two years, loudly releasing and exhaling”, and something else is continuously developing. And not only people from gamedev could benefit for themselves. In the end, monoliths not only in games remained.

Could it have been on the sidelines of the conference to extort from Alexander the way to cheat at World of Tanks? On the contrary, it was possible to get an explanation from him why this game is protected from cheating better than many others: almost everything is processed on the server and not on the client, so no matter how you try to cheat with the installed application, you can do little.

Smartphones: a pair of paired reports

iOS and Android divided the mobile world into two parts. It's funny that the same number pops up on Heisenbug in connection with the mobile theme: there were just two with the #mobile tag in the top 5 presentations of the audience, and each of them was represented by exactly two speakers.



The first one is “Testing geolocation in Badoo: cones, stones, crutches and a selfie stick” by Alexander Khozi and Nikolay Kozlov . For diteming with the “users nearby” function, geolocation is obviously important, and the careful approach of the speakers to the topic was illustrated by the fact that instead of “GPS” they said “GNSS” (“global navigation satellite system”): GPS is the most popular, but not the only satellite navigation system.

The tricks with geolocation testing begin with the fact that it implies movement in space. If you can call other functions of the application directly in your workplace, then how can you check the movement without leaving the office? As it turned out, the separation on Android and iOS here just affects. Both systems in official emulators can “twist” their position, but if Android can only just set specific coordinates (the default is Icelandic village Dalvik, which gave the name to the Android virtual machine), then iOS also has modes that simulate movement. But if there are not enough standard tools, third-party tools come to the rescue, which were also discussed. And later they also touched on power consumption - those who were in an unfamiliar place with a discharged phone, because they often looked at the map, they understand how sore this geolocation place is.



Just about energy consumption was the second favorite “mobile” report from Alexey Lavrenyuk and Timur Torubarov from Yandex. Moreover, they even began by saying that they had listened to the speech of Alexander and Nikolay. But then they went almost to the format "crazy hands." How to measure the power consumption of an application without spending time discharging a smartphone to zero? Theoretically, there are software ways to look at it, but for several reasons they did not suit it - for example, because of the insufficient frequency of measurements (besides, there the difference between Android and iOS also made itself felt). Among the third-party hardware-projects like Batt0r now also did not find the right one.

As a result, in Yandex they decided to act “hard” and “hardcore” themselves, opened the circuit, made their own Arduino-based circuit for measuring current and wrote their library to work with it - and at the conference they told about it in detail. In general, although Aleksey went far away from his spring report on load testing, here, too, was something to boot.

A / B tests

What could be easier to conduct A / B-test? He launched two different options for different audiences, compared, left in the production showed himself better.

But it was not there. Roman Poborchy in the title of the report loudly stated “Your A / B tests are broken,” and this provocative statement was supported by arguments. Here, for example, one of the first. Usually, when comparing, the results are considered to differ with a probability of 95%, and this number looks solid. But if instead of a single test, a whole series is carried out (“now we choose the best of dozens of options”), then the probability that at least one of them resulted in a random fluctuation increases sharply, which means that it’s very likely that you will consider users liked the "option is the one that actually shot by accident.

It is curious that in the audience reviews, Roman was praised both “for statistics with mathematics” and “for the fact that there were few statistics with mathematics”. There is an explanation for such a paradox: the report spoke about the importance of the statistical approach in A / B tests, but at the same time it was carefully structured so that an unprepared listener would not be buried under a pile of formulas and would not cease to understand anything.

White box

Nikita Makarov (Classmates) participates in the Heisenbug Program Committee, starting with the very first conference. Not surprisingly, he understands what reports the audience is waiting for, and his own turned out to be highly appreciated.

And Nikita is sure that in our time, testers need to understand the code. It is not surprising that his report was devoted to “white boxes”, and he immediately explained what the “black” limit.

Add a less predictable: a curious historical fact from his report. Chaos Monkey is now a well-known tool that randomly disables various elements of a large system, allowing you to find out how the system will react to such a failure. And Netflix, who created the Chaos Monkey, are now considered the main apologists for this approach. But far less it is known that this idea was first implemented in the 70s, and not at the software level, but at the “iron” one. With the development of circuitry, when the circuits became complex and the consequences of failure of any element are not obvious, they were physically excluded from the circuit to check them, thereby causing fault injection .

Error messages

Antonina Khisametdinova from the Pavlova Dog Company had a report on a topic related to testing. It is good to find errors and get rid of them, but inevitably situations arise when the end user encounters them. This means that the search is not limited to the search: you need to understand what the user should see in this case. And with this, everything is sadder now than I would like: many people act according to the principle “that there is nothing to mess with error messages, we need to cut the features”.

For example, what to do if the entire service has failed? It is clear that “to repair as soon as possible” - but is it possible at the same time to inform users of something better than “service is unavailable”? Yes: estimates of the time until recovery of its working capacity can be of help, as well as indicating alternative ways to get what you want, and notifying users before they get into the problem with their nose.

Separately, this is interesting: among other things, Antonina talked about the trust of users in infrastructure technologies like cash desks and ATMs, and that a mistake could undermine this trust. And after the conference, there was a massive failure of cash registers, becoming a clear illustration of such an error.

Nuclear power plants

Finally, there was a completely unexpected report. Vyacheslav Alenkov represented the ASE group of companies related to Rosatom, and nuclear energy is not the first thing that comes to mind at the words “testing conference”. And from the side it would be possible to assume that the construction of a nuclear power plant is a conservative area, where all the main principles were laid many years ago, innovations come unhurriedly, and the waterfall model reigns unshakably. Do we need total digitization of everything and agile when it’s not about a fashionable mobile photo service, but about a huge, complex and very responsible physical object?

It turned out that needed. Without flexibility today, simply nowhere - with the “waterfall” you will find yourself simply uncompetitive on the world market (“ASE” works in a number of countries). And the transition to digital projects occurred precisely for reasons related to testing. The error is much more expensive when it is physically embodied in the form of an incorrectly laid pipe — and the more you can catch them before implementation (for example, in a 3D model), the better. In general, everything is closer to the life of typical Heisenbug members than one might have thought.

Bugs

And finally, after the reports, we will share a couple of funny “bugs” of the past Heisenbug.



First, the badges were not designed for very long inscriptions, and, for example, Speaker Roman Poborchy turned out to be an “independent consultant on.” In general, of course, when organizing a testing conference, you should be prepared for the fact that participants can enter unnaturally long values ​​in any field!

Secondly, there was a very unexpected problem. In one of the halls, the image on the projector sometimes disappeared, and it was not immediately possible to understand that when speakers were walking on the carpets of this room, static was accumulating, which makes itself felt when you touch the laptop. This is really a tricky bug - try to reproduce this, until you understand what it is.

But we carefully fix all the problems that arise so that each subsequent conference will be better. So by the next Heisenbug (Petersburg) these bugs should be corrected. See you there!