Test it all: how the Heisenbug 2018 Piter went

If you try to describe the past Heisenbug in one word, it will be "diversity." Speakers from giant companies and young startups, topics from testing mobile games to testing blockchain, reports with a bunch of code and absolutely no code; finally, there were no reports at all, but the “birds of a feather” session.

Probably the best way to talk about such an event is not to try to find a “common denominator”, but to cite several different examples of what could be heard at the last conference. What we did under the cut.

Fuzzing testing and compilers

')
Do you think that you have a very difficult and responsible job? We thought so about ours too, until we listened to the report of Maxim Kazantsev (Azul Systems) and did not realize what it was like when testing compilers.

First, there, users think “it should always work correctly”: if people are ready to understand and forgive a mistake in a mobile application, the situation “the compiler made a mistake in my program” causes confusion at best. Secondly, from the faith of users "there can not be bugs" the probability of their appearance does not disappear - even on the contrary, in a project of such scale and complexity it is even more difficult to deal with them. And thirdly, the Falcon JIT compiler, which Maxim works on, is now very actively developing - that is, with such complexity and such quality requirements, it is necessary to test very large innovations in a very short time.

But at the same time, the report was devoted not to what people go through, about whose work we usually do not even think about. It was dedicated to Fuzzing testing, which is able to help both when working on compilers, and in completely different projects. Its essence is to automatically generate millions of different tests with an element of randomness, “blindly firing in all directions”: under certain conditions, the probability of getting hit will be higher than it would have been with slow aimed fire.

In general, it is not clear in what time a million monkeys could write War and Peace, but they can test the compiler with proper supervision within a reasonable time.

---

Zeptolab and Autotesting Games

“Mobile game” may not sound as responsibly as “JIT-compiler”, but from the point of view of testing it is also another task. While the “usual” mobile testing uses a whole set of automated tools, in the game many of them turn out to be unsuitable: the interface elements do not have standard view id, on different devices everything can load at completely different speeds, and in general a lot of its complex specifics.

Not surprisingly, game development from the outside may seem unsuitable for automated testing. And when creating the superhit, Cut the Rope of Zeptolab did not help the idea of ​​logging the actions of manual testers: yes, you can record at what point the tap or svayp occurred, but this log should not be reused on a device with a different screen resolution or a less powerful processor.

However, in this Zeptolab did not bury the idea of ​​automation, when working on the game King of Thieves they returned to it - and there they abstracted both from the exact coordinates “which pixel they poked” and from the exact time periods, learning instead to determine the essence of tapas. And now Dmitry Alekseev and Evgeny Shumakov told about it. It is curious that at one of the past Heisenbug, Philippe Keks spoke with the theme “How to teach robots to play games”, but there it was about a game with a very straightforward gameplay (DREG-Racing) - and the specifics of King of Thieves are different. And it is also interesting that the Appium project was useful to the company: its creator Dan Kuyayar also spoke at Heisenbug.

---

Configuration Testing and Developers

For ambitious tasks like “we automate non-automated”, it is easy to forget about less spectacular, but no less necessary. Fortunately, Heisenbug had someone to remind about them.

For example, everyone remembers and thinks about the “core” code from developers, realizing the importance of testing business logic — but what is related to configuration can easily escape from attention as an “insignificant auxiliary entity”. But Ruslan Cheremin reminded that, in general, with this part, everything is in some sense even more complicated. It can also lead to mistakes that cost a business money, and in doing so, it usually depends on the environment. And this means that what seems to be tested “on my machine” may surprise in production.

The report, in fact, developed the theme of Andrei Satarin from the previous Heisenbug (moving from the general to the more specific), and also well illustrated the Heisenbug slogan “about testing not only for testers”. The visitors of our Java conferences have long known Ruslana (we organized the JUG meeting with him back in 2012), and here he gave exactly the look “from the developer’s side,” and his report suggested that the audience did not hear the word “Java” for the first time.

---

Yandex, VK and crowdsourcing

Immediately, two large companies at Heisenbug shared how they use not only regular staff members for testing, but also wider circles: Olga Megorskaya talked about working with the “assessors” of the Yandex.Tolok project, and Anastasia Semenyuk about VK Testers.

What are the advantages of such an approach, what difficulties, and how to overcome these difficulties? For example, Olga said that this has allowed to expand the “throughput” of testing and save on QA-outsourcing, so that many large Yandex projects are already actively using the new process, but it has not been without difficulties too. Using thousands of “non-testers” even for routine tasks means that these tasks need to be described in detail in a language they understand, and the developers were not always ready to do it. As a result, an intermediate layer of “experienced assessors” helps, receiving test cases in an arbitrary form and translating them into a clear detailed algorithm: they just understand what questions other assessors will have.

As many participants of Heisenbug noted in the reviews, these reports were interesting, but not too applicable for "ordinary" companies: when you do not have an army of enthusiastic loyal users, like VK, nor large-scale crowdsourcing project, like Yandex, attempts to do something then a similar may not be appropriate. But just the uniqueness of these situations made the reports themselves unique: if you can listen to a lot of “typical experience” from someone, then something similar can be heard only from these people. As Olga noted, while constructing its processes, Yandex could not even learn from other people's experience, and had to fill all the bumps on its own.

---

Vitaly Friedman and UX

Vitaly Friedman is widely known, but not in the test circles: the website Smashing Magazine he founded for web designers and web developers is very much appreciated in the relevant industry. His reports are also met with a bang, but usually at completely different conferences. However, such important topics for Vitali as UI / UX also require testing, and he spoke to an atypical public.

How does the element of the "carousel" on the Turkish sites? Why is it better not to use it, and if you still use it, how? Which site probably contains the world's longest comparison of product characteristics, and what could be done to make it convenient to use? Why in such comparisons need the ability to change the column in some places? What is the rating for the product is ideal, if "5.0" feels twisted and deceitful? According to which checklist “did we take this into account” is it worth walking through when implementing the “accordion” pattern?

All this does not look like questions for a testing conference - and the report did turn out to be a kind of “offtopic”. However, audience reviews showed that including it in the program was the right decision: many expressed themselves in the spirit of “yes, not about testing, but it was amazing.”

(And if you are interested in the question about the "accordion" - Vitali has a big article about this pattern, and there is the mentioned checklist at the end.)

---

BoF and format experiments

Among the reports there were many interesting things - however, everything about the format of the reports is generally understandable. And there was another format on Heisenbag that was not held at this conference earlier. The evening of the first day, in addition to the party and sports "What? Where? When? ”, The BoF sessions were held (the name of the format was due to the English proverb“ birds of a feather flock together ”, roughly corresponding to“ two pair of boots ”).

What did they represent? Chairs are located in a circle, some of the seats are occupied by speakers, some of the spectators - and the discussion of a predetermined topic begins. Is it often possible to see both Michael Bolton and Simon Stewart participating in the same conversation at once?

There were two sessions: in Russian they talked about testing in production, in English - about the eternal dichotomy "to use a ready-made tool or to cut your bicycle." More viewers gathered Russian-speaking, but both were quite lively.

---

Michael Bolton (and point)

In principle, this could be limited to a name that speaks for itself in the testing community. However, there are cases when a person with a well-deserved reputation is not the brightest speaker. As organizers, we focus on the audience feedback - and when, after the very first Heisenbug, the responses to the Rex Black performance turned out to be not very enthusiastic, they wound it on a mustache. We are glad to inform you that everything is different with Bolton: his closing keynote Testers are their worse enemies has gathered very inspired reviews.

This is probably due to the fact that Bolton was very "alive": he absolutely did not bronze in his regalia, jokes on and off the stage, drags two glasses of beer at the BoF session at once (“I’m very interested in how many glasses can be effectively carry at the same time? ") and immediately creates a relaxing informal atmosphere.

But “informal” does not mean “unprofessional”: in his speech, he thoughtfully walked through what he considered serious problems. “People confuse testing with simple build checks. There is a definition of a program as a “set of instructions for a computer,” and I see a problem in it. It is like giving the word “house” the definition of “building materials collected in a certain way”. It is reasonable to define a house as a place where people live. And the program is like what people use. We are fixated on testing tools, and I’m not against the tools themselves, but we use them as a way to avoid contact with people. ”

---

There was still a lot of interesting things - from the story of Artyom Eroshenko about the next version of the Allure Framework to the talk of Alexey Rodionov about how Petri nets can help with testing. But one could continue for so long that it would be better to dwell on this moment. If you forgot to mention something very important or wrote something incorrectly, we accept bug reports in a personal. And we begin to wait for the next Heisenbug, which will be held in Moscow in December!