The history of one facs Yandex. Navigator. In six acts with prologue and remorse
Fakapy happen and big companies, with testers and strict release procedures. On Monday, such a fix happened with us - we rolled out the Yandex.Navigator version for Android with an unpleasant error: the application kept the microphone on and recorded all the sound in a local file. As a result, the place on the device was quickly clogged up, and it just looked suspicious, although the contents of the file were never transmitted over the network.
Now the error has been fixed, in the version that does not contain it. We found the problem pretty quickly thanks to our users and within a few hours after the release, stopped distributing the update and promptly published a fix.
')
We decided not to be shy, but to share with you the experience that we have learned from this situation. Perhaps this will help you to be better. As usual, the reason was the combination of technological factors and communication between people. Details - under the cut.
First about what was new in this version. To prevent the driver from distracting from the road of interaction with the Navigator interface, we decided to implement his main scenarios without using his hands. To do this, we needed to build in the application voice activation, which allows you to call the voice interface on the command "Yandex". And to make it so that a person can confirm or reject a question from the Navigator by voice. For example, when rebuilding a route.
Yandex has its own SpeechKit recognition technology, which has already been used by the Navigator. But before, it only allowed the application to understand the commands after pressing a button (for example, building a route or searching on an organization or address map). The new version has the necessary voice activation and confirmation feature.
Navigator was the largest and most serious application in which the recognition version with voice control was used. Naturally, in the process of integration with it, the library itself developed and improved. And, of course, this happened in many iterations using the debug version of the library. This is where the confusion began.
When we approached the desired result, we assembled Navi (as we call the application within the team) with the final version of SpeechKit and began to check it for release. Testing went well and we were ready to launch.
When the final tests came to an end, we saw that the latest version of the library was writing too many logs, and they were suspicious. It turned out that we mistakenly took the debug version again. There was little time left, but we hoped that the release version of the library should not differ much from the debug version, and without deep testing we tried to assemble the Navigator with it. Alas, the application began to fall.
The crash occurred in the library, which is responsible for recognizing route confirmation commands. In it, sound processing is carried out in parallel, and for the convenience of writing parallel code and speed, a small set of primitive functions with different platform-dependent implementations are used. With each thread that was created by the library or in which the library was initialized, some data is associated (for example, the memory pool), and they must be initialized. Attempting to access the library from a thread in which this initialization was not performed led to a crash, so it was necessary to work with it from one thread. Unfortunately, this condition was sometimes not met due to an error.
It was relatively easy to fix the error, but there was no time left for testing with the new version, and we decided to roll back to the debug and run with it.
Very little time was left before the scheduled launch time. The next release was already ready, and did not want to pull. At the same time, we spent a lot of effort on testing the application with the mainstream version, as a result of which it became clear that recognition with it works quite well and its only drawback is that everything happens a little slower. Although the release version fix was trivial, its addition to Navi promised us a new testing process and a deadline for an unknown period due to the possibility of finding new bugs.
But the debug version had its own features. We knew that she wrote the error log, and this was another small plus of her, since it allowed us to collect information about failures on the first run. But the logs are not everything. A large amount of test data is critical to achieving maximum quality. For example, you need to understand when the library could not recognize the command, to have logs for the sound.
Another possibility of such a recording was made so that our employees could travel with a special assembly of the Navigator and assemble the test environment in real conditions in order to teach them voice technologies. Therefore, in the debug version, the sound was recorded on the sd card. This logic was turned on by definitions in the code, and was turned off in the release build. It should have been disabled and in debug mode, but no — the header file in which this macro was defined was skipped.
Naturally, the test plan did not describe sound recording and storing it on the device, because this functionality was not planned. A small change in the size of the application during testing was not noticed, since each time the application was called from the background, the file was overwritten again, which did not allow the sample to grow to noticeable values.
But in the process of movement, when the Navigator session does not stop for quite a long time, samples could grow to several gigs. And users noticed this within a few hours after launch. At this point, it became clear that the production got an erroneous code that was planned to be used exclusively for internal testing.
The consequences are already in the newspapers: www.vedomosti.ru/newspaper/articles/2015/09/08/608063-tainii-navigator-yandeksa .
As a result of small errors and assumptions, we received a serious problem, due to which some people, alas, lost confidence in the Navigator and even in general to Yandex. We all and I personally apologize to all those affected by this problem.
We, for ourselves, analyze the situation in detail and write an action plan that is needed so that the similar will never happen again. This is exactly the case when the final rules are written, albeit not in blood, as in aviation, but in pain from the loss of people's trust and reputation. I will not give a plan here; I think everyone can draw conclusions from the situation for himself.
Now the error has been fixed, in the version that does not contain it. We found the problem pretty quickly thanks to our users and within a few hours after the release, stopped distributing the update and promptly published a fix.
')
We decided not to be shy, but to share with you the experience that we have learned from this situation. Perhaps this will help you to be better. As usual, the reason was the combination of technological factors and communication between people. Details - under the cut.
Act One. Prologue. Everything fits perfectly
First about what was new in this version. To prevent the driver from distracting from the road of interaction with the Navigator interface, we decided to implement his main scenarios without using his hands. To do this, we needed to build in the application voice activation, which allows you to call the voice interface on the command "Yandex". And to make it so that a person can confirm or reject a question from the Navigator by voice. For example, when rebuilding a route.
Yandex has its own SpeechKit recognition technology, which has already been used by the Navigator. But before, it only allowed the application to understand the commands after pressing a button (for example, building a route or searching on an organization or address map). The new version has the necessary voice activation and confirmation feature.
Navigator was the largest and most serious application in which the recognition version with voice control was used. Naturally, in the process of integration with it, the library itself developed and improved. And, of course, this happened in many iterations using the debug version of the library. This is where the confusion began.
The second action. Confusion occurs
When we approached the desired result, we assembled Navi (as we call the application within the team) with the final version of SpeechKit and began to check it for release. Testing went well and we were ready to launch.
When the final tests came to an end, we saw that the latest version of the library was writing too many logs, and they were suspicious. It turned out that we mistakenly took the debug version again. There was little time left, but we hoped that the release version of the library should not differ much from the debug version, and without deep testing we tried to assemble the Navigator with it. Alas, the application began to fall.
The third action. Everything goes wrong and we hurry
The crash occurred in the library, which is responsible for recognizing route confirmation commands. In it, sound processing is carried out in parallel, and for the convenience of writing parallel code and speed, a small set of primitive functions with different platform-dependent implementations are used. With each thread that was created by the library or in which the library was initialized, some data is associated (for example, the memory pool), and they must be initialized. Attempting to access the library from a thread in which this initialization was not performed led to a crash, so it was necessary to work with it from one thread. Unfortunately, this condition was sometimes not met due to an error.
It was relatively easy to fix the error, but there was no time left for testing with the new version, and we decided to roll back to the debug and run with it.
The fourth action. Forerunner fakapa
Very little time was left before the scheduled launch time. The next release was already ready, and did not want to pull. At the same time, we spent a lot of effort on testing the application with the mainstream version, as a result of which it became clear that recognition with it works quite well and its only drawback is that everything happens a little slower. Although the release version fix was trivial, its addition to Navi promised us a new testing process and a deadline for an unknown period due to the possibility of finding new bugs.
But the debug version had its own features. We knew that she wrote the error log, and this was another small plus of her, since it allowed us to collect information about failures on the first run. But the logs are not everything. A large amount of test data is critical to achieving maximum quality. For example, you need to understand when the library could not recognize the command, to have logs for the sound.
Another possibility of such a recording was made so that our employees could travel with a special assembly of the Navigator and assemble the test environment in real conditions in order to teach them voice technologies. Therefore, in the debug version, the sound was recorded on the sd card. This logic was turned on by definitions in the code, and was turned off in the release build. It should have been disabled and in debug mode, but no — the header file in which this macro was defined was skipped.
The fifth action. Fakap
Naturally, the test plan did not describe sound recording and storing it on the device, because this functionality was not planned. A small change in the size of the application during testing was not noticed, since each time the application was called from the background, the file was overwritten again, which did not allow the sample to grow to noticeable values.
But in the process of movement, when the Navigator session does not stop for quite a long time, samples could grow to several gigs. And users noticed this within a few hours after launch. At this point, it became clear that the production got an erroneous code that was planned to be used exclusively for internal testing.
Action sixth. Consequences and remorse
The consequences are already in the newspapers: www.vedomosti.ru/newspaper/articles/2015/09/08/608063-tainii-navigator-yandeksa .
As a result of small errors and assumptions, we received a serious problem, due to which some people, alas, lost confidence in the Navigator and even in general to Yandex. We all and I personally apologize to all those affected by this problem.
We, for ourselves, analyze the situation in detail and write an action plan that is needed so that the similar will never happen again. This is exactly the case when the final rules are written, albeit not in blood, as in aviation, but in pain from the loss of people's trust and reputation. I will not give a plan here; I think everyone can draw conclusions from the situation for himself.
Source: https://habr.com/ru/post/266465/
All Articles