Valid sturgeon freshness data

The second freshness is what nonsense! Freshness is only one - the first, it is also the last. And if sturgeon second freshness, it means that it is rotten!

The end of 2012 turned out to be generous for projects with foreign partners. Personally, in addition to expanding the geography of my portfolio, this allowed me to change my attitude towards the usual approaches and methods.

On usability tests, we fix the numerical indicators. Previously recorded time, success, frequency; On the basis of these data, formulas were built in the hope of obtaining some kind of integral indicator of criticality of problems. They even dreamed of finding (or creating) a universal indicator of usability applicable to any system.
')
At some point we began to refuse to fix the time: too many conditions made this indicator unreliable. For example, on tests in the “Think Aloud” format, the method itself introduced a hefty error — until the person states everything that came to his head, a lot of time will pass, and it is thought out loud differently.

Nevertheless, we kept mathematics with all our might. And in the name of mathematics, the test script was strictly observed: after all, if we need to calculate the average or the percentage, then each respondent must go through all the steps of the script, no matter how much this corresponds to the actual user experience.

Sometimes I felt simply ridiculous: it happened that the person on the test had already achieved the goal by some kind of abnormal, but quite effective method, but instead of congratulating him on his victory, I had to beg to go along the path prescribed in the script. “Well, you found the right song / changed the tariff / saved the contract, but let's try to do it differently,” I asked purely for the sake of validity of the data. Fortunately, no one ever sent me away in response to such a request.

World harmony and a tear of the child

One of the foreign partners, who ordered testing for us, sent a sample report in which there were practically no numerical data. There was only one table - with a list of test participants. The criticality of problems was pointed out, but it was derived not by the formula, but expertly. Probably, due to the stereotype of thinking, I nevertheless prepared our typical tabular summary of the results of testing, although I cut it pretty badly - prior to ranking by the frequency of problems. “Thank you,” the partner wrote to me, “but you don’t show these data to the general order. We know that a problem is a problem, and that it doesn’t matter on such a sample, 2 or 3 people have encountered it. And in a large corporation, these numbers can be a reason to dictate what is important and what is not. ”

In fact, a colleague (here it is necessary to clarify that the customer in this case was also a usability company, we were a local contractor for them) voiced to me what is preached in our company: conducting qualitative, not quantitative, research. That is the main thing - to identify the problem, and not to measure and weigh it. Another colleague was right: if you show business figures that are too low or too high, then the most obvious action would seem to change these indicators, and not to solve the problem fundamentally and essentially.


Comics: xkcd.com

In September 2012, our company held the UX Masterclass in Moscow. Among the speakers was Gavin Lee of User Sentric, and among other things, he defended the often contested idea that usability could be measured. For example, he cited the stupid dialog box of the medical program and asked the audience in the hall to estimate how long it would take to respond, and what the probability of an error was.



Of course, you can measure everything. Including the costs of the deceased as a result of patient error, and even that, is it worth all the world's harmony of tears tortured child.

Of course, here I will be rightly reminded of the shortage of resources that can be spent on corrections in the interface - in my opinion, only for the distribution of these resources and carry out measurements, and not for the interface itself. I do not have an answer to this remark. I cannot say whether it is worth spending oncology more or less than on cardiovascular diseases, under the pretext that cancer patients suffer more or that there are more cores in number.

Is it possible to find out the truth from non-valid data?

Another foreign client was represented by a whole team: developer, manager, UX specialist. When discussing the work, the developer told us sternly, almost rudely: “You in our industry do not understand a damn, therefore give us raw data, without any ratings, rankings or conclusions. We'll figure it out. ” This client was present at all tests and actively participated in them. For example, this way: in the course of the test I asked to change the conditions of the task. Our UX-colleague sighed: no valid data remained. I almost flinched when I heard this, because the wording almost word for word repeated those words that I had heard at the beginning of my usability path.

But the rudeness of the developer caused me more positive feelings than the sigh of a colleague. It was clear that the person is a fan of the product and wants to make it perfect. He does not care how many times this or that problem manifested itself. If there is a problem, the product needs to be improved. Criticality, it seems, did not interest him much, just as a demanding cook does not care about the degree of freshness of sturgeon. It should be fresh, and that's it. The accuracy of measuring rottenness is completely irrelevant when it comes to a dinner party.

I have no doubt that the developer set priorities for himself and, maybe, even used numbers. In the end, the expert assessment (perhaps the second most frequent method after usability testing) is determined by the expert’s criticality, and no one reproaches it with a disability. The more problems we reveal, even if they are insignificant or completely random, the more holistic our vision of the product will be.

There came another foreign voice. True, its owner was not our client or partner, but he is an authority on UX. This is Jared Spool. First, he talked about the " death from a thousand cuts " - when a lot of seemingly minor problems eventually lead to serious dissatisfaction with the product. Secondly, he criticized the hard scenario approach to testing for the fact that because of him we get the same result programmed by us, and not real information about the user experience.

Instead, Spoole offers improvisation - not to follow the script strictly, but to adapt to the expectations and goals of the user. Graphs and charts will go in this case to Woland. In my opinion, in most cases there is a road to them. It is when viewing graphs and diagrams that the focus of our discipline is blurred: it begins to seem that a problem occurs where indicators have risen or dropped to a certain level, and not where the user, instead of his main job, is forced to engage in a war with the interface. This sacrifice (the impossibility of collecting valid data) is worth it, if in return we find out whether a sturgeon really seems fresh to a person sitting in front of us it is convenient to work with the product, or he simply performs the tasks that we gave him.

Author: Anton Alyabyev, analyst at UIDesign Group.

PS By the way, this week our company is 10 years old. There is something to tell here, so we decided to describe the entire history of the UIDesign Group (from part-time work on weekends to entering the international market) in a series of articles that we will post on our “regular” blog (not on a habr because not quite the format). If suddenly someone will be interested - you can read the first article on this link .