DIY: Mobile Testing

When our company was invited to speak at the Mobile Developer & Business Days conference with the topic “Features of rapid testing of mobile interfaces,” we agreed without hesitation. What, then, we tested a lot of this stuff. But I just introduced myself to the picture: here I am setting out these very features, and they ask me to tell about some project, expressively fast ... Actually, we all passed the tests very quickly. The lion's share of time was usually spent on paperwork, coordination, recruit. However, it was too late to refuse to speak. Therefore, I had to admit on one of the first slides that we have all the tests fast, and the method of speed to build on what can be omitted, both in conducting the test and analyzing the results.

Story

On the left in the picture above - my grandmother, here she is 25-30 years old. On the right is the Motorola phone, which we bought to our grandmother when she was 80 years old. Then, probably, a passage is expected about how much mobile technology has covered all layers of users. Or how hard it is for older people to use modern mobile technology. This is all true, but it was hard not for my grandmother, but for me.
')
My grandmother voluntarily limited her competence in using the phone to calls, using it for hours and charging the battery. But she sometimes forgot about the battery, and the phone lost its clock. Of course, all debugging and preventive work went to me, as to the closest grandson. So, I spent up to 10 minutes to set the time. More truly, on search in the menu of the necessary point. At first I tried to include intuition and logic, choosing the most likely points. Then he moved on to complete exhaustive search, except perhaps for games. Grandma waved her hand - "Well, to hell with it." But I could not destroy the image of a caring grandson and a modern person, I fought to the end.

What is significant, the same pattern was repeated with each next need to set the time. When another grandson was added to the script (my grandmother has eight), the result did not improve.

Not such a big problem to throw the phone in the trash, the main task he solved - you could call from the house to go home. But the next time I chose a mobile phone after such a blow to the self-esteem inflicted by Motorola, I would look at its phones last. By the way, after some time Motorola itself left the European and Russian market. Is that why? .. And those who read Cooper will recall his assessment of the Razr model - an excellent industrial design and Frankenstein instead of firmware.

If the developers had spent a day or two testing my grandmother’s phone menu, then perhaps by rearranging a few lines of code they would have earned him the reputation of a normal phone and would get a loyal customer. But alas.

***
There are a lot of phone manufacturers, software developers are legion. There are SDK, emulators, test automation tools - but only functional ones. You can force the robot to “push” the buttons and pull the controls of your software, either randomly or according to a recorded script. But user testing tools are few. And they are focused mainly on the tests of sites opened from the phone, but not on the applications and, even more so, not on the firmware itself.

The hardware part of the toolkit is also a solid DYI: regular webcams, improvised holders for them, searching for a position so that there is no glare and the camera does not obscure the view to the user. Here is an article in confirmation - that even profile specialists are forced to invent and use improvised means. And here is one of the few examples of quasi-industrial assistant, created specifically for mobile research - http://www.mrtappy.com/ . And then for 295 ye you will receive not a product yet, but a designer:

What will you get in the box?



Even our western colleague said that this is unreasonably expensive, although in general the cost of usability research in the West is higher.

But any developer understands that power is knowledge! Functional code can be written in notepad, if you know how to write code. Also in testing - if you know what to look at, then a productive testing session is possible in the subway over the shoulder of a fellow traveler.

What to test?

Everything. If you do not have enough time, then test everything that you doubt. And just in case, what you are most confident about. Confidence breeds blindness and leads to annoying omissions.

When to test?

Once you understand exactly what you want to give to users. Perhaps at the same time find out if they want to get it.

You can start even earlier - if you just can not understand what exactly you want to give to users. In the process of communicating with them, your idea may ripen or fade.

Do not wait!

Do not wait for the release, design or approval of the stakeholders. Firstly, often neither the release nor the design changes the user experience enough to devalue the test results on the beta version. We once waited for a release from the customer for about half a year. The product was already hanging in appstore, collecting negative reviews, but we were not allowed to sign off. When it was given, we saw that practically nothing had changed in the product.

Secondly, for sure, for any stage of the project you can come up with a suitable testing method. A paper prototype will endure everything. Even if he does little, he will take a little away.

Where to find users?

If you work in any office center, then look for them in the corridor. If you have your own building, use colleagues who are not involved in the project. If everyone is working on the same project, contact your relatives and friends.

This method does not help if the target audience must have specific skills. For example, an application for a tobacco supply manager will put the average user at a dead end in business terminology. Although as a crash test, you can go for it - for a good business application, an untrained user will be able to at least express hypotheses about his purpose. For the bad - do not understand anything at all.

From which side to go to the user?

So, you have a product or product idea, users and the determination to present the product to users. From which side to go?

In general, the user has three sides: desire, opinion and experience. Usability usually comes from the side of experience. Desire and opinion are the path of marketing, and bad marketing works only on opinion, and good marketing also on desire. Why is that?

Opinion about the product - you can have without using the product and not wanting it. That is, it may be unfounded, unmotivated, unreliable. It is difficult to isolate reasonable, motivated and reliable opinions, but we are not telepaths.

Desire - you can have, without possessing neither experience nor opinion. But desire is valuable in itself, since it is precisely it that turns into money.

Experience - you can have only if you use the product. Desire is not required.

The desire, though not necessary, is generally much easier to test the product on motivated users - they are more cheerful, more talkative, more sincere. They also have flaws - motivated users will “swallow” some of the problems without noticing them.

Looking ahead, already somewhere in the distance gave, we can say that Success = strong desire + successful experience. How to add at least one known value to this equation? In our case, this is clearly not an experience, so we will think about desire.

On tests, one thing can often be noticed: if a test is conducted by a girl, and the respondent is a man, the latter is inclined to “wrestle” somewhat, to exaggerate the subjective assessment of their success in performing tasks. In this case, extraneous motivation works - not to the product itself or to the goals, with the help of its achievable. But still a person will enthusiastically poke at all buttons, trying to achieve the goal. How to put this into practice quick testing? Very simply, a little bit to rectify the situation with my grandmother.

You dress up in a blonde, go out into the corridor, ask for the first oncoming young man, “Oh, you know, I can't put the watch on my phone ... Can you help me?” And smile pitifully. Done, you are a motivated user with a specific goal.

Get ready for the negative

Although in real projects we fix both disadvantages and merits, for therapeutic purposes, take advice: look only for problems. Do not expect users to praise. The more swearing and bewilderment you got, the better you did usability testing.
The picture where the developer is watching the tests and sputtering in a scream “Where did you get these idiots?” Is true.



Be prepared for the fact that users are stupid. But you make a product for them. You might even expect to get money from them. Perhaps you expect to do this regularly. Therefore be indulgent. It doesn't matter if the interface is bad or the user is stupid. It is important that they find a common language. If you have the strength and knowledge to create smart, solvent and eager users for your product - ok, go ahead! If not, customize your interface.

Do not suggest

Although you really can get caught by not very clever users, they still can not tell and give instructions. Your task is not to force the user to go to the end and hear his opinion (this is to the marketing). Your task is to understand whether the user will be able to independently reach your goal through your interface.

Set a goal

The goal is to live outside the interface, outside the product. At the same time, the goal is not to “press the last button on the last step of setting up the phone,” the goal is to “get a phone that is suitable for solving problems in real life” —for example, showing the correct time.

If you do not have time to compose plausible plots (about grandma and setting the clock), you can stay within the interface. Just formulate the tasks with the words "Suppose you want to change the time on the phone." If the user accepted the assumption that he wants something, he thereby entered the game and began to portray at least some motivated person.

Make sure that after the words “you want”, the instruction “... go to the“ settings ”section, go to the“ general ”item, then select the“ main ”sub-item ...

How to fix the results?

First, what is the result of testing? Simply put, usability is a balance between the number of steps to the goal and the complexity of these steps. To fix the number of steps is easy, but the complexity is difficult. Although they say that the quality of the interface is measured in decibels and the censorship of expressions, it is still suggested that time should be considered an indicator of complexity, based on the logic “if it is difficult, you have to think long”. Accordingly, in fast testing it will be enough to have such a matrix:



As you can see, there is even some kind of diagnostics, indicating what type of problems the user has encountered.

You can fix these values ​​directly with your head, pen in a notebook, web or action camera (you can hide it in the head of the blonde who asks a young man to help her with the phone). If you are able to write a simple logger that saves the number of clicks (clicks, tapes, swipes and other user movements) and the duration of the session, then you will get a primitive, but automated usability meter *.

* Of course, there are limitations and exceptions. Such a matrix is ​​poorly applicable to entertainment items: if I tapped the phone screen twice, then I stared at the screen for an hour and a half, and then turned it off, it does not necessarily mean that I couldn’t manage the task and cut off the devil’s device. It is likely that I watched the movie, and the player was so comfortable that it did not force me to twist the sound, turn off the subtitles and do other stuff that is not relevant to my task to watch 3-4 series of some sitcom.

At this, perhaps everyone will be happy to answer questions in the comments. Thanks for attention!

Author: Anton Alyabyev, UIDG analyst.

PS And here is the very presentation that was discussed at the beginning of the topic.