How I tried to improve Halo 2, but I almost ruined it.

After the release in 2004, Halo 2 instantly became the most popular multiplayer game on Xbox Live. She retained this title for almost two years, and you can even say with confidence that the main reason for the survival of Xbox Live in her early years was the huge popularity of this single game. Over the entire six-year life of the game, more than 6.6 million players have played more than 499 million hours in online multiplayer Halo 2 . The Bungie development team took a huge risk and created a new type of online games, which was a huge success and delighted millions of people.

And therefore I am glad that I did not kill this game in the bud.

During development, Halo 2 received the code name "Prophets" ("Prophets"), because a new alien race was added to the Halo universe. At that time, most researchers at Microsoft provided support for three to five games, but because this project was so important for the first Xbox, two full-time researchers were given to help game developers to work full time — me and Randy Pagulyan . We were both experienced scientists with doctoral degrees in experimental psychology and had long been in the Games Games Research division of Microsoft. Our task was to use qualitative and quantitative techniques, for example, usability studies, game testing (playtesting) and surveys, which allow development teams to understand how their games will be accepted after graduation.

This is a story about the time when I did not succeed in becoming a prophet, and my attempts to project research data for the future led to a conflict between the Microsoft research department and the Bungie design team. When they talk about the research of users of games, they usually mention only cases where we were right, cases where the data allowed us to correct the design of games. But this story is not about that. It’s about how two otherwise competent researchers made wrong conclusions about the innovative design element of the game and gave bad recommendations, and how the game in spite of it won success.
')

Innovation

Before Halo 2 , most online games did not use automatic player selection (matchmaking). Lobby was used to find other people for playing online. The players had a lobby list, they read their brief descriptions and choose the right one. If it turned out that the lobby is occupied by hooligans or more talented players, then you could go back and choose a new lobby, more suitable for you.

A huge advantage of such lobby systems was control. The creator of the lobby had the ability to flexibly customize the gameplay, allowing only those cards and modes that he liked, and throwing out users who did not play according to his rules. It was customary to see the lobby, proudly declared: "without snipers" or "_name of a specific map_, every man for himself."

The system proposed in Halo 2 took away all choices from players, replacing it with a system in which players could choose only the general type of match (i.e. Free for all, Big Team Battle, etc.), after which Bungie chose map, type of game and opponents. The figure below shows a screenshot of the almost finished pre-release version of the Optimatch hopper. (The terminology we used in the game has not been preserved.)



Here’s how GameSpy described the Halo 2 system in an article before the game’s release:

“An interesting twist - all game modes, maps, vehicles and almost everything else will be determined by Bungie.

Although it may seem strange at first, it is a good idea for many reasons. By ensuring the same optimization of all players for one type of game, Bungie will be able to guarantee that all games will run smoothly. Also, the company will be able to be sure of the correctness of the ratings, because almost everyone will play on the same cards with approximately the same number of players. They will be able to post updates on Xbox Live at any time, and everyone will start playing new games. The rating system is determined by the type of matches, that is, you can, for example, be at 25th place in Assault mode, but only at 78th in Slayer ... "

This description today seems everyday and obvious, but it happened because the system was so successful that it became a new standard for multiplayer games. The victory of Halo 2 was so overwhelming that it’s hard to imagine how online games worked before.

And again I repeat - I am glad that I did not manage to destroy it.

Research

Our research challenge was to make sure the players understood the new paradigm. It was so different from what was implemented in the first Halo that we wanted to introduce real players as early as possible, starting with paper prototypes and text descriptions. Players were shown descriptions and interface sketches of several options for how to play multiplayer games, including a new matchmaking system, but without the traditional lobbies created by users.

The reaction of the overwhelming number of participants was: "We understand it, but it is disgusting." The players with whom we spoke almost unanimously reported that they wanted to have the level of personal control that the lobby system gave them. They believed that the advantages of the new matchmaking system are not worth giving up the old one. Today it is difficult to imagine it, but then the principle “press one button and trust us” seemed used to choosing the players themselves too abnormal and controlling.

Considering ourselves a true voice of users, Randy and I came to Bungie and told them that the players were disgusted with the new design, and that they needed to think about other ways to implement multiplayer. Designers stood their ground, arguing that their vision of the future was better than the status quo, and history proved that they were absolutely right. Players love the new system and it has become the gold standard for online gaming. Studying matchmaking Halo 2 was the only major failure in my career (at least for now).

Errors

What happened? How did it happen that the research results were in such a contradiction with what actually happened after the release? The answer lies in two interrelated errors: one was made by the participants in the study, the other by the researchers themselves.

The participants' mistake was a cognitive distortion of “prediction of affect” (affective forecasting): they assumed that they would feel in a hypothetical situation. There is a huge amount of literature on how wrong people can assess the emotional impact of hypothetical situations on them. Even serious, life-changing events, such as disability or winning the lottery, are difficult to evaluate abstractly.

If you asked our research team at that time, you would get the answer: "Of course, people are not able to predict their emotions well, but we are not doing this." Initially, our study posed such a question: can players understand this system? This is an acceptable question for research, which we were able to answer with a sure yes. But when participants also began to express their opinions about the system, we regarded these opinions not as guesses, but as truth.

The snag was that research participants never played online shooter with real matchmaking. Again, today it seems strange, because matchmaking has become a standard feature of every multiplayer online game. But at that time, participants played multiplayer only over a local network or, at best, over a student hostel network. We conducted research in 2003, when less than 16% of Americans had broadband access. In fact, we asked participants to compare their known experience (lobby) with the unknown (fair and accurate matchmaking among a large online community). They understood the advantages and disadvantages of the existing system. In the case of the proposed matchmaking system, they truly could only understand what they are being deprived of. Because of this, the proposed system seemed a much worse change than it actually was.

Therefore, when research participants told us that they would not like to use such a system, we, the researchers, made their own mistake and presented their comments as an accurate reflection of the feelings of the majority of future players after they actually played the game. However, after bitter disputes, Bungie decided to ignore our objections and move on with its new matchmaking system, which turned out to be absolutely the right choice.

Of course, when the designers of Bungie rejected these teams of researchers, they did not point out methodological errors and did not expose cognitive distortion. They had a uniquely clear design vision based on solid principles, and during the ensuing heated discussions inside the studio, they gained battle confidence in the development and design managers that this is the way to create an excellent multiplayer game. One of them secretly told me later that no results of our research would convince them to change their course. Almost always, when I transfer research data to developers and they reject it in a similar way, I after the release of the game in a professional tone state: “Well, I told you so” But in the case of Halo 2 , the opposite side was right, and thanks to it the world of games became better.

We, the researchers, should have presented the results more differentiated. Our data was correct only if you looked at them from a certain angle: “That's what some players will say when they hear about the system for the first time.” Based on this, we could work with the team on testing various ways of familiarizing players with the system in order to improve their first impression and increase the speed with which players would realize the true value of this system. In a fundamental sense, the data were not bad, we just had to consider their nominal value.

Lessons

Since then, I have thought a lot about this incident, so I will list some lessons learned from it. I hope this story will allow others not to commit specifically these mistakes and start making their own, more interesting and new mistakes.

Lesson 1: Sometimes researchers must lose out on arguments.

UX researchers tend to believe that they discover the Truth with a capital "I". This can lead to great friction when other participants in the development process refuse to accept our conclusions. In general, we are usually right, but false-positive, false-negative and simply incorrect results are always possible.

The opinion of researchers of game users always has a significant weight in the development process, even when we are no more ideal than all other employees. We must actively defend our understanding of UX, but this does not mean that we must always win. In fact, I want to state that, like the players in our games, the researchers must have an ideal level of failures. If every study is equally successful, it will simply mean that we are not innovative enough or do not take on complex research topics. We need to take risks, and this means that sometimes we have to lose.

Lesson 2: Error does not mean the end of work with researchers.

The Halo 2 user research was a very hard job. Microsoft has made a big bet on this project, allocating to it such a resource of researchers, which at another time could support half a dozen games. Studio Bungie, well known for carefully selecting its partners, made a “leap of faith”, giving us an unprecedented level of access to its development process during one of the most difficult crunches in its history. Randy and I had a huge responsibility: we had to influence user feedback on the design of the game, which is very important for many people. Faced with unambiguous assessments of the participants, horrified by the fact that the design will negatively affect the impressions of millions of players, we decided that we need to take the fight. But we were wrong and we lost.

This story happened in 2003, in the middle of the Halo 2 development cycle, and we conducted many other successful game studies. After its release, Randy and I plunged into the process of supporting the sequel, Halo 3 , which has become one of the most successful research projects on game users. The same two researchers, the same designers, the same franchise, but our work as a result turned out to be on the cover of Wired magazine and became an important milestone that made the user research a must for the gaming industry. In fact, this study and its failure made a direct contribution to future success. Bungie management concluded that research needs to be more integrated with the work of designers so that this does not happen anymore, and this integration was the main reason for our success with Halo 3 . A few years later, Bungie herself even hired me to create and manage her internal research department.

The researchers are not perfect, but our partners do not need us to be perfect. They need us to honestly and fully convey to them the opinions of the players, so that we push ourselves to innovation and risk, to admit our mistakes and adapt to them.

Lesson 3: research and design operate on the same field, but according to different rules

Players can be judged only by their own feelings: or by their gaming experience, or by what they see right in front of the laboratory. Since the work of the researcher is to enhance the voice of the players, we have the same limitations. Our prophecies are true no more than what the players react to.

Designers have no such limitation. They come up with ideas that have little or nothing to do with what it was before. Because of this, such ideas can be difficult to test and validate. There are ways to evaluate new ideas, but as researchers we must be aware that these methods are much more risky than other tools and measure our conclusions.

It's funny that a good counter-example of the presentation of new gameplay elements was clearly demonstrated by the Bungie team itself during the development of Halo 2 . The matchmaking system discussed in the article was just one part of a much more extensive list of multiplayer features introduced in Halo 2 , and many other ideas met with the same resistance. To convince Microsoft's skeptical guide, the design team created a video simulation of how the game will look like with friends in the finished product. Although this decision would not have worked with matchmaking, it was an example of how much effort and imagination is required to convey the vision of a new gameplay.

Lesson 4: Making Methodological Mistakes and Confronting Their Consequences - The Best Way to Understand the Structure of Research

Of course, I knew about the problems with the prediction of affect even before this study. But I still allowed myself to get carried away with the confident opinion of the participants about the new system and took their predictions as a fact. Having once made such a mistake and faced with its humiliating consequences, I became much less inclined to repeat it.

Of course, memorizing the principles of planning a qualitative research is correct and good, but you will never feel them the hard way until you break these principles and personally encounter the results. It doesn’t matter whether you broke the rules intentionally, or partly the circumstances were to blame. When you have to throw days or weeks of hard work into a bucket, it leaves scars reminiscent of yourself. Going beyond the flags may be the best way to understand why these flags are generally needed.

But you will also find that sometimes ... retribution does not occur. Some rules of qualitative research are the laws of nature, while others are merely recommendations. The study of game users is an applied area, work in it is carried out at breakneck speed, limited resources and in conditions of confusion. Not every study will be a gem of planning. Managers push for changes in research methods, participants skip meetings, equipment breaks down, and the task of the researcher is to choose the method of adaptation to the conditions with the least damage. Understanding which principles allow flexibility and which cannot be violated, improves us as researchers, allows us to adapt and bring maximum benefit to developers and their games.

Finally

Innovations in design require risk, and in this case, the risk assumed by the Bungie design team has proven itself many times. An important task of user research in the design process is to reduce this design risk. Research allows designers to try new things, while at the same time finding and eliminating potential problems before they begin to interfere with real players. But within our profession, we also take on our own small risks, taking advantage of personal opinion in choosing the structure of research and in whether it is worth fighting for the solution of the problems identified. In this case, our particular choice did not justify itself, but this does not negate the fact that research risks are necessary. A good researcher should always use his experience and sense to select the appropriate level of risk, optimally suitable for designers and for the benefit of the players.

I am especially grateful to Randy Pagulaian for helping to create this article and for the courage that allowed him to share this burden of public shame with me. I also thank Chris Butcher, David Candland, Curtis Creamer, Max Hoberman, and Jason Jones for helping me brush up on this story and advise me on how to best explain it.

About the author: John Hopson has been in the gaming industry for 16 years. He was a lead researcher in such projects and game series as Halo, Age of Empires, Destiny, World of Warcraft, Overwatch and Hearthstone. He has a doctorate in experimental psychology from Duke University, is the author of numerous articles on the topics at the intersection of psychology and games, including the famous "Behavioral Game Design". John is currently the head of analytics at ArenaNet. The opinions expressed in the article are not the official position of ArenaNet or its employees.