How EA made life difficult for us, or how we fixed a bug 12 years ago

Sometimes bugs creep into programs. Moreover, they creep in so that they can only be detected after many, many years after their release, when it is already unprofitable to repair them. Sometimes these bugs are too critical to ignore. Therefore, under the cut, I’ll tell you how we fixed one such critical bug in an old racer. But at the same time I will clearly demonstrate what a float is bad for, what the consequences might be and how to deal with it.

Lyrics

It's about the game Need for Speed: Most Wanted. This game is very popular and extremely loved by many gamers, enthusiasts, and many are considered almost the best in the series. Even if you are not a gambler, you probably have heard about this game in particular, or about the series as a whole. But first things first.

I am a speedraner. Passing the game for speed. I had the opportunity to be one of the pioneers in speeding racing games, so at the same time I am one of the “global moderators” of the NFS series runaway community. The Czech share with me was nicknamed Ewil.
')
Why fate? Because at one point a person came to our server discord and said that all our track records were wrong and that we were noobs. Reluctantly, suppressing a baguette from what seemed to be a baseless accusation and fighting the language barrier (this person speaks English at a very bad level), we began to figure out what was wrong with our records. From the fragments of speech, we realized that there is a certain “timebug” in the game, which makes IGT wrong. Ewil reviewed some of the records and counted the time with his hands. It turned out we did not lie. The IGT records differed sharply from the RTA . It was not a couple of milliseconds, which I can also decide the outcome of the record, and in some places the difference reached even a few seconds (!).

We began to look for the cause and consequences of this phenomenon. Long before that, it was in my personal interest that I tried to “pull out” of the IGT game. My attempt was unsuccessful, and I somehow scored on this idea. We could not find any information on the Internet, except for some English page with a very short description, without any evidence base. Search on YouTube also did not bring results, but records that read “No TimeBug” were found.

A little later, I met SpeedyHeart, and she suggested to me that in the game time counts as a float. It all began to clear up, and we are slowly moving from a dull introduction to a fierce action!

How it works

Armed with Cheat Engine, OllyDbg, DxWND, and NFS: MostWanted version 1.3, I climbed into my memory. I dug up something like this (the picture is clickable):



We are interested in the last three addresses. They store in themselves IGT for different situations. Why they float - Black Box alone knows ... But they don’t! Float, he has the same accuracy as a shotgun, and maybe even worse.


Actually, a little about the timers themselves. Timers store time in seconds, i.e. the whole part is the number of full seconds. Two of these timers, namely Global IGT and Race IGT, are periodically reset. For Global IGT, this happens at the moment of entering the main menu, and Race IGT is reset when the race is restarted. IGT is calculated through Global IGT, and at some point in time it lacks accuracy. Because of this, time is considered wrong.

At this stage I was interested in several questions:

1. Since there is a difference in time, is the gameplay different with and without a bug? It is logical to assume that if IGT accelerates, then in general the game should become “faster”
2. What is the scope of this bug? How it will behave at different values ​​of the timer, and how the game will react to it.

The answer to question number 1 was found very quickly. I just took and changed the testimony of Global IGT to 300000.0 and got what I got. The time has almost doubled (!), But this did not affect the physics and behavior of the game. For fun, I also tried other timers, but for some reason they do not affect anything. Actually, if the gameplay accelerated with the acceleration of time, then in our world of speed-driving it is considered to be completely legitimate. All the same, we love bugs. But such a deal did not suit anyone.

I went a little deeper and found the answer to question 2. As soon as the Global IGT reaches 524288, the time in the game completely stops. It confuses the game a little and it starts to behave badly to do interesting things. For example, it does not allow the race to start after restart, tightly blocking the game (you can exit it only through the task manager or Alt + F4). The gap / lag from rivals stops working. And if you lose the race, the game sends you to float freely in the world.

Float - hell perfectionist.

Although not in the literal sense, but still. For a visual demonstration, I wrote a small program that will help me assess the sadness of what is happening.

Before the actual code, I'll write a little about the goal. It is known that the game "blocks" the physics update cycle at 120 times per second (again, thanks to SpeedyHeart for the information). However, vsync chops off the physics update even more, up to 60 times per second. Accordingly, we simply take the float variable and cyclically add 1/60 of a second there. Then we calculate how many steps we have achieved the result, and how many steps we had to achieve this result. We will also do everything cyclically for different random variables and calculate the average error in the calculations. Any deviations in 2 or less steps (33ms) will be considered insignificant, because the game shows the time to hundredths of a second.

#include <stdio.h> #include <stdlib.h> #include <time.h> #include <math.h> #include <conio.h> #define REPEATS 1000 //   #define ESCAPE 27 //   ESCAPE #define TEST_TIME 2 // ,    #define START_TIME 0 //     void main( ) { time_t t; srand( time( &t ) ); while ( true ) { float diffs[ REPEATS ]; int frame_diffs[ REPEATS ]; //   for ( int i = 0; i < REPEATS; i++ ) { int limit = rand( ) % TEST_TIME + 1;//   - , // +1     0 limit *= 60; //    limit += (START_TIME * 60); //    float t = 0.0f + (START_TIME*60); //    float step = 1.0f / 60.0f; //     60  int steps = 0; while ( t < limit ) { steps++; t += step; } //        double expectation = (double)(limit - START_TIME*60)/ ( 1.0 / 60.0 ); printf("%f\n", t ); printf("Difference = %f; steps = %d\n", t - limit, steps ); printf( "Expected steps = %f; frames dropped = %d\n", expectation, (int)expectation - (int)steps ); diffs[i] = fabs( t - limit ); frame_diffs[ i ] = (int)expectation - (int)steps; } //     float sum = 0; int frame_sum = 0; for ( int j = 0; j < REPEATS; j++ ) { sum += diffs[ j ]; frame_sum += frame_diffs[ j ]; } printf( "Avg. time difference = %f, avg. frame difference = %d\n", sum / REPEATS, frame_sum / REPEATS ); //   "any key" ,   "ESCAPE"  printf( "Press any key to continue, press esc to quit\n" ); if ( getch() == ESCAPE ) break; } } 

By changing the values ​​of START_TIME and TEST_TIME we can get the statistics we need. In general, while START_TIME does not exceed 15 minutes, the usual 2-minute check-in will be “free” from the bug. The difference is not critical in the game, 1-2 frames:



16 minutes turned out to be a “critical point”, when time floats mercilessly:



The moment is also interesting that, depending on START_TIME, the “side” of the error will change. For example, after an hour and a half of continuous gameplay, time will start to flow slower than it should:



Having played around with the values ​​a little more, I appreciated that in the resulting program “time” flows in much the same way as in the game. This was confirmed practically - any records recorded “from the main menu” during the first 15 minutes of the gameplay were clean. With START_TIME close to 300,000 seconds, the number of steps was almost two times less than expected. When START_TIME is larger than the magic constant 524288, the program stopped working. All this confirmed that the time counting process was copied correctly.

Eliminate unwanted behavior

Now that the problem is known and its behavior, we can fix it. You only need to rewrite Global IGT whenever the player starts the race again. This can be learned quite simply - at this moment the Race IGT is reset. But there is a problem.

There are two editions of the game: NFS: Most Wanted and NFS: Most Wanted Black Edition. The second edition includes two additional cars and two tracks, and the 69th test. But technically, these are two completely different games! Their executable files are different. In addition, there is a patch 1.3 ... Which is different for each edition. As a result, we have 4 different versions of the game that need to be supported. This fact makes the “right” way overly complex and unjustified. In an amicable way, you need to slightly tweak the file being launched and reset the counter there, but ... Edit 4 different executables that are also packed and protected from debugging ... It’s better to just write a simple program that will track the state of the timers in realtime and reset them if necessary. We will write on C #.



This is the architecture I have sketched. GameProcess is a helper class that simplifies access to the read / write process memory. GameHolder is the heart of the program. He will initialize GameProcess, and when “hooking” the process will determine the version of the game and create the necessary instance of the heir to Game. Since the “fix” logic does not differ from version to version, it is better to put it into one class.

How do we determine the version? Simply - by the size of the main module. I specially implemented imageSize. And in order not to litter the code with magic constants, we write down enum:

 enum ProcessSizes { MW13 = 0x00678e4e } 

The remaining versions will add as they fall into my hands.

isUnknown is responsible for whether we were able to determine the version or not. Of all the class, we are interested only in the Refresh method, here it is:

  public void Refresh() { if(!process.IsOpen) { // In cases when the process is not open, but the game exists // The process had either crashed, either was exited on purpose if(game != null) Console.WriteLine("Process lost (quit?)"); game = null; return; } if(isUnknown) // If we couldn't determine game version, do nothing { return; } // If process is opened, but the game doesn't exist, we need to create it if(process.IsOpen && game == null) { Console.WriteLine("Opened process, size = 0x{0:X}", process.ImageSize); switch((ProcessSizes)process.ImageSize) // Guessing version { case ProcessSizes.MW13: game = new MW.MW13(process); break; default: Console.WriteLine("Unknown game type"); isUnknown = false; break; } } // At last, update game game.Update(); } 

The fix logic came out quite simple:

  public abstract class Game { private float lastTime; private GameProcess game; /// <summary> /// Synch-timer's address /// </summary> protected int raceIgtAddress; /// <summary> /// Timer-to-sync address /// </summary> protected int globalIgtAddress; private void ResetTime() { byte[] data = { 0, 0, 0, 0 }; game.WriteMemory(globalIgtAddress, data); } public void Update() { float tmp = game.ReadFloat(raceIgtAddress); if (tmp < lastTime) { ResetTime(); Console.WriteLine("Timer reset"); } lastTime = tmp; } public Game(GameProcess proc) { game = proc; lastTime = -2; // Why not anyway } } 

The matter remains for the small: to implement the version by setting the necessary values ​​in the constructor for the corresponding protected variables. In Maine, we simply throw the update cycle into a separate trade and forget about it. Oh yeah, due to the nature of the Nvidia cards and the implementation features of the NFS game installer, we will take the name of the process as input so that we can customize.

  class Program { static void Run(object proc) { GameHolder holder = new GameHolder((string)proc); while (true) { Thread.Sleep(100); holder.Refresh(); } } static void Main(string[] args) { Thread t = new Thread(new ParameterizedThreadStart(Run)); t.Start(args[0]); Console.WriteLine("Press any key at any time to close"); Console.ReadKey(); t.Abort(); } } 

This fix ends. Compile, run, and forget about the timebag, yay! ^ _ ^ The image is clickable.



In fact, this bug is not going anywhere. If one race does not physically fit into the framework of 15 minutes, then nothing can be done about it. But such races in the game as much as one, and that from the police.
Full source code on githaba .

Summary

This is how one small bug spoiled life for us. But it could have been avoided if the Black Box had used double at one time, but no. By the way, this is a vivid example of how “once written” results in a bunch of undetectable / rolling bugs. Timebug has been in every Black Box game ever since . In Carbon, ProStreet and even Undercover. In the latter, they changed the logic of counting IGT, but these three timers are still there, and rounding errors lead to strange consequences. SpeedyHeart promised to make a video review of all the information found in the process, so we are waiting.

What did this situation teach me? I do not know. I already understood that using float for serious calculations is a dubious idea. But now I can better imagine exactly how all this will work in practice. However, it turned out to be amusing that such a serious company could make such a serious mistake, and not even notice it for several years in a row.

It seems to me that for this task (IGT calculation) you need to use the following path: set the timestamp at the beginning of the race, and then deduct it from the current time. Moreover, arithmetic operations should be avoided, even over integers. 1/60 of a second is 16, (6) milliseconds, so in the case of an integer, we will naively throw back 0, (6) with each addition, which will lead to inaccuracies in the calculation.

In some foreseeable future, I will try to write a fix to other versions. At this I have everything, thank you for your attention.

UPD : Corrected the link to githab makes it possible for you to move to a new name.
UPD2 : Rolled out the second part, those interested can read .