How to write your own C ++ game engine

Translation of an article by Jeff Preshing (Jeff Preshing) How to Write Your Own C ++ Game Engine .

How to write your own C ++ game engine

Recently, I have been busy writing a C ++ game engine. I use it to create a small mobile game Hop Out . Here is the video recorded from my iPhone 6. (You can turn on the sound!)

Hop Out is the game that I want to play myself: a retro arcade game with 3D cartoon graphics. The goal of the game is to repaint each of the platforms, as in Q * Bert.

Hop Out is still in development, but the engine that powers it is starting to take on a mature shape, so I decided to share here some tips on engine development.

Why would anyone want to write a game engine? There are many possible reasons:

• You are a craftsman. You like to build systems from scratch and see how they come to life.
• You want to learn more about game development. I’ve been in the gaming industry for 14 years and still trying to figure it out. I was not even sure that I could write the engine from scratch, because it is so different from the daily work duties of a programmer in a large studio. I wanted to check.
• You like the feeling of control. To organize the code exactly the way you want, and always knowing where everything is - it brings pleasure.
• You are inspired by classic game engines such as AGI (1984), id Tech 1 (1993), Build (1995), and industry giants like Unity and Unreal.
• You believe that we, the gaming industry, have to dump the cover of mystery from the engine development process. We have not really mastered the art of game development - where there! The more carefully we consider this process, the higher our chances of improving it.

The gaming platforms in 2017 — mobile, consoles and PCs — are very powerful and in many ways resemble each other. Development of the game engine has ceased to be a struggle with a weak and rare iron, as it was in the past. In my opinion, now it’s rather a struggle with the complexity of your own work . You can easily create a monster! That's why all the tips in this article revolve around how to keep your code manageable. I combined them into three groups:

1. Use an iterative approach
2. Think twice before generalizing too much
3. Realize that serialization is a vast topic.

These tips apply to any game engine. I'm not going to tell you how to write a shader, what an octree is or how to add physics. I believe you already know that you should know this - and in many ways these topics depend on the type of game you want to do. Instead, I deliberately chose topics that are not covered extensively - topics that I find most interesting when I try to dispel the veil of secrecy over anything.

Use an iterative approach

My first advice is to keep anything (whatever!) To work without stopping, then repeat.

If possible, start with a sample application that initializes the device and draws something on the screen. In this case, I downloaded the SDL , opened Xcode-iOS/Test/TestiPhoneOS.xcodeproj , then launched the example testgles2 on my iPhone.

Voila! I have a great spinning cube using OpenGL ES 2.0.

My next step was to download a 3D model of Mario made by someone. I quickly wrote a draft OBJ file loader — this format is not so complicated — and I corrected the example so that it draws Mario instead of a cube. I also integrated SDL_Image to load textures.

Then I implemented two stick controls to move Mario. (At first I considered the idea of ​​creating a dual-stick shooter. However, not with Mario).

The next thing I wanted to do was get acquainted with skeletal animation, so I opened Blender , created a tentacle model, and tied a skeleton of two bones to it, which oscillated here and there.

By that time, I abandoned the OBJ format and wrote a Python script to export my own JSON files from Blender. These JSON files described the skinned mesh, the skeleton, and the animation data. I downloaded these files into the game using the C ++ JSON library .

As soon as everything worked, I returned to Blender and created a more developed character (This was the first three-dimensional person made and engraved by me. I was very proud of him.)

Over the next few months, I took these steps:

• He began to allocate functions of working with vectors and matrices into his own library of three-dimensional mathematics.
• Replaced .xcodeproj with CMake project
• I made the engine run on both Windows and iOS, because I like working in Visual Studio.
• Began to move the code to separate libraries "engine" and "game". Over time, I divided them into even smaller libraries.
• I wrote a separate application to convert my JSON files to binary data that the game can load directly.
• At some point, removed all the SDL libraries from the iOS build. (The build for Windows still uses SDL.)

The key point is this: I didn’t plan the engine architecture before I started programming . It was a conscious choice. Instead, I just wrote as simple as possible code that implements the next part of the functionality, then I looked at it to see which architecture arose naturally. By "engine architecture" I mean the set of modules that make up the game engine, the dependencies between these modules and the API for interacting with each module.

This approach is iterative because it focuses on small practical results. It works well when writing a game engine, because at every step you have a working program. If something goes wrong when you extract the code to a new module, you can always compare the changes with the code that worked before. Of course, I assume that you are using some kind of version control system .

It may seem that with this approach a lot of time is wasted, because you always write bad code, which you then need to rewrite completely. But most of the changes are moving the code from one .cpp file to another, extracting function definitions to .h files or other equally simple actions. Determining where that should lie is a difficult task, and it is easier to solve it when the code already exists.

I bet that more time is spent with the opposite approach: try to think in advance about the architecture that will do everything you need. Two of my favorite articles about the dangers of excessive engineering - The Vicious Circle of Generalization by Tomash Dabrowski and Joel Spolsky's Don't Let Architecture Astronauts Scare You .

I'm not saying that you should not solve problems on paper before you run into them in code. I also do not claim that you should not decide in advance what functionality you need. For example, I knew from the very beginning that I wanted the engine to load all the resources in the background thread. I just did not try to design or implement this functionality until my engine started loading at least some resources.

The iterative approach gave me a much more elegant architecture than I could have imagined looking at a blank sheet of paper. The iOS build of my engine today consists 100% of the original code, including its own math library, container templates, reflection / serialization system, rendering framework, physics, and audio mixer. I had reasons to write each of these modules myself, but for you this may not be necessary. Instead, there are many excellent open source libraries and licenses that may be suitable for your engine. GLM , Bullet Physics and STB headers are just some of the interesting examples.

Think twice before generalizing too much

As programmers, we strive to avoid duplication of code, and we like it when the code follows a uniform style. Nevertheless, I think it is useful not to let these instincts control all decisions.

Break DRY principle from time to time.

I will give an example: my engine contains several template classes of smart pointers that are close in spirit to std::shared_ptr . Each of them helps to avoid memory leaks, speaking a wrapper around a raw pointer.

• Owned<> for dynamically allocated objects that have a single owner.
• Reference<> uses reference counting to allow an object to have multiple owners.
• audio::AppOwned<> used by code outside the audio mixer. This allows gaming systems to own objects that the audio mixer uses, such as the voice that is currently being played.
• audio::AudioHandle<> uses a reference counting system internal to the audio mixer.

It may seem that some of these classes duplicate the functionality of others, violating the principle of DRY . In fact, at the beginning of development, I tried to reuse the existing Reference<> class as much as possible. However, I found out that the lifetime of an audio object is subject to special rules: if an object has finished playing a fragment and the game does not own a pointer to this object, you can immediately place it in the queue for deletion. If the game captured a pointer, then the audio object should not be deleted. And if the game captured a pointer, but the owner of the pointer is destroyed before the replay is over, it must be canceled. Instead of complicating Reference<> , I decided that it would be more practical to introduce separate classes of templates.

95% of the time reusing existing code is the right way. But if it starts to shackle you, or you find that you complicate something that was once simple, ask yourself: should not this part of the code base really be split in two.

Using different calling conventions is fine

One of the things I don’t like in Java is that it forces you to define every function within a class. In my opinion, it is meaningless. This may give your code a more uniform look, but also encourages over-complication and does not support the iterative approach I described earlier.

In my C ++ engine, some functions belong to classes, and some do not. For example, each opponent in the game is a class, and most of the opponent's behavior is implemented in this class, as one would expect. On the other hand, sphere casts in my engine are executed by calling sphereCast() , functions in physics . sphereCast() does not belong to any class - it is just part of the physics module. I have a build system that manages dependencies between modules, which keeps the code sufficient (for me) well organized. Wrapping this function in an arbitrary class in no way improves the organization of the code.

And then there is dynamic dispatch , which is a form of polymorphism . Often we need to call the function of an object, not knowing the exact type of this object. The first impulse of a C ++ programmer is to define an abstract base class with virtual functions, then reload these functions in a derived class. It works, but this is only one of the techniques. There are other dynamic dispatching methods that do not add as much additional code, or have other advantages:

• C ++ 11 introduced std::function , and this is a convenient way to store callback functions. You can also write your own version of std::function , which does not cause so much pain when you enter it in the debugger.
• Many callback functions can be implemented using a pair of pointers: a function pointer and an opaque argument. Only explicit casting inside the callback function is required. This is often found in pure C libraries.
• Sometimes the base type is known at compile time, and you can bind a function call without any runtime overhead. Turf , the library I use in my game engine, relies heavily on this method. Take a look at turf::Mutex for an example. This is just a typedef over platform-specific classes.
• Sometimes the most direct way is to create and maintain a table of raw function pointers on your own. I used this approach in my audio mixer and serialization system. The Python interpreter also makes full use of this technique, as will be shown below.
• You can even store function pointers in a hash table, using function names as keys. I use this technique to dispatch input events, such as multitouch events. This is part of the strategy for recording game input and playing it in the replay system.

Dynamic scheduling is an extensive topic. I only superficially talked about it to show how many ways there are to implement it. The more extensible low-level code you write - which is not uncommon for the game engine - the more often you find yourself studying alternatives. If you're not used to programming in this form, the Python interpreter written in C is a great example to learn. It implements a powerful object model: each PyObject points to a PyTypeObject , and each PyTypeObjet contains a table of function pointers for dynamic dispatching. The Defining New Types document is a good starting point if you want to get into the details right away.

Realize serialization is a broad topic.

Serialization is the conversion of runtime objects to a sequence of bytes and vice versa. In other words, saving and loading data.

For many, if not most, engines, game content is created in various editable ones, such as .png , .json , .blend or proprietary formats, then it is eventually converted into platform-specific game formats that the engine can quickly load. The last application in this process is often called "cooker". Cooker can be integrated into another tool or even distributed among several machines. Usually, a cooker and a number of tools are developed and maintained in tandem with the game engine itself.

When preparing such a pipeline, the choice of file formats at each of the stages remains with you. You can define several proprietary formats, and they can evolve as you add functionality to the engine. While they are evolving, you may need to maintain the compatibility of some programs with previously saved files. No matter in what format, in the end you have to serialize them in C ++.

In C ++, there are countless ways to organize serialization. One of the fairly obvious ones is to add the save and load functions to the classes that you want to serialize. You can achieve backward compatibility by storing the version number in the file header, then passing that number to each load function. This works, although the code can become cumbersome.

  void load(InStream& in, u32 fileVersion) { //   - in >> m_position; in >> m_direction; //           2. if (fileVersion >= 2) { in >> m_velocity; } } 

You can write more flexible, less error-prone serialization code by taking advantage of reflection — namely, creating runtime data describing the location of your C ++ types. To get a quick idea of ​​how reflection can help with serialization, take a look at how Blender does it, an open source project.

When you build Blender from source, many steps are taken. First, the makesdna utility compiles and runs. This utility parses a set of C header files in the Blender source tree, and then displays a brief summary of all the defined types in its own format, known as SDNA . These SDNA data serve as reflection data . SDNA is then bundled with Blender itself, and saved with each .blend file that Blender writes. From this point on, every time Blender loads a .blend file, it compares the SDNA .blend files from the SDNA linked to the current version at runtime and uses common serialization code to handle all the differences. This strategy gives Blender an impressive range of backward and forward compatibility. You can still download version 1.0 files in the latest version of Blender, and new .blend files can be downloaded in older versions.

Like Blender, many game engines — and related tools — create and use their own reflection data. There are many ways to do this: you can parse your own C / C ++ source code to extract type information, as Blender does. You can create a separate data description language and write a tool for generating type descriptions and C ++ reflection data from this language. You can use preprocessor macros and C ++ templates to generate reflection data at runtime. And as soon as you have these reflections at your fingertips, countless ways are opened to write a common serializer on top of all this.

Sure, I miss a lot of detail. In this article, I just wanted to show that there are many ways to serialize data, some of which are very complex. Programmers simply do not discuss serialization as much as other engine systems, even though most other systems depend on it. For example, out of 96 programmer reports of GDC 2017 , I counted 31 reports on graphics, 11 on online, 10 on tools, 3 on physics, 2 on audio — and only one on serialization itself .

At a minimum, try to imagine how complex your requirements will be. If you make a little game like Flappy Bird, with a few assets, you probably won't have to think much about serialization. You can probably download textures directly from a PNG and that will be enough. If you need a compact binary format with backward compatibility, but you do not want to develop your own - take a look at third-party libraries, such as Cereal or Boost.Serialization . I don’t think that Google Protocol Buffers are ideal for serializing game resources, but they are still worth exploring.

Writing a game engine - even a small one - is a big enterprise. I could say a lot more, but honestly, the most useful advice I can come up with for an article of such length: work iteratively, resist slightly the code compilation, and remember that serialization is an extensive topic, so you will need to choose the right strategy . My experience shows that each of these points can become a stumbling block if it is ignored.

I like to compare observations on this topic, so I'm very interested to hear from other developers. If you wrote the engine, did your experience lead to the same conclusions? And if you don’t write or are still going, your thoughts are interesting to me too. What do you think is a good learning resource? What aspects still seem mysterious to you? Feel free to leave comments below or contact me via Twitter .