How we do World of Warships: export automation and content verification

After the premiere closed World of Warships shows at gamescom and Igromir, the official launch of the game is getting closer and closer. Now closed alpha testing is in full swing, and we, the developers of Lesta Studio , St. Petersburg division of Wargaming, still have a whole bunch of questions to solve. At the same time a lot of obstacles still managed to leave behind. Below is a story about how we adapted the exporter of our engine to the needs of "Ships" and lined up the content verification process.

Standard engine delivery

Any engine includes a toolkit for the exporter of 3D-models from 3D-editors in its own data format. Our BigWorld, on the basis of which World of Tanks is made, is no exception. It supports export from 3D Max and Maya. Virtually any game project requires the adaptation of standard exporters to the specifics of the project. In our project, the specifics are the models of ships.
')
The first version of an adapted exporter from Maya simply “recruited” him to recognize the more complex structure of the ship scene. A little Python control code was added to the existing C ++ code, as well as a plugin for Maya with a UI on wxWidget. It looked like this:


UI adapted exporter

The resulting tool had a lot of flaws.

The export could be carried out only with the participation of the user who was supposed to "tell" the exporter what the scene contains. The use of this tool in the automation of various processes, for example, automatic verification of content at the assembly stage of the distribution, was out of the question.

The exporter demanded knowledge of far from obvious parameters from the user, worked slowly, practically did not support scene verification, and also required a huge amount of resources for support.

Architecture was a major issue for future functionality. The export was actually an atomic operation (a set of spaghetti functions) that translated data from one structure (loaded Maya scene) to another structure (BigWorld) directly into physical files. When serializers and business logic are implemented “in monolith”, and the data model is simply missing, it is impossible to add data processing (pre / post-processing), as well as reuse (code reuse) serializers and data model in other tools that implement their own business. -logics. It was impossible to build more complex content production processes.

Over time, it became almost impossible to build up new functionality in the existing code. It was decided to rewrite the exporter from scratch, laying in it a new architecture.

Harsh weekdays

The level of our project has increased the requirements for quality, complexity and volume of content. Over the past couple of years, our studio has grown a lot. We have the opportunity to allocate a sufficient amount of resources for tasks related to the production of content. Professionals came to us who have a great background in architecture development, C ++ / C # technologies. At the same time, for exporter’s developers, this was the first experience of using Python and Maya API. This introduced additional risks that had to be taken into account.

We estimated the exporter refactoring at two to three man-months. Without optimism in gamedev can not be.

To the risks we attributed:

• lack of formal requirements;
• Python proficiency;
• Maya API complexity;
• refactoring of primitive processing algorithms.

A lot of actual time was spent collecting requirements from non-formalized sources, such as developers, who became managers, old-timers, torsion fields, and the code of an existing exporter. These nuggets of knowledge were formalized and recorded in the form of requirements, specifications, and UML diagrams in Confluence.

The first prototypes showed the need to use the concept of namespaces and Python modules (__init__.py). Also, a mechanism was developed that allows using the functionality from the C ++ libraries (.pyd) transparently.

On the complexity and intricacies of the Maya API, you can write a separate book. Any functionality required prototyping, consultations with 3D-artists and with the developers of the engine (rendering).

The standard exporter had its own implementation of a large number of algorithms, for example, polyangulation triangulation, calculation of nested nodes transformation matrices, etc. We abandoned them in favor of using Maya API, which greatly increased the exporter’s performance.

It’s time for the laws of Murphy to add a rule that any project you have in mind will necessarily be implemented within no more than "x3" from the planned one, if you don’t give it up.

The result was worth our effort. In the end, even our main artist responsible for exporting models, after a couple of months of operation, found our exporter "almost perfect."

Look under the hood

Our studio actively uses Python scripts. We tried to implement on it and the whole exporter. Naturally, Python is not suitable for processing large binary data such as vertex containers (vertex buffer), vertex index containers (index buffer), etc. The data model and serializers of similar containers were implemented in C ++ as a library (.pyd) which naturally fit into the Python data model. All business logic was implemented in Python.

The exporter's framework was planned to be used not only for the task of “manual” export from Maya, but also for any tasks where its functionality could be reused, for example, to automate content verification. From any developed toolkit, we require the presence of interfaces (API) for Python, the command line (command line) and the UI tools.

Architecture

The architecture of the exporter framework is modular, layered. There are physical and logical layers, as well as a domain layer. Each layer contains separate modules: a data model, business logic, serializers, as well as converters that can convert the data model of one layer into the data model of another layer. Physical and logical layers actually implement an analogue of the ORM-architecture.

The architecture of the domain layer is designed for convenient processing of the data model by business logic. It is completely isolated and does not contain any assumptions about how it will be serialized into physical storage.


Exporter Layered Architecture

Export process

Layered architecture introduces certain features of the export process. In fact, we will deserialize two (or more) models from various sources (Maya and BigWorld Engine). After this, merge of these models into one new one takes place. Next, the new model is serialized in the BigWorld-Engine-format.


Export process

Flexibility of content production

The implemented architecture allows you to simply build complex processes of content production. For example, our ship’s primary model technologically consists of three separate Maya scenes, each of which is simultaneously being developed by different departments:

• The first scene contains a visual model and a collision model.
• The second scene contains a ballistic model (ballistic model).
• The third contains effect ports.

In addition to this, the engine toolkit (editors) adds (edits) its own data in the derived engine format model (fourth scene).

The exporter easily solves the nontrivial task of combining all four scenes into one resulting ship model.

Content verification

The content verification system allows us to search for content errors both in each layer (source) separately, and in the resulting content. The number of verifiers now reaches several dozen. Automatic content verification is built into the distribution build process (build), which makes it possible to eliminate the human factor as much as possible and ensure the integrity and technical purity of the content.


Maya ship model verification example

Content budgets and duck in the bathroom

An important component of the content verification process is the verification of budgets, for example, verification of polygon budget. The figure below shows, in particular, information on the number of triangles for a visual model for each model:


Maya ui plugin

A vivid illustration of the need for such verification is a bike, told to me by colleagues about a previous project. On the map there was an area built up with non-destructible houses. As soon as the camera turned its attention to this area, the FPS immediately fell wildly. After studying the problem, it turned out that inside one of the houses there was a bath in which a small “plastic” duck swam. All this would have looked like an amusing prank of artists, if it were not for the fact that the duck's model contained about a million polygons.

In practice, it is very difficult to comply with the budget value. Many models are objectively exceptions. Setting the budget value in the form of a range also does not solve the problem, since with time polygonage models simply begin to strive for the upper value of the range. In our case, we plan to change the personal budgets of those models that do not match the standard budget of this type of model.

Content processing

At each export stage, processing (pre / post-processing) is required. For example, before converting the Maya layer's logical data model into the domain model for air defense guns, a preliminary rotation of the cannon skeleton by 45 degrees along the Y axis and removal of the skeleton is required. Our architecture allowed us to transparently embed various processing at any stage of export.



An example of a model before and after pre-processing

X64 support

Quite recently, our artists massively switched from 32-bit Maya 2012 to 64-bit Maya 2014. Since the exporter is almost entirely written in Python, we have almost no problems with x64 support. Only the library (.pyd) implemented in C ++ required a bit of "shamanism".

Now the exporter can be easily used in both x32 and x64 processes, since it defines and loads the required C ++ library build (.pyd).

Card verification

In developing the exporter's architecture, we could not have guessed in advance in which other tools and automations it would be possible to reuse it. Automation of card verification is an example of how the “right” exporter architecture found its application in another tool.

Card verification builds and verifies the dependency graph of a map from other objects located on it. In particular, the map uses visual models of the landscape (stones, icebergs), buildings (houses, hangars, jetties), equipment (airplanes, boats), etc.

The peculiarity of the card verifier is that it can verify not only the presence of files of these visual models, but also the models themselves using the exporter's framework. This made it possible to eliminate the human factor when the Map Development Department (LA) has to “take a word for it” from the 3D Model Development Department (3D Art), which uses technically correct models.

Build distributive

The exporter framework has found its application in the process of preparing the content pack for the distribution. The distribution kit should not get models that:

• are no longer used;
• are still under development;
• intended for future versions of the product.

According to the basic list of game objects (root game objects) it is required to build a dependency graph, which will form a complete list of the required content. There is nothing easier than deserializing a model with the exporter framework and “finding out” what other models will need (content references).

Results

The history of the development of our exporter showed how from a simple highly specialized tool it evolved into a powerful system that solves its immediate tasks, and also found application in other content production processes. The basis of its successful development and reuse is a modular architecture that allows you to use its individual "cubes" to build other systems.

In the near future, the exporter will have another test related to the change of the BigWorld Engine file format. We are sure that the embedded architecture will not experience any difficulties and will be able to support the work both with the existing and with the new file format.