StarCraft II game opened for machine learning agents

The PySC2 editor shows the interpretation of the playing field for a person (left), as well as the color versions of the feature layers on the right. For example, the top row shows signs of surface height, fog of war, slime, camera location and player resources, video

Testing artificial intelligence agents in computer games is an important part of AI training. DeepMind pays great attention to this aspect of learning, using both specially created environments like DeepMind Lab and well-known games like Atari and Go ( AlphaGo system). It is especially important to test agents in such games that were not specifically created for machine learning, but, on the contrary, they were created for people’s games, and people play them well. This is where AI training is most valuable.

Based on these assumptions, DeepMind, in conjunction with Blizzard Entertainment, released the SC2LE toolkit to stimulate AI research on the StarCraft II platform.

StarCraft and StarCraft II are among the most popular games of all time, tournaments on them are over 20 years old. The original version of StarCraft is also used for machine learning and AI research, and the developers show their creations at the annual AIIDE bots competition . Part of the success of the game is associated with a good balance and multi-level gameplay, which simultaneously makes it an ideal option for AI research.
')
So, the main task of the player is to defeat the opponent, but at the same time you need to complete a lot of subtasks: extract resources, build buildings. The game has other qualities that are attractive for the development of AI: for example, a constant large pool of experienced players online, where you can hone your skills.



The SC2LE toolkit includes the following:

• Blizzard-developed Machine Learning APIs , which developers and researchers can use to connect to the game engine. Including first released tools for Linux.
• Training dataset with anonymized game entries . DeepMind promises in the coming weeks to increase the number of records from 65 thousand to 500 thousand.
• The open source version of the DeepMind toolkit is PySC2 to easily use the API and feature layer level with other agents.
• Sets of simple mini-games to test the performance of agents on simple tasks.
• Joint scientific article describing the environment and the basic results of machine learning in mini-games, a description of training with the teacher on the data set of game records and a full-fledged 1v1 game of the agent against the game AI.

StarCraft II will not be an easy game to learn AI. Suffice it to say that more than 300 basic actions are available for the player. Compare this with Atari games, where the number of actions does not exceed ten (such as "up", "down", "left", "right"). In addition, actions in StarCraft have a hierarchical structure, can be changed or supplemented, and many of them require specifying a point on the screen. Simple math shows that even on a fragment of 84 × 84 pixels there are about 100 million possible actions!



“StarCraft is interesting for many reasons,” said Oriol Vinyals, leading DeepMind researcher on the StarCraft II project, and an expert StarCraft II player (he wrote the powerful bots for the game back in his student years). - Memory is a critical factor. What you see at the moment is different from what you saw before, and something specific that happened a minute ago can cause you to change your behavior at the moment. ”

The PySC2 editor provides an easy-to-use interface for connecting agents to the game. As shown in the very first screenshot, the game was laid out into “layers of signs” that are isolated from each other. These are such signs as unit types, visibility on the map, surface height, etc.

The animation below shows some of the mini-games designed to train agents to perform specific actions in the game, such as moving a camera, collecting minerals, or selecting units. The developers at DeepMind are hoping that the community will give up ideas for new mini-games.



The first results show that AI agents do a good job with mini-games, but throughout the whole game, even the best agents like A3C cannot win the built-in AI even at the simplest level. Perhaps collecting more gaming sessions and additional agent training will help correct the situation. At the same time, training on such a large base (500 thousand gaming sessions) will open up fundamentally new research opportunities, such as long-term AI memory and prediction of sequences of events.

The developers at Blizzard say that they themselves are interested in opening the game engine for external AI agents. First, it can make the game more interesting for current players. Secondly, it is important to study the gameplay itself and develop future games.