Steam Protocol 2 and Steam Files - Introduction

Source Pirate Kit and all-all-all ...

Back in 2007, the author of one of the most remarkable programs, the Source Pirate Kit ( SPK ), completely lost the sources of his project along with the rest of the documents - his hard drive died where it was all kept. The purpose of the program to some will be clear from the name (someone even came across it) - the creation of Standalone-versions of games on the Half-Life and Half-Life 2 engines (to put it simply - pirate). Everything in the program was good, but with the loss of the source code, it went into oblivion and the ability to somehow tweak the algorithm of its work - the whole program was a monolithic EXE file and all the necessary auxiliary programs were connected with resources and unpacked if necessary. But - the main functionality was hidden in the program itself ...

Start of development

After seeing this sad news, I, who studied programming amateur for only 2 years, decided to at least try to correct the situation. At that time I was not working for the public - the project was done “for myself”, without publishing it anywhere. While working on it, I gradually studied programming more deeply - I wrote at that time in the Borland Delphi 7 environment (do not shout that C ++ is ideal for such tasks - it’s hard for me even now to jot out a normal user interface, even using QtCreator ). There were disappointments, shoals, etc., but the project gradually grew, eventually getting closer to the state when it could be published on the web ...

First publication and initial development

And then came the moment of truth - in the second half of 2007, I published the first version of the program on the resource, where the Anti-Steam-movement was the most active in our region - csmania.ru (this year the resource stopped working, apparently the administrators abandoned it completely ). I immediately ran into a lot of jambs and bugs - I myself was not really worried about testing programs at that time (don’t forget that I learned programming in general at that time). This gave me an additional incentive to work on the project - after all, if someone found bugs, then it still needs someone! Since then, work on the project was in full swing, gradually covering new areas - first I studied WinAPI, then Assembler (x86), network programming, cryptography and a lot more, going in addition to these areas and not only ... For all the time, the original project evolved from the simplest program without a bunch of jambs to a serious program with support for simple scripts, powerful architecture and complete independence from third-party tools - all the functionality was implemented exclusively by my code. The name was also changed - instead of the Source Pirate Kit , the Universal Pirate Kit was used, since the program was very flexible and could work with any games - it was only necessary to correctly write a couple of files for the program.

Further development

Since all games based on Half-Life and Half-Life 2 , were distributed through Steam , at a certain stage it was decided to look - and how does it work?
Even at the Universal Pirate Kit development stage, I studied and sorted out “under the bone” the formats used by Steam for storing game files - GCF (Grid Cache File, Game Cache File) and NCF (Not-Cache File) . A further development was the attempt to create such files from scratch. This project was implemented almost to the full - even Steam accepted the upgraded and created files. True, the format was not dismantled at 100% - there is one tricky checksum in it, which I could not overcome. Because of it, Steam saw that the file was upgraded and pumped it ...
The next step was to try to understand - “how does Steam load game content?”. In the course of working on this issue, I studied the TCP / IP protocol stack, I became acquainted with the Wireshark and IDA Pro programs ... Back in the beginning, I came across another remarkable resource , whose team also worked on this problem. There I met a lot of great people, communicated very closely with them, and together we were able to overcome all these protocols!
')

New projects

In the course of studying the network protocols of Steam, I (and not only me) had the idea to create at least my own servers, similar to those with which the official client communicates. This idea was very well received in the Russian community and I began to develop (in parallel with the study of the protocols themselves).
Somewhere in the middle of the process with protocols, another crazy idea arose - “why not write your client, since we are writing our own server? All the same, it will be necessary then to separately “break” the game in order to play it ... "
Thus was born another project - SteamLite . It became the pinnacle of my creative work of those times - the modular structure was implemented in it (the following modules were written - UI, FileFormats_ {GCF, NCF}, GameConvertor, Network, Viewer). It was possible to download files from official servers (not all files, many were protected by additional protections on the server side), make and apply patches for them, view and even edit (!!!) the contents of files without unpacking them, automatic “crack” of the game when its launch and much more ... Everything would be fine, but this project (as well as server development) “turned down” and never reached the first normal release due to some of the reasons described below.

Global bummer, gentlemen ...

For all the time of work on these projects (and this is 2007-2011), VALVE did not sit idly by and came up with disgusting for us - a new file format and new network protocols. At the same time, the protection of the games themselves was slightly changing - the main “sagging” of work on full-fledged pirated versions of games was caused by the release of Half-Life 2: Episode 2 with its achievements.
The result of all this was that with the official release of the new file format and the new network protocol (Steam 3) I personally abandoned the whole thing - the desire to disassemble it all from 0 disappeared after the very first attempts - Protocol Buffers were used everywhere, which made it difficult to analyze network packets. And I am glad that other teams have not abandoned work on the new protocol “in the closet” - at the moment everything seems to be working, but nobody is sitting on their own servers / clients ...

And what is the essence, sir?

At the moment I have knowledge of the Steam server infrastructure (at least for the previous version of the protocol), the server-client interaction protocols themselves, the formats of the files used. Some of the formats are used now - for example, BLOB files, ContentDescription Record (CDR), VDF.
I have a desire to publish data on the work on all this heap of formats and protocols, since at the moment this data is also widely available. I just want to sort out all the features in more detail, as well as write sensible articles on this topic for Russian-speaking users.
Here is a sample list of articles that will be published:

• Steam Protocol v2 - structure, description and analysis of server code;
• Steam Files - analysis of various file formats. It will be divided into several parts (GCF + NCF, BLOB + CDR + CDF).

If these articles are needed by the community, they will be published.

PS: I will be glad to hear reasonable criticism in terms of presentation and content - the first article, no experience in writing them.