This winter, or more precisely, on one of the days between Catholic Christmas and New Year, the technical support engineers at Veeam were engaged in unusual tasks: they hunted for a group of hackers called “Veeamonymous”.

About how the guys themselves came up with and spent at work a real quest in reality, with tasks “close to combat,” said
Kirill Stetsko ,
Escalation Engineer .
“Why did you start this at all?”')
- In approximately the same way that people invented Linux at the time - just for fun, for their own pleasure.
We wanted to move, and at the same time we wanted to do something useful, something interesting. Plus, it was necessary to give some emotional relief to the engineers from their everyday work.
- Who suggested this? Whose idea was it?- The idea was our manager Katya Egorova, and then the concept and all further ideas were born together. Originally thought to make a hackathon. But during the development of the concept, the idea turned into a quest, yet the technical support engineer is a different kind of activity than programming.
So, we called friends-comrades-acquaintances, different people helped us with the concept - one person with T2 (second line of support -
editor's note ), one person with T3, a couple of people from the SWAT team (quick response team to urgent cases -
Ed. ). They all gathered together, sat down and tried to come up with tasks for our quest.
- It was very unexpected to find out everything about it, because, as far as I know, the script writers usually work out the quest mechanics, that is, you did not only do this complicated thing, but also with regard to your work, your professional field of activity.- Yes, we wanted to do not just entertainment, but to “pump over” the technical skills of engineers. One of the tasks in our department is the exchange of knowledge and training, but such a quest is an excellent opportunity to give people to “touch” some new techniques for them live.
- How did you invent the tasks?“They were brainstorming.” We had an understanding that we have to do some technical tests, such that they are interesting and at the same time carry new knowledge.
For example, we thought that people should be given to try to sniff traffic, use hex editors, do something for Linux, some slightly deeper things related to our products (Veeam Backup & Replication and others).
Also an important part was the concept. We decided to start from the topic of hackers, anonymous access and the atmosphere of secrecy. The symbol made the mask of Guy Fawkes, and the name came naturally - Veeamonymous.
"In the beginning was the word"
To stir up interest, we decided before the event to arrange a PR company in the quest theme: we hung posters announcing our office. And after a few days, they secretly painted them themselves with spray cans and launched a “duck”, they say, some attackers spoiled the posters, even put a photo with proof….
- So you did it yourself, that is, a team of organizers ?!- Yes, on Friday, at about 9 o'clock, when everyone had already left, we walked and drew a green letter “V” from cylinders.) Many of the participants in the quest never realized who did this - people came to us and asked who spoiled the posters ? Someone very seriously approached this issue and arranged a whole investigation on this topic.
For the quest, we also wrote audio files, “ripped out” sounds: for example, when an engineer logs into our [production CRM] system, then there is an answering machine robot that speaks all sorts of phrases, numbers ... Here we are from those words that he has written down, made up more or less meaningful phrases, well, maybe a little bit of curves - for example, we got “No friends to help you” in the audio file.
The IP address, for example, we represented in binary code, everything, again, with the help of these numbers [spoken by the robot], all kinds of frightening sounds were added. They shot the video themselves: in the video we have a man sitting in a black hood and in the mask of Guy Fawkes, but in fact there is not one person, but three, because two stand behind him and keep the “background” from the blanket :).
- Well, you are confused, just say.- Yes, we caught fire. In general, we first came up with our technical tasks, and then composed a literary-game outline on the subject of what allegedly happened. According to the scenario, the participants hunted for a group of hackers called “Veeamonymous”. The idea was also in the fact that we kind of “break the 4th wall”, that is, we transfer events into reality - we drew from a can, for example.
One of the native speakers of English from our department helped us with the literary processing of the text.
- Wait, but what for a native speaker? Have you done this in English?- Yes, we did for the St. Petersburg and Bucharest offices, so everything was in English.
For the first experience we tried to make everything just work, so the script was linear and fairly simple. Added more entourage: secret texts, ciphers, pictures.

We also used memes: there were a lot of pictures on the topics of investigations, UFOs, some popular horror stories - some teams were distracted by this, tried to find some hidden messages there, apply their knowledge of steganography and other things ... but, of course, there is nothing like that It was.
About thorns
However, in the process of preparation, we also encountered unexpected tasks for ourselves.
Many fought over them and solved any questions that suddenly arose, but about a week before the quest they all thought that everything was gone.
Probably worth a little talk about the technical basis of the quest.
Everything was done on our internal ESXi lab. We had 6 teams, so it was necessary to allocate 6 resource pools. So, for each team we deployed a separate pool with the necessary virtual machines (the same IP). But since all this was on servers that are on the same network, the current configuration of our VLANs did not allow us to isolate the machines in different pools. And, for example, during the test run, we received situations when the machine from one pool connected to the machine from another.
- How could you correct the situation?“At first they thought for a long time, tested all sorts of options with permissions, separate vLANs to machines. In the end, they did this - each team sees only the Veeam Backup server, through which all further work takes place, but does not see the hidden podpulul, which contains:
- several windows machines
- Windows core server
- Linux machine
- VTL (Virtual Tape Library) pair
All pools are assigned a separate group of ports on the vDS switch and their Private VLAN. This double isolation is just necessary to completely eliminate the possibility of network interaction.
About the brave
- Anyone could take part in the quest? How were the teams formed?- It was our first experience of holding such an event, and the capabilities of our laboratory were limited to 6 teams.
At first, as I said, we conducted a PR company: using posters and a newsletter, we were told that a quest would be held. We even had some clues - phrases in binary code were encrypted on the posters themselves. In this way, we became interested in people, and people already agreed among themselves with friends, with friends, and cooperated. As a result, more willing people responded than we had pools, so we had to make a selection: we invented a simple test task and sent it to everyone who responded. It was a logical task, it had to be solved for speed.
The team allowed up to 5 people. The captain was not required there, the idea was in cooperation, in communication with each other. Someone is strong, for example, in Linux, someone is strong in tapes (backups on tapes), and everyone, seeing the task, could invest their efforts in a common solution. Everyone communicated with each other, found a solution.
- And at what point did the event start? Did you have some kind of “hour X”?- Yes, we had a strictly appointed day, we chose it so that there was less workload in the department. Naturally, we notified the team leaders in advance that such and such teams are invited to participate in the quest, and they should be given some relief [regarding loading] on this day. It turned out that it should be the end of the year, December 28, Friday. They expected to take about 5 hours, but all the teams managed faster.
- Everyone was on an equal footing, did everyone have the same tasks based on real-life cases?- Well, yes, each of the compilers took some stories from personal experience. We knew about something that could be in reality, and it would be interesting for a person to “feel” it, look, figure it out. They took some more specific things, such as data recovery from damaged tapes. Someone with clues, but most teams coped on their own.
Or it was necessary to apply the magic of fast scripts - for example, we had a story that a kind of “logical bomb” “tore up” a multivolume archive into random folders on a tree, and it was necessary to collect data. You can do it manually - one by one to find and copy [files], or you can write a script using a mask.
In general, we tried to adhere to the point of view that one problem can be solved in different ways. For example, if you are a little more experienced or you want to “get confused”, then you can solve it faster, and there is a direct solution to the problem “head on” - but at the same time you will spend more time on the task. That is, almost every task had several solutions, and it was interesting which ways teams would choose. So nonlinearity was precisely in the choice of a solution.
By the way, the Linux-task turned out to be the most difficult - only one team decided it by itself, without prompts.
- And you could take tips? How in this quest ??- Yes, it was possible to take, because we understood that people were different, and those who lacked some knowledge could get into the same team, so in order not to delay the passage, and not to lose interest in the competition, we decided that we would hints. For this, each team was watched by a man from the organizers. Well, we made sure that no one cheaters.

About stars
- And there were prizes for the winners?- Yes, we tried to make the most pleasant prizes both for all participants and for the winners: the winners received designer sweatshirts with the Veeam logo and the phrase, encoded in hexadecimal code, in black). All participants received a Guy Fawkes mask and a branded bag with a logo and the same code.
- So you have everything in the real quest was!- Well, we wanted to do a cool, adult thing, and it seems to me that we did it.
- And there is! And what was the reaction in the end to those who participated in this quest? Did you achieve your goals?- Yes, many then approached, saying that they clearly saw their weak points and wanted to tighten them. Someone stopped being afraid of certain technologies - for example, dumping blocks from tapes and trying to get something out of there ... Someone realized that he needed to pull up Linux, and so on. We tried to give a fairly wide range of tasks, but not quite trivial.
Winning team"Who wants, he will achieve!"
- From those who prepared the quest, it took a lot of effort?- In fact yes. But this was most likely due to the fact that we had no experience in preparing such quests, such infrastructures. (Let's make a reservation that this is not our real infrastructure - it just had to perform some kind of gaming functions.)
For us it was a very interesting experience. At first, I was skeptical, because the idea seemed to me even too cool, I thought that it was very difficult to implement. But they started to do it, they started to plow, they started to catch fire, and in the end we did it. And there were even virtually no overlays.
In general, we spent 3 months. For the most part, we invented the concept, discussed what we can implement. In the process, of course, something changed, because we understood that we for some reason do not have the technical ability to do this. On the move, we had to redo something, but so that the whole canvas, history and logic did not break. We tried not only to give a list of technical tasks, but to make it go down in history, to be coherent and logical. The main work was the last month, that is, 3-4 weeks before day X.
- That is, apart from your main activity, did you allocate time for preparation?- We did this in parallel with the main work, yes.
- Are you being asked to do more?- Yes, we have many requests to repeat.
- And you?- We have new ideas, new concepts, we want to attract more people and stretch it in time - both the selection process and the game process itself. In general, we are inspired by the “Cicada” project, it is possible to google it - this is a very cool IT topic, people from all over the world come together to create branches, on forums, they use cipher translation and solve riddles, and so on.
- The idea was excellent, just respect for the idea and implementation, because it is really worth it. I sincerely wish that this enthusiasm is not lost in you, so that all your new projects will also be successful. Thank!
- Yes, but will it be possible to look at an example of a task that you definitely will not reuse?- I suspect that we will not reuse any of them. Therefore, I can tell you about the progress of the quest.
Bonus trackAt the very beginning, players have the name of a virtual machine and credentials from vCenter. Having logged into it, they see this car, but it does not start. Here we must guess that something is wrong with the .vmx file. After downloading it, they see the hint needed for the second step. In fact, it says that the database used by Veeam Backup & Replication is encrypted.
After removing the hint, downloading the .vmx file back and successfully turning on the machine, they see that there is indeed a base64-encrypted base on one of the disks. Accordingly, the task is to decrypt it and get a full-featured Veeam server.
It is a little about a virtualka on which all this occurs. As we remember, according to the plot, the main character of the quest is a rather dark personality and is engaged in something that is clearly not too legal. Therefore, his work computer should have quite a hacker view that we had to create, despite the fact that it is Windows. First of all, the mass of props was added, such as information on major hacks, DDoS attacks and the like. Then they installed all typical software and put different dumps, files with hashes, etc. everywhere. Just like in the movies. Among other things, there were folders named for the principle of closed-case *** and open-case ***
To go further, players need to restore hints from files in backups.
Here it must be said that at the beginning, the players were given quite a bit of information, and most of the data (such as IP, logins and passwords) they receive during the quest, finding hints in backups or files scattered on the machines. Initially, backup files are on the Linux repository, but the folder itself is mounted on the server (mounted) with the noexec flag, so the agent responsible for restoring files cannot start.
After repairing the repository, the participants get access to all the content and can finally restore any information. It remains to understand which one. And for this they just need to examine the files stored on this machine, determine which ones are broken and what needs to be restored.
At this stage, the scenario shifts away from general IT knowledge to specific Veeam functions.
In this particular example (when you know the file name, but you do not know where to look for it), you need to use the search function in Enterprise Manager, and so on. As a result, after the restoration of the entire logical chain, the players have one more login / password and the output of nmap. This brings them to the Windows Core server, and by RDP (so that life does not seem to be honey).
The main feature of this server: with the help of a simple script and several dictionaries, an absolutely meaningless folder and file structure was formed there. And when you login, you receive a welcome message like "A logical bomb exploded here, so you have to collect hints for further steps in pieces."
The following hint was divided into a multi-volume archive (40-50 pieces) and randomly decomposed into these folders. Our intention was that players should show their talents in writing simple PowerShell scripts in order to put together a multi-volume archive using a well-known mask and get the required data. (But it turned out like in that joke - some of the subjects turned out to be unusually physically developed.)
The archive contained a photo of the cassette (with the inscription “Last Supper - Best Moments”), which gave a hint of using the connected tape library, where there was a cassette with a similar name. That's just one problem - it turned out to be inoperable so much that it wasn’t even cataloged. Here began probably the most hardcore part of the quest. We erased the header from the cassette, so in order to restore the data from it, you just have to smash the “raw” blocks and view them in the hex editor to find the markers of the beginning of the files.
We find the marker, look at the offset, multiply the block by its size, add the offset and use the internal tool to try to recover the file from a specific block. If everything is done correctly and the mathematics has come together, then the players have the .wav file in their hands.
In it, using a voice generator, among other things, a binary code is dictated, which is revealed in another IP.
This, it turns out, is a new Windows server, where everything hints at the need to use Wireshark, only it is not there. The main trick is that there are two systems installed on this machine - only the disk from the second is disabled via the device manager offline, and the logical chain makes it necessary to reboot. After that, it turns out that by default a completely different system should be loaded, where Wireshark is installed. And we all this time were on the secondary OS.
There is nothing special to do here, it is enough to turn on capture on a single interface. A relatively careful examination of the dump clearly reveals the left packet sent from the auxiliary machine at regular intervals, in which there is a link to the youtube video, where players are asked to call a certain number. The first caller will listen to the congratulations on the first place, the rest - an invitation to HR (joke)).
By the way, we have open
vacancies for technical support engineers and interns. Welcome to the team!