Analysis of all tasks of the qualifying game Yandex.Root

Tonight ended the first game of the qualifying round of Yandex.Root - the Olympics for Unix engineers and system administrators. It was attended by 456 people from 229 teams, 194 of which completed at least one task. 38 teams coped with all nine.

We are conducting Root for the fourth time, but for the first time we decided to publish the task analysis on Habré. The tasks that we give at the Olympiad are comparable to those that our system administrators regularly solve. In Yandex, almost every day something rolls out and, when something goes wrong, you need to quickly recognize it and respond effectively.


')
In general, contests for system administrators are a much rarer genre than contests of programmers, so in some way we have to be pioneers here. We tried very hard to make the tasks interesting, as well as those that really showed in the participants the qualities that are important in real work. As far as we got it, judge you.

We will be grateful if you tell us about it and share your opinion on how to do better. By the way, if you want, you can try yourself in a real game. The second part of the first round will take place in four days - on Tuesday, April 14, and you can still register for it.

Shannon game

We decided to name all the games in memory of people who contributed to modern technologies that are used in our work. This one is dedicated to Claude Shannon , an engineer and mathematician, who among other things gave us the word "bit." By the way, the root.yandex.ru service itself is running on the compute nodes of the Yandex private cloud.

The goal of the qualifying game is to solve several problems on the virtual machine by changing the configuration of the installed OS: for example, to start the service or adjust the program. Tasks can be both related and independent. It is necessary to “clear the monitoring”, that is, to solve problems with the Critical status (marked in red) inside the virtual machine. Her image can be downloaded in advance, the image is encrypted - the decryption key is published and sent to all participants by email at the beginning of the game.

After decrypting the image, you need to establish a connection to the gaming VPN. To do this, all participants receive a config-file: the captain receives it by mail at the time of approval of the application, team members - when they accept the invitation of the captain. In case of loss, the file can be downloaded again with the “download” command. Each player can connect to the VPN from his virtual machine, so tasks can be distributed within the team and solved in parallel.

Base

ArchLinux is used as the operating system for this game. After the OS is loaded, the user sees the “shannon login:” prompt. The player has no clues regarding the access details. As a rule, in such cases, you need to use the presence of "physical" access to the machine and reset the superuser password:

• reboot, stop the countdown timer in the Grub bootloader;
• press 'e', ​​find the linux line ..., append to the end init = / bin / bash;
• press Ctrl-X, waiting for the download;
• enter the passwd command, set a new password;
• enter the exec /sbin/init command, the download continues.

After performing this procedure, you can log in with a new password.

To solve problems, you need to install some packages, which means you need to prepare the package manager for work:

• synchronize with the external mirror: pacman-Sy , pacman-key --populate ;
• update package manager: pacman -S pacman , pacman-db-upgrade;
• to solve some of the tasks we need tcpdump and strace:
  pacman -S tcpdump strace.

Inside the game image is a game program that runs test scripts (hereinafter referred to as checks) on a remote server. In case of successful verification, the game counts the execution of the task.

To ensure connectivity with a remote server, you need to configure OpenVPN. The organizers have already prepared everything you need - just copy the configuration file (it is attached to the letter from Yandex.Root) in /etc/openvpn/openvpn.conf and execute the systemctl start openvpn@openvpn command.

1.SSL

This task was decided first of all. The team SudariLudari coped with it. The task is to generate a certificate signed by the specified certificate of the certification authority.


Let's write the data from the task into the ca.crt and ca.key files, respectively. Now look at the certificate of the certification center:

 # openssl x509 -in ca.crt -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 12767824280512002782 (0xb1306c894a867ade) Signature Algorithm: sha1WithRSAEncryption Issuer: C=RU, ST=Moscow, L=Moscow, O=Yandex, OU=Root, CN=root.yandex.com Validity Not Before: Apr 6 16:43:09 2015 GMT Not After : Apr 5 16:43:09 2016 GMT Subject: C=RU, ST=Moscow, L=Moscow, O=Yandex, OU=Root, CN=root.yandex.com Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (1024 bit) Modulus: 00:a3:2b:01:a7:04:75:30:42:d4:f3:2d:be:6e:86: b8:7e:79:14:40:43:24:30:1b:30:a0:fa:7e:17:5b: 3b:7c:33:e8:97:fc:e1:b1:e1:00:86:e0:68:ca:86: 2f:4d:ab:38:66:e3:5f:80:8c:f0:79:72:77:54:dc: e7:b9:8f:52:eb:72:f5:af:c5:ed:a7:d7:91:46:ee: 13:d2:2c:6e:5d:1d:ef:7b:a7:3a:63:29:59:9a:f0: e9:1c:ad:47:3a:59:62:1c:71:9e:3c:16:55:76:d5: 90:9e:50:5b:a4:b6:1b:6a:10:30:c4:13:0d:42:c4: 49:f0:1e:02:32:76:7c:a7:97 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 06:FB:29:15:D7:71:15:5B:D5:F1:09:39:DC:2E:32:D5:61:55:FC:DD X509v3 Authority Key Identifier: keyid:06:FB:29:15:D7:71:15:5B:D5:F1:09:39:DC:2E:32:D5:61:55:FC:DD DirName:/C=RU/ST=Moscow/L=Moscow/O=Yandex/OU=Root/CN=root.yandex.com serial:B1:30:6C:89:4A:86:7A:DE X509v3 Basic Constraints: CA:TRUE Signature Algorithm: sha1WithRSAEncryption 9a:f3:64:f2:20:1b:57:8f:98:32:af:fd:a3:19:0c:78:1e:91: c4:bb:3d:9b:a8:e1:63:20:0f:b8:02:3f:75:48:56:49:7f:83: f8:19:d1:71:bb:07:58:ae:2b:6b:d1:bf:ae:23:35:cd:ea:db: e4:10:f5:6a:93:33:dd:f6:8e:37:6c:9d:14:91:81:d2:d8:63: 96:0a:ff:29:77:e0:69:45:e3:a7:5e:50:b2:51:21:b3:28:59: 22:01:01:4d:a9:3e:d0:14:24:92:1e:7e:70:50:2f:26:11:82: 66:a6:a3:af:a9:b6:23:c0:e2:c7:79:85:dc:4c:bb:9d:2c:75: 18:35 

Request a new certificate with the same values:

 # openssl req -out cert.csr -new -nodes Country Name (2 letter code) [AU]:RU State or Province Name (full name) [Some-State]:Moscow Locality Name (eg, city) []: Organization Name (eg, company) [Internet Widgits Pty Ltd]:Yandex Organizational Unit Name (eg, section) []:Root Common Name (eg server FQDN or YOUR name) []:10.0.0.15 Email Address []: Please enter the following 'extra' attributes to be sent with your certificate request A challenge password []: An optional company name []: 

Prepare a structure for the work of the certification center:

 # mkdir /etc/ssl/newcerts # echo 01 > /etc/ssl/serial # touch /etc/ssl/index.txt 

The following command will return an error, the text of which is not so obvious:

 # openssl ca -cert ca.crt -keyfile ca.key -in cert.csr -out cert.crt Using configuration from /etc/ssl/openssl.cnf Check that the request matches the signature Signature ok The stateOrProvinceName field needed to be the same in the CA certificate (Moscow) and the request (Moscow) 

In fact, the problem is that the lines in the certificate
Certification center and request are written in different encodings. To work around this problem, edit the /etc/ssl/openssl.cnf file and change the value of the string_mask parameter in the [req] section to pkix .

It remains to prepare the files for the web server:

 # mv privkey.pem cert.key # cat ca.crt >> cert.crt 

Now we will install the web server (pacman -S nginx) and enable SSL in /etc/nginx/nginx.conf% , uncommenting the appropriate section of the server {} .

Checking:

 # nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful # systemctl restart nginx 

But the decision is not made by the checking system with the diagnostics “SSLv3 is weak” . Change the configuration to the recommended Mozilla :

• mark the options Nginx and Modern, insert the configuration in nginx.conf ;
• generate the Diffie-Hellman parameters: openssl dhparam -out dhparam.pem -outform PEM -2 2048 ;
• restart nginx : systemctl restart nginx.

2.MariaDB repair

This task requires recovery of the database.

There is a MariaDB database in / var / lib / mysql. We had access there with login checker and master key, but something went wrong.

 BTW, the `data` table structure was: +-------+---------+------+-----+---------+-------+ | Field | Type | Null | Key | Default | Extra | +-------+---------+------+-----+---------+-------+ | name | text | YES | | NULL | | | hits | int(11) | YES | | NULL | | | size | int(11) | YES | | NULL | | +-------+---------+------+-----+---------+-------+ 

From the name it is obvious that the database is used MariaDB , known MySQL fork.

Install the DBMS: pacman -S mariadb and try to start systemctl start mysqld . From the logs it is clear that mysqld is looking for files in the wrong place. From the configuration file /etc/mysql/my.cnf can see that the system’s operation in the network mode is broken - skip-networking , bind-address options are added, the wrong datadir value is specified. To save time, we will not attempt to repair the configuration file; instead, replace it with a known working:

 [mysqld] key_buffer_size = 16M max_allowed_packet = 1M table_open_cache = 64 sort_buffer_size = 512K net_buffer_length = 8K read_buffer_size = 256K read_rnd_buffer_size = 512K myisam_sort_buffer_size = 8M tmpdir = '/var/tmp' 

Let's correct the owner of the files - chown -R mysql:mysql /var/lib/mysql , and try to start the database again - systemctl restart mysqld .

Hurray, mysqld works. Let's try to connect:

 # mysql ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO) 

We see that password protection is set, and the password is unknown to us / So, as in the case of the operating system, we will have to reset it:

• stop mysqld: systemctl stop mysqld ;
• prepare the file with the reset instruction: echo "UPDATE mysql.user SET password = PASSWORD('root') where user = 'root';" > /tmp/reset.sql echo "UPDATE mysql.user SET password = PASSWORD('root') where user = 'root';" > /tmp/reset.sql ;
• we start mysqld with loading of our file: mysqld --user=mysql --init-file=/tmp/reset.sql ;
• Now we ’ll stop mysqld in a regular way. To do this, press Ctrl-\ ;
• run unit again - systemctl start mysqld

After performing this procedure, we can already connect to the database with the password root . Let's try: mysql -ppassword -uroot . From the output of the show databases command, we see the existence of the db database, but there is no data table in it. However, this table is on disk (/var/lib/mysql/db/data.ibd) . Not enough table definition (data.frm).

Fortunately, the task contains a hint on how the table looks, which allows you to reconstruct frm without analyzing the system tables. Run the query:

 create table data2 (name text, hits int(11), size int(11)); 

Create a table with the name data does not work, because the system tables still contain its mention. Now connect the data file to our new table. To do this, disable the "empty" ibd-file from the table: alter table data2 discard tablespace; , substitute it filled with mv data.ibd data2.ibd and connect back to alter table data2 import tablespace; .

Since db.data is still listed in the system tables, we cannot make the drop table db.data . You will have to create a temporary database, transfer a new table to it, delete the old one, and then create it again:

 rename table db.data2 to db2.data; drop database db; create database db character set utf8; rename table db2.data to db.data; alter table db.data engine = innodb; 

It remains only to give access to the user: grant all privileges on db.* to 'checker'@'%' identified by 'masterkey'; . Unfortunately, the test fails due to a connection error. A check with tcpdump indicates that mysqld is not responding. Let's check the firewall with iptables-save and find the problem left by the villain — erroneous rules have been added to the nat table. However, removing them briefly restores the network - the rules appear again.

As a rule, periodic actions are caused by the work of crontab . Check (crontab -l, cat /etc/cron.* /etc/crontab /etc/cron.d/*) and delete all the tasks of the current user (crontab -r) .

3.Binary

Run 1.exe

In this task, you need to run an "unusual" program. From the task we know the file name - "1.exe". Find the file on the file system and check its contents:

 # find / -iname 1.exe /root/1/1.exe # file /root/1/1.exe /root/1/1.exe: PE32 executable (console) Intel 80386 Mono/.Net assembly, for MS Windows 

To run .NET applications under GNU / Linux, there is a mono environment. Install it (pacman -S mono) and try to run our program:

 # cd /root/1 # mono 1.exe 

But this task is not as simple as it seems. In return, the game will return:

 Name: Binary Status: uncompleted Output: bad program 

Let's try to understand what the 1.exe program does. Running mono under strace will show that the program listens on a TCP socket and executes the commands sent by the test. Surface traffic analysis using tcpdump shows that the program can read files, access the embedded database and perform calculations. Verification finishes work after performing the calculations, which means that the problem most likely lies in them.

You can often extract additional information by searching for text strings inside executable files. Let's try to use this technique - install the binutils software binutils and apply the strings program to the 1.exe file. One piece of output is similar to the list of libraries used by the program:

 System.Core mscorlib System.Xml dnAnalytics 

All of these libraries, except dnAnalytics , are part of mono - this is easy to check with the help of the package manager (pacman -Ql mono) .

Install the missing library by downloading and unpacking the archive from the official site (bin / *. Dll should be put in / root / 1).
After restarting, the program successfully passes the test.

4.Mongo

In this task, you need to deploy the MongoDB shard storage, writing down the data offered by the organizers.

There is a database in /var/lib/db.tar.gz.

Make a root. Features a collection of 2 shards

First, install mongodb: pacman -S mongodb . The organizers left the archive with the base, unpack it and make an archive copy:

 cd /var/lib/mongodb tar jxf /var/lib/db.tar.bz2 mongod --dbpath db mongodump rm -rf db 

For sharding you will need a special service database called configdb . Create it:

 mkdir -p /data/configdb mongod --configsvr & 

Now run the “sharper” mongos: mongos --configdb localhost & . The task is to raise two shards, so we will prepare two mongod instances:

 mkdir /var/lib/mongodb/s1 /var/lib/mongodb/s2 mongod --dbpath /var/lib/mongodb/s1 --port 30001 --nojournal & mongod --dbpath /var/lib/mongodb/s1 --port 30002 --nojournal & 

And connect them to mongos :

 # mongo mongos> sh.addShard("localhost:30001") mongos> sh.addShard("localhost:30002") mongos> sh.enableSharding("root") 

Now it remains to load the dump back: mognorestore --port 30001 dump/ .

However, this is not enough to solve the problem - the root.features collection will not be evenly shared between two shards. Let's solve this problem by creating an index on the document identifier and turning on the balancer:

 # mongo root mongos> db.features.ensureIndex({"_id":"hashed"}) mongos> sh.shardCollection("root.features", {"_id":"hashed"}) mongos> sh.enableBalancing("root.features") 

Having waited until the collection is redistributed between shards, we will run the test again.

5.Strange Protocol

This task was the most difficult. Actually, as we ourselves predicted.

Set up an echo server on port 13000.

This time we need to start the echo server on port 13000.

This task seems simple - indeed, the simplest implementation of the echo server is already built in, for example, in xinetd . Running tcpdump port 13000 shows that the exchange takes place via the UDP protocol, but setting up the echo-dgram in xinetd does not give the expected result.

Let's take a closer look at the traffic - run tcpdump again, but with the -X option. The latest package seems interesting:

  0x0010: 0a00 000f ebee 32c8 0012 ffd6 656e 6574 ......2.....enet 0x0020: 2065 7272 6f72 .error 

Searching for the word enet leads to a site that describes the implementation of the enet protocol, which allows you to send data streams via UDP without worrying about packet loss (as in TCP).

Further search leads to the pyenet library, binding to the enet for the Python language, which is just right for our task. Let's write a simple program:

 import enet import sys host = enet.Host(enet.Address(b'0.0.0.0', 13000), 100, 0, 0) while True: evt = host.service(0) if evt.type == enet.EVENT_TYPE_RECEIVE: data = evt.packet.data evt.peer.send(0, enet.Packet(data)) 

It remains to install these libraries:

 pacman -S git git clone git://github.com/aresch/pyenet cd pyenet git clone git://github.com/lsalzman/enet pacman -S cython base-devel python setup.py build python setup.py install 

We start our program, and the check is performed successfully this time.

6.File

The organizers hid root.txt somewhere inside / root / file .

There is a / root / file inside your image. Find a good root.txt file and make it available via image_ip / root.txt .

Let's try to understand what is / root / file :

 # file /root/file /root/file: LVM2 PV (Linux Logical Volume Manager), UUID: XT6zLL-YAUv-nmA9-BSrw-2pBV-CTi2-vqKe35, size: 31457280 

It looks like a disk image. Linux has a kernel loop module, which allows you to turn files into block devices. We use it:

 losetup /dev/loop0 /root/file 

Since the LVM volume is located inside the disk image, let's connect it with regular tools:

 # vgchange -ay 1 logical volume(s) in volume group "VolGroup00" now active 

Let's see what's inside:

 mount /dev/mapper/VolGroup00-lv0 /mnt ls /mnt 

See root.txt.gz , unpack: gunzip /mnt/root.txt.gz .

Since we have already installed nginx for setting SSL, we will use it to distribute the file via HTTP:

 umount /mnt mount /dev/mapper/VolGroup00-lv0 /usr/share/nginx/html/ 

Unfortunately, the check fails - we found the wrong root.txt . We will look further. Let's see what our file system is:

 # file -s /dev/dm-0 /dev/dm-0: BTRFS Filesystem sectorsize 4096, nodesize 4096, leafsize 4096) 

Since btrfs has an understandable subvolume , let's look at their list:

 # pacman -S btrfs-progs # btrfs subvolume list /usr/share/nginx/html/ ID 256 gen 14 top level 5 path root ID 257 gen 11 top level 5 path root_1 

It turns out that there is another subvolume - root_1. We mount it exactly:

 umount /usr/share/nginx/html mount -t btrfs -o subvol=root_1 /dev/mapper/VolGroup00-lv0 /usr/share/nginx/html/ 

Now we have found another root.txt.gz file. Unpack it: gunzip /usr/share/nginx/html/root.txt.gz .

This will be the solution to the problem.

7.MariaDB Tuning

While solving the MariaDB repair problem, we repaired the database, but it works too slowly. It is time to fix it.

The repaired MariaDB is slow. Tune it up.

Let's see where our base slows down. Let's turn on the slow query log , where all queries that run for longer than a second will get:

 mysql -u root -ppassword db mysql> set global slow_query_log = ON; mysql> set global long_query_time = 1; 

Run the check and look at the log: tail /var/lib/mysql/shannon-slow.log .

We see the query SELECT COUNT(*) FROM db.data WHERE size < 10; .

Let's look at the query plan:

 mysql> explain SELECT COUNT(*) FROM db.data WHERE size < 10 \G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: data type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 25061163 Extra: Using where 1 row in set (0.00 sec) 

Of course, such a query is executed too slowly - we do not have indices for this field. Add an index: mysql> create index data_size on data(size); .

Restarting the test will show us the same problem for data (hits), which we solve in the same way.

8.HG

In /root/repo mercurial repository, for which we need to correct the story and make it available via http.

There is a HG repository in / root / repo.

Drop all .gz files in all revisions available via ip : 8000 /

First, install mercurial : pacman -S mercurial . You can change the history using the convert module, which is disabled by default. Turn it on:

 # cat <<EOF > ~/.hgrc [extensions] hgext.convert= EOF 

This module can apply file matching rules in the source and target repositories. Such rules are called filemap . We write and apply a rule that throws out a file 2.osm.gz :

 echo 'exclude "2.osm.gz"' > /root/fmap hg convert --filemap ~/fmap /root/repo /root/repo1 

After executing the command, we get the repository / root / repo1 , which is devoid of the 2.osm.gz file in all revisions. It remains to make it accessible from the outside. Mercurial has a built-in web server, which we will use:

 cd /root/repo1 hg serve 

9.Strange File

This task was handled by the largest number of commands - 151. There is a strange tester / file on the file system that no one can change.

We got a strange file in ~ tester / file. No one can change it. Fix it.

Indeed, the file cannot be changed - even from under the root:

 # echo test >> ~tester/file -bash: /home/tester/file: Permission denied 

First, let's see what our file system is:

 # mount | grep ' on / ' /dev/sda2 on / type ext4 (rw,relatime,data=ordered) 

From the ext4 (man 5 ext4) you can find out that files on this file system can have
following attributes:
Let's look at the chattr (1) page, where the behavior of the system for files with the immutable attribute set is described in detail:
The answer is obvious - you need to remove this attribute from the file: chattr -i ~tester/file . Problem solved.