Linux Quest. Congratulations to the winners and tell about solutions

On March 25, we opened registration on Linux Quest , a game for fans and connoisseurs of the Linux operating system. Some statistics: 1117 people registered for the game, 317 of them found at least one key, 241 successfully coped with the task of the first stage, 123 - the second and 70 completed the third stage. Today, our game has come to an end, and we congratulate our winners!

• The first place was taken by Alexander Teldekov.
  Alexander said to himself that he is the most typical sysadmin. He lives in Volgograd, various Unix-like systems have been administrating for about twenty years. I managed to work in Internet providers, bank, system integrator. Now he works remotely in a small company, he is engaged in cloud infrastructure for a large foreign customer. Loves to read, listen to music. About the Game, Alexander said that he liked the game as a whole, he likes such puzzles. In one of the companies at the interview dealt with Hackerrank-like, it was interesting.
• Second place - Roman Suslov.
  Roman from Moscow. He's 37 years old. Linux / Unix engineer works in Jet Infosystems. At work, you have to deal with the administration and troubleshooting of Linux / Unix-systems + SAN. Interests are very different: Linux-systems, programming, reverse engineering, information security, Arduino. About the Game Roman said that the game was generally liked. “I crushed some of my convolutions and distracted myself from the everyday routine of everyday work. :) I would like to have more tasks, but I did not have time to get into the taste as the game was already over. ”
• The third is alex3d.
  Alex lives in Moscow, develops software. "Thanks for the contest, it was interesting to test your google-fu skill."

Also in the ranking of the top 10 players:

• Yevgeniy saldayev
• Markel Mokhnachevsky
• Konstantin Konosov
• Pavel Sergeev
• Vladimir Bovaev
• Ivan Bubnov
• Pavlo Klets

We understand that there are a lot of options for solving all of our tasks; below one of the possible solutions is described.
')

1. The first stage

We called it “Are you exactly an admin?”, Since the task was quite simple - to fix the warm lamp service.

1.1. Interesting Facts:

Two players found the first key in the first 15 minutes of the game, and in the first hour we had three leaders who coped with the task.

1.2. The task

You went to work in a company where for a long time there was no competent information technology specialist. Before you begin to restore order, you need to solve a burning problem that blocks the work of the office.

A cleaning lady hooked a mop to the power cable of the server cabinet. Meals have been restored, but a very important website still does not work. The website is important because the company is not very concerned about information security, and on the main page of this you can find in open text the password of the administrator from the CEO computer.

The other day the password was changed, and the new one was forgotten, the director cannot work. Rumor has it that there were more keys on this machine that can help us in deciphering the backup copy of accounting documents.

Everyone is waiting for a quick solution

1.3. Decision

1. First of all, you need to change the root password in the virtual machine in order to access it. At startup, we note that this is Ubuntu 16.04 Server.

To reset the root password, restart the machine, at boot, when the grub menu is displayed, go to editing the Ubuntu item with the "e" button. Edit the linux line, add init=/bin/bash to the end. Loaded via Ctrl + x, we get bash. Remount root with rw, change password:

 $ mount -o remount,rw /dev/mapper/ubuntu--vg-root $ passwd 

Do not forget about sync, reboot.

2. The condition states that the web server does not work for us, we look:

 $ curl localhost Not Found The requested URL / was not found on this server. Apache/2.4.18 

That is, in fact, Apache is running, but responds with code 404. We are looking at the config:

 $ vim /etc/apache2/sites-enabled/000-default.conf 

Here is the key - StevenPaulSteveJobs.

Check the path /usr/share/WordPress - this is not, but there is /usr/share/wordpress . Edit the configuration and restart the Apache.

 $ systemctl restart apache2 

3. We try again, we get an error:

 Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/share/wordpress/wp-includes/wp-db.php on line 1488 

DB not running?

 $ systemctl status mysql Active: active (running) 

What's the matter? Need to understand. To do this, you need to access MySQL, as described in the documentation . One of the documentation points recommends that we register the skip-grant-tables option in /etc/mysql/mysql.conf.d/mysqld.cnf . Here, too, is the key - AugustaAdaKingByron.

Adjust rights to user 'wp'@'localhost' . We start MySQL, make it available on the network, commenting on the skip-networking option in the config.

4. After the done actions, the web server is started, but the site still does not work, because

 Warning: require_once(/usr/share/wordpress/wp-content/themes/twentysixteen/footer.php): failed to open stream: Permission denied in /usr/share/wordpress/wp-includes/template.php on line 562 

We rule the rights to the file.

 $ chmod 644 /usr/share/wordpress/wp-content/themes/twentysixteen/footer.php 

We update the page, go to the site and find the key - BjarneStroustrup! We found all three keys, our director can work, we decrypted the accounting files. Everyone is happy, and you have a lot of work ahead to build infrastructure, backups and security in the company.

2. The second stage

It was necessary to solve the problem of collecting analytics. Everyone loves analytics - who and from where and in what quantities go. We have come up with a case that all engineers in life can face in one form or another.

2.1. Interesting Facts

One of our players entered the correct key in the first 10 minutes of the game, and in the first hour we had a leader who coped with the task.

2.2. The task

You went to work at the company, managers came to you and asked to find out who sent letters from Africa. You need to build the top 21 recipient addresses for them. The first letters of the recipients' addresses are the key. One thing: the mail server through which the letters were sent, does not load. Everyone is waiting for the prompt resolution of the issue!

2.3. Decision

1. The server is not loaded due to a non-existent swap partition in fstab, while loading, the system tries to mount it and crashes. How to boot?

Download the image, we downloaded CentOS 7, boot from the Live CD / DVD (Troubleshooting -> Rescue), install the system, edit /etc/fstab . Immediately find the first key - GottfriedWilhelm11646Leibniz!

Create a swap:

 $ lvcreate -n swap centos -L 256M $ sync && reboot 

2. There is no password as always, you need to change the root password on the virtual machine. We have already done this in the first task. We change and successfully go to the server, but it immediately goes into reboot. The server is overloaded with such speed that it does not even have time to look at all the logs carefully. How to understand what is happening?

Again we load from livecd, carefully study the system logs and just in case we look at kroner, once such a periodicity. There we find the problem and the second key - Alan1912MathisonTuring!

You need to delete or comment out the line echo b > /proc/sysrq-trigger in /etc/crontab .

3. After that, the server is loaded, and you can perform the task of managers: “What are the addresses of Africa?” This information is generally public. You can find this information on the Internet by the phrases “ip address africa”, “geoip database”. To solve the problem, you can use freely available address allocation bases (geoip). We used the MaxMind GeoLite2 DB , which is available under the Creative Commons Attribution-ShareAlike 4.0 license, as a reference.

Let's try to solve our problem using only Linux system utilities, but in general it can be solved in a huge number of ways: using text filtering utilities and using scripts in various programming languages.

To begin with, we simply get the pairs “IP-sender-recipient” from the mail log /var/log/maillog (we will build the recipients email table - the sender's IP). This can be done with this command:

 $ cat /var/log/maillog | fgrep -e ' connect from' -e 'status=sent' | sed 's/[]<>[]/ /g' | awk '/connect from/ {ip=$11} /status=sent/ {print $10" "ip}' > log1.txt 

And before we continue with the compilation of a database of African addresses, we will look at the top IP addresses of senders.

 $ cat log1.txt | cut -d' ' -f1 | sort | uniq -c | sort -r | head -n 40 5206 L2JhbjAbM67GA99jg@mail.ru 4165 iHKTBkegOQa6fIALq@mail.ru 3739 nHkcBl7BdgXxijSYD7@mail.ru 3405 SMAzPJAzbl9vp4hAXo@mail.ru 3346 xILz6d7P@mail.ru 

Among all, the first three recipients from the top are clearly distinguished by the number of letters. If you grep the IP addresses of the senders that were sent to the addresses from this top-3, you can see a clear predominance of certain networks:

 $ cat log1.txt | fgrep 'L2JhbjAbM67GA99jg@mail.ru' | cut -d' ' -f2 | sort | cut -d'.' -f1 | uniq -c | sort -r | head 831 105 806 41 782 197 664 196 542 154 503 102 266 156 165 45 150 160 108 165 

Most of the 105/8, 41/8, 196 / 8,197 / 8 networks are allocated to AFRINIC, one of the five regional Internet registrars that distribute Internet resources. AFRINIC distributes address space in Africa. A 41/8 refers to the whole AFRINIC.

 https://www.nic.ru/whois/?searchWord=105.0.0.0 https://www.nic.ru/whois/?searchWord=41.0.0.0 

Thus, the answer to the problem, in fact, is in the log itself.

 $ cat log1.txt | fgrep -e '105.' -e '41.' -e '196.' -e '197.' -e '154.' -e '102.' | awk '{print $1}' | sort | uniq -c | sort -r | head -n 21 4209 L2JhbjAbM67GA99jg@mail.ru 3313 iHKTBkegOQa6fIALq@mail.ru 2704 nHkcBl7BdgXxijSYD7@mail.ru 2215 uvRbp1O@mail.ru 1774 sPmMsmmFiV@mail.ru 1448 BtG3aHgQgCKuze2AKuRH@mail.ru 1233 eQpuuQ2uQdbwRL3@mail.ru 958 nJT5dpaBZ@mail.ru 862 ef4WbQiB@mail.ru 762 dQCqKL6eVminFfH7wLA@mail.ru 632 ifq6Rd1HxuCQOdO9@mail.ru 539 cFwm2ssypMmx1sA7@mail.ru 531 twtTnr4G@mail.ru 431 TSrczgYASrR11Hs3qCi@mail.ru 380 o3r3exc3OL@mail.ru 357 rzmjr2VAHK@mail.ru 348 vnPr6YjJ3ndw@mail.ru 312 anOjFXrwOtLP2Rl1Vcz6@mail.ru 289 dvny5zHmRW8fiT@mail.ru 282 sgg9jPxFDYvzw8Kr@mail.ru 274 tKSevzA7GntJ@mail.ru 

At this stage, we get the string "LinuxBenedictTorvadst".

The correct key is "LinusBenedictTorvalds".

The resulting string contains a typo with respect to the correct key in the 3 extreme characters. This is due to the fact that the networks we have chosen are not entirely allocated to African countries and to the way emails are distributed to IP addresses in our log.

With sufficient clarification of the largest networks allocated to African countries, you can get an exact answer .:

 $ cat log1.txt | fgrep -e' '105.{30..255}. -e' '41. -e' '196.{64..47}. -e' '196.{248..132}. -e' '197.{160..31}. -e' '154.{127..255}. -e' '102.{70..255}. -e' '156.{155..255}. | awk '{print $1}' | sort | uniq -c | sort -r | head -n 21 3350 L2JhbjAbM67GA99jg@mail.ru 2662 iHKTBkegOQa6fIALq@mail.ru 2105 nHkcBl7BdgXxijSYD7@mail.ru 1724 uvRbp1O@mail.ru 1376 sPmMsmmFiV@mail.ru 1092 BtG3aHgQgCKuze2AKuRH@mail.ru 849 eQpuuQ2uQdbwRL3@mail.ru 712 nJT5dpaBZ@mail.ru 584 ef4WbQiB@mail.ru 463 dQCqKL6eVminFfH7wLA@mail.ru 365 ifq6Rd1HxuCQOdO9@mail.ru 269 cFwm2ssypMmx1sA7@mail.ru 225 twtTnr4G@mail.ru 168 TSrczgYASrR11Hs3qCi@mail.ru 142 o3r3exc3OL@mail.ru 111 rzmjr2VAHK@mail.ru 96 vnPr6YjJ3ndw@mail.ru 78 anOjFXrwOtLP2Rl1Vcz6@mail.ru 56 lHzWiB7ExvRtSbAcU9@mail.ru 56 dvny5zHmRW8fiT@mail.ru 40 sgg9jPxFDYvzw8Kr@mail.ru 

The task can also be solved in another way.
We download MaxMind, we unpack, and the following three commands also solve our problem.

 $ cat GeoLite2-Country-Locations-ru.csv | grep "" | cut -d',' -f1 > africaIds.txt $ grep -Ff africaIds.txt GeoLite2-Country-Blocks-IPv4.csv | cut -d',' -f1 > africaNetworks.txt $ grepcidr -f africaNetworks.txt log1.txt | cut -d' ' -f1 | sort | uniq -c | sort -r | head -n21 

One way or another, we eventually counted the statistics, and the managers got the data they needed to work!

3. The third stage

The third stage is somewhat similar to the first - you also need to fix the warm lamp service, but everything is more complicated than in the first task.

3.1. Interesting Facts

In the first 15 minutes, three players found the first key, 2 hours and 20 minutes after the start of the stage, our winner coped with the task.

3.2. The task

You went to work at a company where all company documents are stored on the internal Wiki server. Last year, the engineer ordered 3 new drives for the server in addition to one existing one, arguing that for the fault tolerance of the system, you need to put the drives in certain arrays. Unfortunately, a few weeks after their installation, the engineer went to rest in India and did not return.

For several years the server worked without failures, but a couple of days ago the company’s network was hacked. According to the instructions, security officers removed the disks from the server and sent them to you. In the process of transportation, one disk was irretrievably lost.

It is necessary to restore the functionality of the Wiki, primarily interested in the content of wiki pages. Some piece of text that was on one of the pages of this wiki is a password from server 1C and is urgently needed to unlock it.

In addition, somewhere on the wiki pages or elsewhere there were passwords from the log server and the video surveillance server, which are also desirable to recover, without them the investigation of the incident is impossible. As always they are waiting for the prompt solution of the issue!

3.3. Decision

1. We are trying to boot in turn from the disks we have and everywhere we get the same message:

 No bootable medium found! System halted 

It is necessary to boot from something. Again saves the boot from the Live CD / DVD (Troubleshooting -> Rescue). When loading we try to find the boot partition, we don’t find it, we get into the shell. We are trying to learn what and how with the disks. It is known that there are three of them. There are more tools for this in the 7th CentOS version, where there are blkid or lsblk that show us all the information about disks.

How and what we do:

 $ ls /dev/sd* 

Immediately it is clear that

 /dev/sdb1 - ext4 /dev/sdb2 -  lvm /dev/sda1  /dev/sdc1 -   /dev/sda2  /dev/sdc2 -         

Mount sdb1, it is clear that this is the CentOS 6 boot partition.

 $ mkdir /mnt/sdb1 && mount /dev/sdb1 /mnt/sdb1 

Obviously, go to the grub section and find the first key there - James191955Gosling in an unusual file.

2. We study pvs and lvs, as we work with LVM. We see that there should be 2 physical volumes, one is not located and swears at the lost uid. We see that there should be 2 logical volume: root and swap, while root is partially lost (the P attribute for volume). Mount fails, but sorry! We really need it.

There are 2 more disks, we look at them, we collect and we mount:

 $ mdadm --examine --verbose --scan $ mdadm --assemble --verbose --scan $ mkdir /mnt/md127 && mount /dev/md127 /mnt/md127 

We look, it is clear that this is a CentOS 6 boot partition and a duplicate of what is already on /dev/sdb1 , and here again the same key - James191955Gosling!

We look as /dev/md127 .

 $ mdadm --detail /dev/md127 

We see that it should have been assembled from 4 disks, assembled from two /dev/sda1 and /dev/sdc1 , they should have been numbers 2 and 4 in the system. We assume that from /dev/sda2 and /dev/sdc2 also possible to assemble an array. It is not clear why there are no metadata on them, but this is on the conscience of the admin, who is somewhere in Goa. We assume that there should be a RAID10, although there are options. We collect:

 $ mdadm --create --verbose /dev/md0 --assume-clean --level=10 --raid-devices=4 missing /dev/sda2 missing /dev/sdc2 

We look blkid, pvs, lvs. We find that we have collected a physical volume, which we previously lacked.

Immediately fix lvroot, mount it, but first activate VG:

 $ vgchange -ay $ mkdir /mnt/lvroot && mount /dev/mapper/vg_c6m1-lv_root /mnt/lvroot 

And everything is there, including the key DennisBMacAlistairCRitchie - in the root home directory - /root/sweet .

3. We are still trying to revive our server so that it starts up normally. All logical volume from our /dev/md0 (where everything was found) is dragged to /dev/sdb2 , where the entire server initially worked.

 $ pvmove /dev/md0 /dev/sdb2 $ vgreduce vg_c6m1 /dev/md0 

We extinguish the server, remove 1 and 3 discs, leave the second one, boot from the Live CD / DVD into the Rescue. Find the boot partition, restore the bootloader to grub:

 root (hd0,0) setup (hd0) 

We tear off the boot disk and successfully load, but the site does not work.

4. There are two options to start the site: configure Apache from scratch or use nginx already configured in advance with php-fpm:

 $ /etc/init.d/nginx start $ /etc/init.d/php-fpm start 

Finally, you need to start MySQL:

 $ /etc/init.d/mysqld start 

It does not start, and the secret lies in /var/log/mysql . As soon as you solve the problem with MySQL, the site will work, on the main page there will be a key - RichardGCCMatthewGNUStallman! Now we have access to 1C, and employees will be able to receive a salary. And you, as always, have a lot of work ahead on building infrastructure and security in the company.

We can also once again share a list of books that helped us and our participants to prepare for the game: linux.mail.ru/books .

Thank you for being with us! Follow the announcements of the next games!