There was no sadness, updates pumped up (Arch)

Based on this post .

About Archlinux goes a lot of rumors, including not entirely truthful. In particular, well-established public opinion says that Arch often breaks during updates, as the bleeding edge. In practice, this is one of the most enduring and maintainable systems, which can live for years without reinstalling, while updating almost every day.

Sometimes, about one or two times a year, problems still arise. Some malicious bug manages to leak into the repository, after the update the system breaks down, and you as a user can do little about it - you need to roll back.

Situation

With the next update, everything breaks down to crap dog . The level of possible problems varies from "fonts become ugly" to "network stopped working", and even "the kernel does not see disk partitions." Many users of Archa are able to deal with this problem anyway, and in this article I will tell you how I do it.

Return control system

For repairs, we need a console and a working network. If there is - well. If not, there is an easy and fast way to get them guaranteed. Boot from the bootable USB flash drive, mount our disk partitions, read the system:

arch-chroot /mnt 

Everything, we are in the console of our system, and we have a network.

And the presence of a bootable flash drive in archiver's arsenal is almost mandatory.

Repair algorithm

1. We are convinced that the problem is insoluble on its own.
2. We determine the cause of the problem and the date of the problem update.
3. Substitute /etc/pacman.d/mirrorlist
4. Perform pacman -Syyuu

The system is repaired!

How it works

There is a special update server called Arch Linux Archive (formerly called Arch Linux Rollback Machine), which stores snapshots of all repositories for each individual date.

You can select the desired date by specifying the server name in the mirrorlist file in a special way. The easiest way is to backup the old file, create a new one and add a single line there (in the example, December 1, 2017 is set)

 ## ## Arch Linux repository mirrorlist ## Generated on 2042-01-01 ## Server=https://archive.archlinux.org/repos/2017/12/01/$repo/os/$arch 

After that, the pacman-Syyuu command will forcefully update the package database to the one that corresponds to the selected date, and forcefully reinstall all packages in the system, the versions of which do not correspond to this date.

Our task is to determine the nearest date, rolling back on which the working system gives, find out which packages were updated the next day, and find out which of these packages caused a breakdown .

Let's get started!

Determining the desired date

If the system was updated not so long ago, you can step by step reduce the date in the mirrorlist, each time performing pacman -Syyuu and checking if the problem has disappeared. The advantage of this approach is that it is highly likely to immediately calculate a specific package and add it to /etc/pacman.conf in the IgnorePkg line - until better times.

If a month has passed since the last update, say, a month, then iteratively divide the time interval in half. First, roll back a fortnight. If the problem has disappeared, then we are updated a quarter of a month in advance; if not, then a quarter of a month ago. And so on. This method allows you to determine the exact moment of failure that occurred during the last year, in just 9 date changes.

Identifying the culprit package

So, let's say - we have not been updated for a week, pacman reports that there are 200 non-updated packages. After the upgrade, the system crashes.

Roll back for a day, several packages are "updated" to an earlier version. Check - the system is still broken.

Roll back for another day, a few more packages reduce the version. The system is broken.

Another day, the next three packages are "updated" and vice versa hurray, the problem has disappeared!

Now we know for sure that one of these three packages is to blame. Suppose it is linux , linux-headers and gnome-shell . Since linux-headers pull after linux , we don’t take them into account, this is a dependency. So we have only two options.

Next we add the candidates in /etc/pacman.conf one by one.

Let's start with the linux package:

 # Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup IgnorePkg = linux #IgnoreGroup = 

After that, we update the system through pacman -Syu and see if the problem has disappeared. If it disappeared, it is precisely because the package-culprit is written in pacman.conf as forbidden to upgrade. If it does not disappear, it rolls back to the working system again, enter the next candidate in IgnorePkg and repeat the cycle.

Item which must be first

And whose, actually, is a bug? This article is devoted to problems at the level of the entire distribution. There is no point in rolling back to old versions of packages if the problem is not caused by their updates. Therefore, the first thing we do is google the error in all possible ways, setting up the output only for fresh records. If the problem is common, then with great probability you will stumble upon a fresh thread on any official distribution forum where you will find all the details - the exact date of the problem update, the name of the problem package, and even technical reasons (which actually broke under the hood).

Take the example above. Suppose we found out that our system does not boot into graphics mode after updating the gnome-shell package to version 3.26.2

What to do next?

First we make the system work - we roll back to the date a day before the harmful update and add the gnome-shell to IgnorePkg (alternatively, add the whole gnome group to the IgnoreGroup so that the gnome is not partially updated).

Next, we are trying to find the answer to the question in Google - "is this my personal problem or do others have it, too?" If there is no such problem on the Internet, it may be a sign that the bug is too fresh. We are waiting for confidence for several days, sometimes repeating search queries in different combinations. If the problem is that the problem package really climbed into the repository - it will definitely come out on the Internet ! Archa has a huge user base, and people will write bug reports and ask questions on the forums.

If within a few days you do not see any mention of the problem, I have bad news for you. With a probability close to 100%, you broke the system yourself. Remember your actions, analyze logs, understand what happened. In general, such a case (when some kind of update leads to a breakdown, but this is not a bug) is typical of situations that are always covered in the news in the heading "when updating, manual intervention is required." This happens when the next update introduces such a major change to the system that the means of the update itself cannot be correctly implemented.