Shot myself in the (kernel upgrade) foot

2021-03-25

About an one and half hours ago, my workstation didn't come back after a reboot. GRUB worked and disk unlock worked. But as soon as the machine started to start lxdm, I got core dumps.

First steps to resolution: Boot some live system and have a look what the machine says:

cryptsetup open /dev/sda20 cryptroot
mount /dev/mapper/cryptroot /mnt
chroot /mnt
journalctl --since=today

Journalctl showed me:

Mar 25 20:38:50 detritus2 systemd-coredump[655]: Process 648 (lxdm-greeter-gt) of user 976 dumped core.
(...)
Mar 25 20:45:23 detritus2 systemd-coredump[648]: Process 641 (lxdm-greeter-gt) of user 976 dumped core.

Luckily I remembered that I had recently messed with my Kernel update settings. I had this great idea of putting my workstation's Archlinux kernel into IgnorePkg to avoid near-daily reboots after every kernel update. Turns out, Archlinux documentation saying "partial upgrades are unsupported" all over the place has some reason behind it and my idea wasn't as great as I thought.

So I booted an Archlinux live system, opened my LUKS disk, mounted it into /mnt (forgot boot, of course), re-created initramfs, reinstalled GRUB and rebooted. Behavior remained the same (of course), as I had forgotten that bloody boot partition.

On the second attempt (boot into Arch live, open LUKS, mount /mnt, mount /mnt/boot (!), rebuild initramfs, install GRUB, reboot), my system was usable again.

List of commands for recovery:

cryptsetup open /dev/sda20 cryptroot
mount /dev/mapper/cryptroot /mnt
mount /dev/sda1 /mnt/boot
arch-chroot /mnt
pacman -Syu
mkinitcpio -P
grub-mkconfig -o /boot/grub/grub.cfg
exit (to leave chroot)
reboot

Yeah, I admit, that was some huge told-you-so, dear Archlinux documentation maintainers...