Kernel 6.1.0-55 freezes Debian 12: how to check your servers and update
After the recent wave of leaks and breaches, we install all updates on our infrastructure servers and staging automatically and reboot them automatically, too. On November 16–17, that's how Debian 12 kernel 6.1.0-55 arrived, and all ten servers that booted into it froze: some after a couple of hours, others after 30. It didn't reach production: there, instead of an automatic reboot, we have an alert that a server needs one.
The cause is an ext4 bug, and a fix is already out. We're sharing what we ran into so you can be more careful. If you run Debian 12, check your kernel now.
- Kernel
6.1.0-55(linux 6.1.192-1, released on November 15) has an ext4 bug. A few hours after boot, disk writes stall while the network keeps working. - The fix is kernel
6.1.0-56(linux 6.1.192-2) inbookworm-security. Thelinux-image-amd64metapackage already installs it. - If a server is running
-55right now, don't runapton it: installing packages triggers the freeze by itself. Boot into-54first. - Roll out updates, kernels especially, in waves: staging and a couple of servers first, the rest a day later.
What broke
On November 15, Debian LTS published DLA-4906-1, a linux 6.1.192-1 kernel update for Debian 12 with security fixes. On the system, this is the linux-image-6.1.0-55-amd64 package.
In this kernel, ext4 dereferences a null pointer in ext4_mb_load_buddy_gfp (#1154101, severity critical; a similar report is #1154153). The crashing task holds a jbd2 journal handle. The journal commit waits for it forever, and every process that writes to the root filesystem hangs in the D state.
Meanwhile, the server looks alive. The network works, ping replies, and services that are already running keep serving requests as long as they don't need to write to disk. Everything that writes stalls: logs, databases, package installation.
The fix is linux 6.1.192-2, the linux-image-6.1.0-56-amd64 package in bookworm-security. The bug was closed in the Debian tracker on November 16, and the linux-image-amd64 metapackage already points to -56. The -55 package hasn't been removed from the repository: if you install it explicitly by name, it will install.
What happened on our servers
On our infrastructure servers and staging, unattended-upgrades installs updates with automatic reboots (Automatic-Reboot): on some, right after the update; on others, at night on a schedule. On November 16–17, it installed 6.1.0-55 and rebooted the servers.
All ten servers that booted into -55 froze: infrastructure machines, part of staging, and CI runners. They included both virtual machines and bare metal, so both are affected. One runner froze three times on November 16, the rest 2–30 hours after boot. Servers that got the update later received -56 right away and weren't affected.
Check your servers
uname -r # which kernel is running
dpkg -l 'linux-image-6.1.0-*' | grep '^ii' # which kernels are installed
-
6.1.0-55-amd64is running. The server can freeze at any moment. Don't install packages on it; go straight to If the server is already on -55. -
-55is installed, but a different kernel is running.-55will boot on the next reboot. Install the fix and remove-55before rebooting:apt updateapt install linux-image-amd64apt purge linux-image-6.1.0-55-amd64 -
-55isn't installed. A regular update will install-56directly.
If the server is already on -55
Don't install packages on -55: apt rebuilds the initramfs, writes to disk, and triggers the freeze. That's how two of our servers froze a second time. This order worked for us:
-
If the server is frozen, hard-reset it through your provider's panel or API. A regular
rebootcan hang too. -
Right after boot, while the server still responds, select the old kernel for one boot and reboot. Don't write anything else to disk:
grub-reboot 'Advanced options for Debian GNU/Linux>Debian GNU/Linux, with Linux 6.1.0-54-amd64'grub-editenv list # should show next_entry=Advanced options…systemctl rebootMenu entry names depend on
GRUB_DISTRIBUTOR; you can list yours withgrep -E "menuentry |submenu " /boot/grub/grub.cfg. You don't need to changeGRUB_DEFAULT: booting into-54is a one-time thing. -
On
-54, install the fixed kernel, remove-55, and reboot. GRUB will boot the new kernel by default:apt updateapt install linux-image-amd64apt purge linux-image-6.1.0-55-amd64systemctl reboot -
Check
uname -r: it should be6.1.0-56-amd64.
If the server freezes before you can log in over SSH, your provider's console (VNC) is the only option: in the GRUB menu, select "Advanced options for Debian GNU/Linux" and kernel 6.1.0-54. We didn't try to save two such servers and reinstalled them with Debian 13.
Kernel 6.12 or Debian 13
bookworm-security also has a kernel from the 6.12 branch, the same one as in Debian 13: linux-image-6.12-amd64 (6.12.117-1~deb12u1). Follow the same steps, just with linux-image-6.12-amd64 instead of linux-image-amd64:
apt install linux-image-6.12-amd64
We installed 6.12 on our Debian 12 servers. The ones we repaired have already booted into it, and the rest will switch at their next planned reboot. We haven't seen any problems so far, but our servers have been on 6.12 for less than two days, so it's too early to talk about stability.
Debian 12 is already in LTS. If your servers are due to move to Debian 13 anyway, this is a reason not to put it off.
Roll out updates in waves
The kernel bug is only half the problem. The worse part is that the buggy kernel reached a dozen servers within a day and rebooted them by itself. Automatic-Reboot is convenient as long as updates are safe. When they aren't, a single bug becomes an outage of the whole fleet.
- Staging and a couple of servers first, then the rest. Wait at least a day between stages: the
-55bug showed up 2–30 hours after boot, so a couple of hours wouldn't have caught it. - Don't reboot the whole fleet at once. Use a different
Automatic-Reboot-Timefor different groups of servers, or reboot manually after checking on staging. - Track which kernel is running. node_exporter exports the
node_uname_infometric with areleaselabel: it shows where a new kernel has booted, and you can alert on a specific version.
We weren't the only ones with update-related outages this week. On November 17, some servers at Timeweb Cloud, a Russian cloud provider, lost network connectivity for almost two hours (every location except Novosibirsk was affected): during an update of the hypervisors' network stack, the dynamic routing service didn't fully restart. The same day, iCloud, the App Store, and Apple Pay were down, and Apple named an internal update as the cause. The causes are different and unrelated to the Debian kernel, but all three outages started with an update. After its review, Timeweb promised to increase the intervals between update stages.
Leave feedback
If you enjoy using Tuna, or on the contrary you are not happy with something, please leave feedback.
News
We announce new features and posts on our Telegram channel. Subscribe so you don't miss anything.
Help
We value our users and carefully review every request. If you have any problems with tuna, please contact us in one of the following ways: