ba-auto-daily/docs/incident/nik-gpu-ethernet-incident-report.md
Nik Afiq 598e1ef65b docs: add nik-gpu ethernet incident report (2026-09-24)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 20:56:10 +09:00

6.2 KiB
Raw Blame History

Incident Report: nik-gpu — Total Network Loss After Kernel Update

Host: nik-gpu (Ryzen 9 3900X / RTX 2080 SUPER / Ubuntu 24.04) Date: September 24, 2026 Duration: ~00:45–01:25 (≈40 minutes) Severity: Full network loss (wired + WiFi) — host completely offline Status: Resolved


Summary

Both network interfaces on nik-gpu — the onboard Realtek RTL8111/8168/8211/8411 Gigabit Ethernet and the Intel Wi-Fi 6 AX200 — silently lost their kernel drivers after a kernel update to 6.8.0-142-generic. The box appeared to have no NIC at all. Root cause: the linux-modules-extra-6.8.0-142-generic package, which contains the Realtek r8169 driver, was never installed for that kernel version — almost certainly due to an interrupted or incomplete apt upgrade. It was not a cable, DNS, or hardware fault.


Timeline

Time (JST) Event
00:45 curl fails to resolve any host, including google.com and ipconfig.me — looked like a DNS issue at first.
00:46 ip a / ip route reveal the real problem: only lo and docker0 exist. No physical NIC is present at all — not a DNS issue.
00:54 lspci confirms both NICs are physically detected by the kernel (Realtek RTL8111 + Intel AX200), but lshw -C network shows both as UNCLAIMED — no driver bound. lsmod | grep r8169 returns nothing; sudo modprobe r8169 fails: FATAL: Module r8169 not found in directory /lib/modules/6.8.0-142-generic.
~00:55 /lib/modules/6.8.0-142-generic/kernel/drivers/net/ethernet/realtek/ contains only the legacy 8139cp/8139too modules — no r8169. du -sh on the whole modules directory shows only 57M (abnormally small; a healthy one is hundreds of MB). Multiple older kernels (117 through 139) are still installed and available as fallbacks.
00:59–01:06 Attempt to reboot into the older, known-good 6.8.0-139-generic kernel. First reboot lands in a bare grub> rescue shell instead of the graphical boot menu (GRUB failed to load its menu automatically). A second attempt to boot the default kernel from here lands back on 142, not 139.
~01:06 Switch tactics: use sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-139-generic" to force the next boot only to 139, sidestepping the unreliable GRUB menu entirely.
01:07 Reboot succeeds — uname -r confirms 6.8.0-139-generic. r8169 is loaded, enp9s0 is up with a DHCP lease (10.10.40.12/24), and even k3s networking (flannel.1, cni0) is back. Network restored via fallback kernel.
01:12–01:13 With network access back, attempt to repair 142: sudo apt-get install --reinstall linux-modules-6.8.0-142-generic initially fails on a syntax error (--reinstall placed before install). Corrected command runs, apt reports "already the newest version," and a subsequent proper install --reinstall run completes cleanly — but /lib/modules/6.8.0-142-generic is still stubbornly 57M. The linux-modules package itself was never actually broken.
01:18 Root cause identified: dpkg -l | grep 6.8.0-142 shows linux-image, linux-modules, and both nvidia-595-open module packages installed for 142 — but linux-modules-extra-6.8.0-142-generic is missing entirely. That package (not linux-modules) is what actually ships r8169 and most other hardware drivers. Installing it (sudo apt-get install linux-modules-extra-6.8.0-142-generic) brings the modules directory to 180M, and r8169.ko.zst now appears in the Realtek driver folder.
01:25 Reboot back into 142 (now the default). uname -r confirms 6.8.0-142-generic, r8169 is loaded, enp9s0 has its DHCP lease, and curl ifconfig.me returns a real public IP. Fully resolved.

Root Cause

The 142 kernel's linux-image and linux-modules (core) packages installed correctly, but linux-modules-extra-6.8.0-142-generic was never installed. Ubuntu splits kernel modules into two packages:

  • linux-modules-<ver> — a small core set (~57M); does not include most hardware drivers.
  • linux-modules-extra-<ver> — the much larger package (~180M+ here) containing most out-of-tree/hardware drivers, including Realtek's r8169 used by this board's onboard ethernet.

Because linux-modules-extra was absent for 142, the kernel had no driver to bind to either NIC — both showed as UNCLAIMED in lshw, and neither appeared in ip a at all. This produced misleading symptoms early on (curl DNS failures looked like a DNS problem, then like a possible cable/hardware problem) before the real, narrower cause was isolated.

The most likely trigger is an interrupted or incomplete apt upgrade at some point when 142 was installed (e.g., a closed SSH session or power interruption mid-upgrade), which let the smaller linux-modules package complete but not the much larger linux-modules-extra package.


Fix Applied

sudo apt-get install linux-modules-extra-6.8.0-142-generic
sudo reboot

Confirmed post-fix:

uname -r            # 6.8.0-142-generic
lsmod | grep r8169   # r8169 loaded
ip a                 # enp9s0 UP with DHCP lease
curl ifconfig.me     # returns public IP — full connectivity confirmed

Notes / Follow-ups

  • Not a cable or DNS issue, despite the initial symptoms pointing that way. The curl DNS failures were a downstream effect of having no network interface at all, not a resolver problem.
  • grub-reboot proved more reliable than catching the GRUB menu — the graphical/text boot menu didn't appear reliably on this box during this incident (landed in a bare grub> rescue shell once). Setting a one-time next- boot target via grub-reboot "<menu path>" avoided fighting menu timing entirely and is worth defaulting to for any future kernel-fallback situations on this host.
  • Worth spot-checking other kernel versions on this box (and other machines with a similar setup) for the same gap — run dpkg -l | grep linux-modules-extra alongside dpkg -l | grep linux-image to confirm every installed kernel has its matching -extra package, since this class of partial-upgrade gap won't surface again until the next apt upgrade installs a new kernel and this one repeats.