ba-auto-daily/docs/incident/nik-gpu-ethernet-incident-report.md
Nik Afiq 598e1ef65b docs: add nik-gpu ethernet incident report (2026-09-24)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 20:56:10 +09:00

100 lines
6.2 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Incident Report: nik-gpu — Total Network Loss After Kernel Update
**Host:** `nik-gpu` (Ryzen 9 3900X / RTX 2080 SUPER / Ubuntu 24.04)
**Date:** September 24, 2026
**Duration:** ~00:45–01:25 (≈40 minutes)
**Severity:** Full network loss (wired + WiFi) — host completely offline
**Status:** Resolved
---
## Summary
Both network interfaces on `nik-gpu` — the onboard Realtek RTL8111/8168/8211/8411
Gigabit Ethernet and the Intel Wi-Fi 6 AX200 — silently lost their kernel drivers
after a kernel update to `6.8.0-142-generic`. The box appeared to have no NIC at
all. Root cause: the `linux-modules-extra-6.8.0-142-generic` package, which
contains the Realtek `r8169` driver, was never installed for that kernel version —
almost certainly due to an interrupted or incomplete `apt` upgrade. It was not a
cable, DNS, or hardware fault.
---
## Timeline
| Time (JST) | Event |
|---|---|
| 00:45 | `curl` fails to resolve *any* host, including `google.com` and `ipconfig.me` — looked like a DNS issue at first. |
| 00:46 | `ip a` / `ip route` reveal the real problem: only `lo` and `docker0` exist. **No physical NIC is present at all** — not a DNS issue. |
| 00:54 | `lspci` confirms both NICs are physically detected by the kernel (Realtek RTL8111 + Intel AX200), but `lshw -C network` shows both as **UNCLAIMED** — no driver bound. `lsmod \| grep r8169` returns nothing; `sudo modprobe r8169` fails: `FATAL: Module r8169 not found in directory /lib/modules/6.8.0-142-generic`. |
| ~00:55 | `/lib/modules/6.8.0-142-generic/kernel/drivers/net/ethernet/realtek/` contains only the legacy `8139cp`/`8139too` modules — no `r8169`. `du -sh` on the whole modules directory shows only **57M** (abnormally small; a healthy one is hundreds of MB). Multiple older kernels (117 through 139) are still installed and available as fallbacks. |
| 00:59–01:06 | Attempt to reboot into the older, known-good `6.8.0-139-generic` kernel. First reboot lands in a bare `grub>` rescue shell instead of the graphical boot menu (GRUB failed to load its menu automatically). A second attempt to boot the default kernel from here lands back on `142`, not `139`. |
| ~01:06 | Switch tactics: use `sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-139-generic"` to force the *next* boot only to `139`, sidestepping the unreliable GRUB menu entirely. |
| 01:07 | Reboot succeeds — `uname -r` confirms `6.8.0-139-generic`. `r8169` is loaded, `enp9s0` is up with a DHCP lease (`10.10.40.12/24`), and even k3s networking (`flannel.1`, `cni0`) is back. **Network restored via fallback kernel.** |
| 01:12–01:13 | With network access back, attempt to repair `142`: `sudo apt-get install --reinstall linux-modules-6.8.0-142-generic` initially fails on a syntax error (`--reinstall` placed before `install`). Corrected command runs, `apt` reports "already the newest version," and a subsequent proper `install --reinstall` run completes cleanly — but `/lib/modules/6.8.0-142-generic` is still stubbornly **57M**. The `linux-modules` package itself was never actually broken. |
| 01:18 | Root cause identified: `dpkg -l \| grep 6.8.0-142` shows `linux-image`, `linux-modules`, and both `nvidia-595-open` module packages installed for `142` — but **`linux-modules-extra-6.8.0-142-generic` is missing entirely**. That package (not `linux-modules`) is what actually ships `r8169` and most other hardware drivers. Installing it (`sudo apt-get install linux-modules-extra-6.8.0-142-generic`) brings the modules directory to **180M**, and `r8169.ko.zst` now appears in the Realtek driver folder. |
| 01:25 | Reboot back into `142` (now the default). `uname -r` confirms `6.8.0-142-generic`, `r8169` is loaded, `enp9s0` has its DHCP lease, and `curl ifconfig.me` returns a real public IP. **Fully resolved.** |
---
## Root Cause
The `142` kernel's `linux-image` and `linux-modules` (core) packages installed
correctly, but **`linux-modules-extra-6.8.0-142-generic` was never installed**.
Ubuntu splits kernel modules into two packages:
- `linux-modules-<ver>` — a small core set (~57M); does **not** include most
hardware drivers.
- `linux-modules-extra-<ver>` — the much larger package (~180M+ here) containing
most out-of-tree/hardware drivers, **including Realtek's `r8169`** used by this
board's onboard ethernet.
Because `linux-modules-extra` was absent for `142`, the kernel had no driver to
bind to either NIC — both showed as `UNCLAIMED` in `lshw`, and neither appeared
in `ip a` at all. This produced misleading symptoms early on (`curl` DNS failures
looked like a DNS problem, then like a possible cable/hardware problem) before the
real, narrower cause was isolated.
The most likely trigger is an interrupted or incomplete `apt` upgrade at some
point when `142` was installed (e.g., a closed SSH session or power interruption
mid-upgrade), which let the smaller `linux-modules` package complete but not the
much larger `linux-modules-extra` package.
---
## Fix Applied
```bash
sudo apt-get install linux-modules-extra-6.8.0-142-generic
sudo reboot
```
Confirmed post-fix:
```bash
uname -r # 6.8.0-142-generic
lsmod | grep r8169 # r8169 loaded
ip a # enp9s0 UP with DHCP lease
curl ifconfig.me # returns public IP — full connectivity confirmed
```
---
## Notes / Follow-ups
- **Not a cable or DNS issue**, despite the initial symptoms pointing that way.
The `curl` DNS failures were a downstream effect of having no network interface
at all, not a resolver problem.
- **`grub-reboot` proved more reliable than catching the GRUB menu** — the
graphical/text boot menu didn't appear reliably on this box during this
incident (landed in a bare `grub>` rescue shell once). Setting a one-time next-
boot target via `grub-reboot "<menu path>"` avoided fighting menu timing
entirely and is worth defaulting to for any future kernel-fallback situations
on this host.
- **Worth spot-checking other kernel versions on this box** (and other machines
with a similar setup) for the same gap — run
`dpkg -l | grep linux-modules-extra` alongside `dpkg -l | grep linux-image` to
confirm every installed kernel has its matching `-extra` package, since this
class of partial-upgrade gap won't surface again until the next `apt upgrade`
installs a new kernel and this one repeats.