Compare commits
No commits in common. "598e1ef65b054809c00bccc05bda99d5b73e04ef" and "756abe99a87bf8669b2fb457e561c6b86f2ee590" have entirely different histories.
598e1ef65b
...
756abe99a8
1
.gitignore
vendored
1
.gitignore
vendored
@ -6,4 +6,3 @@ __pycache__/
|
|||||||
scratchpad/
|
scratchpad/
|
||||||
/PAUSED
|
/PAUSED
|
||||||
/.env
|
/.env
|
||||||
.DS_Store
|
|
||||||
|
|||||||
@ -141,7 +141,7 @@ PRESETS = {
|
|||||||
# ],
|
# ],
|
||||||
"q4h": [
|
"q4h": [
|
||||||
"login", "cafe", "event_sweep", "mailbox", "stamina",
|
"login", "cafe", "event_sweep", "mailbox", "stamina",
|
||||||
"event_sweep", "story_sweep", "story_sweep_hard", "exit_game"
|
"event_sweep", "exit_game"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -1,99 +0,0 @@
|
|||||||
# Incident Report: nik-gpu — Total Network Loss After Kernel Update
|
|
||||||
|
|
||||||
**Host:** `nik-gpu` (Ryzen 9 3900X / RTX 2080 SUPER / Ubuntu 24.04)
|
|
||||||
**Date:** September 24, 2026
|
|
||||||
**Duration:** ~00:45–01:25 (≈40 minutes)
|
|
||||||
**Severity:** Full network loss (wired + WiFi) — host completely offline
|
|
||||||
**Status:** Resolved
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Summary
|
|
||||||
|
|
||||||
Both network interfaces on `nik-gpu` — the onboard Realtek RTL8111/8168/8211/8411
|
|
||||||
Gigabit Ethernet and the Intel Wi-Fi 6 AX200 — silently lost their kernel drivers
|
|
||||||
after a kernel update to `6.8.0-142-generic`. The box appeared to have no NIC at
|
|
||||||
all. Root cause: the `linux-modules-extra-6.8.0-142-generic` package, which
|
|
||||||
contains the Realtek `r8169` driver, was never installed for that kernel version —
|
|
||||||
almost certainly due to an interrupted or incomplete `apt` upgrade. It was not a
|
|
||||||
cable, DNS, or hardware fault.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Timeline
|
|
||||||
|
|
||||||
| Time (JST) | Event |
|
|
||||||
|---|---|
|
|
||||||
| 00:45 | `curl` fails to resolve *any* host, including `google.com` and `ipconfig.me` — looked like a DNS issue at first. |
|
|
||||||
| 00:46 | `ip a` / `ip route` reveal the real problem: only `lo` and `docker0` exist. **No physical NIC is present at all** — not a DNS issue. |
|
|
||||||
| 00:54 | `lspci` confirms both NICs are physically detected by the kernel (Realtek RTL8111 + Intel AX200), but `lshw -C network` shows both as **UNCLAIMED** — no driver bound. `lsmod \| grep r8169` returns nothing; `sudo modprobe r8169` fails: `FATAL: Module r8169 not found in directory /lib/modules/6.8.0-142-generic`. |
|
|
||||||
| ~00:55 | `/lib/modules/6.8.0-142-generic/kernel/drivers/net/ethernet/realtek/` contains only the legacy `8139cp`/`8139too` modules — no `r8169`. `du -sh` on the whole modules directory shows only **57M** (abnormally small; a healthy one is hundreds of MB). Multiple older kernels (117 through 139) are still installed and available as fallbacks. |
|
|
||||||
| 00:59–01:06 | Attempt to reboot into the older, known-good `6.8.0-139-generic` kernel. First reboot lands in a bare `grub>` rescue shell instead of the graphical boot menu (GRUB failed to load its menu automatically). A second attempt to boot the default kernel from here lands back on `142`, not `139`. |
|
|
||||||
| ~01:06 | Switch tactics: use `sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-139-generic"` to force the *next* boot only to `139`, sidestepping the unreliable GRUB menu entirely. |
|
|
||||||
| 01:07 | Reboot succeeds — `uname -r` confirms `6.8.0-139-generic`. `r8169` is loaded, `enp9s0` is up with a DHCP lease (`10.10.40.12/24`), and even k3s networking (`flannel.1`, `cni0`) is back. **Network restored via fallback kernel.** |
|
|
||||||
| 01:12–01:13 | With network access back, attempt to repair `142`: `sudo apt-get install --reinstall linux-modules-6.8.0-142-generic` initially fails on a syntax error (`--reinstall` placed before `install`). Corrected command runs, `apt` reports "already the newest version," and a subsequent proper `install --reinstall` run completes cleanly — but `/lib/modules/6.8.0-142-generic` is still stubbornly **57M**. The `linux-modules` package itself was never actually broken. |
|
|
||||||
| 01:18 | Root cause identified: `dpkg -l \| grep 6.8.0-142` shows `linux-image`, `linux-modules`, and both `nvidia-595-open` module packages installed for `142` — but **`linux-modules-extra-6.8.0-142-generic` is missing entirely**. That package (not `linux-modules`) is what actually ships `r8169` and most other hardware drivers. Installing it (`sudo apt-get install linux-modules-extra-6.8.0-142-generic`) brings the modules directory to **180M**, and `r8169.ko.zst` now appears in the Realtek driver folder. |
|
|
||||||
| 01:25 | Reboot back into `142` (now the default). `uname -r` confirms `6.8.0-142-generic`, `r8169` is loaded, `enp9s0` has its DHCP lease, and `curl ifconfig.me` returns a real public IP. **Fully resolved.** |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Root Cause
|
|
||||||
|
|
||||||
The `142` kernel's `linux-image` and `linux-modules` (core) packages installed
|
|
||||||
correctly, but **`linux-modules-extra-6.8.0-142-generic` was never installed**.
|
|
||||||
Ubuntu splits kernel modules into two packages:
|
|
||||||
|
|
||||||
- `linux-modules-<ver>` — a small core set (~57M); does **not** include most
|
|
||||||
hardware drivers.
|
|
||||||
- `linux-modules-extra-<ver>` — the much larger package (~180M+ here) containing
|
|
||||||
most out-of-tree/hardware drivers, **including Realtek's `r8169`** used by this
|
|
||||||
board's onboard ethernet.
|
|
||||||
|
|
||||||
Because `linux-modules-extra` was absent for `142`, the kernel had no driver to
|
|
||||||
bind to either NIC — both showed as `UNCLAIMED` in `lshw`, and neither appeared
|
|
||||||
in `ip a` at all. This produced misleading symptoms early on (`curl` DNS failures
|
|
||||||
looked like a DNS problem, then like a possible cable/hardware problem) before the
|
|
||||||
real, narrower cause was isolated.
|
|
||||||
|
|
||||||
The most likely trigger is an interrupted or incomplete `apt` upgrade at some
|
|
||||||
point when `142` was installed (e.g., a closed SSH session or power interruption
|
|
||||||
mid-upgrade), which let the smaller `linux-modules` package complete but not the
|
|
||||||
much larger `linux-modules-extra` package.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Fix Applied
|
|
||||||
|
|
||||||
```bash
|
|
||||||
sudo apt-get install linux-modules-extra-6.8.0-142-generic
|
|
||||||
sudo reboot
|
|
||||||
```
|
|
||||||
|
|
||||||
Confirmed post-fix:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uname -r # 6.8.0-142-generic
|
|
||||||
lsmod | grep r8169 # r8169 loaded
|
|
||||||
ip a # enp9s0 UP with DHCP lease
|
|
||||||
curl ifconfig.me # returns public IP — full connectivity confirmed
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Notes / Follow-ups
|
|
||||||
|
|
||||||
- **Not a cable or DNS issue**, despite the initial symptoms pointing that way.
|
|
||||||
The `curl` DNS failures were a downstream effect of having no network interface
|
|
||||||
at all, not a resolver problem.
|
|
||||||
- **`grub-reboot` proved more reliable than catching the GRUB menu** — the
|
|
||||||
graphical/text boot menu didn't appear reliably on this box during this
|
|
||||||
incident (landed in a bare `grub>` rescue shell once). Setting a one-time next-
|
|
||||||
boot target via `grub-reboot "<menu path>"` avoided fighting menu timing
|
|
||||||
entirely and is worth defaulting to for any future kernel-fallback situations
|
|
||||||
on this host.
|
|
||||||
- **Worth spot-checking other kernel versions on this box** (and other machines
|
|
||||||
with a similar setup) for the same gap — run
|
|
||||||
`dpkg -l | grep linux-modules-extra` alongside `dpkg -l | grep linux-image` to
|
|
||||||
confirm every installed kernel has its matching `-extra` package, since this
|
|
||||||
class of partial-upgrade gap won't surface again until the next `apt upgrade`
|
|
||||||
installs a new kernel and this one repeats.
|
|
||||||
Loading…
x
Reference in New Issue
Block a user