fix: make pia-wg PreDown idempotent to prevent restart failures
Some checks failed
validate / lint (push) Failing after 1s
Some checks failed
validate / lint (push) Failing after 1s
Enabling MSS clamping exposed a general hazard: systemctl restart wg-quick@pia-wg tears down the live interface using whatever PreDown rules are on disk right now, not what the interface was actually brought up with. If Ansible had already rewritten pia-wg.conf with a changed rule, the live interface (built under the old rules) won't have whatever the new PreDown line tries to delete - iptables -D fails, wg-quick's own set -e aborts the whole down/up sequence, and the interface is left orphaned (never reaches its own built-in ip link delete step), which then makes the following wg-quick up fail too with "already exists". Confirmed against wg-quick's real source (execute_hooks() runs each hook via `(eval "$hook")`) that appending `2>/dev/null || true` to each PreDown iptables -D line makes wg-quick treat it as succeeded regardless of prior state - verified by rendering the template with Jinja2 directly before trusting it against the live host. This prevents the failure mode entirely for any future PostUp/PreDown content change, not just this one. Add INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md covering all four issues from this troubleshooting window: the zombie WireGuard tunnel, a qBittorrent 5.2.0 stale-lock crash loop, the PMTU black hole this fix addresses, and the restart failure hit while deploying it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
8629a36278
commit
4d613c9c06
244
INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md
Normal file
244
INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md
Normal file
@ -0,0 +1,244 @@
|
||||
# Incident report: VLAN 50 egress outage, qBittorrent crash loop, and a PMTU black hole
|
||||
|
||||
**Date:** 2026-08-25 to 2026-08-27 (discovered/resolved across this window; underlying triggers occurred 2026-08-24/25)
|
||||
**Systems:** `minisforum` (PIA WireGuard gateway), `nik-debian` (VLAN 50 workloads: `browser-vpn-proxy`, `qbittorrent`, `jdownloader`)
|
||||
**Status:** All four issues resolved live, with code fixes committed for three of them (qBittorrent, MSS clamping, `wg-quick` restart idempotency). The remaining gap (PIA tunnel handshake-death alerting/remediation) is documented with prevention options, deliberately not yet implemented — see "Open follow-up" below.
|
||||
|
||||
## Summary
|
||||
|
||||
Four **independent** problems, found across the same troubleshooting session:
|
||||
|
||||
1. **minisforum's PIA WireGuard tunnel (`pia-wg`) silently died** — the interface, systemd unit, and routes all stayed "up" and looked healthy, but the encrypted session with PIA's server had stopped working hours earlier. This black-holed all VLAN 50 egress (`browser-vpn-proxy`, `qbittorrent`, `jdownloader`) — safely (no leak to `eth0`), but silently (nothing paged anyone).
|
||||
2. **qBittorrent (5.2.0) got stuck in an internal crash loop** on `nik-debian`, unrelated to the tunnel issue and pre-dating it. A known upstream bug in qBittorrent's single-instance lock file handling caused it to restart every ~1-2 seconds, forever, without ever opening its WebUI.
|
||||
3. **A pre-existing PMTU black hole** between `pia-wg` (MTU `1420`) and the rest of the path caused real downloads to hang indefinitely once the tunnel itself was working again — small requests succeeded, but any connection needing a larger response (like a real download's TLS handshake) silently never got one. This was a known, anticipated gap (`pia_mss_clamp_enabled` existed in the role from the start, deliberately left off pending real evidence) — now confirmed and fixed.
|
||||
4. **Deploying the fix for #3 broke `wg-quick@pia-wg`'s restart**, because tearing down the live interface with newly-changed `PreDown` rules it never actually had installed made the whole teardown abort partway, orphaning the interface. Fixed at the root by making `PreDown` idempotent — a general hazard for any future config change, not specific to MSS clamping.
|
||||
|
||||
All four were found during the same troubleshooting session because a user report ("the proxy isn't working after I rebooted nik-debian") triggered an investigation broad enough to surface all of them in turn — #3 only became visible *after* #1 was fixed (a dead tunnel has no traffic to black-hole), and #4 only surfaced while actually deploying #3's fix. **None of the four share a root cause** — see "Were these related?" below.
|
||||
|
||||
## Timeline
|
||||
|
||||
All timestamps as observed from their own source; see the timezone caveat below before doing precise cross-host arithmetic.
|
||||
|
||||
| When (as logged) | Source | Event |
|
||||
| --- | --- | --- |
|
||||
| 2026-08-24 23:07:21Z | `nik-debian` node `Ready` condition | `nik-debian` rejoins the cluster after a reboot |
|
||||
| 2026-08-24 23:31:28 (pod-local clock) | `browser-vpn-proxy` container log | Fresh pod starts, SOCKS5 proxy begins accepting connections |
|
||||
| 2026-08-24 23:32:08 (pod-local clock) | `qbittorrent`/`jdownloader` pod logs | Both pods' init containers complete, main containers start |
|
||||
| ~2026-08-25 08:35 (pod-local clock, per file mtimes) | qBittorrent's own log-rotation file timestamps | qBittorrent's internal crash loop begins (first evidence of rapid log rotation) — **this predates the tunnel's death below** |
|
||||
| ~2026-08-25 13:15 JST (back-calculated from the healthcheck's own handshake-age readings) | minisforum, `pia-gateway-healthcheck` | `pia-wg`'s WireGuard session actually goes dead (last real handshake) |
|
||||
| 2026-08-25 11:04-11:05 (pod-local clock) | `browser-vpn-proxy` container log | First visible symptoms: `broken pipe`, then `connection timed out` reaching an external site |
|
||||
| 2026-08-25 20:42-20:46 JST | minisforum, `journalctl -u pia-gateway-healthcheck` | Healthcheck has been correctly detecting `FAIL handshake: ...s old, exceeds max 180s` every 60s the entire time — just never surfaced to a human |
|
||||
| 2026-08-25 (user report) | User | Reports proxy not working after a reboot of `nik-debian` |
|
||||
| 2026-08-25, live troubleshooting | This session | Both root causes diagnosed and fixed (see below) |
|
||||
|
||||
**Timezone caveat:** minisforum's `systemctl`/`journalctl` output explicitly shows `JST`. The container/pod log timestamps pasted during this investigation did not carry an explicit timezone marker, so their exact offset from minisforum's JST clock isn't confirmed here — treat cross-host time deltas in this report as approximate, not to-the-second precise. The *relative* ordering of events on the same host (e.g. "qBittorrent's loop started before the tunnel died") is solid, since the reasoning for that only compares timestamps that share a clock.
|
||||
|
||||
## Were these related?
|
||||
|
||||
No — evidence points at three independent causes, not a shared trigger:
|
||||
|
||||
- qBittorrent's crash loop started (~08:35, pod-local clock) well before the PIA tunnel actually died (~13:15 JST, minisforum's clock) and well before egress failures were even visible (~11:04, pod-local clock, on a *different* pod).
|
||||
- The qBittorrent crash loop's own root cause (a stale single-instance lock file) has nothing to do with WireGuard, PIA, or networking at all.
|
||||
- The PMTU black hole (Incident 3) is a pre-existing structural gap in the tunnel's own MTU handling — not something the reboot, the tunnel death, or qBittorrent's bug caused. It was simply **not visible** while the tunnel was dead (Incident 1) — no working tunnel meant no traffic to black-hole — and only surfaced once Incident 1 was fixed and something tried to move a large enough response across the path again.
|
||||
- All three problems happened to be sitting there, undetected, until a user report about a fourth, unrelated-seeming thing (`browser-vpn-proxy` after a reboot) triggered an investigation broad enough to notice all of them in sequence.
|
||||
|
||||
The reboot of `nik-debian` itself is **not** the confirmed trigger for any of the three:
|
||||
- The PIA tunnel runs on a completely different host (minisforum), which was not rebooted.
|
||||
- qBittorrent's crash loop started roughly 9 hours *after* the reboot, not immediately following it — something else (unidentified — see below) caused qBittorrent's underlying unclean shutdown that left the stale lock file in the first place. The reboot is a plausible but unconfirmed contributor; it was not caught in the act.
|
||||
- The PMTU black hole is a property of `pia-wg`'s own fixed `mtu 1420` versus the rest of the path — a static configuration characteristic, not something a reboot would change.
|
||||
|
||||
## Incident 1: PIA WireGuard tunnel silently died ("zombie tunnel")
|
||||
|
||||
### Symptom
|
||||
|
||||
All three VLAN 50 workloads lost egress simultaneously. `qbittorrent`'s and `jdownloader`'s sidecar containers failed with `OSError: [Errno 101] Network unreachable` — an immediate "no route" response, not a timeout. `browser-vpn-proxy` showed the same underlying cause but surfaced as a plain timeout from outside the cluster.
|
||||
|
||||
### Root cause
|
||||
|
||||
`wg-quick@pia-wg.service` was `active (exited)` (this is *normal* for `wg-quick` — it's a one-shot config script, not a long-running daemon) and the `pia-wg` interface existed with its usual `PostUp`-installed routes and iptables rules. But `sudo wg show pia-wg` showed:
|
||||
|
||||
```
|
||||
latest handshake: 7 hours, 29 minutes, 29 seconds ago
|
||||
persistent keepalive: every 25 seconds
|
||||
```
|
||||
|
||||
WireGuard's own protocol re-handshakes roughly every 2 minutes under normal operation, and a 25-second keepalive is configured specifically to keep that happening even with no application traffic. A handshake this stale means the actual encrypted session with PIA's server (`69.33.198.63:1337`) had been dead for hours — the local interface, routes, and firewall rules `wg-quick` set up were all still present and "correct," but nothing was actually getting through. `ip route show table pia` confirmed the route was the normal *working* route (`default dev pia-wg scope link`), **not** the kill switch's `unreachable` route — meaning the kill switch's own "interface down → block" mechanism never triggered, because the interface never actually went down. It just stopped working while looking fine.
|
||||
|
||||
**What actually broke the session was not conclusively identified.** `ping -c3 69.33.198.63` succeeded cleanly (0% loss, ~70ms) from minisforum's normal path, ruling out "minisforum's WAN is broken" or "that IP is completely unreachable." The most likely explanations, in rough order of plausibility, none confirmed:
|
||||
1. **PIA's registered-key expiry.** A plain `sudo systemctl restart wg-quick@pia-wg` (which only recreates the local interface with the *same* key) did **not** produce a new handshake even after 10+ seconds — strongly suggesting PIA's server no longer recognized this specific registered public key. Re-registering from scratch (`-e pia_force_reregister=true`, generating a fresh keypair and calling PIA's `addKey` API again) *did* restore a working tunnel immediately. This is the strongest evidence: a fresh registration worked where a bare restart did not.
|
||||
2. A server-side outage specific to that one PIA edge server (`69.33.198.63`) — plausible, but the successful fresh re-registration landed on a *different* exit IP in the same Hong Kong region, so this wasn't independently distinguished from (1).
|
||||
3. A transient NAT/UDP path issue on minisforum's own WAN — considered less likely given `ping` (a different protocol, admittedly) worked fine, and a restart-without-reregistration should have been enough to recover from a purely local NAT hiccup, but wasn't.
|
||||
|
||||
### How to reproduce
|
||||
|
||||
Not independently reproduced on demand — this was diagnosed from a live occurrence, not triggered experimentally. The **diagnostic signature**, if it happens again, is:
|
||||
```bash
|
||||
sudo wg show pia-wg # interface "looks" fine, but check "latest handshake"
|
||||
ip route show table pia # shows the *working* route, not "unreachable" — the kill switch did NOT engage
|
||||
sudo systemctl restart wg-quick@pia-wg
|
||||
sudo wg show pia-wg latest-handshakes # if this stays at 0 (never) for more than ~15-20s after a restart, a bare restart isn't enough — re-registration is likely needed
|
||||
```
|
||||
|
||||
### How it was fixed (this occurrence)
|
||||
|
||||
```bash
|
||||
ansible-playbook -i ansible/inventory.yaml \
|
||||
ansible/playbooks/pia-gateway.yaml -K -J \
|
||||
-e pia_force_reregister=true
|
||||
```
|
||||
|
||||
This hit a real, separate, now-fixed bug on the way: `ansible/roles/pia-gateway/tasks/register.yaml`'s `when:` conditions used `pia_force_reregister`/`pia_force_key_rotation` directly in boolean expressions. Passed via the CLI's plain `key=value` form (`-e pia_force_reregister=true`, exactly what the role's own README documents), Ansible sets the variable as the **string** `"true"`, not a real boolean — and recent `ansible-core` rejects using a string directly in a `when:` boolean expression (`Conditional result (True) was derived from value of type 'str'`). Fixed by wrapping every use in `| bool` (commit `e757850`), which correctly coerces both real booleans (the `defaults/main.yaml` case) and `"true"`/`"false"` strings (the CLI-override case). Verified with an isolated local test playbook exercising both cases before trusting it against the live host.
|
||||
|
||||
### Known gap / prevention (not yet implemented — deferred by request)
|
||||
|
||||
`ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2` (run via `pia-gateway-healthcheck.timer`, every 60s) **already correctly detected this failure the entire time** — every single run logged `FAIL handshake: ...s old, exceeds max 180s`. The script is explicitly designed as "observability only, no remediation" (its own header comment says so), which is a deliberate prior design choice, not an oversight. The actual gap is that nothing consumes that detection — no alert, no dashboard, nothing — so a real, correctly-detected outage sat unnoticed until a user happened to test the affected service directly.
|
||||
|
||||
Options, not yet decided:
|
||||
|
||||
1. **Alert only (recommended by this session, not yet chosen by the user):** keep the "no remediation" design, but have the healthcheck (or a small wrapper) notify a human after a few consecutive failures. Two ways to actually send it:
|
||||
- Via `alert-bridge.home.arpa`'s `POST /alerts` (already exists, already designed for exactly this — a per-caller bearer token instead of distributing the raw Discord webhook secret to every host). Its exact request schema lives in the private `gitea.nik4nao.com/nik/alert-bridge` source, not visible from outside — confirm it before wiring anything.
|
||||
- A direct `curl` to the Discord webhook URL from minisforum. Simpler, no schema to reverse-engineer, but means minisforum needs its own copy of the raw webhook secret rather than a scoped token — a step away from the existing pattern.
|
||||
2. **Auto-restart on sustained failure:** after N consecutive failed checks, automatically `systemctl restart wg-quick@pia-wg`. Fixes it without a human noticing, but is a real architecture change to a script deliberately built to never remediate — and masks a recurring problem instead of surfacing it, if the underlying cause (e.g. PIA-side key expiry) keeps happening.
|
||||
3. **Both:** attempt auto-restart, but still alert either way, so there's always a record even when unattended recovery works.
|
||||
4. Given evidence pointed at *registration* expiry rather than a simple interface restart being sufficient, any remediation path should probably escalate to `pia_force_reregister=true`-equivalent logic, not just a bare `wg-quick` restart, if a restart-only attempt doesn't restore a fresh handshake within a short grace period.
|
||||
|
||||
## Incident 2: qBittorrent 5.2.0 stale-lock crash loop
|
||||
|
||||
### Symptom
|
||||
|
||||
`qbittorrent` pod showed `1/2 Ready` — the `qbittorrent` container was `Running` (no restarts, no crash reported at the Kubernetes level) but its readiness probe failed continuously: `dial tcp <pod-ip>:8080: connect: connection refused`. This reproduced identically on a **freshly recreated pod** (2.5 minutes old), ruling out anything specific to the original (12-hour-old, post-reboot) pod instance.
|
||||
|
||||
### Root cause
|
||||
|
||||
`kubectl exec ... ps aux` showed `qbittorrent-nox` genuinely running, consuming ~80% CPU. Its own log file (`/config/qBittorrent/logs/qbittorrent.log` — the container's stdout only ever showed the linuxserver.io init banner, qBittorrent logs to a file, not stdout) showed:
|
||||
|
||||
```
|
||||
(N) 2026-08-25T21:08:00 - qBittorrent termination initiated
|
||||
(N) 2026-08-25T21:08:00 - qBittorrent is now ready to exit
|
||||
(N) 2026-08-25T21:08:01 - qBittorrent v5.2.0 started. Process ID: 1376
|
||||
(N) 2026-08-25T21:08:01 - Using config directory: /config/qBittorrent
|
||||
(N) 2026-08-25T21:08:01 - qBittorrent termination initiated
|
||||
(N) 2026-08-25T21:08:01 - qBittorrent is now ready to exit
|
||||
(N) 2026-08-25T21:08:03 - qBittorrent v5.2.0 started. Process ID: 1384
|
||||
...
|
||||
```
|
||||
|
||||
A self-perpetuating crash loop, cycling every 1-2 seconds (PID incrementing by 8 each time — matching the container's process-spawning pattern), with the process **voluntarily** exiting immediately after every start (no crash, no signal — it logs its own "termination initiated"). The `/config/qBittorrent/` directory contained a directory's worth of hundreds of rotated log files (`qbittorrent.log.bak1` through `.bak414`+), each ~66-67KB, many piling up within minutes of each other — direct evidence of sustained rapid restart cycling, not a one-off.
|
||||
|
||||
This exactly matches a known, confirmed upstream bug: **[qbittorrent/qBittorrent#24164](https://github.com/qbittorrent/qBittorrent/issues/24164)** (and its duplicate, #24185). qBittorrent 5.2.0 switched its single-instance lock from the old fcntl-based `QtLockedFile` (a 0-byte lock file) to Qt's `QLockFile`, which expects the lock file to contain a PID + hostname. If the *previous* shutdown wasn't clean and left a lock file the new version can't parse/verify, 5.2.0 assumes another instance holds it, tries to hand off over the (also stale) `ipc-socket`, that fails too, and it exits — then immediately restarts (supervised by the container's own `s6` process manager) and repeats forever.
|
||||
|
||||
**What triggered the original unclean shutdown is not confirmed.** The lock/IPC files were both dated `2026-08-25 08:05` (pod-local clock), about 30 minutes before the crash-loop pattern is first visible in the rotated-log timestamps (`08:35`) — consistent with *something* restarting the qBittorrent process around `08:05` without a clean shutdown, but what that something was (a Kubernetes-level event, an OOM condition, a manual action, something else) wasn't caught in the act and isn't identified here.
|
||||
|
||||
One structural factor worth naming even though it's not fully confirmed: qBittorrent's lock file embeds a **hostname**, and every Kubernetes pod restart gets a **new** hostname (the pod name). That's exactly the kind of environment where a lock's embedded hostname can't reliably be trusted to "match" across restarts — plausibly why this surfaces more readily here than it would on a stable bare-metal or VM host with a fixed hostname, though this wasn't independently isolated as *the* determining factor versus the unclean-shutdown precondition alone.
|
||||
|
||||
### How to reproduce
|
||||
|
||||
Not reproduced from a clean slate in this session — confirmed via matching log signatures against the upstream issue, not an independent from-scratch trigger. Based on the upstream report, the reproduction conditions are: qBittorrent 5.2.0 starts with a `lockfile`/`ipc-socket` already present in its config directory from a *previous*, unclean shutdown (e.g. the container was SIGKILLed rather than given a chance to shut down gracefully). The diagnostic signature if it happens again:
|
||||
```bash
|
||||
kubectl -n downloads logs <qbittorrent-pod> -c qbittorrent -c ip-reporter --tail=5 # stdout goes quiet right after the LSIO init banner
|
||||
kubectl -n downloads exec <qbittorrent-pod> -c qbittorrent -- ps aux # qbittorrent-nox IS running, high CPU, PID climbing on repeat checks
|
||||
kubectl -n downloads exec <qbittorrent-pod> -c qbittorrent -- tail -n 20 /config/qBittorrent/logs/qbittorrent.log # the "started / termination initiated / ready to exit" cycle
|
||||
```
|
||||
|
||||
### How it was fixed
|
||||
|
||||
**This occurrence (live remediation):**
|
||||
```bash
|
||||
kubectl -n downloads exec <pod> -c qbittorrent -- rm -f /config/qBittorrent/lockfile /config/qBittorrent/ipc-socket
|
||||
```
|
||||
The next `s6`-supervised restart (within 1-2 seconds, given the loop's own cycle time) found no lock file, acquired a fresh one, and started normally.
|
||||
|
||||
**Prevention (implemented, committed):** added a new `qbittorrent-clear-stale-lock` init container to `manifests/media/qbittorrent.yaml` that unconditionally removes `lockfile`/`ipc-socket` before qBittorrent starts, on every pod start. This is safe specifically because this Deployment uses `strategy: Recreate` with a single replica and a `ReadWriteOnce` PVC — Kubernetes itself guarantees the previous instance is fully terminated before a new pod's init containers ever run, so there is no scenario in this specific topology where the lock could be legitimately held by a still-running peer. (This reasoning would **not** automatically transfer to a `RollingUpdate` strategy or multiple replicas — don't copy this pattern there without re-checking the concurrency assumption.)
|
||||
|
||||
### Residual risk
|
||||
|
||||
The upstream issue references a fix PR (#24218) intended to make qBittorrent itself clean up a stale lock automatically. Whether the pinned image (`lscr.io/linuxserver/qbittorrent:5.2.0`) includes that fix wasn't checked — if it doesn't, our init-container workaround remains necessary until either the image is updated past the point where the upstream fix landed, or that's independently confirmed. What caused the *original* unclean shutdown (the actual precondition for hitting this bug at all) remains unidentified — the fix here prevents the *symptom* (permanent crash loop) from persisting, but doesn't address whatever causes qBittorrent's shutdowns to be unclean in the first place, if that keeps happening.
|
||||
|
||||
## Incident 3: PMTU black hole on the `pia-wg` forward path
|
||||
|
||||
### Symptom
|
||||
|
||||
After Incident 1 was fixed (fresh PIA registration, working tunnel), JDownloader still couldn't actually download anything: a link showed status "Running" but no progress and no speed, indefinitely. The app itself was healthy, and basic connectivity worked (its own self-update fetched a small file from an external CDN without any problem).
|
||||
|
||||
### Root cause
|
||||
|
||||
JDownloader's own per-host plugin log (`subyshare.com_jd.plugins.hoster.SubyShareCom.log.0`) showed the real error: connecting to the actual download server succeeded, JDownloader sent its TLS `ClientHello`, and then nothing came back —
|
||||
|
||||
```
|
||||
javax.net.ssl.SSLException: Read timed out
|
||||
Caused by: java.net.SocketTimeoutException: Read timed out
|
||||
ConnectIP: sbs237.sbsf.tech/195.201.192.190:182
|
||||
```
|
||||
|
||||
— a 60-second read timeout waiting for the server's handshake response. Small round-trips (a version-check HTTP fetch, plain ICMP pings) worked fine throughout; only a connection needing to move a larger response hung.
|
||||
|
||||
This is the textbook signature of a path-MTU (PMTU) black hole. `ansible/roles/pia-gateway/templates/pia-wg.conf.j2` sets `pia-wg`'s interface MTU to `1420` (WireGuard's own encapsulation overhead requires a lower MTU than the outer network's `1500`) — confirmed directly, not assumed, via a real test: `ping -M do -s 1450 -c 3 1.1.1.1` from a VLAN 50 pod returned
|
||||
|
||||
```
|
||||
From 10.10.40.53 icmp_seq=1 Frag needed and DF set (mtu = 1420)
|
||||
```
|
||||
|
||||
i.e. minisforum correctly tells the sender to fragment/shrink, and the pod's own kernel correctly honors that — **our own side of the path relays PMTU discovery correctly.** That means the actual black hole is further out: somewhere between minisforum's tunnel and the real download server (PIA's own network, or the server's own path), where an equivalent "fragmentation needed" notification would need to travel much further back to the *server* (not us) and evidently isn't making it, so the server's larger response packets just silently vanish instead of ever being resent smaller.
|
||||
|
||||
This gap was **anticipated from the start**, not newly introduced: `ansible/roles/pia-gateway/defaults/main.yaml` already had a `pia_mss_clamp_enabled` flag and the corresponding `iptables -t mangle ... TCPMSS --clamp-mss-to-pmtu` rule already written in the WireGuard config template — deliberately left `false` by `plan.md`'s own explicit instruction ("do not guess this... flip only after ... MTU testing shows it's actually needed"). It most likely wasn't caught by earlier canary/qBittorrent/browser-vpn-proxy testing because those tests mostly exercised small request/response pairs (`curl ifconfig.me`, `dig`, small API calls) — this only surfaces on a connection that needs to move a larger response, like a real file download's TLS handshake.
|
||||
|
||||
### How to reproduce
|
||||
|
||||
```bash
|
||||
kubectl -n downloads exec vlan50-canary -c netshoot -- ping -M do -s 1392 -c 3 1.1.1.1 # works — under the real MTU
|
||||
kubectl -n downloads exec vlan50-canary -c netshoot -- ping -M do -s 1450 -c 3 1.1.1.1 # fails locally with "Frag needed... mtu = 1420" — this is the confirmation test, not a guess
|
||||
```
|
||||
Application-level symptom: any download/request whose response exceeds ~1420 bytes (accounting for WireGuard/IP/TCP overhead) will hang indefinitely rather than erroring cleanly, while small requests keep working — making this easy to miss if testing only checks basic connectivity.
|
||||
|
||||
### How it was fixed
|
||||
|
||||
Enabled the already-built toggle rather than writing new mechanism: `ansible/roles/pia-gateway/defaults/main.yaml`'s `pia_mss_clamp_enabled` flipped from `false` to `true`. This clamps the TCP MSS on SYN packets outbound on `pia-wg` for `10.10.50.0/24` traffic, so both ends of any new TCP connection negotiate a small-enough segment size from the start — sidestepping the need for a working PMTU-discovery ICMP round-trip entirely. Applying it needed a second, unplanned fix — see Incident 4 immediately below, discovered in the process of actually deploying this one. Confirmed working: after Incident 4's recovery, all three VLAN 50 workloads' egress checks passed and the specific JDownloader download that had been stuck actually completed.
|
||||
|
||||
### Residual risk
|
||||
|
||||
MSS clamping fixes new TCP connections going forward; it doesn't retroactively fix anything already stuck — confirmed in practice, the originally-stuck download needed a manual retry after the fix landed. The scope is intentionally narrow (`10.10.50.0/24` outbound on `pia-wg` only, SYN packets only) to match the existing kill-switch/firewall pattern of "own chain, only touch what's ours" — this should not affect any other traffic on minisforum.
|
||||
|
||||
## Incident 4: enabling MSS clamping broke `wg-quick` restart (discovered while fixing Incident 3)
|
||||
|
||||
### Symptom
|
||||
|
||||
Running the Ansible playbook to actually apply the MSS-clamping default change (no `-e` override, since the change was now a committed default) completed with `changed=0` — the config file was never rewritten at all, because `pia-wg.conf` already existed and the registration block that writes it is gated behind `pia_force_reregister or pia_force_key_rotation or not pia_wg_conf_stat.stat.exists`; a plain re-run with the file already present skips that entire block. Re-running with `-e pia_force_reregister=true` this time got further, but then failed outright:
|
||||
```
|
||||
RUNNING HANDLER [pia-gateway : Restart pia-wg]
|
||||
[ERROR]: Task failed: Module failed: Unable to restart service wg-quick@pia-wg: Job for wg-quick@pia-wg.service failed because the control process exited with error code.
|
||||
```
|
||||
|
||||
### Root cause
|
||||
|
||||
Two separate, now both understood:
|
||||
|
||||
1. **The config-write gate.** The role currently bundles "rewrite `pia-wg.conf` from the template" together with "actually re-register with PIA" under one `when:` condition — there's no path to pick up a template-only change (like flipping `pia_mss_clamp_enabled`) without forcing a full PIA re-registration too. Not fixed here; noted as a design gap (see Open follow-up).
|
||||
2. **`systemctl restart` tore down the live interface using rules it never had.** `journalctl` showed exactly why: `PreDown`'s new `iptables -t mangle -D FORWARD ... TCPMSS` line failed with `iptables: Bad rule (does a matching rule exist in that chain?)` — because the *currently running* interface had been brought up under the *old* config (before MSS clamping existed), so its `PostUp` never added that mangle rule in the first place. Confirmed against `wg-quick`'s real source (`execute_hooks()` in `wireguard-tools`'s `linux.bash`): each hook line runs via `(eval "$hook")`, and `wg-quick`'s own `set -e` aborts the entire down/up sequence on the first failing hook. Since that failing line was the *last* `PreDown` line, everything before it (the kill-switch's `unreachable` route seed, the `MASQUERADE`/`ACCEPT` rule deletes) had already run successfully — confirmed live: `ip route show table pia` showed `unreachable default` (fail-closed, not leaking) — but `wg-quick`'s own built-in `ip link delete dev pia-wg` step, which normally runs *after* all custom `PreDown` lines, never got reached. The interface was left orphaned: still present at the kernel level, but stripped of its firewall rules. The subsequent `wg-quick up` then failed too, immediately, with `` `pia-wg' already exists``, since `ip link add` can't create an interface that's still there.
|
||||
|
||||
This is a **general** hazard, not specific to MSS clamping: Ansible's `template` task overwrites `pia-wg.conf` on disk before the "Restart pia-wg" handler fires, so *any* future change to what `PostUp`/`PreDown` actually do will hit the identical failure the next time the config changes while the tunnel is already up.
|
||||
|
||||
### How to reproduce
|
||||
|
||||
Change any `PostUp`/`PreDown` rule content in `pia-wg.conf.j2` (add, remove, or reorder a rule) while `pia-wg` is currently running under the *old* version of that content, then `systemctl restart wg-quick@pia-wg` (or let Ansible's handler do it). If any new/changed `PreDown` line tries to `-D` something the live interface never actually added, the restart fails and the interface is left orphaned.
|
||||
|
||||
### How it was fixed
|
||||
|
||||
**This occurrence (live recovery):**
|
||||
```bash
|
||||
sudo ip link delete dev pia-wg
|
||||
sudo systemctl start wg-quick@pia-wg
|
||||
```
|
||||
Directly removing the orphaned interface (the exact step the aborted teardown was one line away from doing itself) let a fresh `wg-quick up` succeed cleanly, this time actually applying the new `PostUp` rules including the MSS clamp. Confirmed via `iptables -t mangle -S FORWARD` showing the rule actually installed, and a handshake timestamp matching the moment of the restart (not stale).
|
||||
|
||||
**Prevention (implemented, committed):** made every `PreDown` `iptables -D` line in `pia-wg.conf.j2` tolerate the rule it's removing not existing (`2>/dev/null || true`), verified safe against `wg-quick`'s real source (a hook line's compound exit status becomes 0, so `set -e` no longer treats "nothing to delete" as fatal) and verified by rendering the template directly with Jinja2 before trusting it. The first `PreDown` line (`ip route replace unreachable default table pia`) didn't need this — `route replace` doesn't fail due to prior state the way `-D` does. This makes `PreDown` fully idempotent, so this exact failure mode can't recur for *any* future `PostUp`/`PreDown` content change — no separate "self-heal after the fact" layer needed, since the restart now just succeeds correctly every time.
|
||||
|
||||
### Residual risk
|
||||
|
||||
None identified for this specific failure mode going forward — the fix addresses the root cause directly rather than adding a recovery layer on top of a still-fragile restart. The separate config-write gating issue (item 1 above) remains open.
|
||||
|
||||
## Open follow-up
|
||||
|
||||
- **Decide and implement PIA tunnel remediation/alerting** (Incident 1's "Known gap" section above) — explicitly deferred by the user to a later date, options documented above.
|
||||
- **Consider splitting "rewrite config from template" from "re-register with PIA" in the role** (Incident 4's root cause #1) — currently there's no way to pick up a template-only default change without forcing a full re-registration.
|
||||
- Confirm whether a newer `lscr.io/linuxserver/qbittorrent` tag already includes the upstream fix for #24164, as a potential alternative/complement to the init-container workaround.
|
||||
- If it recurs, try to catch what actually triggers qBittorrent's unclean shutdown in the first place (Incident 2's root cause section) — nothing in this investigation identified it.
|
||||
@ -171,6 +171,25 @@ real.
|
||||
side relays PMTU discovery fine, so the black hole is further out, on
|
||||
PIA's network or the remote server's own path, where clamping the MSS
|
||||
up front avoids needing that ICMP round-trip at all).
|
||||
- `pia-wg.conf`'s `PreDown` lines are deliberately tolerant of the rule
|
||||
they're removing not existing (`iptables -D ... 2>/dev/null || true`)
|
||||
— enabling MSS clamping live (above) exposed why this matters:
|
||||
`systemctl restart wg-quick@pia-wg` tears down the *currently running*
|
||||
interface using whatever `PreDown` lines are on disk *right now* — if
|
||||
Ansible already rewrote the config with a new/changed rule before the
|
||||
restart handler fires, the live interface (brought up under the *old*
|
||||
rules) won't have whatever the new `PreDown` line is trying to delete.
|
||||
`wg-quick`'s own `execute_hooks()` aborts the entire down/up sequence
|
||||
on the first failing hook (confirmed against its real source, not
|
||||
assumed), so one non-idempotent `-D` used to turn any future
|
||||
`PostUp`/`PreDown` content change into a broken restart *and* an
|
||||
orphaned interface (the never-reached built-in `ip link delete` step
|
||||
left `pia-wg` existing but unconfigured, which then made the
|
||||
following `wg-quick up` fail too with `` `pia-wg' already exists``) —
|
||||
requiring manual recovery (`ip link delete dev pia-wg` before a fresh
|
||||
`systemctl start`). Every `-D` line now tolerates this by design, so a
|
||||
config change that alters `PostUp`/`PreDown` content can never break
|
||||
a restart this way again — confirmed live, 2026-08-27.
|
||||
- Installs an observability-only health check (`pia-gateway-healthcheck
|
||||
.timer`, every `pia_healthcheck_interval_sec`) that logs interface,
|
||||
handshake age, rule/route, and firewall-policy state to the journal. It
|
||||
|
||||
@ -29,12 +29,29 @@ PostUp = iptables -t nat -A POSTROUTING -s {{ pia_vlan50_subnet }} -o %i -j MASQ
|
||||
PostUp = iptables -t mangle -A FORWARD -s {{ pia_vlan50_subnet }} -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
|
||||
{% endif %}
|
||||
|
||||
# PreDown's iptables -D lines are deliberately tolerant (2>/dev/null ||
|
||||
# true) of the rule they target not existing — confirmed against
|
||||
# wg-quick's own source (execute_hooks() runs each line via `(eval
|
||||
# "$hook")`, and wg-quick's top-level `set -e` treats that subshell's
|
||||
# exit code as fatal for the WHOLE up/down sequence): without this, a
|
||||
# single -D failing (e.g. the currently-live interface was brought up
|
||||
# under an OLDER config that never added the rule this line is now
|
||||
# trying to remove — root-caused live, 2026-08-27, when enabling MSS
|
||||
# clamping for the first time made this exact thing happen) aborts
|
||||
# every PreDown line after it, including wg-quick's own built-in `ip
|
||||
# link delete`, leaving an orphaned interface that then makes the
|
||||
# following `wg-quick up` fail too ("pia-wg already exists") — turning
|
||||
# any future PostUp/PreDown content change into a two-step manual
|
||||
# recovery (`ip link delete dev pia-wg` before a fresh start) instead
|
||||
# of a clean, unattended `systemctl restart`. `ip route replace` (the
|
||||
# first PreDown line) doesn't need this — replace never fails due to
|
||||
# prior state the way -D does.
|
||||
PreDown = ip route replace unreachable default table {{ pia_route_table_name }}
|
||||
PreDown = iptables -t nat -D POSTROUTING -s {{ pia_vlan50_subnet }} -o %i -j MASQUERADE
|
||||
PreDown = iptables -D {{ pia_iptables_chain }} -d {{ pia_vlan50_subnet }} -i %i -m state --state ESTABLISHED,RELATED -j ACCEPT
|
||||
PreDown = iptables -D {{ pia_iptables_chain }} -s {{ pia_vlan50_subnet }} -o %i -j ACCEPT
|
||||
PreDown = iptables -t nat -D POSTROUTING -s {{ pia_vlan50_subnet }} -o %i -j MASQUERADE 2>/dev/null || true
|
||||
PreDown = iptables -D {{ pia_iptables_chain }} -d {{ pia_vlan50_subnet }} -i %i -m state --state ESTABLISHED,RELATED -j ACCEPT 2>/dev/null || true
|
||||
PreDown = iptables -D {{ pia_iptables_chain }} -s {{ pia_vlan50_subnet }} -o %i -j ACCEPT 2>/dev/null || true
|
||||
{% if pia_mss_clamp_enabled %}
|
||||
PreDown = iptables -t mangle -D FORWARD -s {{ pia_vlan50_subnet }} -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
|
||||
PreDown = iptables -t mangle -D FORWARD -s {{ pia_vlan50_subnet }} -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu 2>/dev/null || true
|
||||
{% endif %}
|
||||
|
||||
[Peer]
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user