homelab/INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md

Incident report: VLAN 50 egress outage, qBittorrent crash loop, and a PMTU black hole

Date: 2026-08-25 to 2026-08-27 (discovered/resolved across this window; underlying triggers occurred 2026-08-24/25) Systems: minisforum (PIA WireGuard gateway), nik-debian (VLAN 50 workloads: browser-vpn-proxy, qbittorrent, jdownloader) Status: All four issues resolved live, with code fixes committed for three of them (qBittorrent, MSS clamping, wg-quick restart idempotency). The remaining gap (PIA tunnel handshake-death alerting/remediation) is documented with prevention options, deliberately not yet implemented — see "Open follow-up" below.

Summary

Four independent problems, found across the same troubleshooting session:

1. minisforum's PIA WireGuard tunnel (pia-wg) silently died — the interface, systemd unit, and routes all stayed "up" and looked healthy, but the encrypted session with PIA's server had stopped working hours earlier. This black-holed all VLAN 50 egress (browser-vpn-proxy, qbittorrent, jdownloader) — safely (no leak to eth0), but silently (nothing paged anyone).
2. qBittorrent (5.2.0) got stuck in an internal crash loop on nik-debian, unrelated to the tunnel issue and pre-dating it. A known upstream bug in qBittorrent's single-instance lock file handling caused it to restart every ~1-2 seconds, forever, without ever opening its WebUI.
3. A pre-existing PMTU black hole between pia-wg (MTU 1420) and the rest of the path caused real downloads to hang indefinitely once the tunnel itself was working again — small requests succeeded, but any connection needing a larger response (like a real download's TLS handshake) silently never got one. This was a known, anticipated gap (pia_mss_clamp_enabled existed in the role from the start, deliberately left off pending real evidence) — now confirmed and fixed.
4. Deploying the fix for #3 broke wg-quick@pia-wg's restart, because tearing down the live interface with newly-changed PreDown rules it never actually had installed made the whole teardown abort partway, orphaning the interface. Fixed at the root by making PreDown idempotent — a general hazard for any future config change, not specific to MSS clamping.

All four were found during the same troubleshooting session because a user report ("the proxy isn't working after I rebooted nik-debian") triggered an investigation broad enough to surface all of them in turn — #3 only became visible after #1 was fixed (a dead tunnel has no traffic to black-hole), and #4 only surfaced while actually deploying #3's fix. None of the four share a root cause — see "Were these related?" below.

Timeline

All timestamps as observed from their own source; see the timezone caveat below before doing precise cross-host arithmetic.

When (as logged)	Source	Event
2026-08-24 23:07:21Z	nik-debian node Ready condition	nik-debian rejoins the cluster after a reboot
2026-08-24 23:31:28 (pod-local clock)	browser-vpn-proxy container log	Fresh pod starts, SOCKS5 proxy begins accepting connections
2026-08-24 23:32:08 (pod-local clock)	qbittorrent/jdownloader pod logs	Both pods' init containers complete, main containers start
~2026-08-25 08:35 (pod-local clock, per file mtimes)	qBittorrent's own log-rotation file timestamps	qBittorrent's internal crash loop begins (first evidence of rapid log rotation) — this predates the tunnel's death below
~2026-08-25 13:15 JST (back-calculated from the healthcheck's own handshake-age readings)	minisforum, pia-gateway-healthcheck	pia-wg's WireGuard session actually goes dead (last real handshake)
2026-08-25 11:04-11:05 (pod-local clock)	browser-vpn-proxy container log	First visible symptoms: broken pipe, then connection timed out reaching an external site
2026-08-25 20:42-20:46 JST	minisforum, journalctl -u pia-gateway-healthcheck	Healthcheck has been correctly detecting FAIL handshake: ...s old, exceeds max 180s every 60s the entire time — just never surfaced to a human
2026-08-25 (user report)	User	Reports proxy not working after a reboot of nik-debian
2026-08-25, live troubleshooting	This session	Both root causes diagnosed and fixed (see below)

Timezone caveat: minisforum's systemctl/journalctl output explicitly shows JST. The container/pod log timestamps pasted during this investigation did not carry an explicit timezone marker, so their exact offset from minisforum's JST clock isn't confirmed here — treat cross-host time deltas in this report as approximate, not to-the-second precise. The relative ordering of events on the same host (e.g. "qBittorrent's loop started before the tunnel died") is solid, since the reasoning for that only compares timestamps that share a clock.

Were these related?

No — evidence points at three independent causes, not a shared trigger:

• qBittorrent's crash loop started (~08:35, pod-local clock) well before the PIA tunnel actually died (~13:15 JST, minisforum's clock) and well before egress failures were even visible (~11:04, pod-local clock, on a different pod).
• The qBittorrent crash loop's own root cause (a stale single-instance lock file) has nothing to do with WireGuard, PIA, or networking at all.
• The PMTU black hole (Incident 3) is a pre-existing structural gap in the tunnel's own MTU handling — not something the reboot, the tunnel death, or qBittorrent's bug caused. It was simply not visible while the tunnel was dead (Incident 1) — no working tunnel meant no traffic to black-hole — and only surfaced once Incident 1 was fixed and something tried to move a large enough response across the path again.
• All three problems happened to be sitting there, undetected, until a user report about a fourth, unrelated-seeming thing (browser-vpn-proxy after a reboot) triggered an investigation broad enough to notice all of them in sequence.

The reboot of nik-debian itself is not the confirmed trigger for any of the three:

• The PIA tunnel runs on a completely different host (minisforum), which was not rebooted.
• qBittorrent's crash loop started roughly 9 hours after the reboot, not immediately following it — something else (unidentified — see below) caused qBittorrent's underlying unclean shutdown that left the stale lock file in the first place. The reboot is a plausible but unconfirmed contributor; it was not caught in the act.
• The PMTU black hole is a property of pia-wg's own fixed mtu 1420 versus the rest of the path — a static configuration characteristic, not something a reboot would change.

Incident 1: PIA WireGuard tunnel silently died ("zombie tunnel")

Symptom

All three VLAN 50 workloads lost egress simultaneously. qbittorrent's and jdownloader's sidecar containers failed with OSError: [Errno 101] Network unreachable — an immediate "no route" response, not a timeout. browser-vpn-proxy showed the same underlying cause but surfaced as a plain timeout from outside the cluster.

Root cause

wg-quick@pia-wg.service was active (exited) (this is normal for wg-quick — it's a one-shot config script, not a long-running daemon) and the pia-wg interface existed with its usual PostUp-installed routes and iptables rules. But sudo wg show pia-wg showed:

latest handshake: 7 hours, 29 minutes, 29 seconds ago
persistent keepalive: every 25 seconds

WireGuard's own protocol re-handshakes roughly every 2 minutes under normal operation, and a 25-second keepalive is configured specifically to keep that happening even with no application traffic. A handshake this stale means the actual encrypted session with PIA's server (69.33.198.63:1337) had been dead for hours — the local interface, routes, and firewall rules wg-quick set up were all still present and "correct," but nothing was actually getting through. ip route show table pia confirmed the route was the normal working route (default dev pia-wg scope link), not the kill switch's unreachable route — meaning the kill switch's own "interface down → block" mechanism never triggered, because the interface never actually went down. It just stopped working while looking fine.

What actually broke the session was not conclusively identified. ping -c3 69.33.198.63 succeeded cleanly (0% loss, ~70ms) from minisforum's normal path, ruling out "minisforum's WAN is broken" or "that IP is completely unreachable." The most likely explanations, in rough order of plausibility, none confirmed:

1. PIA's registered-key expiry. A plain sudo systemctl restart wg-quick@pia-wg (which only recreates the local interface with the same key) did not produce a new handshake even after 10+ seconds — strongly suggesting PIA's server no longer recognized this specific registered public key. Re-registering from scratch (-e pia_force_reregister=true, generating a fresh keypair and calling PIA's addKey API again) did restore a working tunnel immediately. This is the strongest evidence: a fresh registration worked where a bare restart did not.
2. A server-side outage specific to that one PIA edge server (69.33.198.63) — plausible, but the successful fresh re-registration landed on a different exit IP in the same Hong Kong region, so this wasn't independently distinguished from (1).
3. A transient NAT/UDP path issue on minisforum's own WAN — considered less likely given ping (a different protocol, admittedly) worked fine, and a restart-without-reregistration should have been enough to recover from a purely local NAT hiccup, but wasn't.

How to reproduce

Not independently reproduced on demand — this was diagnosed from a live occurrence, not triggered experimentally. The diagnostic signature, if it happens again, is:

sudo wg show pia-wg          # interface "looks" fine, but check "latest handshake"
ip route show table pia      # shows the *working* route, not "unreachable" — the kill switch did NOT engage
sudo systemctl restart wg-quick@pia-wg
sudo wg show pia-wg latest-handshakes   # if this stays at 0 (never) for more than ~15-20s after a restart, a bare restart isn't enough — re-registration is likely needed

How it was fixed (this occurrence)

ansible-playbook -i ansible/inventory.yaml \
  ansible/playbooks/pia-gateway.yaml -K -J \
  -e pia_force_reregister=true

This hit a real, separate, now-fixed bug on the way: ansible/roles/pia-gateway/tasks/register.yaml's when: conditions used pia_force_reregister/pia_force_key_rotation directly in boolean expressions. Passed via the CLI's plain key=value form (-e pia_force_reregister=true, exactly what the role's own README documents), Ansible sets the variable as the string "true", not a real boolean — and recent ansible-core rejects using a string directly in a when: boolean expression (Conditional result (True) was derived from value of type 'str'). Fixed by wrapping every use in | bool (commit e757850), which correctly coerces both real booleans (the defaults/main.yaml case) and "true"/"false" strings (the CLI-override case). Verified with an isolated local test playbook exercising both cases before trusting it against the live host.

Known gap / prevention (not yet implemented — deferred by request)

ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2 (run via pia-gateway-healthcheck.timer, every 60s) already correctly detected this failure the entire time — every single run logged FAIL handshake: ...s old, exceeds max 180s. The script is explicitly designed as "observability only, no remediation" (its own header comment says so), which is a deliberate prior design choice, not an oversight. The actual gap is that nothing consumes that detection — no alert, no dashboard, nothing — so a real, correctly-detected outage sat unnoticed until a user happened to test the affected service directly.

Options, not yet decided:

1. Alert only (recommended by this session, not yet chosen by the user): keep the "no remediation" design, but have the healthcheck (or a small wrapper) notify a human after a few consecutive failures. Two ways to actually send it:
   • Via alert-bridge.home.arpa's POST /alerts (already exists, already designed for exactly this — a per-caller bearer token instead of distributing the raw Discord webhook secret to every host). Its exact request schema lives in the private gitea.nik4nao.com/nik/alert-bridge source, not visible from outside — confirm it before wiring anything.
   • A direct curl to the Discord webhook URL from minisforum. Simpler, no schema to reverse-engineer, but means minisforum needs its own copy of the raw webhook secret rather than a scoped token — a step away from the existing pattern.
2. Auto-restart on sustained failure: after N consecutive failed checks, automatically systemctl restart wg-quick@pia-wg. Fixes it without a human noticing, but is a real architecture change to a script deliberately built to never remediate — and masks a recurring problem instead of surfacing it, if the underlying cause (e.g. PIA-side key expiry) keeps happening.
3. Both: attempt auto-restart, but still alert either way, so there's always a record even when unattended recovery works.
4. Given evidence pointed at registration expiry rather than a simple interface restart being sufficient, any remediation path should probably escalate to pia_force_reregister=true-equivalent logic, not just a bare wg-quick restart, if a restart-only attempt doesn't restore a fresh handshake within a short grace period.

Incident 2: qBittorrent 5.2.0 stale-lock crash loop

Symptom

qbittorrent pod showed 1/2 Ready — the qbittorrent container was Running (no restarts, no crash reported at the Kubernetes level) but its readiness probe failed continuously: dial tcp <pod-ip>:8080: connect: connection refused. This reproduced identically on a freshly recreated pod (2.5 minutes old), ruling out anything specific to the original (12-hour-old, post-reboot) pod instance.

Root cause

kubectl exec ... ps aux showed qbittorrent-nox genuinely running, consuming ~80% CPU. Its own log file (/config/qBittorrent/logs/qbittorrent.log — the container's stdout only ever showed the linuxserver.io init banner, qBittorrent logs to a file, not stdout) showed:

(N) 2026-08-25T21:08:00 - qBittorrent termination initiated
(N) 2026-08-25T21:08:00 - qBittorrent is now ready to exit
(N) 2026-08-25T21:08:01 - qBittorrent v5.2.0 started. Process ID: 1376
(N) 2026-08-25T21:08:01 - Using config directory: /config/qBittorrent
(N) 2026-08-25T21:08:01 - qBittorrent termination initiated
(N) 2026-08-25T21:08:01 - qBittorrent is now ready to exit
(N) 2026-08-25T21:08:03 - qBittorrent v5.2.0 started. Process ID: 1384
...

A self-perpetuating crash loop, cycling every 1-2 seconds (PID incrementing by 8 each time — matching the container's process-spawning pattern), with the process voluntarily exiting immediately after every start (no crash, no signal — it logs its own "termination initiated"). The /config/qBittorrent/ directory contained a directory's worth of hundreds of rotated log files (qbittorrent.log.bak1 through .bak414+), each ~66-67KB, many piling up within minutes of each other — direct evidence of sustained rapid restart cycling, not a one-off.

This exactly matches a known, confirmed upstream bug: qbittorrent/qBittorrent#24164 (and its duplicate, #24185). qBittorrent 5.2.0 switched its single-instance lock from the old fcntl-based QtLockedFile (a 0-byte lock file) to Qt's QLockFile, which expects the lock file to contain a PID + hostname. If the previous shutdown wasn't clean and left a lock file the new version can't parse/verify, 5.2.0 assumes another instance holds it, tries to hand off over the (also stale) ipc-socket, that fails too, and it exits — then immediately restarts (supervised by the container's own s6 process manager) and repeats forever.

What triggered the original unclean shutdown is not confirmed. The lock/IPC files were both dated 2026-08-25 08:05 (pod-local clock), about 30 minutes before the crash-loop pattern is first visible in the rotated-log timestamps (08:35) — consistent with something restarting the qBittorrent process around 08:05 without a clean shutdown, but what that something was (a Kubernetes-level event, an OOM condition, a manual action, something else) wasn't caught in the act and isn't identified here.

One structural factor worth naming even though it's not fully confirmed: qBittorrent's lock file embeds a hostname, and every Kubernetes pod restart gets a new hostname (the pod name). That's exactly the kind of environment where a lock's embedded hostname can't reliably be trusted to "match" across restarts — plausibly why this surfaces more readily here than it would on a stable bare-metal or VM host with a fixed hostname, though this wasn't independently isolated as the determining factor versus the unclean-shutdown precondition alone.

How to reproduce

Not reproduced from a clean slate in this session — confirmed via matching log signatures against the upstream issue, not an independent from-scratch trigger. Based on the upstream report, the reproduction conditions are: qBittorrent 5.2.0 starts with a lockfile/ipc-socket already present in its config directory from a previous, unclean shutdown (e.g. the container was SIGKILLed rather than given a chance to shut down gracefully). The diagnostic signature if it happens again:

kubectl -n downloads logs <qbittorrent-pod> -c qbittorrent -c ip-reporter --tail=5   # stdout goes quiet right after the LSIO init banner
kubectl -n downloads exec <qbittorrent-pod> -c qbittorrent -- ps aux                  # qbittorrent-nox IS running, high CPU, PID climbing on repeat checks
kubectl -n downloads exec <qbittorrent-pod> -c qbittorrent -- tail -n 20 /config/qBittorrent/logs/qbittorrent.log   # the "started / termination initiated / ready to exit" cycle

How it was fixed

This occurrence (live remediation):

kubectl -n downloads exec <pod> -c qbittorrent -- rm -f /config/qBittorrent/lockfile /config/qBittorrent/ipc-socket

The next s6-supervised restart (within 1-2 seconds, given the loop's own cycle time) found no lock file, acquired a fresh one, and started normally.

Prevention (implemented, committed): added a new qbittorrent-clear-stale-lock init container to manifests/media/qbittorrent.yaml that unconditionally removes lockfile/ipc-socket before qBittorrent starts, on every pod start. This is safe specifically because this Deployment uses strategy: Recreate with a single replica and a ReadWriteOnce PVC — Kubernetes itself guarantees the previous instance is fully terminated before a new pod's init containers ever run, so there is no scenario in this specific topology where the lock could be legitimately held by a still-running peer. (This reasoning would not automatically transfer to a RollingUpdate strategy or multiple replicas — don't copy this pattern there without re-checking the concurrency assumption.)

Residual risk

The upstream issue references a fix PR (#24218) intended to make qBittorrent itself clean up a stale lock automatically. Whether the pinned image (lscr.io/linuxserver/qbittorrent:5.2.0) includes that fix wasn't checked — if it doesn't, our init-container workaround remains necessary until either the image is updated past the point where the upstream fix landed, or that's independently confirmed. What caused the original unclean shutdown (the actual precondition for hitting this bug at all) remains unidentified — the fix here prevents the symptom (permanent crash loop) from persisting, but doesn't address whatever causes qBittorrent's shutdowns to be unclean in the first place, if that keeps happening.

Incident 3: PMTU black hole on the pia-wg forward path

Symptom

After Incident 1 was fixed (fresh PIA registration, working tunnel), JDownloader still couldn't actually download anything: a link showed status "Running" but no progress and no speed, indefinitely. The app itself was healthy, and basic connectivity worked (its own self-update fetched a small file from an external CDN without any problem).

Root cause

JDownloader's own per-host plugin log (subyshare.com_jd.plugins.hoster.SubyShareCom.log.0) showed the real error: connecting to the actual download server succeeded, JDownloader sent its TLS ClientHello, and then nothing came back —

javax.net.ssl.SSLException: Read timed out
Caused by: java.net.SocketTimeoutException: Read timed out
ConnectIP: sbs237.sbsf.tech/195.201.192.190:182

— a 60-second read timeout waiting for the server's handshake response. Small round-trips (a version-check HTTP fetch, plain ICMP pings) worked fine throughout; only a connection needing to move a larger response hung.

This is the textbook signature of a path-MTU (PMTU) black hole. ansible/roles/pia-gateway/templates/pia-wg.conf.j2 sets pia-wg's interface MTU to 1420 (WireGuard's own encapsulation overhead requires a lower MTU than the outer network's 1500) — confirmed directly, not assumed, via a real test: ping -M do -s 1450 -c 3 1.1.1.1 from a VLAN 50 pod returned

From 10.10.40.53 icmp_seq=1 Frag needed and DF set (mtu = 1420)

i.e. minisforum correctly tells the sender to fragment/shrink, and the pod's own kernel correctly honors that — our own side of the path relays PMTU discovery correctly. That means the actual black hole is further out: somewhere between minisforum's tunnel and the real download server (PIA's own network, or the server's own path), where an equivalent "fragmentation needed" notification would need to travel much further back to the server (not us) and evidently isn't making it, so the server's larger response packets just silently vanish instead of ever being resent smaller.

This gap was anticipated from the start, not newly introduced: ansible/roles/pia-gateway/defaults/main.yaml already had a pia_mss_clamp_enabled flag and the corresponding iptables -t mangle ... TCPMSS --clamp-mss-to-pmtu rule already written in the WireGuard config template — deliberately left false by plan.md's own explicit instruction ("do not guess this... flip only after ... MTU testing shows it's actually needed"). It most likely wasn't caught by earlier canary/qBittorrent/browser-vpn-proxy testing because those tests mostly exercised small request/response pairs (curl ifconfig.me, dig, small API calls) — this only surfaces on a connection that needs to move a larger response, like a real file download's TLS handshake.

How to reproduce

kubectl -n downloads exec vlan50-canary -c netshoot -- ping -M do -s 1392 -c 3 1.1.1.1   # works — under the real MTU
kubectl -n downloads exec vlan50-canary -c netshoot -- ping -M do -s 1450 -c 3 1.1.1.1   # fails locally with "Frag needed... mtu = 1420" — this is the confirmation test, not a guess

Application-level symptom: any download/request whose response exceeds ~1420 bytes (accounting for WireGuard/IP/TCP overhead) will hang indefinitely rather than erroring cleanly, while small requests keep working — making this easy to miss if testing only checks basic connectivity.

How it was fixed

Enabled the already-built toggle rather than writing new mechanism: ansible/roles/pia-gateway/defaults/main.yaml's pia_mss_clamp_enabled flipped from false to true. This clamps the TCP MSS on SYN packets outbound on pia-wg for 10.10.50.0/24 traffic, so both ends of any new TCP connection negotiate a small-enough segment size from the start — sidestepping the need for a working PMTU-discovery ICMP round-trip entirely. Applying it needed a second, unplanned fix — see Incident 4 immediately below, discovered in the process of actually deploying this one. Confirmed working: after Incident 4's recovery, all three VLAN 50 workloads' egress checks passed and the specific JDownloader download that had been stuck actually completed.

Residual risk

MSS clamping fixes new TCP connections going forward; it doesn't retroactively fix anything already stuck — confirmed in practice, the originally-stuck download needed a manual retry after the fix landed. The scope is intentionally narrow (10.10.50.0/24 outbound on pia-wg only, SYN packets only) to match the existing kill-switch/firewall pattern of "own chain, only touch what's ours" — this should not affect any other traffic on minisforum.

Incident 4: enabling MSS clamping broke wg-quick restart (discovered while fixing Incident 3)

Symptom

Running the Ansible playbook to actually apply the MSS-clamping default change (no -e override, since the change was now a committed default) completed with changed=0 — the config file was never rewritten at all, because pia-wg.conf already existed and the registration block that writes it is gated behind pia_force_reregister or pia_force_key_rotation or not pia_wg_conf_stat.stat.exists; a plain re-run with the file already present skips that entire block. Re-running with -e pia_force_reregister=true this time got further, but then failed outright:

RUNNING HANDLER [pia-gateway : Restart pia-wg]
[ERROR]: Task failed: Module failed: Unable to restart service wg-quick@pia-wg: Job for wg-quick@pia-wg.service failed because the control process exited with error code.

Root cause

Two separate, now both understood:

1. The config-write gate. The role currently bundles "rewrite pia-wg.conf from the template" together with "actually re-register with PIA" under one when: condition — there's no path to pick up a template-only change (like flipping pia_mss_clamp_enabled) without forcing a full PIA re-registration too. Not fixed here; noted as a design gap (see Open follow-up).
2. systemctl restart tore down the live interface using rules it never had. journalctl showed exactly why: PreDown's new iptables -t mangle -D FORWARD ... TCPMSS line failed with iptables: Bad rule (does a matching rule exist in that chain?) — because the currently running interface had been brought up under the old config (before MSS clamping existed), so its PostUp never added that mangle rule in the first place. Confirmed against wg-quick's real source (execute_hooks() in wireguard-tools's linux.bash): each hook line runs via (eval "$hook"), and wg-quick's own set -e aborts the entire down/up sequence on the first failing hook. Since that failing line was the last PreDown line, everything before it (the kill-switch's unreachable route seed, the MASQUERADE/ACCEPT rule deletes) had already run successfully — confirmed live: ip route show table pia showed unreachable default (fail-closed, not leaking) — but wg-quick's own built-in ip link delete dev pia-wg step, which normally runs after all custom PreDown lines, never got reached. The interface was left orphaned: still present at the kernel level, but stripped of its firewall rules. The subsequent wg-quick up then failed too, immediately, with `pia-wg' already exists, since ip link add can't create an interface that's still there.

This is a general hazard, not specific to MSS clamping: Ansible's template task overwrites pia-wg.conf on disk before the "Restart pia-wg" handler fires, so any future change to what PostUp/PreDown actually do will hit the identical failure the next time the config changes while the tunnel is already up.

How to reproduce

Change any PostUp/PreDown rule content in pia-wg.conf.j2 (add, remove, or reorder a rule) while pia-wg is currently running under the old version of that content, then systemctl restart wg-quick@pia-wg (or let Ansible's handler do it). If any new/changed PreDown line tries to -D something the live interface never actually added, the restart fails and the interface is left orphaned.

How it was fixed

This occurrence (live recovery):

sudo ip link delete dev pia-wg
sudo systemctl start wg-quick@pia-wg

Directly removing the orphaned interface (the exact step the aborted teardown was one line away from doing itself) let a fresh wg-quick up succeed cleanly, this time actually applying the new PostUp rules including the MSS clamp. Confirmed via iptables -t mangle -S FORWARD showing the rule actually installed, and a handshake timestamp matching the moment of the restart (not stale).

Prevention (implemented, committed): made every PreDown iptables -D line in pia-wg.conf.j2 tolerate the rule it's removing not existing (2>/dev/null || true), verified safe against wg-quick's real source (a hook line's compound exit status becomes 0, so set -e no longer treats "nothing to delete" as fatal) and verified by rendering the template directly with Jinja2 before trusting it. The first PreDown line (ip route replace unreachable default table pia) didn't need this — route replace doesn't fail due to prior state the way -D does. This makes PreDown fully idempotent, so this exact failure mode can't recur for any future PostUp/PreDown content change — no separate "self-heal after the fact" layer needed, since the restart now just succeeds correctly every time.

Residual risk

None identified for this specific failure mode going forward — the fix addresses the root cause directly rather than adding a recovery layer on top of a still-fragile restart. The separate config-write gating issue (item 1 above) remains open.

Open follow-up

• Decide and implement PIA tunnel remediation/alerting (Incident 1's "Known gap" section above) — explicitly deferred by the user to a later date, options documented above.
• Consider splitting "rewrite config from template" from "re-register with PIA" in the role (Incident 4's root cause #1) — currently there's no way to pick up a template-only default change without forcing a full re-registration.
• Confirm whether a newer lscr.io/linuxserver/qbittorrent tag already includes the upstream fix for #24164, as a potential alternative/complement to the init-container workaround.
• If it recurs, try to catch what actually triggers qBittorrent's unclean shutdown in the first place (Incident 2's root cause section) — nothing in this investigation identified it.