Found live, 2026-09-22: after fixing pia-killswitch.sh's FORWARD-jump
repositioning and getting one pia-gateway register attempt past the
safety check, the very next attempt - roughly an hour later, no
reboot, no UFW reload, no script content change in between - hit the
identical "FORWARD jump to kill-switch chain is not first" failure
again. This disproves the earlier assumption that kube-router/dockerd
reassert their own position-1 jumps only at boot/restart scale:
kube-router resyncs on ordinary Kubernetes API events, and this
cluster's CronJobs (ca-sync, porkbun-ddns, gitea-backup) each
create/delete a Pod every few minutes - plausible triggers for
kube-router's own reconciliation loop, and therefore plausible
triggers for this drift, recurring on a timescale of minutes.
The reposition logic itself (previous commit) is correct - it just
only ran on the existing triggers (boot, UFW reload, an
Ansible-detected script content change), none of which fired between
the two attempts.
Adds pia-killswitch-reassert.timer/.service: re-runs the already fully
idempotent pia-killswitch.sh every pia_healthcheck_interval_sec (same
cadence as the health check), giving it regular chances to run instead
of only sporadic ones. Deliberately separate from
pia-gateway-recover.timer and NOT gated by pia_auto_recovery_enabled -
this is the kill switch continuing to assert its own already-
established, narrowly-scoped state (source rule, table default,
FORWARD jump position), the same kind of thing it already does
unconditionally at boot and after a UFW reload, just on a schedule
too - not the higher-level automatic recovery/registration logic the
plan was deliberately cautious about running unattended.
Unblocked live in the meantime with a manual
`sudo systemctl restart pia-killswitch.service`.
Offline checks: yamllint, ansible-playbook --syntax-check, rendered
templates (Jinja2 + bash -n on the updated script), ansible-lint
(diffed against main - no new finding categories), full pia_gateway.py
test suite (94/94, unaffected by this change) all pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live on the first deployment of the auto-recovery controller
(2026-09-22): pia-gateway register correctly refused with "FORWARD
jump to kill-switch chain is not first". Root cause confirmed on
minisforum via iptables -S FORWARD/-S KUBE-ROUTER-FORWARD/-S
DOCKER-USER/-S DOCKER-FORWARD: the PIA-VLAN50 jump was 4th, behind
kube-router's and dockerd's own jumps, which each independently
reassert position 1 on their own resyncs/restarts (kube-router had
even left a duplicate of its own rule behind). pia-killswitch.sh.j2
only checked whether its jump existed (iptables -C) before deciding
not to touch anything — existence isn't proof of position, so once
pushed down it stayed down.
Confirmed safe to reposition unconditionally by reading the actual
content of what's ahead of it rather than assuming: KUBE-ROUTER-FORWARD
only matches specific pod /32 addresses, DOCKER-USER was empty, and
DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50
arrives via enp1s0) - none can intercept VLAN 50 traffic regardless of
order. Fixed the script itself (delete + reinsert at position 1 if the
current first rule isn't ours) rather than loosening the Python
controller's forward_jump_first safety check, which was working
correctly when it refused to register against the drifted state.
Verified offline: rendered template passes bash -n, and the exact
live topology (rule present but 4th) reproduced against a bash mock
of iptables, confirming reposition-when-drifted, idempotency when
already first, and insert-when-absent all behave correctly. No real
iptables/wg/ip on this dev machine to test further; not yet
re-applied or re-verified against minisforum.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Serialize every mutating command (recover, renew, register,
rewrite-config, pause, resume, state-reset) on the shared lock,
acquired BEFORE state is read rather than after: reading first and
locking only around the write let a slower command silently clobber
whatever a faster one wrote in between — in particular, an operator
pause could be lost to an in-flight recovery pass that read state
before the pause ran and only saved its own copy back afterwards.
- Fix emergency_close() to reposition its VLAN 50 DROP to the first
effective FORWARD position when it exists but has been pushed down
by something else (e.g. an out-of-band ufw reload) — a bare `-C`
existence check said nothing about position.
- Tighten wait_for_verified()'s handshake freshness check from >= to >
since_epoch: wg show and since_epoch are both whole seconds, so a
handshake at exactly the same second as since_epoch is
indistinguishable from a stale, pre-activation one and must not
count as proof of a new one.
- Fix the Ansible config-validation task to fail on any exit code
outside status's three documented health exits (0/1/2), not just on
5 — verified live against the real program (no wg/ip/iptables on
this dev machine reproduces the "nothing configured yet" first-run
case; status correctly returns 1, and broken/missing config.json
correctly returns 5 and is now caught).
- Correct the live fault-injection drill in the role README: port
1337 is only the addKey registration API port, not the tunnel's own
WireGuard UDP endpoint (derive that from `wg show pia-wg endpoints`
instead), and make the packet-capture check direction-aware
(`tcpdump -Q out`) so it can't conflate a real egress leak with
VLAN 50's normal inbound arrival on enp1s0.
Added 6 tests (86 total, all passing): emergency-close repositioning
past a preceding ACCEPT, pause/resume blocking on an in-flight
lock holder and surviving into a subsequent recovery pass, and the
handshake boundary at exactly/one-second-either-side of since_epoch.
yamllint, ansible-playbook --syntax-check, and ansible-lint (diffed
against main — no new finding categories) all pass. No live changes;
nothing pushed to main.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Enabling MSS clamping exposed a general hazard: systemctl restart
wg-quick@pia-wg tears down the live interface using whatever PreDown
rules are on disk right now, not what the interface was actually
brought up with. If Ansible had already rewritten pia-wg.conf with a
changed rule, the live interface (built under the old rules) won't
have whatever the new PreDown line tries to delete - iptables -D fails,
wg-quick's own set -e aborts the whole down/up sequence, and the
interface is left orphaned (never reaches its own built-in ip link
delete step), which then makes the following wg-quick up fail too with
"already exists".
Confirmed against wg-quick's real source (execute_hooks() runs each
hook via `(eval "$hook")`) that appending `2>/dev/null || true` to
each PreDown iptables -D line makes wg-quick treat it as succeeded
regardless of prior state - verified by rendering the template with
Jinja2 directly before trusting it against the live host. This
prevents the failure mode entirely for any future PostUp/PreDown
content change, not just this one.
Add INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md covering all four
issues from this troubleshooting window: the zombie WireGuard tunnel,
a qBittorrent 5.2.0 stale-lock crash loop, the PMTU black hole this
fix addresses, and the restart failure hit while deploying it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Registers minisforum as a PIA WireGuard peer for VPN VLAN 50, with a
boot-ordered kill switch (dedicated PIA-VLAN50 iptables chain + a
terminal unreachable route in a dedicated routing table), multi-region
addKey fallback (Hong Kong -> Taiwan -> JP Tokyo, each region's full
server list, in order), and an observability-only health check.
Verified live against minisforum: registration succeeds, wg-quick@pia-wg
is up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>