Found live, 2026-09-22: after fixing pia-killswitch.sh's FORWARD-jump
repositioning and getting one pia-gateway register attempt past the
safety check, the very next attempt - roughly an hour later, no
reboot, no UFW reload, no script content change in between - hit the
identical "FORWARD jump to kill-switch chain is not first" failure
again. This disproves the earlier assumption that kube-router/dockerd
reassert their own position-1 jumps only at boot/restart scale:
kube-router resyncs on ordinary Kubernetes API events, and this
cluster's CronJobs (ca-sync, porkbun-ddns, gitea-backup) each
create/delete a Pod every few minutes - plausible triggers for
kube-router's own reconciliation loop, and therefore plausible
triggers for this drift, recurring on a timescale of minutes.
The reposition logic itself (previous commit) is correct - it just
only ran on the existing triggers (boot, UFW reload, an
Ansible-detected script content change), none of which fired between
the two attempts.
Adds pia-killswitch-reassert.timer/.service: re-runs the already fully
idempotent pia-killswitch.sh every pia_healthcheck_interval_sec (same
cadence as the health check), giving it regular chances to run instead
of only sporadic ones. Deliberately separate from
pia-gateway-recover.timer and NOT gated by pia_auto_recovery_enabled -
this is the kill switch continuing to assert its own already-
established, narrowly-scoped state (source rule, table default,
FORWARD jump position), the same kind of thing it already does
unconditionally at boot and after a UFW reload, just on a schedule
too - not the higher-level automatic recovery/registration logic the
plan was deliberately cautious about running unattended.
Unblocked live in the meantime with a manual
`sudo systemctl restart pia-killswitch.service`.
Offline checks: yamllint, ansible-playbook --syntax-check, rendered
templates (Jinja2 + bash -n on the updated script), ansible-lint
(diffed against main - no new finding categories), full pia_gateway.py
test suite (94/94, unaffected by this change) all pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live, 2026-09-22, on the addKey step of the first successful
kill-switch-check pass: every candidate failed with
kind=tls detail="certificate verification failed". Root-caused on
minisforum: PIA's own CA certificate (files/pia-ca.crt) has a CA:TRUE
Basic Constraints extension that isn't marked critical - a real RFC
5280 non-conformance, confirmed not a stale vendored copy (a fresh
fetch from pia-foss/manual-connections is byte-identical, same
sha256 already recorded in the file's own header, same defect).
Recent OpenSSL (3.2+; confirmed via a direct repro against
minisforum's 3.5.7 - SSLCertVerificationError, verify_message
"Basic Constraints of CA cert not marked critical") rejects this by
default, a check promoted from X509_STRICT-only into the baseline
chain-verification path. openssl s_client (LibreSSL, this dev
machine) and curl both accept the identical cert/chain/CA file - only
Python's ssl module, via this host's system OpenSSL, rejects it. This
went unnoticed until now because the three prior successful
registrations all went through curl --cacert (the legacy Ansible
path); this was the first time the Python ssl code path ran for real.
Python's stdlib ssl module has no custom verify-callback hook, so
there's no clean way to tolerate just this one known,
third-party-controlled defect while keeping full validation for
everything else. Rather than weakening verify_mode or adding a new
dependency (pyOpenSSL, for a real callback - considered, rejected as
a bigger lift than necessary) to route around a gap in a library we
don't control, addKey now shells out to curl
(HttpClient.get_pinned_curl, curl added to this role's apt list) -
the same tool PIA's own reference client already uses for this exact
endpoint, with --cacert and never -k/--insecure, so this is a
different, already-proven-correct TLS backend for the one endpoint
that needs it, not a downgrade. serverlist/token are unaffected
(public WebPKI certs) and still use the stdlib ssl path.
Verified: a real subprocess curl call against the actual failing
candidate (Server-11669-10a / 69.33.198.97) gets a genuine HTTP
response from PIA, proving TLS validation succeeds via this path.
Also found and handled: curl's own exit code for a --fail-triggered
rejection isn't fully portable (macOS/SecureTransport curl reports 56
where GNU/OpenSSL curl reports 22 for the same case) - classification
now also matches curl's own error text as a second signal, not just
the exit code.
Added 8 tests (94 total, all passing) covering the curl invocation
shape, stdin token delivery (no trailing newline, never in argv), and
exit-code + text-based error classification including the
cross-platform exit-code case found live.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live on the first deployment of the auto-recovery controller
(2026-09-22): pia-gateway register correctly refused with "FORWARD
jump to kill-switch chain is not first". Root cause confirmed on
minisforum via iptables -S FORWARD/-S KUBE-ROUTER-FORWARD/-S
DOCKER-USER/-S DOCKER-FORWARD: the PIA-VLAN50 jump was 4th, behind
kube-router's and dockerd's own jumps, which each independently
reassert position 1 on their own resyncs/restarts (kube-router had
even left a duplicate of its own rule behind). pia-killswitch.sh.j2
only checked whether its jump existed (iptables -C) before deciding
not to touch anything — existence isn't proof of position, so once
pushed down it stayed down.
Confirmed safe to reposition unconditionally by reading the actual
content of what's ahead of it rather than assuming: KUBE-ROUTER-FORWARD
only matches specific pod /32 addresses, DOCKER-USER was empty, and
DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50
arrives via enp1s0) - none can intercept VLAN 50 traffic regardless of
order. Fixed the script itself (delete + reinsert at position 1 if the
current first rule isn't ours) rather than loosening the Python
controller's forward_jump_first safety check, which was working
correctly when it refused to register against the drifted state.
Verified offline: rendered template passes bash -n, and the exact
live topology (rule present but 4th) reproduced against a bash mock
of iptables, confirming reposition-when-drifted, idempotency when
already first, and insert-when-absent all behave correctly. No real
iptables/wg/ip on this dev machine to test further; not yet
re-applied or re-verified against minisforum.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Serialize every mutating command (recover, renew, register,
rewrite-config, pause, resume, state-reset) on the shared lock,
acquired BEFORE state is read rather than after: reading first and
locking only around the write let a slower command silently clobber
whatever a faster one wrote in between — in particular, an operator
pause could be lost to an in-flight recovery pass that read state
before the pause ran and only saved its own copy back afterwards.
- Fix emergency_close() to reposition its VLAN 50 DROP to the first
effective FORWARD position when it exists but has been pushed down
by something else (e.g. an out-of-band ufw reload) — a bare `-C`
existence check said nothing about position.
- Tighten wait_for_verified()'s handshake freshness check from >= to >
since_epoch: wg show and since_epoch are both whole seconds, so a
handshake at exactly the same second as since_epoch is
indistinguishable from a stale, pre-activation one and must not
count as proof of a new one.
- Fix the Ansible config-validation task to fail on any exit code
outside status's three documented health exits (0/1/2), not just on
5 — verified live against the real program (no wg/ip/iptables on
this dev machine reproduces the "nothing configured yet" first-run
case; status correctly returns 1, and broken/missing config.json
correctly returns 5 and is now caught).
- Correct the live fault-injection drill in the role README: port
1337 is only the addKey registration API port, not the tunnel's own
WireGuard UDP endpoint (derive that from `wg show pia-wg endpoints`
instead), and make the packet-capture check direction-aware
(`tcpdump -Q out`) so it can't conflate a real egress leak with
VLAN 50's normal inbound arrival on enp1s0.
Added 6 tests (86 total, all passing): emergency-close repositioning
past a preceding ACCEPT, pause/resume blocking on an in-flight
lock holder and surviving into a subsequent recovery pass, and the
handshake boundary at exactly/one-second-either-side of since_epoch.
yamllint, ansible-playbook --syntax-check, and ansible-lint (diffed
against main — no new finding categories) all pass. No live changes;
nothing pushed to main.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Enabling MSS clamping exposed a general hazard: systemctl restart
wg-quick@pia-wg tears down the live interface using whatever PreDown
rules are on disk right now, not what the interface was actually
brought up with. If Ansible had already rewritten pia-wg.conf with a
changed rule, the live interface (built under the old rules) won't
have whatever the new PreDown line tries to delete - iptables -D fails,
wg-quick's own set -e aborts the whole down/up sequence, and the
interface is left orphaned (never reaches its own built-in ip link
delete step), which then makes the following wg-quick up fail too with
"already exists".
Confirmed against wg-quick's real source (execute_hooks() runs each
hook via `(eval "$hook")`) that appending `2>/dev/null || true` to
each PreDown iptables -D line makes wg-quick treat it as succeeded
regardless of prior state - verified by rendering the template with
Jinja2 directly before trusting it against the live host. This
prevents the failure mode entirely for any future PostUp/PreDown
content change, not just this one.
Add INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md covering all four
issues from this troubleshooting window: the zombie WireGuard tunnel,
a qBittorrent 5.2.0 stale-lock crash loop, the PMTU black hole this
fix addresses, and the restart failure hit while deploying it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Registers minisforum as a PIA WireGuard peer for VPN VLAN 50, with a
boot-ordered kill switch (dedicated PIA-VLAN50 iptables chain + a
terminal unreachable route in a dedicated routing table), multi-region
addKey fallback (Hong Kong -> Taiwan -> JP Tokyo, each region's full
server list, in order), and an observability-only health check.
Verified live against minisforum: registration succeeds, wg-quick@pia-wg
is up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>