8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| fb9ecffa9f |
fix: tolerate content-verified kube-router jumps ahead of the kill switch
Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:
19:38:58 3rd unhealthy check -> restart
19:39:09 recovered - route open, handshake fresh
19:40:03 drift again -> emergency close -> route forced closed,
counter reset to 0
19:41:08 drift again -> emergency close again
VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.
This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.
Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.
Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.
pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|||
| 89d289addf |
feat: add periodic kill-switch reassertion timer
Found live, 2026-09-22: after fixing pia-killswitch.sh's FORWARD-jump repositioning and getting one pia-gateway register attempt past the safety check, the very next attempt - roughly an hour later, no reboot, no UFW reload, no script content change in between - hit the identical "FORWARD jump to kill-switch chain is not first" failure again. This disproves the earlier assumption that kube-router/dockerd reassert their own position-1 jumps only at boot/restart scale: kube-router resyncs on ordinary Kubernetes API events, and this cluster's CronJobs (ca-sync, porkbun-ddns, gitea-backup) each create/delete a Pod every few minutes - plausible triggers for kube-router's own reconciliation loop, and therefore plausible triggers for this drift, recurring on a timescale of minutes. The reposition logic itself (previous commit) is correct - it just only ran on the existing triggers (boot, UFW reload, an Ansible-detected script content change), none of which fired between the two attempts. Adds pia-killswitch-reassert.timer/.service: re-runs the already fully idempotent pia-killswitch.sh every pia_healthcheck_interval_sec (same cadence as the health check), giving it regular chances to run instead of only sporadic ones. Deliberately separate from pia-gateway-recover.timer and NOT gated by pia_auto_recovery_enabled - this is the kill switch continuing to assert its own already- established, narrowly-scoped state (source rule, table default, FORWARD jump position), the same kind of thing it already does unconditionally at boot and after a UFW reload, just on a schedule too - not the higher-level automatic recovery/registration logic the plan was deliberately cautious about running unattended. Unblocked live in the meantime with a manual `sudo systemctl restart pia-killswitch.service`. Offline checks: yamllint, ansible-playbook --syntax-check, rendered templates (Jinja2 + bash -n on the updated script), ansible-lint (diffed against main - no new finding categories), full pia_gateway.py test suite (94/94, unaffected by this change) all pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| 99f6ef84f0 |
fix: use curl for addKey — PIA's CA cert fails modern OpenSSL by default
Found live, 2026-09-22, on the addKey step of the first successful kill-switch-check pass: every candidate failed with kind=tls detail="certificate verification failed". Root-caused on minisforum: PIA's own CA certificate (files/pia-ca.crt) has a CA:TRUE Basic Constraints extension that isn't marked critical - a real RFC 5280 non-conformance, confirmed not a stale vendored copy (a fresh fetch from pia-foss/manual-connections is byte-identical, same sha256 already recorded in the file's own header, same defect). Recent OpenSSL (3.2+; confirmed via a direct repro against minisforum's 3.5.7 - SSLCertVerificationError, verify_message "Basic Constraints of CA cert not marked critical") rejects this by default, a check promoted from X509_STRICT-only into the baseline chain-verification path. openssl s_client (LibreSSL, this dev machine) and curl both accept the identical cert/chain/CA file - only Python's ssl module, via this host's system OpenSSL, rejects it. This went unnoticed until now because the three prior successful registrations all went through curl --cacert (the legacy Ansible path); this was the first time the Python ssl code path ran for real. Python's stdlib ssl module has no custom verify-callback hook, so there's no clean way to tolerate just this one known, third-party-controlled defect while keeping full validation for everything else. Rather than weakening verify_mode or adding a new dependency (pyOpenSSL, for a real callback - considered, rejected as a bigger lift than necessary) to route around a gap in a library we don't control, addKey now shells out to curl (HttpClient.get_pinned_curl, curl added to this role's apt list) - the same tool PIA's own reference client already uses for this exact endpoint, with --cacert and never -k/--insecure, so this is a different, already-proven-correct TLS backend for the one endpoint that needs it, not a downgrade. serverlist/token are unaffected (public WebPKI certs) and still use the stdlib ssl path. Verified: a real subprocess curl call against the actual failing candidate (Server-11669-10a / 69.33.198.97) gets a genuine HTTP response from PIA, proving TLS validation succeeds via this path. Also found and handled: curl's own exit code for a --fail-triggered rejection isn't fully portable (macOS/SecureTransport curl reports 56 where GNU/OpenSSL curl reports 22 for the same case) - classification now also matches curl's own error text as a second signal, not just the exit code. Added 8 tests (94 total, all passing) covering the curl invocation shape, stdin token delivery (no trailing newline, never in argv), and exit-code + text-based error classification including the cross-platform exit-code case found live. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| de5ce7bf52 |
fix: reposition pia-killswitch's FORWARD jump, not just check it exists
Found live on the first deployment of the auto-recovery controller (2026-09-22): pia-gateway register correctly refused with "FORWARD jump to kill-switch chain is not first". Root cause confirmed on minisforum via iptables -S FORWARD/-S KUBE-ROUTER-FORWARD/-S DOCKER-USER/-S DOCKER-FORWARD: the PIA-VLAN50 jump was 4th, behind kube-router's and dockerd's own jumps, which each independently reassert position 1 on their own resyncs/restarts (kube-router had even left a duplicate of its own rule behind). pia-killswitch.sh.j2 only checked whether its jump existed (iptables -C) before deciding not to touch anything — existence isn't proof of position, so once pushed down it stayed down. Confirmed safe to reposition unconditionally by reading the actual content of what's ahead of it rather than assuming: KUBE-ROUTER-FORWARD only matches specific pod /32 addresses, DOCKER-USER was empty, and DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50 arrives via enp1s0) - none can intercept VLAN 50 traffic regardless of order. Fixed the script itself (delete + reinsert at position 1 if the current first rule isn't ours) rather than loosening the Python controller's forward_jump_first safety check, which was working correctly when it refused to register against the drifted state. Verified offline: rendered template passes bash -n, and the exact live topology (rule present but 4th) reproduced against a bash mock of iptables, confirming reposition-when-drifted, idempotency when already first, and insert-when-absent all behave correctly. No real iptables/wg/ip on this dev machine to test further; not yet re-applied or re-verified against minisforum. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| 87a9303cc3 |
fix: address code review on pia-gateway auto-recovery controller
Some checks failed
validate / lint (push) Failing after 2s
- Serialize every mutating command (recover, renew, register, rewrite-config, pause, resume, state-reset) on the shared lock, acquired BEFORE state is read rather than after: reading first and locking only around the write let a slower command silently clobber whatever a faster one wrote in between — in particular, an operator pause could be lost to an in-flight recovery pass that read state before the pause ran and only saved its own copy back afterwards. - Fix emergency_close() to reposition its VLAN 50 DROP to the first effective FORWARD position when it exists but has been pushed down by something else (e.g. an out-of-band ufw reload) — a bare `-C` existence check said nothing about position. - Tighten wait_for_verified()'s handshake freshness check from >= to > since_epoch: wg show and since_epoch are both whole seconds, so a handshake at exactly the same second as since_epoch is indistinguishable from a stale, pre-activation one and must not count as proof of a new one. - Fix the Ansible config-validation task to fail on any exit code outside status's three documented health exits (0/1/2), not just on 5 — verified live against the real program (no wg/ip/iptables on this dev machine reproduces the "nothing configured yet" first-run case; status correctly returns 1, and broken/missing config.json correctly returns 5 and is now caught). - Correct the live fault-injection drill in the role README: port 1337 is only the addKey registration API port, not the tunnel's own WireGuard UDP endpoint (derive that from `wg show pia-wg endpoints` instead), and make the packet-capture check direction-aware (`tcpdump -Q out`) so it can't conflate a real egress leak with VLAN 50's normal inbound arrival on enp1s0. Added 6 tests (86 total, all passing): emergency-close repositioning past a preceding ACCEPT, pause/resume blocking on an in-flight lock holder and surviving into a subsequent recovery pass, and the handshake boundary at exactly/one-second-either-side of since_epoch. yamllint, ansible-playbook --syntax-check, and ansible-lint (diffed against main — no new finding categories) all pass. No live changes; nothing pushed to main. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| 4d613c9c06 |
fix: make pia-wg PreDown idempotent to prevent restart failures
Some checks failed
validate / lint (push) Failing after 1s
Enabling MSS clamping exposed a general hazard: systemctl restart wg-quick@pia-wg tears down the live interface using whatever PreDown rules are on disk right now, not what the interface was actually brought up with. If Ansible had already rewritten pia-wg.conf with a changed rule, the live interface (built under the old rules) won't have whatever the new PreDown line tries to delete - iptables -D fails, wg-quick's own set -e aborts the whole down/up sequence, and the interface is left orphaned (never reaches its own built-in ip link delete step), which then makes the following wg-quick up fail too with "already exists". Confirmed against wg-quick's real source (execute_hooks() runs each hook via `(eval "$hook")`) that appending `2>/dev/null || true` to each PreDown iptables -D line makes wg-quick treat it as succeeded regardless of prior state - verified by rendering the template with Jinja2 directly before trusting it against the live host. This prevents the failure mode entirely for any future PostUp/PreDown content change, not just this one. Add INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md covering all four issues from this troubleshooting window: the zombie WireGuard tunnel, a qBittorrent 5.2.0 stale-lock crash loop, the PMTU black hole this fix addresses, and the restart failure hit while deploying it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| 663337f818 |
feat: add workaround for qBittorrent 5.2.0 stale lock issue with init container
Some checks failed
validate / lint (push) Failing after 1s
|
|||
| 325d3bc5c7 |
feat: add pia-gateway role for minisforum PIA WireGuard egress
Registers minisforum as a PIA WireGuard peer for VPN VLAN 50, with a boot-ordered kill switch (dedicated PIA-VLAN50 iptables chain + a terminal unreachable route in a dedicated routing table), multi-region addKey fallback (Hong Kong -> Taiwan -> JP Tokyo, each region's full server list, in order), and an observability-only health check. Verified live against minisforum: registration succeeds, wg-quick@pia-wg is up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |