3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| fb9ecffa9f |
fix: tolerate content-verified kube-router jumps ahead of the kill switch
Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:
19:38:58 3rd unhealthy check -> restart
19:39:09 recovered - route open, handshake fresh
19:40:03 drift again -> emergency close -> route forced closed,
counter reset to 0
19:41:08 drift again -> emergency close again
VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.
This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.
Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.
Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.
pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|||
| 99f6ef84f0 |
fix: use curl for addKey — PIA's CA cert fails modern OpenSSL by default
Found live, 2026-09-22, on the addKey step of the first successful kill-switch-check pass: every candidate failed with kind=tls detail="certificate verification failed". Root-caused on minisforum: PIA's own CA certificate (files/pia-ca.crt) has a CA:TRUE Basic Constraints extension that isn't marked critical - a real RFC 5280 non-conformance, confirmed not a stale vendored copy (a fresh fetch from pia-foss/manual-connections is byte-identical, same sha256 already recorded in the file's own header, same defect). Recent OpenSSL (3.2+; confirmed via a direct repro against minisforum's 3.5.7 - SSLCertVerificationError, verify_message "Basic Constraints of CA cert not marked critical") rejects this by default, a check promoted from X509_STRICT-only into the baseline chain-verification path. openssl s_client (LibreSSL, this dev machine) and curl both accept the identical cert/chain/CA file - only Python's ssl module, via this host's system OpenSSL, rejects it. This went unnoticed until now because the three prior successful registrations all went through curl --cacert (the legacy Ansible path); this was the first time the Python ssl code path ran for real. Python's stdlib ssl module has no custom verify-callback hook, so there's no clean way to tolerate just this one known, third-party-controlled defect while keeping full validation for everything else. Rather than weakening verify_mode or adding a new dependency (pyOpenSSL, for a real callback - considered, rejected as a bigger lift than necessary) to route around a gap in a library we don't control, addKey now shells out to curl (HttpClient.get_pinned_curl, curl added to this role's apt list) - the same tool PIA's own reference client already uses for this exact endpoint, with --cacert and never -k/--insecure, so this is a different, already-proven-correct TLS backend for the one endpoint that needs it, not a downgrade. serverlist/token are unaffected (public WebPKI certs) and still use the stdlib ssl path. Verified: a real subprocess curl call against the actual failing candidate (Server-11669-10a / 69.33.198.97) gets a genuine HTTP response from PIA, proving TLS validation succeeds via this path. Also found and handled: curl's own exit code for a --fail-triggered rejection isn't fully portable (macOS/SecureTransport curl reports 56 where GNU/OpenSSL curl reports 22 for the same case) - classification now also matches curl's own error text as a second signal, not just the exit code. Added 8 tests (94 total, all passing) covering the curl invocation shape, stdin token delivery (no trailing newline, never in argv), and exit-code + text-based error classification including the cross-platform exit-code case found live. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| 87a9303cc3 |
fix: address code review on pia-gateway auto-recovery controller
Some checks failed
validate / lint (push) Failing after 2s
- Serialize every mutating command (recover, renew, register, rewrite-config, pause, resume, state-reset) on the shared lock, acquired BEFORE state is read rather than after: reading first and locking only around the write let a slower command silently clobber whatever a faster one wrote in between — in particular, an operator pause could be lost to an in-flight recovery pass that read state before the pause ran and only saved its own copy back afterwards. - Fix emergency_close() to reposition its VLAN 50 DROP to the first effective FORWARD position when it exists but has been pushed down by something else (e.g. an out-of-band ufw reload) — a bare `-C` existence check said nothing about position. - Tighten wait_for_verified()'s handshake freshness check from >= to > since_epoch: wg show and since_epoch are both whole seconds, so a handshake at exactly the same second as since_epoch is indistinguishable from a stale, pre-activation one and must not count as proof of a new one. - Fix the Ansible config-validation task to fail on any exit code outside status's three documented health exits (0/1/2), not just on 5 — verified live against the real program (no wg/ip/iptables on this dev machine reproduces the "nothing configured yet" first-run case; status correctly returns 1, and broken/missing config.json correctly returns 5 and is now caught). - Correct the live fault-injection drill in the role README: port 1337 is only the addKey registration API port, not the tunnel's own WireGuard UDP endpoint (derive that from `wg show pia-wg endpoints` instead), and make the packet-capture check direction-aware (`tcpdump -Q out`) so it can't conflate a real egress leak with VLAN 50's normal inbound arrival on enp1s0. Added 6 tests (86 total, all passing): emergency-close repositioning past a preceding ACCEPT, pause/resume blocking on an in-flight lock holder and surviving into a subsequent recovery pass, and the handshake boundary at exactly/one-second-either-side of since_epoch. yamllint, ansible-playbook --syntax-check, and ansible-lint (diffed against main — no new finding categories) all pass. No live changes; nothing pushed to main. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |