Compare commits

..

6 Commits

Author SHA1 Message Date
80bf24cf0b docs: record step 3 as verified live over a full 25.5-hour window
pia_forward_bypass_chains held up against real production conditions,
not just a short recheck: 1528 healthy passes, 0 SAFETY DRIFT events,
0 safety_drift classifications since the fix was deployed
(2026-09-22 20:37 -> 2026-09-23 22:02). kube-router kept resyncing
throughout; the bypass tolerance handled it silently every time.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 22:09:13 +09:00
fb9ecffa9f fix: tolerate content-verified kube-router jumps ahead of the kill switch
Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:

  19:38:58  3rd unhealthy check -> restart
  19:39:09  recovered - route open, handshake fresh
  19:40:03  drift again -> emergency close -> route forced closed,
            counter reset to 0
  19:41:08  drift again -> emergency close again

VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.

This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.

Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.

Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.

pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 19:55:47 +09:00
89d289addf feat: add periodic kill-switch reassertion timer
Found live, 2026-09-22: after fixing pia-killswitch.sh's FORWARD-jump
repositioning and getting one pia-gateway register attempt past the
safety check, the very next attempt - roughly an hour later, no
reboot, no UFW reload, no script content change in between - hit the
identical "FORWARD jump to kill-switch chain is not first" failure
again. This disproves the earlier assumption that kube-router/dockerd
reassert their own position-1 jumps only at boot/restart scale:
kube-router resyncs on ordinary Kubernetes API events, and this
cluster's CronJobs (ca-sync, porkbun-ddns, gitea-backup) each
create/delete a Pod every few minutes - plausible triggers for
kube-router's own reconciliation loop, and therefore plausible
triggers for this drift, recurring on a timescale of minutes.

The reposition logic itself (previous commit) is correct - it just
only ran on the existing triggers (boot, UFW reload, an
Ansible-detected script content change), none of which fired between
the two attempts.

Adds pia-killswitch-reassert.timer/.service: re-runs the already fully
idempotent pia-killswitch.sh every pia_healthcheck_interval_sec (same
cadence as the health check), giving it regular chances to run instead
of only sporadic ones. Deliberately separate from
pia-gateway-recover.timer and NOT gated by pia_auto_recovery_enabled -
this is the kill switch continuing to assert its own already-
established, narrowly-scoped state (source rule, table default,
FORWARD jump position), the same kind of thing it already does
unconditionally at boot and after a UFW reload, just on a schedule
too - not the higher-level automatic recovery/registration logic the
plan was deliberately cautious about running unattended.

Unblocked live in the meantime with a manual
`sudo systemctl restart pia-killswitch.service`.

Offline checks: yamllint, ansible-playbook --syntax-check, rendered
templates (Jinja2 + bash -n on the updated script), ansible-lint
(diffed against main - no new finding categories), full pia_gateway.py
test suite (94/94, unaffected by this change) all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 19:08:28 +09:00
99f6ef84f0 fix: use curl for addKey — PIA's CA cert fails modern OpenSSL by default
Found live, 2026-09-22, on the addKey step of the first successful
kill-switch-check pass: every candidate failed with
kind=tls detail="certificate verification failed". Root-caused on
minisforum: PIA's own CA certificate (files/pia-ca.crt) has a CA:TRUE
Basic Constraints extension that isn't marked critical - a real RFC
5280 non-conformance, confirmed not a stale vendored copy (a fresh
fetch from pia-foss/manual-connections is byte-identical, same
sha256 already recorded in the file's own header, same defect).
Recent OpenSSL (3.2+; confirmed via a direct repro against
minisforum's 3.5.7 - SSLCertVerificationError, verify_message
"Basic Constraints of CA cert not marked critical") rejects this by
default, a check promoted from X509_STRICT-only into the baseline
chain-verification path. openssl s_client (LibreSSL, this dev
machine) and curl both accept the identical cert/chain/CA file - only
Python's ssl module, via this host's system OpenSSL, rejects it. This
went unnoticed until now because the three prior successful
registrations all went through curl --cacert (the legacy Ansible
path); this was the first time the Python ssl code path ran for real.

Python's stdlib ssl module has no custom verify-callback hook, so
there's no clean way to tolerate just this one known,
third-party-controlled defect while keeping full validation for
everything else. Rather than weakening verify_mode or adding a new
dependency (pyOpenSSL, for a real callback - considered, rejected as
a bigger lift than necessary) to route around a gap in a library we
don't control, addKey now shells out to curl
(HttpClient.get_pinned_curl, curl added to this role's apt list) -
the same tool PIA's own reference client already uses for this exact
endpoint, with --cacert and never -k/--insecure, so this is a
different, already-proven-correct TLS backend for the one endpoint
that needs it, not a downgrade. serverlist/token are unaffected
(public WebPKI certs) and still use the stdlib ssl path.

Verified: a real subprocess curl call against the actual failing
candidate (Server-11669-10a / 69.33.198.97) gets a genuine HTTP
response from PIA, proving TLS validation succeeds via this path.
Also found and handled: curl's own exit code for a --fail-triggered
rejection isn't fully portable (macOS/SecureTransport curl reports 56
where GNU/OpenSSL curl reports 22 for the same case) - classification
now also matches curl's own error text as a second signal, not just
the exit code.

Added 8 tests (94 total, all passing) covering the curl invocation
shape, stdin token delivery (no trailing newline, never in argv), and
exit-code + text-based error classification including the
cross-platform exit-code case found live.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 18:48:48 +09:00
7b78e4a88e fix: surface the real OpenSSL error instead of a generic TLS message
_request()'s ssl.SSLCertVerificationError handler hardcoded "certificate
verification failed" regardless of cause, discarding e.verify_message/
e.reason (standard OpenSSL diagnostic text - never a secret) - exactly
what's needed to tell a wrong CA, expired cert, clock skew, and a
hostname mismatch apart. Found while diagnosing a live addKey TLS
failure on minisforum that doesn't reproduce from this machine (same
CA file, same pinned IP, same SNI hostname - openssl s_client and a
direct repro of get_pinned() both succeed here), so the next attempt
needs the real reason, not a collapsed generic string. Also stopped
discarding detail on the generic SSLError/OSError branches.

Verified: forcing a real verification failure (wrong CA) now reports
"unable to get local issuer certificate" instead of the old generic
text. Full test suite still passes (86/86).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 18:30:35 +09:00
de5ce7bf52 fix: reposition pia-killswitch's FORWARD jump, not just check it exists
Found live on the first deployment of the auto-recovery controller
(2026-09-22): pia-gateway register correctly refused with "FORWARD
jump to kill-switch chain is not first". Root cause confirmed on
minisforum via iptables -S FORWARD/-S KUBE-ROUTER-FORWARD/-S
DOCKER-USER/-S DOCKER-FORWARD: the PIA-VLAN50 jump was 4th, behind
kube-router's and dockerd's own jumps, which each independently
reassert position 1 on their own resyncs/restarts (kube-router had
even left a duplicate of its own rule behind). pia-killswitch.sh.j2
only checked whether its jump existed (iptables -C) before deciding
not to touch anything — existence isn't proof of position, so once
pushed down it stayed down.

Confirmed safe to reposition unconditionally by reading the actual
content of what's ahead of it rather than assuming: KUBE-ROUTER-FORWARD
only matches specific pod /32 addresses, DOCKER-USER was empty, and
DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50
arrives via enp1s0) - none can intercept VLAN 50 traffic regardless of
order. Fixed the script itself (delete + reinsert at position 1 if the
current first rule isn't ours) rather than loosening the Python
controller's forward_jump_first safety check, which was working
correctly when it refused to register against the drifted state.

Verified offline: rendered template passes bash -n, and the exact
live topology (rule present but 4th) reproduced against a bash mock
of iptables, confirming reposition-when-drifted, idempotency when
already first, and insert-when-absent all behave correctly. No real
iptables/wg/ip on this dev machine to test further; not yet
re-applied or re-verified against minisforum.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 16:46:30 +09:00
11 changed files with 765 additions and 62 deletions

View File

@ -189,6 +189,76 @@ real.
(`forward_jump_first`) was working as intended when it refused to
register against the drifted state — this is a fix to the thing it
checks, not a loosening of the check itself.
**This recurs far more often than "boot-scale," and the fix above only
helped once.** The reposition logic is correct, but it only *ran* on
the existing triggers (boot, a UFW reload, an Ansible-detected content
change to the script). Confirmed live: after that fix landed and got
one `pia-gateway register` attempt past this check, the very next
attempt — roughly an hour later, no reboot, no UFW reload, no script
change in between — hit the identical failure again. kube-router
resyncs on ordinary Kubernetes API events (this cluster's CronJobs —
`ca-sync`, `porkbun-ddns`, `gitea-backup` — each create and delete a
Pod every few minutes, each a plausible trigger), not just at
boot/restart scale as originally assumed. `pia-killswitch-reassert
.timer` (every `pia_healthcheck_interval_sec`, unconditional, not
gated by `pia_auto_recovery_enabled`) exists specifically to close this
gap by giving the already-correct, already-idempotent script regular
chances to run instead of only sporadic ones.
**The reassert timer alone wasn't the full fix either — it created a
new failure mode.** Confirmed live, 2026-09-22, minutes after enabling
`pia-gateway-recover.timer` for the first time (rollout step 3):
`pia-gateway-recover` and `pia-killswitch-reassert` are two
independently-scheduled 60s timers with no coordination between them.
kube-router turned out to resync (and re-push our jump down) roughly
every 60-90s on this cluster — far more often than "rare, boot-scale"
— so a `recover` pass would periodically sample `iptables -S FORWARD`
in the window between a kube-router push-down and the next reassert
fix, correctly detect real (if momentary) drift, and call
`emergency_close()` — which unconditionally forces table `pia`'s route
to `unreachable` and resets `consecutive_failures` to 0 in the same
step. The result was a self-reinforcing outage loop, observed directly
in the journal:
```
19:38:58 3rd consecutive unhealthy check → recovery_cycle → restart
19:39:09 recovered via restart (10s) — route open, handshake fresh
19:40:03 drift detected again ("FORWARD jump ... not first")
→ emergency close → route forced closed, counter reset to 0
19:41:08 drift again → emergency close again → route stays closed
```
VLAN 50 was open for roughly 10 seconds out of every 60-90 second
cycle. Three consecutive real failures were needed to trigger the
restart that reopens the route, but a fresh drift-and-emergency-close
kept arriving before three could accumulate — the counter reset every
time. This is not a bug in the detection (the drift was real, if
transient) — it's that treating *every* momentary reordering as a full
safety emergency was stricter than the actual invariant required,
given kube-router's own resync cadence on this specific host.
**Fix: `pia_forward_bypass_chains` (`Classifier
._forward_jump_effectively_first` in `files/pia_gateway.py`).** The
safety check now tolerates a configured, named FORWARD chain sitting
ahead of our jump — but only for as long as that chain's *own current
content* re-verifies, live, on every single check, as structurally
unable to match `10.10.50.0/24` (every `-s`/`-d` value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct the checker doesn't explicitly recognize fails
closed, not open). This is not a one-time trust of kube-router — if
its rules ever changed to include something that could actually
intercept VLAN 50 traffic, or an unrecognized chain/rule appeared
ahead of ours instead, it still fails closed exactly as before. Scoped
to `KUBE-ROUTER-FORWARD` only (not `DOCKER-USER`/`DOCKER-FORWARD` —
see `pia_forward_bypass_chains`'s own comment in `defaults/main.yaml`
for why only kube-router earned the exception). An emergency close and
a hard stop remain the outcome for anything not on this narrow,
content-reverified allowlist. Immediate live mitigation while this fix
was being built: `sudo systemctl stop pia-gateway-recover.timer`, then
a manual `sudo systemctl restart wg-quick@pia-wg` to reopen the route
(`pia-killswitch-reassert.timer` was left running throughout — it was
never the problem).
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
top of that closed baseline: default route via `pia-wg` in the `pia`
table, ACCEPT + established/related return inserted into the
@ -268,10 +338,14 @@ handled automatically — see the next section.
## Automatic recovery
`files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway`
(root, `0700`, standard library only — runs on the host's `python3.13`).
It is **one** implementation of PIA registration shared by the unattended
timers and by this role's own registration step, so the two can never
drift. Layout on the host:
(root, `0700`, runs on the host's `python3.13`). Almost entirely standard
library — the one exception is `addKey` specifically, which shells out to
`curl` (now an explicit `apt` dependency of this role); see
`HttpClient.get_pinned_curl`'s own docstring and "PIA's CA certificate and
why addKey uses curl, not Python's `ssl`" below for why. It is **one**
implementation of PIA registration shared by the unattended timers and by
this role's own registration step, so the two can never drift. Layout on
the host:
| Path | Purpose |
| --- | --- |
@ -284,6 +358,65 @@ drift. Layout on the host:
| `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) |
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
| `pia-killswitch-reassert.timer` → `.service` | every `pia_healthcheck_interval_sec`, re-runs `pia-killswitch.sh` (idempotent). **Not** gated by `pia_auto_recovery_enabled` — see the "FORWARD jump's position" note above and the service unit's own header for why |
### PIA's CA certificate and why addKey uses curl, not Python's `ssl`
Found live, 2026-09-22, on the very first `pia-gateway register` attempt
after the FORWARD-jump fix above got past the kill-switch check:
`addKey` failed against every candidate with `kind=tls detail=certificate
verification failed`. Root-caused directly on minisforum:
```
$ sudo python3 -c '...ctx.wrap_socket(sock, server_hostname="Server-11669-10a")...'
SSLCertVerificationError verify_message= Basic Constraints of CA cert not marked critical
```
PIA's own CA certificate (`files/pia-ca.crt`) has a `CA:TRUE` Basic
Constraints extension that isn't flagged `critical` — a real RFC 5280
non-conformance, confirmed **not** a stale vendored copy: a fresh fetch
from `pia-foss/manual-connections` on GitHub is byte-identical (same
`32e9b1d1...` SHA-256 already recorded in the file's own header) and has
the identical defect. This is PIA's problem, not ours, and not fixable by
re-vendoring. Recent OpenSSL (3.2+; confirmed live against minisforum's
`3.5.7`) rejects this by default — a check that used to be opt-in only
under `X509_V_FLAG_X509_STRICT` and has since been promoted into the
default chain-verification path. `openssl s_client` on this repo's own
dev machine (LibreSSL) and this same host's `curl` both accept the exact
same certificate/chain/CA file without complaint — only Python's `ssl`
module, using this host's system OpenSSL, rejects it. That's also why
this went unnoticed until the auto-recovery work: the legacy Ansible
path's three successful registrations (2026-08-24, 08-25, and this
morning) all went through `curl --cacert` (`addkey-attempt.yaml`), never
through Python's `ssl` module — this was the *first* time that code path
ever ran for real.
Python's stdlib `ssl` module has no custom-verify-callback hook (a
long-standing, deliberate upstream gap — unlike Go's
`tls.Config.VerifyPeerCertificate` or OpenSSL's own C API), so there is
no clean way to tolerate just this one known, third-party-controlled
defect while still fully validating everything else (chain, expiry,
hostname/SAN). Rather than weakening `verify_mode` for the whole
connection or adding a new dependency (`pyOpenSSL`, for a real verify
callback — considered, rejected as a bigger lift than necessary) to work
around a gap in a library we don't control, `addKey` specifically now
shells out to `curl` (`HttpClient.get_pinned_curl`, `curl` added to this
role's `apt` install list) — the same tool PIA's own reference client
(`connect_to_wireguard_with_token.sh`) already uses for this exact
endpoint, with `--cacert` and never `-k`/`--insecure`, so this is a
different, already-proven-correct TLS backend for the one endpoint that
needs it, not a security downgrade. `serverlist` and the token endpoint
are unaffected (they present public WebPKI certs, which don't have this
problem) and still use the stdlib `ssl` path. Verified: a real subprocess
`curl` call from this repo's dev machine against the actual failing
candidate (`Server-11669-10a` / `69.33.198.97`) gets a genuine HTTP
response from PIA (401, expected — a dummy token was used), proving TLS
validation succeeds via this path where it failed via Python's `ssl`.
curl's own exit codes for a `--fail`-triggered rejection aren't fully
portable either — confirmed live: a curl built against SecureTransport
(this dev machine) reports exit 56, not the "usual" 22 GNU/OpenSSL curl
uses for the identical case — so classification checks curl's own error
text as a second signal alongside the exit code, not the exit code alone.
### Status classes
@ -466,18 +599,83 @@ external secret broker — not a 24-hour token.
- `unknown` (a read command failing) is logged/alerted but never acted
on, by design.
### Rollout status (2026-09-22)
### Rollout status (2026-09-23)
Step 1 (baseline) and the first attempt at step 2 have been run live.
Step 2 failed on the first try: `pia-gateway register` correctly refused
with `safety invariants failing ... FORWARD jump to kill-switch chain is
not first` — a real, pre-existing drift (see the "FORWARD jump's
Steps 1-3 are done and verified live. Step 3 required two attempts —
the first caused a real (safely fail-closed) outage, root-caused and
fixed same-day; the second was verified against a full day of real
production conditions, not just a short window.
- **Step 2, attempt 1** failed: `pia-gateway register` correctly refused
with `safety invariants failing ... FORWARD jump to kill-switch chain
is not first` — a real, pre-existing drift (see the "FORWARD jump's
position is actively re-asserted" note above), not a false alarm from
this session's work, and not the "session dies" handshake issue either.
Root-caused and fixed in `pia-killswitch.sh.j2`; not yet re-applied or
re-verified live. **Use `iptables -S FORWARD` (no `head`) for the
baseline capture below** — the original `head -3` suggestion is what
hid this the first time; the drift was 4 rules in, not at the top.
this session's work, and not the "session dies" handshake issue
either. Root-caused and fixed in `pia-killswitch.sh.j2`.
- **Attempt 2**, after that fix, got past the kill-switch check and
failed differently: `addKey` rejected every candidate with
`kind=tls detail=certificate verification failed`. Root-caused and
fixed — see "PIA's CA certificate and why addKey uses curl, not
Python's `ssl`" above.
- **Attempt 3**, roughly an hour after attempt 1's fix with no reboot,
UFW reload, or script change in between, hit the *identical* FORWARD-
jump failure again — proving the drift recurs on a timescale of
minutes under this cluster's normal CronJob/pod churn, not just at
boot-scale events as originally assumed. Unblocked live with a manual
`sudo systemctl restart pia-killswitch.service`; `pia-killswitch
-reassert.timer` added as the durable fix.
- **Attempt 4** succeeded cleanly end to end: kill-switch check passed,
`addKey` accepted (`hk`/`Server-11669-10a`), and a **fresh handshake
was verified** (age 10s) — the first-ever successful shared-path
registration. `pia-gateway status` confirmed `healthy`/`safety_ok:
True`, `iptables -S FORWARD` confirmed the jump first, and
`pia-killswitch-reassert.timer` confirmed cycling correctly (7
successful runs, ~60s apart, in the journal). **Step 2 is done.**
- **Step 3** (`ansible-playbook ... -K -J`, no override — enables
`pia-gateway-recover.timer` for the first time) deployed cleanly
(`Run the shared PIA registration transaction` correctly skipped, a
verified baseline already existed) but caused a **real live outage**
within minutes: `pia-gateway-recover` and `pia-killswitch-reassert`,
two independent 60s timers, raced — kube-router's actual resync
cadence (~60-90s) turned out to be far more frequent than assumed, and
every momentary drift-then-self-correct was treated as a full safety
emergency, forcibly closing the route and resetting the 3-strikes
counter before it could accumulate. VLAN 50 was open for ~10s out of
every 60-90s cycle for several minutes, confirmed via Discord alerts
from `alert-bridge` (working end to end, first real trigger) and the
full `pia-gateway-recover`/`pia-killswitch-reassert` journals.
Mitigated live: `sudo systemctl stop pia-gateway-recover.timer` +
manual `sudo systemctl restart wg-quick@pia-wg`. Root-caused and fixed
same-day — see the "reassert timer alone wasn't the full fix" addendum
above (`pia_forward_bypass_chains`, content-verified live on every
check).
- **Step 3, re-attempt**, with the bypass fix deployed (`ansible-playbook
... -K -J`; config re-rendered with `forward_bypass_chains:
["KUBE-ROUTER-FORWARD"]`, `rewrite-config` correctly re-rendered
`pia-wg.conf` from the config-only change without a PIA call since a
baseline already existed, recovery timer re-enabled). Left running
unattended for a full **~25.5 hours** before re-checking, deliberately
— the original incident took several minutes and multiple cycles to
manifest, so a short recheck window would have proven nothing.
Confirmed via exact counts, not inference: `journalctl -u
pia-gateway-recover --since "2026-09-22 20:37:00"` showed **1528**
`class=healthy` passes (matches the expected ~1530 at 60s cadence —
the timer ran continuously the whole window), **0** `SAFETY DRIFT`
lines, **0** `class=safety_drift` classifications. `pia-gateway
status` at recheck time: `healthy`, `safety_ok: True`,
`emergency_close_active: False`, `consecutive_failures: 0`,
`last recovery: 1d ago restart` — the last restart was the one from
*before* this fix; none has been needed since. **Step 3 is done.**
kube-router did keep resyncing during this window (that's the whole
premise of the fix); the bypass tolerance handled it silently every
time, with no observable effect on the tunnel.
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
— the original `head -3` suggestion is what hid the first issue; the
drift was 4 rules in, not at the top. **Use `head -2`, not `head -1`**,
for any quick re-check — `-S FORWARD`'s first line is always the `-P`
policy line, never a real rule (this cost a round-trip during the
investigation above).
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`,
@ -499,8 +697,20 @@ hid this the first time; the drift was 4 rules in, not at the top.
should then get past registration. Confirm with `iptables -S FORWARD`
afterwards that the jump is genuinely first, not just that the retry
succeeded.
3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy
pass makes no changes (`journalctl -u pia-gateway-recover -n 5`).
3. Enable: `ansible-playbook ... -K -J` (recovery on; also redeploys
`pia_forward_bypass_chains` into `config.json` if not already applied
— confirm with `sudo pia-gateway --config /etc/pia-gateway/config.json
status` or just `cat /etc/pia-gateway/config.json`). **Watch longer
than one pass** — the live incident here took 2-3 cycles (several
minutes) to manifest, not the first check. `journalctl -u
pia-gateway-recover -f` for at least 5-10 minutes; expect `class=
healthy` throughout, with **zero** `SAFETY DRIFT` lines, even while
`pia-killswitch-reassert` keeps cycling in parallel
(`journalctl -u pia-killswitch-reassert -f` alongside it). If a
`bypass chain KUBE-ROUTER-FORWARD content no longer verified safe`
reason ever appears, treat it as real — it means kube-router's rules
changed to include something that could actually match VLAN 50
traffic, not a false positive to relax further.
4. Drill the silent death. First get the **live WireGuard UDP endpoint**
from the interface itself — **not** port 1337, which is only the
`addKey` registration API port and has nothing to do with the tunnel's
@ -575,8 +785,10 @@ touching `wg0`, k3s, or the host default route):
```bash
sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-killswitch-reassert.timer pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \
/etc/systemd/system/pia-killswitch-reassert.service \
/etc/systemd/system/pia-killswitch-reassert.timer \
/etc/systemd/system/pia-gateway-healthcheck.service \
/etc/systemd/system/pia-gateway-healthcheck.timer \
/etc/systemd/system/pia-gateway-recover.service \

View File

@ -94,6 +94,35 @@ pia_iptables_chain: PIA-VLAN50
# {{ pia_wg_interface }} rather than enumerating the normal interface.
pia_lan_interface: enp1s0
# FORWARD chains the controller may tolerate sitting AHEAD of our own
# kill-switch jump, on the condition that their OWN current content is
# re-verified — live, on every check, never a one-time trust — to be
# structurally unable to match {{ pia_vlan50_subnet }} traffic. Empty (the
# safe default for any other host) means the literal, original "must be
# absolutely first" check. Set to ["KUBE-ROUTER-FORWARD"] here because
# k3s's kube-router network-policy controller and dockerd both also
# insert unconditional jumps near the top of FORWARD, independently of
# this role, and — confirmed live, 2026-09-22 — kube-router resyncs (and
# re-asserts its own top-of-chain position) roughly every 60-90s on this
# cluster's normal CronJob/pod churn, not just at boot/restart scale as
# first assumed. Treating every one of those transient reorderings as a
# full safety-drift emergency (which unconditionally closes table pia's
# route) caused a real live outage: a repeating cycle of recover-succeeds
# -> drift-detected-within-a-minute -> emergency-close -> counter reset
# -> repeat, which kept VLAN 50 open for only seconds at a time. Only
# kube-router is listed — NOT dockerd's DOCKER-USER/DOCKER-FORWARD —
# because kube-router was the only one observed reasserting position
# repeatedly (dockerd sets its hooks up once at daemon start, not on a
# recurring cycle), and DOCKER-FORWARD itself jumps to three further
# nested chains (DOCKER-CT, DOCKER-INTERNAL, DOCKER-BRIDGE) never fully
# content-verified — safer to leave those as a hard drift trigger than
# assume they're equally inert. See
# Classifier._forward_jump_effectively_first's own docstring in
# files/pia_gateway.py for exactly what "re-verified safe" requires of a
# bypassed chain's content.
pia_forward_bypass_chains:
- KUBE-ROUTER-FORWARD
# Reverse-path filtering, applied to {{ pia_wg_interface }} only. The
# kernel default on minisforum (conf.default.rp_filter) is already loose
# (2) at time of writing, so new interfaces inherit loose mode — this task

View File

@ -35,6 +35,7 @@ import datetime as _dt
import fcntl
import http.client
import io
import ipaddress
import json
import os
import random
@ -119,6 +120,7 @@ class Config:
vlan50_subnet: str = "10.10.50.0/24"
iptables_chain: str = "PIA-VLAN50"
lan_interface: str = "enp1s0"
forward_bypass_chains: list = dataclasses.field(default_factory=list)
mss_clamp_enabled: bool = True
region_candidates: list = dataclasses.field(default_factory=lambda: ["hk", "taiwan", "japan"])
handshake_max_age_sec: int = 180
@ -509,6 +511,59 @@ def _is_emergency_rule(rule: str) -> bool:
return f"--comment {EMERGENCY_COMMENT}" in rule or f'--comment "{EMERGENCY_COMMENT}"' in rule
_COMMENT_ONLY_RE = re.compile(r'-m comment --comment "[^"]*"')
def _forward_bypass_target(rule: str) -> Optional[str]:
"""If `rule` is an UNCONDITIONAL FORWARD-level jump to another chain
(optionally with only a cosmetic -m comment match, nothing that
actually narrows which packets it applies to), return that chain's
name; otherwise None. A rule with any real matcher (-s/-d/-p/-i/-o/
any -m other than comment) is deliberately never treated as a bypass
candidate here, however harmless it might look — that judgment is
made once, by a human, for the caller's own forward_bypass_chains
list, not inferred from the FORWARD-level rule shape.
"""
stripped = _COMMENT_ONLY_RE.sub("", rule).split()
if len(stripped) == 4 and stripped[0] == "-A" and stripped[1] == "FORWARD" and stripped[2] == "-j":
return stripped[3]
return None
def _cidr_overlaps(value: str, subnet: str) -> bool:
"""True if `value` could overlap `subnet`; conservative (True) on
anything unparseable rather than silently treating it as disjoint."""
try:
return ipaddress.ip_network(value, strict=False).overlaps(ipaddress.ip_network(subnet))
except ValueError:
return True
def _bypass_chain_rule_is_safe(rule: str, vlan50_subnet: str) -> bool:
"""Conservative check for one rule inside a configured bypass chain:
True only if this rule is PROVABLY unable to match traffic sourced
from vlan50_subnet — every -s/-d value on it is a CIDR disjoint from
vlan50_subnet, there is at least one such value (a bare/unconditional
rule is never safe), and there is no negation or other construct this
function doesn't explicitly recognize (fails closed, not open, on
anything it can't prove).
"""
if "!" in rule.split():
return False
tokens = rule.split()
addrs: list[str] = []
i = 0
while i < len(tokens):
if tokens[i] in ("-s", "-d") and i + 1 < len(tokens):
addrs.append(tokens[i + 1])
i += 2
continue
i += 1
if not addrs:
return False
return all(not _cidr_overlaps(a, vlan50_subnet) for a in addrs)
class Classifier:
def __init__(self, cfg: Config, system: System):
self.cfg = cfg
@ -517,6 +572,34 @@ class Classifier:
def _rc(self, argv: list[str]) -> CmdResult:
return self.sys.run(argv, timeout=20)
def _forward_jump_effectively_first(self, effective_rules: list[str]) -> tuple[bool, list[str]]:
"""Walk FORWARD from the top. Tolerate a configured bypass chain
(cfg.forward_bypass_chains) sitting ahead of our own jump ONLY
while that chain's OWN current content is re-verified, live, on
THIS call to be unable to match VLAN 50 traffic — never a one-time
trust. Anything else ahead of our jump (an unrecognized rule, or a
bypass-listed chain whose content no longer verifies safe) fails
closed. Returns (ok, extra_reasons).
"""
cfg = self.cfg
expected_jump = f"-A FORWARD -s {cfg.vlan50_subnet} -j {cfg.iptables_chain}"
extra: list[str] = []
for r in effective_rules:
if r.strip() == expected_jump:
return True, extra
target = _forward_bypass_target(r)
if target is None or target not in cfg.forward_bypass_chains:
return False, extra
chain = self._rc(["iptables", "-S", target])
if chain.rc != 0:
extra.append(f"bypass chain {target} unreadable")
return False, extra
chain_rules = [l for l in chain.out.splitlines() if l.startswith(f"-A {target}")]
if not chain_rules or not all(_bypass_chain_rule_is_safe(cr, cfg.vlan50_subnet) for cr in chain_rules):
extra.append(f"bypass chain {target} content no longer verified safe")
return False, extra
return False, extra
def classify(self, state: dict) -> Status:
cfg = self.cfg
now = self.sys.now()
@ -636,6 +719,7 @@ class Classifier:
fwd = self._rc(["iptables", "-S", "FORWARD"])
jump_first = False
emergency_present = False
jump_bypass_reasons: list[str] = []
if fwd.rc == 0:
rules = [l for l in fwd.out.splitlines() if l.startswith("-A FORWARD")]
effective = []
@ -644,15 +728,15 @@ class Classifier:
emergency_present = True
continue
effective.append(r)
expected_jump = f"-A FORWARD -s {cfg.vlan50_subnet} -j {cfg.iptables_chain}"
jump_first = bool(effective) and effective[0].strip() == expected_jump
jump_first, jump_bypass_reasons = self._forward_jump_effectively_first(effective)
else:
unknown = True
checks["forward_jump_first"] = jump_first
checks["emergency_rule_present"] = emergency_present
if not jump_first:
safety_ok = False
reasons.append("FORWARD jump to kill-switch chain is not first")
detail = f" ({'; '.join(jump_bypass_reasons)})" if jump_bypass_reasons else ""
reasons.append(f"FORWARD jump to kill-switch chain is not first{detail}")
# -- chain contents
chain = self._rc(["iptables", "-S", cfg.iptables_chain])
@ -751,33 +835,14 @@ class Classifier:
# --------------------------------------------------------------------------
class _PinnedHTTPSConnection(http.client.HTTPSConnection):
"""Connect to a fixed IP while verifying TLS against the hostname (SNI)."""
def __init__(self, host: str, ip: str, port: int, context: ssl.SSLContext, connect_timeout: float, total_timeout: float):
super().__init__(host, port, timeout=connect_timeout, context=context)
self._pinned_ip = ip
self._total_timeout = total_timeout
def connect(self) -> None: # noqa: D401
sock = socket.create_connection((self._pinned_ip, self.port), self.timeout)
sock.settimeout(self._total_timeout)
self.sock = self._context.wrap_socket(sock, server_hostname=self.host)
class HttpClient:
def __init__(self, cfg: Config):
def __init__(self, cfg: Config, system: Optional[System] = None):
self.cfg = cfg
self.system = system or System()
def _system_ctx(self) -> ssl.SSLContext:
return ssl.create_default_context()
def _pia_ctx(self) -> ssl.SSLContext:
ctx = ssl.create_default_context(cafile=self.cfg.ca_file)
ctx.check_hostname = True
ctx.verify_mode = ssl.CERT_REQUIRED
return ctx
def _request(self, conn: http.client.HTTPSConnection, method: str, path: str, body: Optional[bytes], headers: dict) -> tuple[int, bytes]:
try:
conn.request(method, path, body=body, headers=headers)
@ -785,13 +850,22 @@ class HttpClient:
data = resp.read(4_000_000)
return resp.status, data
except ssl.SSLCertVerificationError as e:
raise ApiError("tls", "certificate verification failed") from e
# verify_message/reason are standard OpenSSL diagnostic text
# (e.g. "unable to get local issuer certificate", "certificate
# has expired", a hostname mismatch) — never a secret, and
# exactly what's needed to tell "wrong CA", "expired cert",
# "clock skew", and "hostname mismatch" apart instead of
# collapsing all of them into one opaque string.
detail = e.verify_message or e.reason or str(e) or "certificate verification failed"
raise ApiError("tls", detail) from e
except ssl.SSLError as e:
raise ApiError("tls", e.__class__.__name__) from e
detail = getattr(e, "reason", None) or str(e) or e.__class__.__name__
raise ApiError("tls", f"{e.__class__.__name__}: {detail}") from e
except socket.timeout as e:
raise ApiError("timeout", "request timed out") from e
except (OSError, http.client.HTTPException) as e:
raise ApiError("network", e.__class__.__name__) from e
detail = str(e) or e.__class__.__name__
raise ApiError("network", f"{e.__class__.__name__}: {detail}") from e
finally:
with contextlib.suppress(Exception):
conn.close()
@ -816,12 +890,68 @@ class HttpClient:
}
return self._request(conn, "POST", u.path, body, headers)
def get_pinned(self, hostname: str, ip: str, port: int, path: str, query: dict[str, str]) -> tuple[int, bytes]:
conn = _PinnedHTTPSConnection(
hostname, ip, port, self._pia_ctx(), self.cfg.http_connect_timeout_sec, self.cfg.http_total_timeout_sec
)
full = path + "?" + urllib.parse.urlencode(query)
return self._request(conn, "GET", full, None, {"User-Agent": "pia-gateway"})
def get_pinned_curl(self, hostname: str, ip: str, port: int, path: str, token: str, pubkey: str) -> tuple[int, bytes]:
"""Pinned-IP, SNI-validated, CA-pinned GET — for PIA's addKey only.
This shells out to curl instead of using the stdlib `ssl` module.
Confirmed live (2026-09-22): PIA's own CA certificate (byte-
identical to a fresh fetch from PIA's own repo — not a stale
vendored copy) has a non-critical Basic Constraints extension on
its CA cert, a real RFC 5280 non-conformance in a CA we don't
control. Recent OpenSSL (3.2+; confirmed on minisforum's 3.5.7 via
a direct repro — SSLCertVerificationError, verify_message
"Basic Constraints of CA cert not marked critical") rejects that
in its default chain verification, a check promoted from
X509_STRICT-only to the default path in recent OpenSSL releases.
Python's stdlib `ssl` module has no custom verify-callback hook to
selectively tolerate just this one known, third-party-controlled
defect while still fully validating everything else (chain,
expiry, hostname/SAN) — that gap is long-standing and
deliberate upstream, not something to route around by weakening
verify_mode. curl's TLS stack on this host does not enforce this
specific check and is what PIA's own reference implementation
(connect_to_wireguard_with_token.sh) already uses for this exact
endpoint — reusing it here with --cacert (never -k/--insecure) is
not a security downgrade, just a different, already-proven-correct
TLS backend for the one endpoint that needs it. serverlist and
token still use the stdlib ssl path above against public WebPKI
certs, which do not have this problem.
"""
argv = [
"curl", "--silent", "--show-error", "--fail", "--location", "-G",
"--connect-timeout", str(self.cfg.http_connect_timeout_sec),
"--max-time", str(self.cfg.http_total_timeout_sec),
"--cacert", self.cfg.ca_file,
"--connect-to", f"{hostname}::{ip}:",
"--data-urlencode", "pt@-",
"--data-urlencode", f"pubkey={pubkey}",
f"https://{hostname}:{port}{path}",
]
r = self.system.run(argv, stdin=token, timeout=self.cfg.http_total_timeout_sec + 10)
if r.rc == 0:
return 200, r.out.encode("utf-8")
err = (r.err or "").strip()
# Exit codes are the primary signal, but not fully portable across
# curl builds/TLS backends — confirmed live: a curl built against
# SecureTransport reported exit 56 (CURLE_RECV_ERROR) for the exact
# same --fail-triggered HTTP rejection GNU/OpenSSL curl reports as
# 22. curl's own error text ("returned error: NNN" for --fail,
# "timed out" for CURLE_OPERATION_TIMEDOUT, TLS-related wording for
# cert/handshake problems) is a second, more portable signal — used
# here as a fallback classifier alongside the exit code, not a
# replacement for it.
if r.rc == 127:
raise ApiError("network", "curl not found on PATH")
if r.rc == 28 or "timed out" in err.lower():
raise ApiError("timeout", f"curl: connection or transfer timed out: {err}")
if r.rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83) or "ssl" in err.lower() or "certificate" in err.lower():
raise ApiError("tls", f"curl TLS error (exit {r.rc}): {err}")
if r.rc == 22 or "returned error:" in err:
# --fail suppresses the body on a non-2xx response, so there is
# no status code or body to parse here — same limitation the
# legacy curl-based Ansible path already had.
raise ApiError("http", f"curl: non-2xx HTTP response (exit {r.rc}): {err}")
raise ApiError("network", f"curl exit {r.rc}: {err}")
@dataclasses.dataclass
@ -933,7 +1063,7 @@ class PiaApi:
raise ApiError("http", f"token HTTP {last_status}")
def add_key(self, cand: Candidate, token: str, pubkey: str) -> PeerInfo:
status, data = self.http.get_pinned(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", {"pt": token, "pubkey": pubkey})
status, data = self.http.get_pinned_curl(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", token, pubkey)
if status != 200:
raise ApiError("http", f"addKey HTTP {status}")
if not data.strip():
@ -1124,7 +1254,7 @@ class Controller:
self.cfg = cfg
self.sys = system
self.log = log
self.http = http or HttpClient(cfg)
self.http = http or HttpClient(cfg, system)
self._creds = creds
self.store = StateStore(cfg, system)
self.classifier = Classifier(cfg, system)

View File

@ -21,6 +21,9 @@
# killswitch script changed in the same run), pia-wg's own PostUp
# needs to run last so its "open" route is the final word, not
# killswitch's route-table seeding.
# 4. "Restart pia-killswitch reassertion timer" only restarts the
# *timer* (a scheduling change, e.g. its interval), never runs the
# script itself — no ordering constraint against the others.
- name: Reload systemd daemon
ansible.builtin.systemd:
@ -35,6 +38,11 @@
name: pia-killswitch.service
state: restarted
- name: Restart pia-killswitch reassertion timer
ansible.builtin.systemd:
name: pia-killswitch-reassert.timer
state: restarted
- name: Restart pia-wg
ansible.builtin.systemd:
name: "wg-quick@{{ pia_wg_interface }}"

View File

@ -27,10 +27,16 @@
- wireguard-tools
- iproute2
- python3
- curl
state: present
update_cache: true
# python3-requests is no longer needed: the controller uses only the
# standard library.
# standard library, EXCEPT for the addKey endpoint specifically, which
# shells out to curl — see HttpClient.get_pinned_curl's own docstring
# for why (a non-critical Basic Constraints extension on PIA's own CA
# cert that recent OpenSSL rejects by default and Python's ssl module
# has no way to selectively tolerate). curl is virtually always already
# present on Debian, but declared explicitly rather than assumed.
- name: Pin reverse-path filtering for {{ pia_wg_interface }}
ansible.posix.sysctl:

View File

@ -40,3 +40,35 @@
name: pia-killswitch.service
enabled: true
state: started
- name: Deploy the kill-switch periodic reassertion systemd service
ansible.builtin.template:
src: pia-killswitch-reassert.service.j2
dest: /etc/systemd/system/pia-killswitch-reassert.service
mode: "0644"
owner: root
group: root
notify:
- Reload systemd daemon
- Restart pia-killswitch reassertion timer
- name: Deploy the kill-switch periodic reassertion systemd timer
ansible.builtin.template:
src: pia-killswitch-reassert.timer.j2
dest: /etc/systemd/system/pia-killswitch-reassert.timer
mode: "0644"
owner: root
group: root
notify:
- Reload systemd daemon
- Restart pia-killswitch reassertion timer
- name: Enable and start the kill-switch reassertion timer
ansible.builtin.systemd:
name: pia-killswitch-reassert.timer
enabled: true
state: started
daemon_reload: true
# Unconditional — not gated by pia_auto_recovery_enabled. See the
# service unit's own header for why this is scoped differently from
# the automatic recovery/registration timers.

View File

@ -10,6 +10,7 @@
"vlan50_subnet": {{ pia_vlan50_subnet | to_json }},
"iptables_chain": {{ pia_iptables_chain | to_json }},
"lan_interface": {{ pia_lan_interface | to_json }},
"forward_bypass_chains": {{ pia_forward_bypass_chains | to_json }},
"mss_clamp_enabled": {{ pia_mss_clamp_enabled | bool | to_json }},
"region_candidates": {{ pia_region_candidates | to_json }},
"handshake_max_age_sec": {{ pia_healthcheck_handshake_max_age_sec | int }},

View File

@ -0,0 +1,30 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Periodically re-runs the kill-switch seed script (pia-killswitch.sh) so
# its invariants — source rule, table default, and specifically the
# FORWARD jump's position — are re-asserted on a schedule, not only at
# the external trigger points that used to be the only times it ran
# (boot, a UFW reload, or an Ansible-detected content change to the
# script itself). Needed because other host components that also manage
# iptables — kube-router and dockerd — reassert their OWN position-1
# jumps independently, and confirmed live (2026-09-22) to do so more
# often than originally assumed: the FORWARD jump drifted twice within
# about an hour under this cluster's ordinary CronJob/pod churn (ca-sync,
# porkbun-ddns, gitea-backup, ...), not just at boot-scale events.
#
# The script itself is already fully idempotent (safe to run repeatedly
# — see its own header comment); this unit just gives it more chances to
# run. Deliberately separate from pia-gateway-recover.timer and NOT
# gated by pia_auto_recovery_enabled: this is the kill switch continuing
# to assert its own already-established, narrowly-scoped state — the
# same kind of thing it already does unconditionally at boot and after a
# UFW reload, just on a schedule too — not the higher-level automatic
# recovery/registration logic the plan was deliberately cautious about
# running unattended.
[Unit]
Description=Periodically re-assert the PIA gateway kill switch's FORWARD jump position
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pia-killswitch.sh
LimitCORE=0

View File

@ -0,0 +1,11 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
[Unit]
Description=Run the PIA gateway kill-switch reassertion every {{ pia_healthcheck_interval_sec }}s
[Timer]
OnBootSec=30
OnUnitActiveSec={{ pia_healthcheck_interval_sec }}
AccuracySec=5s
[Install]
WantedBy=timers.target

View File

@ -1,12 +1,16 @@
#!/bin/bash
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Runs once at boot, before wg-quick@{{ pia_wg_interface }}.service
# (enforced by the systemd drop-in in pia-killswitch.service's
# Before=/wg-quick's Requires=After=), and again any time UFW is
# reloaded (see handlers/main.yaml — "Reload ufw" fires before "Restart
# pia-killswitch" so a reload can never leave a permanent gap). Seeds the
# closed state that PIA gateway's kill switch relies on:
# Runs at boot, before wg-quick@{{ pia_wg_interface }}.service (enforced
# by the systemd drop-in in pia-killswitch.service's Before=/wg-quick's
# Requires=After=); any time UFW is reloaded (see handlers/main.yaml —
# "Reload ufw" fires before "Restart pia-killswitch" so a reload can
# never leave a permanent gap); and — since 2026-09-22, after the FORWARD
# jump was found drifting twice within about an hour under this
# cluster's ordinary CronJob/pod churn, far more often than a boot/reload
# -only cadence catches — every {{ pia_healthcheck_interval_sec }}s via
# pia-killswitch-reassert.timer. Seeds the closed state that PIA
# gateway's kill switch relies on:
#
# - a source rule sending {{ pia_vlan50_subnet }} to the dedicated
# "{{ pia_route_table_name }}" table, evaluated after the kernel's

View File

@ -88,6 +88,7 @@ class FakeSystem(pg.System):
self.stdins: list[str] = []
self.genkey_queue = [PRIV_B]
self.jitter = 0.5
self.extra_chains: dict = {} # name -> list[str] | None ("-A <name> ..." lines, or None = missing)
@staticmethod
def open_chain():
@ -206,6 +207,12 @@ class FakeSystem(pg.System):
if self.chain_rules is None:
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
return pg.CmdResult(0, f"-N {CHAIN}\n" + "\n".join(self.chain_rules) + "\n", "")
if a[2] in self.extra_chains:
rules = self.extra_chains[a[2]]
if rules is None:
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
return pg.CmdResult(0, f"-N {a[2]}\n" + "\n".join(rules) + "\n", "")
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
if a[:2] == ["iptables", "-C"]:
rule = "-A " + " ".join(a[2:])
return pg.CmdResult(0 if rule in self.forward_rules else 1, "", "")
@ -255,8 +262,8 @@ class FakeHttp:
return 200, val
return 200, json.dumps({"token": val}).encode()
def get_pinned(self, hostname, ip, port, path, query):
self.calls.append(("addkey", hostname, ip, port, path, dict(query)))
def get_pinned_curl(self, hostname, ip, port, path, token, pubkey):
self.calls.append(("addkey", hostname, ip, port, path, {"pt": token, "pubkey": pubkey}))
kind, val = self.addkey.get(hostname, ("http", 502))
if kind == "err":
raise pg.ApiError(val, "fake")
@ -495,6 +502,139 @@ class TestSafetyDrift(Base):
self.assertEqual(self.http.calls, [])
KUBE_ROUTER_FORWARD_JUMP = '-A FORWARD -m comment --comment "kube-router netpol - TEMCG2JMHZYE7H7T" -j KUBE-ROUTER-FORWARD'
KUBE_ROUTER_SAFE_RULES = [
'-A KUBE-ROUTER-FORWARD -d 10.42.0.39/32 -m comment --comment "pod X" -j KUBE-POD-FW-AAAA',
'-A KUBE-ROUTER-FORWARD -s 10.42.0.39/32 -m comment --comment "pod X" -j KUBE-POD-FW-AAAA',
'-A KUBE-ROUTER-FORWARD -d 10.42.1.5/32 -m physdev --physdev-is-bridged -m comment --comment "pod Y" -j KUBE-POD-FW-BBBB',
]
class TestForwardBypassChains(Base):
"""The content-verified bypass mechanism for kube-router's own FORWARD
jump — found necessary live, 2026-09-22, after the plain reposition
fix alone caused a repeating outage loop: kube-router resyncs (and
re-pushes our jump down) roughly every 60-90s on this cluster, far
more often than assumed, and treating every transient reordering as a
full safety-drift emergency (closing the route each time) kept the
tunnel down far more than it was up.
"""
def classify_with_forward(self, forward_rules, bypass_chains=(), extra_chains=None):
self.cfg.forward_bypass_chains = list(bypass_chains)
self.sysm.forward_rules = list(forward_rules)
if extra_chains:
self.sysm.extra_chains.update(extra_chains)
return self.ctl.classifier.classify(pg.default_state())
def test_disabled_by_default_kube_router_ahead_still_blocks(self):
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=(), # default: nothing configured
)
self.assertFalse(st.safety_ok)
self.assertIn("FORWARD jump to kill-switch chain is not first", " ".join(st.reasons))
def test_configured_bypass_with_verified_safe_content_tolerated(self):
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
)
self.assertTrue(st.safety_ok, st.reasons)
def test_duplicate_bypass_jump_tolerated(self):
# The exact live anomaly: kube-router left two copies of its own
# jump behind after a resync.
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
)
self.assertTrue(st.safety_ok, st.reasons)
def test_bypass_chain_picking_up_a_matching_source_still_blocks(self):
dangerous = KUBE_ROUTER_SAFE_RULES + [f"-A KUBE-ROUTER-FORWARD -s {SUBNET} -j ACCEPT"]
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
)
self.assertFalse(st.safety_ok)
self.assertIn("no longer verified safe", " ".join(st.reasons))
def test_bypass_chain_with_bare_unconditional_rule_blocks(self):
dangerous = KUBE_ROUTER_SAFE_RULES + ["-A KUBE-ROUTER-FORWARD -j ACCEPT"]
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
)
self.assertFalse(st.safety_ok)
def test_bypass_chain_with_negated_match_blocks(self):
dangerous = [f'-A KUBE-ROUTER-FORWARD ! -s {SUBNET} -j ACCEPT']
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
)
self.assertFalse(st.safety_ok)
def test_unrecognized_chain_ahead_still_blocks(self):
# DOCKER-USER is not on the allowlist even when KUBE-ROUTER-FORWARD is.
st = self.classify_with_forward(
["-A FORWARD -j DOCKER-USER", f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"DOCKER-USER": []},
)
self.assertFalse(st.safety_ok)
def test_forward_level_rule_with_real_matcher_not_treated_as_bypass(self):
# Even though KUBE-ROUTER-FORWARD is allowlisted and its content is
# safe, a FORWARD-level rule that itself carries a real matcher
# (not just a cosmetic comment) is never treated as a bypass jump.
st = self.classify_with_forward(
[f"-A FORWARD -s 1.2.3.0/24 -j KUBE-ROUTER-FORWARD", f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
)
self.assertFalse(st.safety_ok)
def test_missing_bypass_chain_blocks(self):
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": None},
)
self.assertFalse(st.safety_ok)
self.assertIn("unreadable", " ".join(st.reasons))
def test_end_to_end_recover_does_not_emergency_close_on_tolerated_drift(self):
# The actual live incident, reproduced end to end: kube-router's
# jump sits ahead of ours, content-verified-safe, and the recovery
# timer must NOT declare safety drift or touch the route.
self.cfg.forward_bypass_chains = ["KUBE-ROUTER-FORWARD"]
self.sysm.forward_rules = [KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"]
self.sysm.extra_chains["KUBE-ROUTER-FORWARD"] = KUBE_ROUTER_SAFE_RULES
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_OK)
self.assertFalse(self.state()["emergency_close_active"])
self.assertEqual(self.sysm.table_default, "open")
self.assertEqual(self.sysm.restart_count, 0)
def test_end_to_end_recover_still_emergency_closes_on_real_drift(self):
# Regression guard: an ACTUALLY unrecognized rule ahead of ours
# must still trigger the full emergency-close path, even with a
# bypass list configured for something else.
self.cfg.forward_bypass_chains = ["KUBE-ROUTER-FORWARD"]
self.sysm.forward_rules = ["-A FORWARD -j DOCKER-USER", f"-A FORWARD -s {SUBNET} -j {CHAIN}"]
self.sysm.extra_chains["DOCKER-USER"] = []
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_SAFETY)
self.assertTrue(self.state()["emergency_close_active"])
class TestRecoveryStateMachine(Base):
def test_healthy_runs_are_idempotent_and_do_not_mutate(self):
self.run_recover_n(3)
@ -1227,6 +1367,106 @@ def dataclass_dict(cfg):
return dataclasses.asdict(cfg)
class FakeCurlSystem(pg.System):
"""Fakes just enough of System.run to exercise get_pinned_curl's argv
construction, stdin handling, and exit-code classification, without a
real curl binary or network access."""
def __init__(self, result=None):
self.result = result or pg.CmdResult(0, '{"status":"OK"}', "")
self.calls = []
self.stdins = []
def run(self, argv, stdin=None, timeout=60):
self.calls.append(list(argv))
if stdin is not None:
self.stdins.append(stdin)
return self.result
class TestPinnedCurlAddKey(unittest.TestCase):
def setUp(self):
self.cfg = pg.Config(ca_file="/etc/wireguard/pia-ca.crt", http_connect_timeout_sec=10, http_total_timeout_sec=30)
def test_argv_shape_matches_the_proven_legacy_curl_invocation(self):
sysm = FakeCurlSystem()
http_client = pg.HttpClient(self.cfg, sysm)
status, data = http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
self.assertEqual(status, 200)
self.assertEqual(data, b'{"status":"OK"}')
argv = sysm.calls[0]
self.assertEqual(argv[0], "curl")
self.assertIn("--fail", argv)
self.assertIn("-G", argv)
self.assertIn("--cacert", argv)
self.assertEqual(argv[argv.index("--cacert") + 1], "/etc/wireguard/pia-ca.crt")
self.assertIn("--connect-to", argv)
self.assertEqual(argv[argv.index("--connect-to") + 1], "Server-11669-10a::69.33.198.97:")
self.assertIn("pt@-", argv)
self.assertIn(f"pubkey={PUB_A}", argv)
self.assertEqual(argv[-1], "https://Server-11669-10a:1337/addKey")
# never -k/--insecure — this must still be a real, validating request
self.assertNotIn("-k", argv)
self.assertNotIn("--insecure", argv)
def test_token_passed_via_stdin_with_no_trailing_newline_and_never_in_argv(self):
sysm = FakeCurlSystem()
http_client = pg.HttpClient(self.cfg, sysm)
http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
self.assertEqual(sysm.stdins, [SECRET_TOKEN]) # exact match: no appended "\n"
for arg in sysm.calls[0]:
self.assertNotIn(SECRET_TOKEN, arg)
def test_timeout_exit_code_classified(self):
sysm = FakeCurlSystem(pg.CmdResult(28, "", "curl: (28) Connection timed out"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "timeout")
def test_tls_exit_codes_classified_with_stderr_detail(self):
for rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83):
with self.subTest(rc=rc):
sysm = FakeCurlSystem(pg.CmdResult(rc, "", "curl: (60) SSL certificate problem"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "tls")
self.assertIn("SSL certificate problem", cm.exception.detail)
def test_http_failure_exit_code_classified(self):
sysm = FakeCurlSystem(pg.CmdResult(22, "", "curl: (22) The requested URL returned error: 401"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "http")
def test_nonstandard_exit_code_with_fail_text_still_classified_as_http(self):
# Observed live: curl built against SecureTransport reports exit 56
# (CURLE_RECV_ERROR), not the "usual" 22, for the same --fail
# -triggered HTTP rejection — the error text is the portable signal.
sysm = FakeCurlSystem(pg.CmdResult(56, "", "curl: (56) The requested URL returned error: 401"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "http")
def test_curl_missing_classified_as_network_not_a_crash(self):
sysm = FakeCurlSystem(pg.CmdResult(127, "", "curl: not found"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "network")
def test_unclassified_exit_code_falls_back_to_network_with_stderr(self):
sysm = FakeCurlSystem(pg.CmdResult(7, "", "curl: (7) Couldn't connect to server"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "network")
self.assertIn("Couldn't connect", cm.exception.detail)
class TestHttpHelpers(unittest.TestCase):
def test_key_and_ip_validators(self):
self.assertTrue(pg.valid_wg_key(PUB_A))