Compare commits

..

4 Commits

Author SHA1 Message Date
89d289addf feat: add periodic kill-switch reassertion timer
Found live, 2026-09-22: after fixing pia-killswitch.sh's FORWARD-jump
repositioning and getting one pia-gateway register attempt past the
safety check, the very next attempt - roughly an hour later, no
reboot, no UFW reload, no script content change in between - hit the
identical "FORWARD jump to kill-switch chain is not first" failure
again. This disproves the earlier assumption that kube-router/dockerd
reassert their own position-1 jumps only at boot/restart scale:
kube-router resyncs on ordinary Kubernetes API events, and this
cluster's CronJobs (ca-sync, porkbun-ddns, gitea-backup) each
create/delete a Pod every few minutes - plausible triggers for
kube-router's own reconciliation loop, and therefore plausible
triggers for this drift, recurring on a timescale of minutes.

The reposition logic itself (previous commit) is correct - it just
only ran on the existing triggers (boot, UFW reload, an
Ansible-detected script content change), none of which fired between
the two attempts.

Adds pia-killswitch-reassert.timer/.service: re-runs the already fully
idempotent pia-killswitch.sh every pia_healthcheck_interval_sec (same
cadence as the health check), giving it regular chances to run instead
of only sporadic ones. Deliberately separate from
pia-gateway-recover.timer and NOT gated by pia_auto_recovery_enabled -
this is the kill switch continuing to assert its own already-
established, narrowly-scoped state (source rule, table default,
FORWARD jump position), the same kind of thing it already does
unconditionally at boot and after a UFW reload, just on a schedule
too - not the higher-level automatic recovery/registration logic the
plan was deliberately cautious about running unattended.

Unblocked live in the meantime with a manual
`sudo systemctl restart pia-killswitch.service`.

Offline checks: yamllint, ansible-playbook --syntax-check, rendered
templates (Jinja2 + bash -n on the updated script), ansible-lint
(diffed against main - no new finding categories), full pia_gateway.py
test suite (94/94, unaffected by this change) all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 19:08:28 +09:00
99f6ef84f0 fix: use curl for addKey — PIA's CA cert fails modern OpenSSL by default
Found live, 2026-09-22, on the addKey step of the first successful
kill-switch-check pass: every candidate failed with
kind=tls detail="certificate verification failed". Root-caused on
minisforum: PIA's own CA certificate (files/pia-ca.crt) has a CA:TRUE
Basic Constraints extension that isn't marked critical - a real RFC
5280 non-conformance, confirmed not a stale vendored copy (a fresh
fetch from pia-foss/manual-connections is byte-identical, same
sha256 already recorded in the file's own header, same defect).
Recent OpenSSL (3.2+; confirmed via a direct repro against
minisforum's 3.5.7 - SSLCertVerificationError, verify_message
"Basic Constraints of CA cert not marked critical") rejects this by
default, a check promoted from X509_STRICT-only into the baseline
chain-verification path. openssl s_client (LibreSSL, this dev
machine) and curl both accept the identical cert/chain/CA file - only
Python's ssl module, via this host's system OpenSSL, rejects it. This
went unnoticed until now because the three prior successful
registrations all went through curl --cacert (the legacy Ansible
path); this was the first time the Python ssl code path ran for real.

Python's stdlib ssl module has no custom verify-callback hook, so
there's no clean way to tolerate just this one known,
third-party-controlled defect while keeping full validation for
everything else. Rather than weakening verify_mode or adding a new
dependency (pyOpenSSL, for a real callback - considered, rejected as
a bigger lift than necessary) to route around a gap in a library we
don't control, addKey now shells out to curl
(HttpClient.get_pinned_curl, curl added to this role's apt list) -
the same tool PIA's own reference client already uses for this exact
endpoint, with --cacert and never -k/--insecure, so this is a
different, already-proven-correct TLS backend for the one endpoint
that needs it, not a downgrade. serverlist/token are unaffected
(public WebPKI certs) and still use the stdlib ssl path.

Verified: a real subprocess curl call against the actual failing
candidate (Server-11669-10a / 69.33.198.97) gets a genuine HTTP
response from PIA, proving TLS validation succeeds via this path.
Also found and handled: curl's own exit code for a --fail-triggered
rejection isn't fully portable (macOS/SecureTransport curl reports 56
where GNU/OpenSSL curl reports 22 for the same case) - classification
now also matches curl's own error text as a second signal, not just
the exit code.

Added 8 tests (94 total, all passing) covering the curl invocation
shape, stdin token delivery (no trailing newline, never in argv), and
exit-code + text-based error classification including the
cross-platform exit-code case found live.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 18:48:48 +09:00
7b78e4a88e fix: surface the real OpenSSL error instead of a generic TLS message
_request()'s ssl.SSLCertVerificationError handler hardcoded "certificate
verification failed" regardless of cause, discarding e.verify_message/
e.reason (standard OpenSSL diagnostic text - never a secret) - exactly
what's needed to tell a wrong CA, expired cert, clock skew, and a
hostname mismatch apart. Found while diagnosing a live addKey TLS
failure on minisforum that doesn't reproduce from this machine (same
CA file, same pinned IP, same SNI hostname - openssl s_client and a
direct repro of get_pinned() both succeed here), so the next attempt
needs the real reason, not a collapsed generic string. Also stopped
discarding detail on the generic SSLError/OSError branches.

Verified: forcing a real verification failure (wrong CA) now reports
"unable to get local issuer certificate" instead of the old generic
text. Full test suite still passes (86/86).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 18:30:35 +09:00
de5ce7bf52 fix: reposition pia-killswitch's FORWARD jump, not just check it exists
Found live on the first deployment of the auto-recovery controller
(2026-09-22): pia-gateway register correctly refused with "FORWARD
jump to kill-switch chain is not first". Root cause confirmed on
minisforum via iptables -S FORWARD/-S KUBE-ROUTER-FORWARD/-S
DOCKER-USER/-S DOCKER-FORWARD: the PIA-VLAN50 jump was 4th, behind
kube-router's and dockerd's own jumps, which each independently
reassert position 1 on their own resyncs/restarts (kube-router had
even left a duplicate of its own rule behind). pia-killswitch.sh.j2
only checked whether its jump existed (iptables -C) before deciding
not to touch anything — existence isn't proof of position, so once
pushed down it stayed down.

Confirmed safe to reposition unconditionally by reading the actual
content of what's ahead of it rather than assuming: KUBE-ROUTER-FORWARD
only matches specific pod /32 addresses, DOCKER-USER was empty, and
DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50
arrives via enp1s0) - none can intercept VLAN 50 traffic regardless of
order. Fixed the script itself (delete + reinsert at position 1 if the
current first rule isn't ours) rather than loosening the Python
controller's forward_jump_first safety check, which was working
correctly when it refused to register against the drifted state.

Verified offline: rendered template passes bash -n, and the exact
live topology (rule present but 4th) reproduced against a bash mock
of iptables, confirming reposition-when-drifted, idempotency when
already first, and insert-when-absent all behave correctly. No real
iptables/wg/ip on this dev machine to test further; not yet
re-applied or re-verified against minisforum.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 16:46:30 +09:00
9 changed files with 460 additions and 53 deletions

View File

@ -163,6 +163,48 @@ real.
Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW
**input** allow for 53/tcp+udp — it's locally-terminated traffic, not **input** allow for 53/tcp+udp — it's locally-terminated traffic, not
routed — see `tasks/firewall.yaml`. routed — see `tasks/firewall.yaml`.
**The FORWARD jump's position is actively re-asserted on every run, not
just seeded once** — found live, 2026-09-22, on the very first
deployment of the auto-recovery controller. minisforum's `FORWARD`
chain default policy is `DROP` (kube-router's doing, not this role's —
correcting an assumption an earlier version of this doc made), and
kube-router and `dockerd` each independently reassert their **own**
jump at position 1 whenever they resync or restart, silently pushing
this rule down without removing it (kube-router doesn't even dedupe
its own older copy when it does — a duplicate of *its* rule was found
live too). The script used to check only whether its rule *existed*
(`iptables -C`) before deciding not to touch anything; existence isn't
proof of position, so once pushed down it stayed down until something
else fixed it. It now checks the actual first rule and repositions
(delete + reinsert at 1) if it isn't this one. Confirmed safe to do
unconditionally by reading kube-router's and Docker's actual chain
contents rather than assuming: `KUBE-ROUTER-FORWARD` only matches
specific pod `/32` addresses (disjoint from `10.10.50.0/24`),
`DOCKER-USER` was empty, and `DOCKER-FORWARD`'s one direct rule is
scoped to `-i docker0` (VLAN 50 traffic arrives via `enp1s0`) — none
of them can intercept VLAN 50 traffic regardless of order, so moving
this rule back to position 1 relative to them changes nothing for pod
or Docker-bridge traffic. The controller's own safety check
(`forward_jump_first`) was working as intended when it refused to
register against the drifted state — this is a fix to the thing it
checks, not a loosening of the check itself.
**This recurs far more often than "boot-scale," and the fix above only
helped once.** The reposition logic is correct, but it only *ran* on
the existing triggers (boot, a UFW reload, an Ansible-detected content
change to the script). Confirmed live: after that fix landed and got
one `pia-gateway register` attempt past this check, the very next
attempt — roughly an hour later, no reboot, no UFW reload, no script
change in between — hit the identical failure again. kube-router
resyncs on ordinary Kubernetes API events (this cluster's CronJobs —
`ca-sync`, `porkbun-ddns`, `gitea-backup` — each create and delete a
Pod every few minutes, each a plausible trigger), not just at
boot/restart scale as originally assumed. `pia-killswitch-reassert
.timer` (every `pia_healthcheck_interval_sec`, unconditional, not
gated by `pia_auto_recovery_enabled`) exists specifically to close this
gap by giving the already-correct, already-idempotent script regular
chances to run instead of only sporadic ones.
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on - `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
top of that closed baseline: default route via `pia-wg` in the `pia` top of that closed baseline: default route via `pia-wg` in the `pia`
table, ACCEPT + established/related return inserted into the table, ACCEPT + established/related return inserted into the
@ -242,10 +284,14 @@ handled automatically — see the next section.
## Automatic recovery ## Automatic recovery
`files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway` `files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway`
(root, `0700`, standard library only — runs on the host's `python3.13`). (root, `0700`, runs on the host's `python3.13`). Almost entirely standard
It is **one** implementation of PIA registration shared by the unattended library — the one exception is `addKey` specifically, which shells out to
timers and by this role's own registration step, so the two can never `curl` (now an explicit `apt` dependency of this role); see
drift. Layout on the host: `HttpClient.get_pinned_curl`'s own docstring and "PIA's CA certificate and
why addKey uses curl, not Python's `ssl`" below for why. It is **one**
implementation of PIA registration shared by the unattended timers and by
this role's own registration step, so the two can never drift. Layout on
the host:
| Path | Purpose | | Path | Purpose |
| --- | --- | | --- | --- |
@ -258,6 +304,65 @@ drift. Layout on the host:
| `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) | | `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) |
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run | | `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` | | `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
| `pia-killswitch-reassert.timer` → `.service` | every `pia_healthcheck_interval_sec`, re-runs `pia-killswitch.sh` (idempotent). **Not** gated by `pia_auto_recovery_enabled` — see the "FORWARD jump's position" note above and the service unit's own header for why |
### PIA's CA certificate and why addKey uses curl, not Python's `ssl`
Found live, 2026-09-22, on the very first `pia-gateway register` attempt
after the FORWARD-jump fix above got past the kill-switch check:
`addKey` failed against every candidate with `kind=tls detail=certificate
verification failed`. Root-caused directly on minisforum:
```
$ sudo python3 -c '...ctx.wrap_socket(sock, server_hostname="Server-11669-10a")...'
SSLCertVerificationError verify_message= Basic Constraints of CA cert not marked critical
```
PIA's own CA certificate (`files/pia-ca.crt`) has a `CA:TRUE` Basic
Constraints extension that isn't flagged `critical` — a real RFC 5280
non-conformance, confirmed **not** a stale vendored copy: a fresh fetch
from `pia-foss/manual-connections` on GitHub is byte-identical (same
`32e9b1d1...` SHA-256 already recorded in the file's own header) and has
the identical defect. This is PIA's problem, not ours, and not fixable by
re-vendoring. Recent OpenSSL (3.2+; confirmed live against minisforum's
`3.5.7`) rejects this by default — a check that used to be opt-in only
under `X509_V_FLAG_X509_STRICT` and has since been promoted into the
default chain-verification path. `openssl s_client` on this repo's own
dev machine (LibreSSL) and this same host's `curl` both accept the exact
same certificate/chain/CA file without complaint — only Python's `ssl`
module, using this host's system OpenSSL, rejects it. That's also why
this went unnoticed until the auto-recovery work: the legacy Ansible
path's three successful registrations (2026-08-24, 08-25, and this
morning) all went through `curl --cacert` (`addkey-attempt.yaml`), never
through Python's `ssl` module — this was the *first* time that code path
ever ran for real.
Python's stdlib `ssl` module has no custom-verify-callback hook (a
long-standing, deliberate upstream gap — unlike Go's
`tls.Config.VerifyPeerCertificate` or OpenSSL's own C API), so there is
no clean way to tolerate just this one known, third-party-controlled
defect while still fully validating everything else (chain, expiry,
hostname/SAN). Rather than weakening `verify_mode` for the whole
connection or adding a new dependency (`pyOpenSSL`, for a real verify
callback — considered, rejected as a bigger lift than necessary) to work
around a gap in a library we don't control, `addKey` specifically now
shells out to `curl` (`HttpClient.get_pinned_curl`, `curl` added to this
role's `apt` install list) — the same tool PIA's own reference client
(`connect_to_wireguard_with_token.sh`) already uses for this exact
endpoint, with `--cacert` and never `-k`/`--insecure`, so this is a
different, already-proven-correct TLS backend for the one endpoint that
needs it, not a security downgrade. `serverlist` and the token endpoint
are unaffected (they present public WebPKI certs, which don't have this
problem) and still use the stdlib `ssl` path. Verified: a real subprocess
`curl` call from this repo's dev machine against the actual failing
candidate (`Server-11669-10a` / `69.33.198.97`) gets a genuine HTTP
response from PIA (401, expected — a dummy token was used), proving TLS
validation succeeds via this path where it failed via Python's `ssl`.
curl's own exit codes for a `--fail`-triggered rejection aren't fully
portable either — confirmed live: a curl built against SecureTransport
(this dev machine) reports exit 56, not the "usual" 22 GNU/OpenSSL curl
uses for the identical case — so classification checks curl's own error
text as a second signal alongside the exit code, not the exit code alone.
### Status classes ### Status classes
@ -440,10 +545,37 @@ external secret broker — not a 24-hour token.
- `unknown` (a read command failing) is logged/alerted but never acted - `unknown` (a read command failing) is logged/alerted but never acted
on, by design. on, by design.
### Rollout (nothing below has been run live yet) ### Rollout status (2026-09-22)
Step 1 (baseline) and three attempts at step 2 have been run live.
- **Attempt 1** failed: `pia-gateway register` correctly refused with
`safety invariants failing ... FORWARD jump to kill-switch chain is
not first` — a real, pre-existing drift (see the "FORWARD jump's
position is actively re-asserted" note above), not a false alarm from
this session's work, and not the "session dies" handshake issue
either. Root-caused and fixed in `pia-killswitch.sh.j2`.
- **Attempt 2**, after that fix, got past the kill-switch check and
failed differently: `addKey` rejected every candidate with
`kind=tls detail=certificate verification failed`. Root-caused and
fixed — see "PIA's CA certificate and why addKey uses curl, not
Python's `ssl`" above.
- **Attempt 3**, roughly an hour after attempt 1's fix with no reboot,
UFW reload, or script change in between, hit the *identical* FORWARD-
jump failure again — proving the drift recurs on a timescale of
minutes under this cluster's normal CronJob/pod churn, not just at
boot-scale events as originally assumed. Unblocked live with a manual
`sudo systemctl restart pia-killswitch.service`; the durable fix is
`pia-killswitch-reassert.timer` (added, not yet applied or verified
live) — see the "FORWARD jump's position" note's "recurs far more
often" addendum above.
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
— the original `head -3` suggestion is what hid the first issue; the
drift was 4 rules in, not at the top.
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show 1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
table pia`, `iptables -S FORWARD | head -3`, `iptables -S PIA-VLAN50`, table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`,
`journalctl -u pia-gateway-healthcheck -n 5`, `curl --socks5-hostname `journalctl -u pia-gateway-healthcheck -n 5`, `curl --socks5-hostname
10.10.40.20:1080 https://ifconfig.me` from the LAN, `wg show wg0`, 10.10.40.20:1080 https://ifconfig.me` from the LAN, `wg show wg0`,
`kubectl get nodes`. Confirm console access. `kubectl get nodes`. Confirm console access.
@ -454,7 +586,14 @@ external secret broker — not a 24-hour token.
verified baseline yet, this run **registers through the shared path** verified baseline yet, this run **registers through the shared path**
(same-key, explicit restart + handshake verification) — watch the (same-key, explicit restart + handshake verification) — watch the
task output; if it fails, `-e pia_registration_backend=legacy` task output; if it fails, `-e pia_registration_backend=legacy`
restores the previous behaviour. restores the previous behaviour. If it fails specifically with
`FORWARD jump to kill-switch chain is not first`, that's the drift
above — the fixed template should correct it via the normal "Restart
pia-killswitch" handler on the same run (the template content
changing is enough to trigger it); re-running the identical command
should then get past registration. Confirm with `iptables -S FORWARD`
afterwards that the jump is genuinely first, not just that the retry
succeeded.
3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy 3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy
pass makes no changes (`journalctl -u pia-gateway-recover -n 5`). pass makes no changes (`journalctl -u pia-gateway-recover -n 5`).
4. Drill the silent death. First get the **live WireGuard UDP endpoint** 4. Drill the silent death. First get the **live WireGuard UDP endpoint**
@ -531,8 +670,10 @@ touching `wg0`, k3s, or the host default route):
```bash ```bash
sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-killswitch-reassert.timer pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \ sudo rm -f /etc/systemd/system/pia-killswitch.service \
/etc/systemd/system/pia-killswitch-reassert.service \
/etc/systemd/system/pia-killswitch-reassert.timer \
/etc/systemd/system/pia-gateway-healthcheck.service \ /etc/systemd/system/pia-gateway-healthcheck.service \
/etc/systemd/system/pia-gateway-healthcheck.timer \ /etc/systemd/system/pia-gateway-healthcheck.timer \
/etc/systemd/system/pia-gateway-recover.service \ /etc/systemd/system/pia-gateway-recover.service \

View File

@ -751,33 +751,14 @@ class Classifier:
# -------------------------------------------------------------------------- # --------------------------------------------------------------------------
class _PinnedHTTPSConnection(http.client.HTTPSConnection):
"""Connect to a fixed IP while verifying TLS against the hostname (SNI)."""
def __init__(self, host: str, ip: str, port: int, context: ssl.SSLContext, connect_timeout: float, total_timeout: float):
super().__init__(host, port, timeout=connect_timeout, context=context)
self._pinned_ip = ip
self._total_timeout = total_timeout
def connect(self) -> None: # noqa: D401
sock = socket.create_connection((self._pinned_ip, self.port), self.timeout)
sock.settimeout(self._total_timeout)
self.sock = self._context.wrap_socket(sock, server_hostname=self.host)
class HttpClient: class HttpClient:
def __init__(self, cfg: Config): def __init__(self, cfg: Config, system: Optional[System] = None):
self.cfg = cfg self.cfg = cfg
self.system = system or System()
def _system_ctx(self) -> ssl.SSLContext: def _system_ctx(self) -> ssl.SSLContext:
return ssl.create_default_context() return ssl.create_default_context()
def _pia_ctx(self) -> ssl.SSLContext:
ctx = ssl.create_default_context(cafile=self.cfg.ca_file)
ctx.check_hostname = True
ctx.verify_mode = ssl.CERT_REQUIRED
return ctx
def _request(self, conn: http.client.HTTPSConnection, method: str, path: str, body: Optional[bytes], headers: dict) -> tuple[int, bytes]: def _request(self, conn: http.client.HTTPSConnection, method: str, path: str, body: Optional[bytes], headers: dict) -> tuple[int, bytes]:
try: try:
conn.request(method, path, body=body, headers=headers) conn.request(method, path, body=body, headers=headers)
@ -785,13 +766,22 @@ class HttpClient:
data = resp.read(4_000_000) data = resp.read(4_000_000)
return resp.status, data return resp.status, data
except ssl.SSLCertVerificationError as e: except ssl.SSLCertVerificationError as e:
raise ApiError("tls", "certificate verification failed") from e # verify_message/reason are standard OpenSSL diagnostic text
# (e.g. "unable to get local issuer certificate", "certificate
# has expired", a hostname mismatch) — never a secret, and
# exactly what's needed to tell "wrong CA", "expired cert",
# "clock skew", and "hostname mismatch" apart instead of
# collapsing all of them into one opaque string.
detail = e.verify_message or e.reason or str(e) or "certificate verification failed"
raise ApiError("tls", detail) from e
except ssl.SSLError as e: except ssl.SSLError as e:
raise ApiError("tls", e.__class__.__name__) from e detail = getattr(e, "reason", None) or str(e) or e.__class__.__name__
raise ApiError("tls", f"{e.__class__.__name__}: {detail}") from e
except socket.timeout as e: except socket.timeout as e:
raise ApiError("timeout", "request timed out") from e raise ApiError("timeout", "request timed out") from e
except (OSError, http.client.HTTPException) as e: except (OSError, http.client.HTTPException) as e:
raise ApiError("network", e.__class__.__name__) from e detail = str(e) or e.__class__.__name__
raise ApiError("network", f"{e.__class__.__name__}: {detail}") from e
finally: finally:
with contextlib.suppress(Exception): with contextlib.suppress(Exception):
conn.close() conn.close()
@ -816,12 +806,68 @@ class HttpClient:
} }
return self._request(conn, "POST", u.path, body, headers) return self._request(conn, "POST", u.path, body, headers)
def get_pinned(self, hostname: str, ip: str, port: int, path: str, query: dict[str, str]) -> tuple[int, bytes]: def get_pinned_curl(self, hostname: str, ip: str, port: int, path: str, token: str, pubkey: str) -> tuple[int, bytes]:
conn = _PinnedHTTPSConnection( """Pinned-IP, SNI-validated, CA-pinned GET — for PIA's addKey only.
hostname, ip, port, self._pia_ctx(), self.cfg.http_connect_timeout_sec, self.cfg.http_total_timeout_sec
) This shells out to curl instead of using the stdlib `ssl` module.
full = path + "?" + urllib.parse.urlencode(query) Confirmed live (2026-09-22): PIA's own CA certificate (byte-
return self._request(conn, "GET", full, None, {"User-Agent": "pia-gateway"}) identical to a fresh fetch from PIA's own repo — not a stale
vendored copy) has a non-critical Basic Constraints extension on
its CA cert, a real RFC 5280 non-conformance in a CA we don't
control. Recent OpenSSL (3.2+; confirmed on minisforum's 3.5.7 via
a direct repro — SSLCertVerificationError, verify_message
"Basic Constraints of CA cert not marked critical") rejects that
in its default chain verification, a check promoted from
X509_STRICT-only to the default path in recent OpenSSL releases.
Python's stdlib `ssl` module has no custom verify-callback hook to
selectively tolerate just this one known, third-party-controlled
defect while still fully validating everything else (chain,
expiry, hostname/SAN) — that gap is long-standing and
deliberate upstream, not something to route around by weakening
verify_mode. curl's TLS stack on this host does not enforce this
specific check and is what PIA's own reference implementation
(connect_to_wireguard_with_token.sh) already uses for this exact
endpoint — reusing it here with --cacert (never -k/--insecure) is
not a security downgrade, just a different, already-proven-correct
TLS backend for the one endpoint that needs it. serverlist and
token still use the stdlib ssl path above against public WebPKI
certs, which do not have this problem.
"""
argv = [
"curl", "--silent", "--show-error", "--fail", "--location", "-G",
"--connect-timeout", str(self.cfg.http_connect_timeout_sec),
"--max-time", str(self.cfg.http_total_timeout_sec),
"--cacert", self.cfg.ca_file,
"--connect-to", f"{hostname}::{ip}:",
"--data-urlencode", "pt@-",
"--data-urlencode", f"pubkey={pubkey}",
f"https://{hostname}:{port}{path}",
]
r = self.system.run(argv, stdin=token, timeout=self.cfg.http_total_timeout_sec + 10)
if r.rc == 0:
return 200, r.out.encode("utf-8")
err = (r.err or "").strip()
# Exit codes are the primary signal, but not fully portable across
# curl builds/TLS backends — confirmed live: a curl built against
# SecureTransport reported exit 56 (CURLE_RECV_ERROR) for the exact
# same --fail-triggered HTTP rejection GNU/OpenSSL curl reports as
# 22. curl's own error text ("returned error: NNN" for --fail,
# "timed out" for CURLE_OPERATION_TIMEDOUT, TLS-related wording for
# cert/handshake problems) is a second, more portable signal — used
# here as a fallback classifier alongside the exit code, not a
# replacement for it.
if r.rc == 127:
raise ApiError("network", "curl not found on PATH")
if r.rc == 28 or "timed out" in err.lower():
raise ApiError("timeout", f"curl: connection or transfer timed out: {err}")
if r.rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83) or "ssl" in err.lower() or "certificate" in err.lower():
raise ApiError("tls", f"curl TLS error (exit {r.rc}): {err}")
if r.rc == 22 or "returned error:" in err:
# --fail suppresses the body on a non-2xx response, so there is
# no status code or body to parse here — same limitation the
# legacy curl-based Ansible path already had.
raise ApiError("http", f"curl: non-2xx HTTP response (exit {r.rc}): {err}")
raise ApiError("network", f"curl exit {r.rc}: {err}")
@dataclasses.dataclass @dataclasses.dataclass
@ -933,7 +979,7 @@ class PiaApi:
raise ApiError("http", f"token HTTP {last_status}") raise ApiError("http", f"token HTTP {last_status}")
def add_key(self, cand: Candidate, token: str, pubkey: str) -> PeerInfo: def add_key(self, cand: Candidate, token: str, pubkey: str) -> PeerInfo:
status, data = self.http.get_pinned(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", {"pt": token, "pubkey": pubkey}) status, data = self.http.get_pinned_curl(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", token, pubkey)
if status != 200: if status != 200:
raise ApiError("http", f"addKey HTTP {status}") raise ApiError("http", f"addKey HTTP {status}")
if not data.strip(): if not data.strip():
@ -1124,7 +1170,7 @@ class Controller:
self.cfg = cfg self.cfg = cfg
self.sys = system self.sys = system
self.log = log self.log = log
self.http = http or HttpClient(cfg) self.http = http or HttpClient(cfg, system)
self._creds = creds self._creds = creds
self.store = StateStore(cfg, system) self.store = StateStore(cfg, system)
self.classifier = Classifier(cfg, system) self.classifier = Classifier(cfg, system)

View File

@ -21,6 +21,9 @@
# killswitch script changed in the same run), pia-wg's own PostUp # killswitch script changed in the same run), pia-wg's own PostUp
# needs to run last so its "open" route is the final word, not # needs to run last so its "open" route is the final word, not
# killswitch's route-table seeding. # killswitch's route-table seeding.
# 4. "Restart pia-killswitch reassertion timer" only restarts the
# *timer* (a scheduling change, e.g. its interval), never runs the
# script itself — no ordering constraint against the others.
- name: Reload systemd daemon - name: Reload systemd daemon
ansible.builtin.systemd: ansible.builtin.systemd:
@ -35,6 +38,11 @@
name: pia-killswitch.service name: pia-killswitch.service
state: restarted state: restarted
- name: Restart pia-killswitch reassertion timer
ansible.builtin.systemd:
name: pia-killswitch-reassert.timer
state: restarted
- name: Restart pia-wg - name: Restart pia-wg
ansible.builtin.systemd: ansible.builtin.systemd:
name: "wg-quick@{{ pia_wg_interface }}" name: "wg-quick@{{ pia_wg_interface }}"

View File

@ -27,10 +27,16 @@
- wireguard-tools - wireguard-tools
- iproute2 - iproute2
- python3 - python3
- curl
state: present state: present
update_cache: true update_cache: true
# python3-requests is no longer needed: the controller uses only the # python3-requests is no longer needed: the controller uses only the
# standard library. # standard library, EXCEPT for the addKey endpoint specifically, which
# shells out to curl — see HttpClient.get_pinned_curl's own docstring
# for why (a non-critical Basic Constraints extension on PIA's own CA
# cert that recent OpenSSL rejects by default and Python's ssl module
# has no way to selectively tolerate). curl is virtually always already
# present on Debian, but declared explicitly rather than assumed.
- name: Pin reverse-path filtering for {{ pia_wg_interface }} - name: Pin reverse-path filtering for {{ pia_wg_interface }}
ansible.posix.sysctl: ansible.posix.sysctl:

View File

@ -40,3 +40,35 @@
name: pia-killswitch.service name: pia-killswitch.service
enabled: true enabled: true
state: started state: started
- name: Deploy the kill-switch periodic reassertion systemd service
ansible.builtin.template:
src: pia-killswitch-reassert.service.j2
dest: /etc/systemd/system/pia-killswitch-reassert.service
mode: "0644"
owner: root
group: root
notify:
- Reload systemd daemon
- Restart pia-killswitch reassertion timer
- name: Deploy the kill-switch periodic reassertion systemd timer
ansible.builtin.template:
src: pia-killswitch-reassert.timer.j2
dest: /etc/systemd/system/pia-killswitch-reassert.timer
mode: "0644"
owner: root
group: root
notify:
- Reload systemd daemon
- Restart pia-killswitch reassertion timer
- name: Enable and start the kill-switch reassertion timer
ansible.builtin.systemd:
name: pia-killswitch-reassert.timer
enabled: true
state: started
daemon_reload: true
# Unconditional — not gated by pia_auto_recovery_enabled. See the
# service unit's own header for why this is scoped differently from
# the automatic recovery/registration timers.

View File

@ -0,0 +1,30 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Periodically re-runs the kill-switch seed script (pia-killswitch.sh) so
# its invariants — source rule, table default, and specifically the
# FORWARD jump's position — are re-asserted on a schedule, not only at
# the external trigger points that used to be the only times it ran
# (boot, a UFW reload, or an Ansible-detected content change to the
# script itself). Needed because other host components that also manage
# iptables — kube-router and dockerd — reassert their OWN position-1
# jumps independently, and confirmed live (2026-09-22) to do so more
# often than originally assumed: the FORWARD jump drifted twice within
# about an hour under this cluster's ordinary CronJob/pod churn (ca-sync,
# porkbun-ddns, gitea-backup, ...), not just at boot-scale events.
#
# The script itself is already fully idempotent (safe to run repeatedly
# — see its own header comment); this unit just gives it more chances to
# run. Deliberately separate from pia-gateway-recover.timer and NOT
# gated by pia_auto_recovery_enabled: this is the kill switch continuing
# to assert its own already-established, narrowly-scoped state — the
# same kind of thing it already does unconditionally at boot and after a
# UFW reload, just on a schedule too — not the higher-level automatic
# recovery/registration logic the plan was deliberately cautious about
# running unattended.
[Unit]
Description=Periodically re-assert the PIA gateway kill switch's FORWARD jump position
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pia-killswitch.sh
LimitCORE=0

View File

@ -0,0 +1,11 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
[Unit]
Description=Run the PIA gateway kill-switch reassertion every {{ pia_healthcheck_interval_sec }}s
[Timer]
OnBootSec=30
OnUnitActiveSec={{ pia_healthcheck_interval_sec }}
AccuracySec=5s
[Install]
WantedBy=timers.target

View File

@ -1,12 +1,16 @@
#!/bin/bash #!/bin/bash
# Managed by Ansible (role: pia-gateway). Do not edit by hand. # Managed by Ansible (role: pia-gateway). Do not edit by hand.
# #
# Runs once at boot, before wg-quick@{{ pia_wg_interface }}.service # Runs at boot, before wg-quick@{{ pia_wg_interface }}.service (enforced
# (enforced by the systemd drop-in in pia-killswitch.service's # by the systemd drop-in in pia-killswitch.service's Before=/wg-quick's
# Before=/wg-quick's Requires=After=), and again any time UFW is # Requires=After=); any time UFW is reloaded (see handlers/main.yaml —
# reloaded (see handlers/main.yaml — "Reload ufw" fires before "Restart # "Reload ufw" fires before "Restart pia-killswitch" so a reload can
# pia-killswitch" so a reload can never leave a permanent gap). Seeds the # never leave a permanent gap); and — since 2026-09-22, after the FORWARD
# closed state that PIA gateway's kill switch relies on: # jump was found drifting twice within about an hour under this
# cluster's ordinary CronJob/pod churn, far more often than a boot/reload
# -only cadence catches — every {{ pia_healthcheck_interval_sec }}s via
# pia-killswitch-reassert.timer. Seeds the closed state that PIA
# gateway's kill switch relies on:
# #
# - a source rule sending {{ pia_vlan50_subnet }} to the dedicated # - a source rule sending {{ pia_vlan50_subnet }} to the dedicated
# "{{ pia_route_table_name }}" table, evaluated after the kernel's # "{{ pia_route_table_name }}" table, evaluated after the kernel's
@ -34,11 +38,16 @@
# source's fate with a broader ACCEPT — the jump is unconditional for # source's fate with a broader ACCEPT — the jump is unconditional for
# this source and the chain itself ends in an unconditional DROP, not # this source and the chain itself ends in an unconditional DROP, not
# a RETURN, so nothing after it in FORWARD is ever consulted for # a RETURN, so nothing after it in FORWARD is ever consulted for
# this source either. # this source either. "First" is actively re-asserted on every run,
# not just seeded once — see the jump-insertion block below for why
# that turned out to matter in practice.
# #
# Idempotent: safe to run more than once (e.g. `systemctl restart # Idempotent: safe to run more than once (e.g. `systemctl restart
# pia-killswitch`, or after this script re-runs following a UFW reload) # pia-killswitch`, or after this script re-runs following a UFW reload)
# without creating duplicate rules or duplicate chains. # without creating duplicate rules or duplicate chains — including
# repositioning the FORWARD jump back to position 1 if something else
# (kube-router, dockerd) has since pushed it down, rather than treating
# its mere existence as proof it's still first.
set -euo pipefail set -euo pipefail
TABLE="{{ pia_route_table_name }}" TABLE="{{ pia_route_table_name }}"
@ -68,8 +77,32 @@ fi
# there is no direct "does this chain exist" check short of grepping -L). # there is no direct "does this chain exist" check short of grepping -L).
iptables -N "${CHAIN}" 2>/dev/null || true iptables -N "${CHAIN}" 2>/dev/null || true
# Exactly one jump from FORWARD into it, at absolute position 1. # Exactly one jump from FORWARD into it, kept at absolute position 1.
if ! iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then # Existence alone ("-C") is not proof of position — confirmed live,
# 2026-09-22: kube-router and dockerd each independently reassert their
# OWN jump at position 1 whenever THEY resync/restart (kube-router
# doesn't even dedupe its own older copy when it does, so duplicates of
# ITS rule can appear too), which silently pushes this one down without
# removing it. A plain "-C, insert only if absent" check then reports
# "already exists" and leaves it stranded wherever it drifted to — this
# is exactly what happened here: the jump was 4th (behind kube-router,
# DOCKER-USER, DOCKER-FORWARD), not 1st, on a host that had been up 200
# days with kube-router/Docker managing FORWARD independently the whole
# time. So check the actual first rule in FORWARD, not just whether ours
# exists somewhere; if it isn't ours, remove it (by spec — iptables -D
# matches wherever it currently sits, not by position) and reinsert at
# 1. Safe to do unconditionally on every run: this rule is narrowly
# {{ pia_vlan50_subnet }}-scoped, so moving it earlier relative to
# kube-router's/Docker's own broader jumps can't affect pod or Docker
# bridge traffic either way — confirmed by reading their actual chain
# contents (KUBE-ROUTER-FORWARD only matches specific pod /32
# addresses, DOCKER-USER was empty, DOCKER-FORWARD's one direct rule is
# scoped to -i docker0), not assumed.
FORWARD_FIRST=$(iptables -S FORWARD | grep '^-A FORWARD' | head -1 || true)
if [[ "${FORWARD_FIRST}" != "-A FORWARD -s ${SUBNET} -j ${CHAIN}" ]]; then
if iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
iptables -D FORWARD -s "${SUBNET}" -j "${CHAIN}"
fi
iptables -I FORWARD 1 -s "${SUBNET}" -j "${CHAIN}" iptables -I FORWARD 1 -s "${SUBNET}" -j "${CHAIN}"
fi fi

View File

@ -255,8 +255,8 @@ class FakeHttp:
return 200, val return 200, val
return 200, json.dumps({"token": val}).encode() return 200, json.dumps({"token": val}).encode()
def get_pinned(self, hostname, ip, port, path, query): def get_pinned_curl(self, hostname, ip, port, path, token, pubkey):
self.calls.append(("addkey", hostname, ip, port, path, dict(query))) self.calls.append(("addkey", hostname, ip, port, path, {"pt": token, "pubkey": pubkey}))
kind, val = self.addkey.get(hostname, ("http", 502)) kind, val = self.addkey.get(hostname, ("http", 502))
if kind == "err": if kind == "err":
raise pg.ApiError(val, "fake") raise pg.ApiError(val, "fake")
@ -1227,6 +1227,106 @@ def dataclass_dict(cfg):
return dataclasses.asdict(cfg) return dataclasses.asdict(cfg)
class FakeCurlSystem(pg.System):
"""Fakes just enough of System.run to exercise get_pinned_curl's argv
construction, stdin handling, and exit-code classification, without a
real curl binary or network access."""
def __init__(self, result=None):
self.result = result or pg.CmdResult(0, '{"status":"OK"}', "")
self.calls = []
self.stdins = []
def run(self, argv, stdin=None, timeout=60):
self.calls.append(list(argv))
if stdin is not None:
self.stdins.append(stdin)
return self.result
class TestPinnedCurlAddKey(unittest.TestCase):
def setUp(self):
self.cfg = pg.Config(ca_file="/etc/wireguard/pia-ca.crt", http_connect_timeout_sec=10, http_total_timeout_sec=30)
def test_argv_shape_matches_the_proven_legacy_curl_invocation(self):
sysm = FakeCurlSystem()
http_client = pg.HttpClient(self.cfg, sysm)
status, data = http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
self.assertEqual(status, 200)
self.assertEqual(data, b'{"status":"OK"}')
argv = sysm.calls[0]
self.assertEqual(argv[0], "curl")
self.assertIn("--fail", argv)
self.assertIn("-G", argv)
self.assertIn("--cacert", argv)
self.assertEqual(argv[argv.index("--cacert") + 1], "/etc/wireguard/pia-ca.crt")
self.assertIn("--connect-to", argv)
self.assertEqual(argv[argv.index("--connect-to") + 1], "Server-11669-10a::69.33.198.97:")
self.assertIn("pt@-", argv)
self.assertIn(f"pubkey={PUB_A}", argv)
self.assertEqual(argv[-1], "https://Server-11669-10a:1337/addKey")
# never -k/--insecure — this must still be a real, validating request
self.assertNotIn("-k", argv)
self.assertNotIn("--insecure", argv)
def test_token_passed_via_stdin_with_no_trailing_newline_and_never_in_argv(self):
sysm = FakeCurlSystem()
http_client = pg.HttpClient(self.cfg, sysm)
http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
self.assertEqual(sysm.stdins, [SECRET_TOKEN]) # exact match: no appended "\n"
for arg in sysm.calls[0]:
self.assertNotIn(SECRET_TOKEN, arg)
def test_timeout_exit_code_classified(self):
sysm = FakeCurlSystem(pg.CmdResult(28, "", "curl: (28) Connection timed out"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "timeout")
def test_tls_exit_codes_classified_with_stderr_detail(self):
for rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83):
with self.subTest(rc=rc):
sysm = FakeCurlSystem(pg.CmdResult(rc, "", "curl: (60) SSL certificate problem"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "tls")
self.assertIn("SSL certificate problem", cm.exception.detail)
def test_http_failure_exit_code_classified(self):
sysm = FakeCurlSystem(pg.CmdResult(22, "", "curl: (22) The requested URL returned error: 401"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "http")
def test_nonstandard_exit_code_with_fail_text_still_classified_as_http(self):
# Observed live: curl built against SecureTransport reports exit 56
# (CURLE_RECV_ERROR), not the "usual" 22, for the same --fail
# -triggered HTTP rejection — the error text is the portable signal.
sysm = FakeCurlSystem(pg.CmdResult(56, "", "curl: (56) The requested URL returned error: 401"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "http")
def test_curl_missing_classified_as_network_not_a_crash(self):
sysm = FakeCurlSystem(pg.CmdResult(127, "", "curl: not found"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "network")
def test_unclassified_exit_code_falls_back_to_network_with_stderr(self):
sysm = FakeCurlSystem(pg.CmdResult(7, "", "curl: (7) Couldn't connect to server"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "network")
self.assertIn("Couldn't connect", cm.exception.detail)
class TestHttpHelpers(unittest.TestCase): class TestHttpHelpers(unittest.TestCase):
def test_key_and_ip_validators(self): def test_key_and_ip_validators(self):
self.assertTrue(pg.valid_wg_key(PUB_A)) self.assertTrue(pg.valid_wg_key(PUB_A))