Compare commits

..

No commits in common. "89d289addf4b3b2ec23f5d452d16ef16839b1816" and "87a9303cc3dfa427a26de5feeb53926d66525958" have entirely different histories.

9 changed files with 53 additions and 460 deletions

View File

@ -163,48 +163,6 @@ real.
Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW
**input** allow for 53/tcp+udp — it's locally-terminated traffic, not **input** allow for 53/tcp+udp — it's locally-terminated traffic, not
routed — see `tasks/firewall.yaml`. routed — see `tasks/firewall.yaml`.
**The FORWARD jump's position is actively re-asserted on every run, not
just seeded once** — found live, 2026-09-22, on the very first
deployment of the auto-recovery controller. minisforum's `FORWARD`
chain default policy is `DROP` (kube-router's doing, not this role's —
correcting an assumption an earlier version of this doc made), and
kube-router and `dockerd` each independently reassert their **own**
jump at position 1 whenever they resync or restart, silently pushing
this rule down without removing it (kube-router doesn't even dedupe
its own older copy when it does — a duplicate of *its* rule was found
live too). The script used to check only whether its rule *existed*
(`iptables -C`) before deciding not to touch anything; existence isn't
proof of position, so once pushed down it stayed down until something
else fixed it. It now checks the actual first rule and repositions
(delete + reinsert at 1) if it isn't this one. Confirmed safe to do
unconditionally by reading kube-router's and Docker's actual chain
contents rather than assuming: `KUBE-ROUTER-FORWARD` only matches
specific pod `/32` addresses (disjoint from `10.10.50.0/24`),
`DOCKER-USER` was empty, and `DOCKER-FORWARD`'s one direct rule is
scoped to `-i docker0` (VLAN 50 traffic arrives via `enp1s0`) — none
of them can intercept VLAN 50 traffic regardless of order, so moving
this rule back to position 1 relative to them changes nothing for pod
or Docker-bridge traffic. The controller's own safety check
(`forward_jump_first`) was working as intended when it refused to
register against the drifted state — this is a fix to the thing it
checks, not a loosening of the check itself.
**This recurs far more often than "boot-scale," and the fix above only
helped once.** The reposition logic is correct, but it only *ran* on
the existing triggers (boot, a UFW reload, an Ansible-detected content
change to the script). Confirmed live: after that fix landed and got
one `pia-gateway register` attempt past this check, the very next
attempt — roughly an hour later, no reboot, no UFW reload, no script
change in between — hit the identical failure again. kube-router
resyncs on ordinary Kubernetes API events (this cluster's CronJobs —
`ca-sync`, `porkbun-ddns`, `gitea-backup` — each create and delete a
Pod every few minutes, each a plausible trigger), not just at
boot/restart scale as originally assumed. `pia-killswitch-reassert
.timer` (every `pia_healthcheck_interval_sec`, unconditional, not
gated by `pia_auto_recovery_enabled`) exists specifically to close this
gap by giving the already-correct, already-idempotent script regular
chances to run instead of only sporadic ones.
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on - `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
top of that closed baseline: default route via `pia-wg` in the `pia` top of that closed baseline: default route via `pia-wg` in the `pia`
table, ACCEPT + established/related return inserted into the table, ACCEPT + established/related return inserted into the
@ -284,14 +242,10 @@ handled automatically — see the next section.
## Automatic recovery ## Automatic recovery
`files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway` `files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway`
(root, `0700`, runs on the host's `python3.13`). Almost entirely standard (root, `0700`, standard library only — runs on the host's `python3.13`).
library — the one exception is `addKey` specifically, which shells out to It is **one** implementation of PIA registration shared by the unattended
`curl` (now an explicit `apt` dependency of this role); see timers and by this role's own registration step, so the two can never
`HttpClient.get_pinned_curl`'s own docstring and "PIA's CA certificate and drift. Layout on the host:
why addKey uses curl, not Python's `ssl`" below for why. It is **one**
implementation of PIA registration shared by the unattended timers and by
this role's own registration step, so the two can never drift. Layout on
the host:
| Path | Purpose | | Path | Purpose |
| --- | --- | | --- | --- |
@ -304,65 +258,6 @@ the host:
| `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) | | `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) |
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run | | `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` | | `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
| `pia-killswitch-reassert.timer` → `.service` | every `pia_healthcheck_interval_sec`, re-runs `pia-killswitch.sh` (idempotent). **Not** gated by `pia_auto_recovery_enabled` — see the "FORWARD jump's position" note above and the service unit's own header for why |
### PIA's CA certificate and why addKey uses curl, not Python's `ssl`
Found live, 2026-09-22, on the very first `pia-gateway register` attempt
after the FORWARD-jump fix above got past the kill-switch check:
`addKey` failed against every candidate with `kind=tls detail=certificate
verification failed`. Root-caused directly on minisforum:
```
$ sudo python3 -c '...ctx.wrap_socket(sock, server_hostname="Server-11669-10a")...'
SSLCertVerificationError verify_message= Basic Constraints of CA cert not marked critical
```
PIA's own CA certificate (`files/pia-ca.crt`) has a `CA:TRUE` Basic
Constraints extension that isn't flagged `critical` — a real RFC 5280
non-conformance, confirmed **not** a stale vendored copy: a fresh fetch
from `pia-foss/manual-connections` on GitHub is byte-identical (same
`32e9b1d1...` SHA-256 already recorded in the file's own header) and has
the identical defect. This is PIA's problem, not ours, and not fixable by
re-vendoring. Recent OpenSSL (3.2+; confirmed live against minisforum's
`3.5.7`) rejects this by default — a check that used to be opt-in only
under `X509_V_FLAG_X509_STRICT` and has since been promoted into the
default chain-verification path. `openssl s_client` on this repo's own
dev machine (LibreSSL) and this same host's `curl` both accept the exact
same certificate/chain/CA file without complaint — only Python's `ssl`
module, using this host's system OpenSSL, rejects it. That's also why
this went unnoticed until the auto-recovery work: the legacy Ansible
path's three successful registrations (2026-08-24, 08-25, and this
morning) all went through `curl --cacert` (`addkey-attempt.yaml`), never
through Python's `ssl` module — this was the *first* time that code path
ever ran for real.
Python's stdlib `ssl` module has no custom-verify-callback hook (a
long-standing, deliberate upstream gap — unlike Go's
`tls.Config.VerifyPeerCertificate` or OpenSSL's own C API), so there is
no clean way to tolerate just this one known, third-party-controlled
defect while still fully validating everything else (chain, expiry,
hostname/SAN). Rather than weakening `verify_mode` for the whole
connection or adding a new dependency (`pyOpenSSL`, for a real verify
callback — considered, rejected as a bigger lift than necessary) to work
around a gap in a library we don't control, `addKey` specifically now
shells out to `curl` (`HttpClient.get_pinned_curl`, `curl` added to this
role's `apt` install list) — the same tool PIA's own reference client
(`connect_to_wireguard_with_token.sh`) already uses for this exact
endpoint, with `--cacert` and never `-k`/`--insecure`, so this is a
different, already-proven-correct TLS backend for the one endpoint that
needs it, not a security downgrade. `serverlist` and the token endpoint
are unaffected (they present public WebPKI certs, which don't have this
problem) and still use the stdlib `ssl` path. Verified: a real subprocess
`curl` call from this repo's dev machine against the actual failing
candidate (`Server-11669-10a` / `69.33.198.97`) gets a genuine HTTP
response from PIA (401, expected — a dummy token was used), proving TLS
validation succeeds via this path where it failed via Python's `ssl`.
curl's own exit codes for a `--fail`-triggered rejection aren't fully
portable either — confirmed live: a curl built against SecureTransport
(this dev machine) reports exit 56, not the "usual" 22 GNU/OpenSSL curl
uses for the identical case — so classification checks curl's own error
text as a second signal alongside the exit code, not the exit code alone.
### Status classes ### Status classes
@ -545,37 +440,10 @@ external secret broker — not a 24-hour token.
- `unknown` (a read command failing) is logged/alerted but never acted - `unknown` (a read command failing) is logged/alerted but never acted
on, by design. on, by design.
### Rollout status (2026-09-22) ### Rollout (nothing below has been run live yet)
Step 1 (baseline) and three attempts at step 2 have been run live.
- **Attempt 1** failed: `pia-gateway register` correctly refused with
`safety invariants failing ... FORWARD jump to kill-switch chain is
not first` — a real, pre-existing drift (see the "FORWARD jump's
position is actively re-asserted" note above), not a false alarm from
this session's work, and not the "session dies" handshake issue
either. Root-caused and fixed in `pia-killswitch.sh.j2`.
- **Attempt 2**, after that fix, got past the kill-switch check and
failed differently: `addKey` rejected every candidate with
`kind=tls detail=certificate verification failed`. Root-caused and
fixed — see "PIA's CA certificate and why addKey uses curl, not
Python's `ssl`" above.
- **Attempt 3**, roughly an hour after attempt 1's fix with no reboot,
UFW reload, or script change in between, hit the *identical* FORWARD-
jump failure again — proving the drift recurs on a timescale of
minutes under this cluster's normal CronJob/pod churn, not just at
boot-scale events as originally assumed. Unblocked live with a manual
`sudo systemctl restart pia-killswitch.service`; the durable fix is
`pia-killswitch-reassert.timer` (added, not yet applied or verified
live) — see the "FORWARD jump's position" note's "recurs far more
often" addendum above.
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
— the original `head -3` suggestion is what hid the first issue; the
drift was 4 rules in, not at the top.
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show 1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`, table pia`, `iptables -S FORWARD | head -3`, `iptables -S PIA-VLAN50`,
`journalctl -u pia-gateway-healthcheck -n 5`, `curl --socks5-hostname `journalctl -u pia-gateway-healthcheck -n 5`, `curl --socks5-hostname
10.10.40.20:1080 https://ifconfig.me` from the LAN, `wg show wg0`, 10.10.40.20:1080 https://ifconfig.me` from the LAN, `wg show wg0`,
`kubectl get nodes`. Confirm console access. `kubectl get nodes`. Confirm console access.
@ -586,14 +454,7 @@ drift was 4 rules in, not at the top.
verified baseline yet, this run **registers through the shared path** verified baseline yet, this run **registers through the shared path**
(same-key, explicit restart + handshake verification) — watch the (same-key, explicit restart + handshake verification) — watch the
task output; if it fails, `-e pia_registration_backend=legacy` task output; if it fails, `-e pia_registration_backend=legacy`
restores the previous behaviour. If it fails specifically with restores the previous behaviour.
`FORWARD jump to kill-switch chain is not first`, that's the drift
above — the fixed template should correct it via the normal "Restart
pia-killswitch" handler on the same run (the template content
changing is enough to trigger it); re-running the identical command
should then get past registration. Confirm with `iptables -S FORWARD`
afterwards that the jump is genuinely first, not just that the retry
succeeded.
3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy 3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy
pass makes no changes (`journalctl -u pia-gateway-recover -n 5`). pass makes no changes (`journalctl -u pia-gateway-recover -n 5`).
4. Drill the silent death. First get the **live WireGuard UDP endpoint** 4. Drill the silent death. First get the **live WireGuard UDP endpoint**
@ -670,10 +531,8 @@ touching `wg0`, k3s, or the host default route):
```bash ```bash
sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-killswitch-reassert.timer pia-gateway-healthcheck.timer sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \ sudo rm -f /etc/systemd/system/pia-killswitch.service \
/etc/systemd/system/pia-killswitch-reassert.service \
/etc/systemd/system/pia-killswitch-reassert.timer \
/etc/systemd/system/pia-gateway-healthcheck.service \ /etc/systemd/system/pia-gateway-healthcheck.service \
/etc/systemd/system/pia-gateway-healthcheck.timer \ /etc/systemd/system/pia-gateway-healthcheck.timer \
/etc/systemd/system/pia-gateway-recover.service \ /etc/systemd/system/pia-gateway-recover.service \

View File

@ -751,14 +751,33 @@ class Classifier:
# -------------------------------------------------------------------------- # --------------------------------------------------------------------------
class _PinnedHTTPSConnection(http.client.HTTPSConnection):
"""Connect to a fixed IP while verifying TLS against the hostname (SNI)."""
def __init__(self, host: str, ip: str, port: int, context: ssl.SSLContext, connect_timeout: float, total_timeout: float):
super().__init__(host, port, timeout=connect_timeout, context=context)
self._pinned_ip = ip
self._total_timeout = total_timeout
def connect(self) -> None: # noqa: D401
sock = socket.create_connection((self._pinned_ip, self.port), self.timeout)
sock.settimeout(self._total_timeout)
self.sock = self._context.wrap_socket(sock, server_hostname=self.host)
class HttpClient: class HttpClient:
def __init__(self, cfg: Config, system: Optional[System] = None): def __init__(self, cfg: Config):
self.cfg = cfg self.cfg = cfg
self.system = system or System()
def _system_ctx(self) -> ssl.SSLContext: def _system_ctx(self) -> ssl.SSLContext:
return ssl.create_default_context() return ssl.create_default_context()
def _pia_ctx(self) -> ssl.SSLContext:
ctx = ssl.create_default_context(cafile=self.cfg.ca_file)
ctx.check_hostname = True
ctx.verify_mode = ssl.CERT_REQUIRED
return ctx
def _request(self, conn: http.client.HTTPSConnection, method: str, path: str, body: Optional[bytes], headers: dict) -> tuple[int, bytes]: def _request(self, conn: http.client.HTTPSConnection, method: str, path: str, body: Optional[bytes], headers: dict) -> tuple[int, bytes]:
try: try:
conn.request(method, path, body=body, headers=headers) conn.request(method, path, body=body, headers=headers)
@ -766,22 +785,13 @@ class HttpClient:
data = resp.read(4_000_000) data = resp.read(4_000_000)
return resp.status, data return resp.status, data
except ssl.SSLCertVerificationError as e: except ssl.SSLCertVerificationError as e:
# verify_message/reason are standard OpenSSL diagnostic text raise ApiError("tls", "certificate verification failed") from e
# (e.g. "unable to get local issuer certificate", "certificate
# has expired", a hostname mismatch) — never a secret, and
# exactly what's needed to tell "wrong CA", "expired cert",
# "clock skew", and "hostname mismatch" apart instead of
# collapsing all of them into one opaque string.
detail = e.verify_message or e.reason or str(e) or "certificate verification failed"
raise ApiError("tls", detail) from e
except ssl.SSLError as e: except ssl.SSLError as e:
detail = getattr(e, "reason", None) or str(e) or e.__class__.__name__ raise ApiError("tls", e.__class__.__name__) from e
raise ApiError("tls", f"{e.__class__.__name__}: {detail}") from e
except socket.timeout as e: except socket.timeout as e:
raise ApiError("timeout", "request timed out") from e raise ApiError("timeout", "request timed out") from e
except (OSError, http.client.HTTPException) as e: except (OSError, http.client.HTTPException) as e:
detail = str(e) or e.__class__.__name__ raise ApiError("network", e.__class__.__name__) from e
raise ApiError("network", f"{e.__class__.__name__}: {detail}") from e
finally: finally:
with contextlib.suppress(Exception): with contextlib.suppress(Exception):
conn.close() conn.close()
@ -806,68 +816,12 @@ class HttpClient:
} }
return self._request(conn, "POST", u.path, body, headers) return self._request(conn, "POST", u.path, body, headers)
def get_pinned_curl(self, hostname: str, ip: str, port: int, path: str, token: str, pubkey: str) -> tuple[int, bytes]: def get_pinned(self, hostname: str, ip: str, port: int, path: str, query: dict[str, str]) -> tuple[int, bytes]:
"""Pinned-IP, SNI-validated, CA-pinned GET — for PIA's addKey only. conn = _PinnedHTTPSConnection(
hostname, ip, port, self._pia_ctx(), self.cfg.http_connect_timeout_sec, self.cfg.http_total_timeout_sec
This shells out to curl instead of using the stdlib `ssl` module. )
Confirmed live (2026-09-22): PIA's own CA certificate (byte- full = path + "?" + urllib.parse.urlencode(query)
identical to a fresh fetch from PIA's own repo — not a stale return self._request(conn, "GET", full, None, {"User-Agent": "pia-gateway"})
vendored copy) has a non-critical Basic Constraints extension on
its CA cert, a real RFC 5280 non-conformance in a CA we don't
control. Recent OpenSSL (3.2+; confirmed on minisforum's 3.5.7 via
a direct repro — SSLCertVerificationError, verify_message
"Basic Constraints of CA cert not marked critical") rejects that
in its default chain verification, a check promoted from
X509_STRICT-only to the default path in recent OpenSSL releases.
Python's stdlib `ssl` module has no custom verify-callback hook to
selectively tolerate just this one known, third-party-controlled
defect while still fully validating everything else (chain,
expiry, hostname/SAN) — that gap is long-standing and
deliberate upstream, not something to route around by weakening
verify_mode. curl's TLS stack on this host does not enforce this
specific check and is what PIA's own reference implementation
(connect_to_wireguard_with_token.sh) already uses for this exact
endpoint — reusing it here with --cacert (never -k/--insecure) is
not a security downgrade, just a different, already-proven-correct
TLS backend for the one endpoint that needs it. serverlist and
token still use the stdlib ssl path above against public WebPKI
certs, which do not have this problem.
"""
argv = [
"curl", "--silent", "--show-error", "--fail", "--location", "-G",
"--connect-timeout", str(self.cfg.http_connect_timeout_sec),
"--max-time", str(self.cfg.http_total_timeout_sec),
"--cacert", self.cfg.ca_file,
"--connect-to", f"{hostname}::{ip}:",
"--data-urlencode", "pt@-",
"--data-urlencode", f"pubkey={pubkey}",
f"https://{hostname}:{port}{path}",
]
r = self.system.run(argv, stdin=token, timeout=self.cfg.http_total_timeout_sec + 10)
if r.rc == 0:
return 200, r.out.encode("utf-8")
err = (r.err or "").strip()
# Exit codes are the primary signal, but not fully portable across
# curl builds/TLS backends — confirmed live: a curl built against
# SecureTransport reported exit 56 (CURLE_RECV_ERROR) for the exact
# same --fail-triggered HTTP rejection GNU/OpenSSL curl reports as
# 22. curl's own error text ("returned error: NNN" for --fail,
# "timed out" for CURLE_OPERATION_TIMEDOUT, TLS-related wording for
# cert/handshake problems) is a second, more portable signal — used
# here as a fallback classifier alongside the exit code, not a
# replacement for it.
if r.rc == 127:
raise ApiError("network", "curl not found on PATH")
if r.rc == 28 or "timed out" in err.lower():
raise ApiError("timeout", f"curl: connection or transfer timed out: {err}")
if r.rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83) or "ssl" in err.lower() or "certificate" in err.lower():
raise ApiError("tls", f"curl TLS error (exit {r.rc}): {err}")
if r.rc == 22 or "returned error:" in err:
# --fail suppresses the body on a non-2xx response, so there is
# no status code or body to parse here — same limitation the
# legacy curl-based Ansible path already had.
raise ApiError("http", f"curl: non-2xx HTTP response (exit {r.rc}): {err}")
raise ApiError("network", f"curl exit {r.rc}: {err}")
@dataclasses.dataclass @dataclasses.dataclass
@ -979,7 +933,7 @@ class PiaApi:
raise ApiError("http", f"token HTTP {last_status}") raise ApiError("http", f"token HTTP {last_status}")
def add_key(self, cand: Candidate, token: str, pubkey: str) -> PeerInfo: def add_key(self, cand: Candidate, token: str, pubkey: str) -> PeerInfo:
status, data = self.http.get_pinned_curl(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", token, pubkey) status, data = self.http.get_pinned(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", {"pt": token, "pubkey": pubkey})
if status != 200: if status != 200:
raise ApiError("http", f"addKey HTTP {status}") raise ApiError("http", f"addKey HTTP {status}")
if not data.strip(): if not data.strip():
@ -1170,7 +1124,7 @@ class Controller:
self.cfg = cfg self.cfg = cfg
self.sys = system self.sys = system
self.log = log self.log = log
self.http = http or HttpClient(cfg, system) self.http = http or HttpClient(cfg)
self._creds = creds self._creds = creds
self.store = StateStore(cfg, system) self.store = StateStore(cfg, system)
self.classifier = Classifier(cfg, system) self.classifier = Classifier(cfg, system)

View File

@ -21,9 +21,6 @@
# killswitch script changed in the same run), pia-wg's own PostUp # killswitch script changed in the same run), pia-wg's own PostUp
# needs to run last so its "open" route is the final word, not # needs to run last so its "open" route is the final word, not
# killswitch's route-table seeding. # killswitch's route-table seeding.
# 4. "Restart pia-killswitch reassertion timer" only restarts the
# *timer* (a scheduling change, e.g. its interval), never runs the
# script itself — no ordering constraint against the others.
- name: Reload systemd daemon - name: Reload systemd daemon
ansible.builtin.systemd: ansible.builtin.systemd:
@ -38,11 +35,6 @@
name: pia-killswitch.service name: pia-killswitch.service
state: restarted state: restarted
- name: Restart pia-killswitch reassertion timer
ansible.builtin.systemd:
name: pia-killswitch-reassert.timer
state: restarted
- name: Restart pia-wg - name: Restart pia-wg
ansible.builtin.systemd: ansible.builtin.systemd:
name: "wg-quick@{{ pia_wg_interface }}" name: "wg-quick@{{ pia_wg_interface }}"

View File

@ -27,16 +27,10 @@
- wireguard-tools - wireguard-tools
- iproute2 - iproute2
- python3 - python3
- curl
state: present state: present
update_cache: true update_cache: true
# python3-requests is no longer needed: the controller uses only the # python3-requests is no longer needed: the controller uses only the
# standard library, EXCEPT for the addKey endpoint specifically, which # standard library.
# shells out to curl — see HttpClient.get_pinned_curl's own docstring
# for why (a non-critical Basic Constraints extension on PIA's own CA
# cert that recent OpenSSL rejects by default and Python's ssl module
# has no way to selectively tolerate). curl is virtually always already
# present on Debian, but declared explicitly rather than assumed.
- name: Pin reverse-path filtering for {{ pia_wg_interface }} - name: Pin reverse-path filtering for {{ pia_wg_interface }}
ansible.posix.sysctl: ansible.posix.sysctl:

View File

@ -40,35 +40,3 @@
name: pia-killswitch.service name: pia-killswitch.service
enabled: true enabled: true
state: started state: started
- name: Deploy the kill-switch periodic reassertion systemd service
ansible.builtin.template:
src: pia-killswitch-reassert.service.j2
dest: /etc/systemd/system/pia-killswitch-reassert.service
mode: "0644"
owner: root
group: root
notify:
- Reload systemd daemon
- Restart pia-killswitch reassertion timer
- name: Deploy the kill-switch periodic reassertion systemd timer
ansible.builtin.template:
src: pia-killswitch-reassert.timer.j2
dest: /etc/systemd/system/pia-killswitch-reassert.timer
mode: "0644"
owner: root
group: root
notify:
- Reload systemd daemon
- Restart pia-killswitch reassertion timer
- name: Enable and start the kill-switch reassertion timer
ansible.builtin.systemd:
name: pia-killswitch-reassert.timer
enabled: true
state: started
daemon_reload: true
# Unconditional — not gated by pia_auto_recovery_enabled. See the
# service unit's own header for why this is scoped differently from
# the automatic recovery/registration timers.

View File

@ -1,30 +0,0 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Periodically re-runs the kill-switch seed script (pia-killswitch.sh) so
# its invariants — source rule, table default, and specifically the
# FORWARD jump's position — are re-asserted on a schedule, not only at
# the external trigger points that used to be the only times it ran
# (boot, a UFW reload, or an Ansible-detected content change to the
# script itself). Needed because other host components that also manage
# iptables — kube-router and dockerd — reassert their OWN position-1
# jumps independently, and confirmed live (2026-09-22) to do so more
# often than originally assumed: the FORWARD jump drifted twice within
# about an hour under this cluster's ordinary CronJob/pod churn (ca-sync,
# porkbun-ddns, gitea-backup, ...), not just at boot-scale events.
#
# The script itself is already fully idempotent (safe to run repeatedly
# — see its own header comment); this unit just gives it more chances to
# run. Deliberately separate from pia-gateway-recover.timer and NOT
# gated by pia_auto_recovery_enabled: this is the kill switch continuing
# to assert its own already-established, narrowly-scoped state — the
# same kind of thing it already does unconditionally at boot and after a
# UFW reload, just on a schedule too — not the higher-level automatic
# recovery/registration logic the plan was deliberately cautious about
# running unattended.
[Unit]
Description=Periodically re-assert the PIA gateway kill switch's FORWARD jump position
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pia-killswitch.sh
LimitCORE=0

View File

@ -1,11 +0,0 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
[Unit]
Description=Run the PIA gateway kill-switch reassertion every {{ pia_healthcheck_interval_sec }}s
[Timer]
OnBootSec=30
OnUnitActiveSec={{ pia_healthcheck_interval_sec }}
AccuracySec=5s
[Install]
WantedBy=timers.target

View File

@ -1,16 +1,12 @@
#!/bin/bash #!/bin/bash
# Managed by Ansible (role: pia-gateway). Do not edit by hand. # Managed by Ansible (role: pia-gateway). Do not edit by hand.
# #
# Runs at boot, before wg-quick@{{ pia_wg_interface }}.service (enforced # Runs once at boot, before wg-quick@{{ pia_wg_interface }}.service
# by the systemd drop-in in pia-killswitch.service's Before=/wg-quick's # (enforced by the systemd drop-in in pia-killswitch.service's
# Requires=After=); any time UFW is reloaded (see handlers/main.yaml — # Before=/wg-quick's Requires=After=), and again any time UFW is
# "Reload ufw" fires before "Restart pia-killswitch" so a reload can # reloaded (see handlers/main.yaml — "Reload ufw" fires before "Restart
# never leave a permanent gap); and — since 2026-09-22, after the FORWARD # pia-killswitch" so a reload can never leave a permanent gap). Seeds the
# jump was found drifting twice within about an hour under this # closed state that PIA gateway's kill switch relies on:
# cluster's ordinary CronJob/pod churn, far more often than a boot/reload
# -only cadence catches — every {{ pia_healthcheck_interval_sec }}s via
# pia-killswitch-reassert.timer. Seeds the closed state that PIA
# gateway's kill switch relies on:
# #
# - a source rule sending {{ pia_vlan50_subnet }} to the dedicated # - a source rule sending {{ pia_vlan50_subnet }} to the dedicated
# "{{ pia_route_table_name }}" table, evaluated after the kernel's # "{{ pia_route_table_name }}" table, evaluated after the kernel's
@ -38,16 +34,11 @@
# source's fate with a broader ACCEPT — the jump is unconditional for # source's fate with a broader ACCEPT — the jump is unconditional for
# this source and the chain itself ends in an unconditional DROP, not # this source and the chain itself ends in an unconditional DROP, not
# a RETURN, so nothing after it in FORWARD is ever consulted for # a RETURN, so nothing after it in FORWARD is ever consulted for
# this source either. "First" is actively re-asserted on every run, # this source either.
# not just seeded once — see the jump-insertion block below for why
# that turned out to matter in practice.
# #
# Idempotent: safe to run more than once (e.g. `systemctl restart # Idempotent: safe to run more than once (e.g. `systemctl restart
# pia-killswitch`, or after this script re-runs following a UFW reload) # pia-killswitch`, or after this script re-runs following a UFW reload)
# without creating duplicate rules or duplicate chains — including # without creating duplicate rules or duplicate chains.
# repositioning the FORWARD jump back to position 1 if something else
# (kube-router, dockerd) has since pushed it down, rather than treating
# its mere existence as proof it's still first.
set -euo pipefail set -euo pipefail
TABLE="{{ pia_route_table_name }}" TABLE="{{ pia_route_table_name }}"
@ -77,32 +68,8 @@ fi
# there is no direct "does this chain exist" check short of grepping -L). # there is no direct "does this chain exist" check short of grepping -L).
iptables -N "${CHAIN}" 2>/dev/null || true iptables -N "${CHAIN}" 2>/dev/null || true
# Exactly one jump from FORWARD into it, kept at absolute position 1. # Exactly one jump from FORWARD into it, at absolute position 1.
# Existence alone ("-C") is not proof of position — confirmed live, if ! iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
# 2026-09-22: kube-router and dockerd each independently reassert their
# OWN jump at position 1 whenever THEY resync/restart (kube-router
# doesn't even dedupe its own older copy when it does, so duplicates of
# ITS rule can appear too), which silently pushes this one down without
# removing it. A plain "-C, insert only if absent" check then reports
# "already exists" and leaves it stranded wherever it drifted to — this
# is exactly what happened here: the jump was 4th (behind kube-router,
# DOCKER-USER, DOCKER-FORWARD), not 1st, on a host that had been up 200
# days with kube-router/Docker managing FORWARD independently the whole
# time. So check the actual first rule in FORWARD, not just whether ours
# exists somewhere; if it isn't ours, remove it (by spec — iptables -D
# matches wherever it currently sits, not by position) and reinsert at
# 1. Safe to do unconditionally on every run: this rule is narrowly
# {{ pia_vlan50_subnet }}-scoped, so moving it earlier relative to
# kube-router's/Docker's own broader jumps can't affect pod or Docker
# bridge traffic either way — confirmed by reading their actual chain
# contents (KUBE-ROUTER-FORWARD only matches specific pod /32
# addresses, DOCKER-USER was empty, DOCKER-FORWARD's one direct rule is
# scoped to -i docker0), not assumed.
FORWARD_FIRST=$(iptables -S FORWARD | grep '^-A FORWARD' | head -1 || true)
if [[ "${FORWARD_FIRST}" != "-A FORWARD -s ${SUBNET} -j ${CHAIN}" ]]; then
if iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
iptables -D FORWARD -s "${SUBNET}" -j "${CHAIN}"
fi
iptables -I FORWARD 1 -s "${SUBNET}" -j "${CHAIN}" iptables -I FORWARD 1 -s "${SUBNET}" -j "${CHAIN}"
fi fi

View File

@ -255,8 +255,8 @@ class FakeHttp:
return 200, val return 200, val
return 200, json.dumps({"token": val}).encode() return 200, json.dumps({"token": val}).encode()
def get_pinned_curl(self, hostname, ip, port, path, token, pubkey): def get_pinned(self, hostname, ip, port, path, query):
self.calls.append(("addkey", hostname, ip, port, path, {"pt": token, "pubkey": pubkey})) self.calls.append(("addkey", hostname, ip, port, path, dict(query)))
kind, val = self.addkey.get(hostname, ("http", 502)) kind, val = self.addkey.get(hostname, ("http", 502))
if kind == "err": if kind == "err":
raise pg.ApiError(val, "fake") raise pg.ApiError(val, "fake")
@ -1227,106 +1227,6 @@ def dataclass_dict(cfg):
return dataclasses.asdict(cfg) return dataclasses.asdict(cfg)
class FakeCurlSystem(pg.System):
"""Fakes just enough of System.run to exercise get_pinned_curl's argv
construction, stdin handling, and exit-code classification, without a
real curl binary or network access."""
def __init__(self, result=None):
self.result = result or pg.CmdResult(0, '{"status":"OK"}', "")
self.calls = []
self.stdins = []
def run(self, argv, stdin=None, timeout=60):
self.calls.append(list(argv))
if stdin is not None:
self.stdins.append(stdin)
return self.result
class TestPinnedCurlAddKey(unittest.TestCase):
def setUp(self):
self.cfg = pg.Config(ca_file="/etc/wireguard/pia-ca.crt", http_connect_timeout_sec=10, http_total_timeout_sec=30)
def test_argv_shape_matches_the_proven_legacy_curl_invocation(self):
sysm = FakeCurlSystem()
http_client = pg.HttpClient(self.cfg, sysm)
status, data = http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
self.assertEqual(status, 200)
self.assertEqual(data, b'{"status":"OK"}')
argv = sysm.calls[0]
self.assertEqual(argv[0], "curl")
self.assertIn("--fail", argv)
self.assertIn("-G", argv)
self.assertIn("--cacert", argv)
self.assertEqual(argv[argv.index("--cacert") + 1], "/etc/wireguard/pia-ca.crt")
self.assertIn("--connect-to", argv)
self.assertEqual(argv[argv.index("--connect-to") + 1], "Server-11669-10a::69.33.198.97:")
self.assertIn("pt@-", argv)
self.assertIn(f"pubkey={PUB_A}", argv)
self.assertEqual(argv[-1], "https://Server-11669-10a:1337/addKey")
# never -k/--insecure — this must still be a real, validating request
self.assertNotIn("-k", argv)
self.assertNotIn("--insecure", argv)
def test_token_passed_via_stdin_with_no_trailing_newline_and_never_in_argv(self):
sysm = FakeCurlSystem()
http_client = pg.HttpClient(self.cfg, sysm)
http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
self.assertEqual(sysm.stdins, [SECRET_TOKEN]) # exact match: no appended "\n"
for arg in sysm.calls[0]:
self.assertNotIn(SECRET_TOKEN, arg)
def test_timeout_exit_code_classified(self):
sysm = FakeCurlSystem(pg.CmdResult(28, "", "curl: (28) Connection timed out"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "timeout")
def test_tls_exit_codes_classified_with_stderr_detail(self):
for rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83):
with self.subTest(rc=rc):
sysm = FakeCurlSystem(pg.CmdResult(rc, "", "curl: (60) SSL certificate problem"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "tls")
self.assertIn("SSL certificate problem", cm.exception.detail)
def test_http_failure_exit_code_classified(self):
sysm = FakeCurlSystem(pg.CmdResult(22, "", "curl: (22) The requested URL returned error: 401"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "http")
def test_nonstandard_exit_code_with_fail_text_still_classified_as_http(self):
# Observed live: curl built against SecureTransport reports exit 56
# (CURLE_RECV_ERROR), not the "usual" 22, for the same --fail
# -triggered HTTP rejection — the error text is the portable signal.
sysm = FakeCurlSystem(pg.CmdResult(56, "", "curl: (56) The requested URL returned error: 401"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "http")
def test_curl_missing_classified_as_network_not_a_crash(self):
sysm = FakeCurlSystem(pg.CmdResult(127, "", "curl: not found"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "network")
def test_unclassified_exit_code_falls_back_to_network_with_stderr(self):
sysm = FakeCurlSystem(pg.CmdResult(7, "", "curl: (7) Couldn't connect to server"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "network")
self.assertIn("Couldn't connect", cm.exception.detail)
class TestHttpHelpers(unittest.TestCase): class TestHttpHelpers(unittest.TestCase):
def test_key_and_ip_validators(self): def test_key_and_ip_validators(self):
self.assertTrue(pg.valid_wg_key(PUB_A)) self.assertTrue(pg.valid_wg_key(PUB_A))