Compare commits
4 Commits
87a9303cc3
...
89d289addf
| Author | SHA1 | Date | |
|---|---|---|---|
| 89d289addf | |||
| 99f6ef84f0 | |||
| 7b78e4a88e | |||
| de5ce7bf52 |
@ -163,6 +163,48 @@ real.
|
||||
Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW
|
||||
**input** allow for 53/tcp+udp — it's locally-terminated traffic, not
|
||||
routed — see `tasks/firewall.yaml`.
|
||||
|
||||
**The FORWARD jump's position is actively re-asserted on every run, not
|
||||
just seeded once** — found live, 2026-09-22, on the very first
|
||||
deployment of the auto-recovery controller. minisforum's `FORWARD`
|
||||
chain default policy is `DROP` (kube-router's doing, not this role's —
|
||||
correcting an assumption an earlier version of this doc made), and
|
||||
kube-router and `dockerd` each independently reassert their **own**
|
||||
jump at position 1 whenever they resync or restart, silently pushing
|
||||
this rule down without removing it (kube-router doesn't even dedupe
|
||||
its own older copy when it does — a duplicate of *its* rule was found
|
||||
live too). The script used to check only whether its rule *existed*
|
||||
(`iptables -C`) before deciding not to touch anything; existence isn't
|
||||
proof of position, so once pushed down it stayed down until something
|
||||
else fixed it. It now checks the actual first rule and repositions
|
||||
(delete + reinsert at 1) if it isn't this one. Confirmed safe to do
|
||||
unconditionally by reading kube-router's and Docker's actual chain
|
||||
contents rather than assuming: `KUBE-ROUTER-FORWARD` only matches
|
||||
specific pod `/32` addresses (disjoint from `10.10.50.0/24`),
|
||||
`DOCKER-USER` was empty, and `DOCKER-FORWARD`'s one direct rule is
|
||||
scoped to `-i docker0` (VLAN 50 traffic arrives via `enp1s0`) — none
|
||||
of them can intercept VLAN 50 traffic regardless of order, so moving
|
||||
this rule back to position 1 relative to them changes nothing for pod
|
||||
or Docker-bridge traffic. The controller's own safety check
|
||||
(`forward_jump_first`) was working as intended when it refused to
|
||||
register against the drifted state — this is a fix to the thing it
|
||||
checks, not a loosening of the check itself.
|
||||
|
||||
**This recurs far more often than "boot-scale," and the fix above only
|
||||
helped once.** The reposition logic is correct, but it only *ran* on
|
||||
the existing triggers (boot, a UFW reload, an Ansible-detected content
|
||||
change to the script). Confirmed live: after that fix landed and got
|
||||
one `pia-gateway register` attempt past this check, the very next
|
||||
attempt — roughly an hour later, no reboot, no UFW reload, no script
|
||||
change in between — hit the identical failure again. kube-router
|
||||
resyncs on ordinary Kubernetes API events (this cluster's CronJobs —
|
||||
`ca-sync`, `porkbun-ddns`, `gitea-backup` — each create and delete a
|
||||
Pod every few minutes, each a plausible trigger), not just at
|
||||
boot/restart scale as originally assumed. `pia-killswitch-reassert
|
||||
.timer` (every `pia_healthcheck_interval_sec`, unconditional, not
|
||||
gated by `pia_auto_recovery_enabled`) exists specifically to close this
|
||||
gap by giving the already-correct, already-idempotent script regular
|
||||
chances to run instead of only sporadic ones.
|
||||
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
|
||||
top of that closed baseline: default route via `pia-wg` in the `pia`
|
||||
table, ACCEPT + established/related return inserted into the
|
||||
@ -242,10 +284,14 @@ handled automatically — see the next section.
|
||||
## Automatic recovery
|
||||
|
||||
`files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway`
|
||||
(root, `0700`, standard library only — runs on the host's `python3.13`).
|
||||
It is **one** implementation of PIA registration shared by the unattended
|
||||
timers and by this role's own registration step, so the two can never
|
||||
drift. Layout on the host:
|
||||
(root, `0700`, runs on the host's `python3.13`). Almost entirely standard
|
||||
library — the one exception is `addKey` specifically, which shells out to
|
||||
`curl` (now an explicit `apt` dependency of this role); see
|
||||
`HttpClient.get_pinned_curl`'s own docstring and "PIA's CA certificate and
|
||||
why addKey uses curl, not Python's `ssl`" below for why. It is **one**
|
||||
implementation of PIA registration shared by the unattended timers and by
|
||||
this role's own registration step, so the two can never drift. Layout on
|
||||
the host:
|
||||
|
||||
| Path | Purpose |
|
||||
| --- | --- |
|
||||
@ -258,6 +304,65 @@ drift. Layout on the host:
|
||||
| `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) |
|
||||
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
|
||||
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
|
||||
| `pia-killswitch-reassert.timer` → `.service` | every `pia_healthcheck_interval_sec`, re-runs `pia-killswitch.sh` (idempotent). **Not** gated by `pia_auto_recovery_enabled` — see the "FORWARD jump's position" note above and the service unit's own header for why |
|
||||
|
||||
### PIA's CA certificate and why addKey uses curl, not Python's `ssl`
|
||||
|
||||
Found live, 2026-09-22, on the very first `pia-gateway register` attempt
|
||||
after the FORWARD-jump fix above got past the kill-switch check:
|
||||
`addKey` failed against every candidate with `kind=tls detail=certificate
|
||||
verification failed`. Root-caused directly on minisforum:
|
||||
|
||||
```
|
||||
$ sudo python3 -c '...ctx.wrap_socket(sock, server_hostname="Server-11669-10a")...'
|
||||
SSLCertVerificationError verify_message= Basic Constraints of CA cert not marked critical
|
||||
```
|
||||
|
||||
PIA's own CA certificate (`files/pia-ca.crt`) has a `CA:TRUE` Basic
|
||||
Constraints extension that isn't flagged `critical` — a real RFC 5280
|
||||
non-conformance, confirmed **not** a stale vendored copy: a fresh fetch
|
||||
from `pia-foss/manual-connections` on GitHub is byte-identical (same
|
||||
`32e9b1d1...` SHA-256 already recorded in the file's own header) and has
|
||||
the identical defect. This is PIA's problem, not ours, and not fixable by
|
||||
re-vendoring. Recent OpenSSL (3.2+; confirmed live against minisforum's
|
||||
`3.5.7`) rejects this by default — a check that used to be opt-in only
|
||||
under `X509_V_FLAG_X509_STRICT` and has since been promoted into the
|
||||
default chain-verification path. `openssl s_client` on this repo's own
|
||||
dev machine (LibreSSL) and this same host's `curl` both accept the exact
|
||||
same certificate/chain/CA file without complaint — only Python's `ssl`
|
||||
module, using this host's system OpenSSL, rejects it. That's also why
|
||||
this went unnoticed until the auto-recovery work: the legacy Ansible
|
||||
path's three successful registrations (2026-08-24, 08-25, and this
|
||||
morning) all went through `curl --cacert` (`addkey-attempt.yaml`), never
|
||||
through Python's `ssl` module — this was the *first* time that code path
|
||||
ever ran for real.
|
||||
|
||||
Python's stdlib `ssl` module has no custom-verify-callback hook (a
|
||||
long-standing, deliberate upstream gap — unlike Go's
|
||||
`tls.Config.VerifyPeerCertificate` or OpenSSL's own C API), so there is
|
||||
no clean way to tolerate just this one known, third-party-controlled
|
||||
defect while still fully validating everything else (chain, expiry,
|
||||
hostname/SAN). Rather than weakening `verify_mode` for the whole
|
||||
connection or adding a new dependency (`pyOpenSSL`, for a real verify
|
||||
callback — considered, rejected as a bigger lift than necessary) to work
|
||||
around a gap in a library we don't control, `addKey` specifically now
|
||||
shells out to `curl` (`HttpClient.get_pinned_curl`, `curl` added to this
|
||||
role's `apt` install list) — the same tool PIA's own reference client
|
||||
(`connect_to_wireguard_with_token.sh`) already uses for this exact
|
||||
endpoint, with `--cacert` and never `-k`/`--insecure`, so this is a
|
||||
different, already-proven-correct TLS backend for the one endpoint that
|
||||
needs it, not a security downgrade. `serverlist` and the token endpoint
|
||||
are unaffected (they present public WebPKI certs, which don't have this
|
||||
problem) and still use the stdlib `ssl` path. Verified: a real subprocess
|
||||
`curl` call from this repo's dev machine against the actual failing
|
||||
candidate (`Server-11669-10a` / `69.33.198.97`) gets a genuine HTTP
|
||||
response from PIA (401, expected — a dummy token was used), proving TLS
|
||||
validation succeeds via this path where it failed via Python's `ssl`.
|
||||
curl's own exit codes for a `--fail`-triggered rejection aren't fully
|
||||
portable either — confirmed live: a curl built against SecureTransport
|
||||
(this dev machine) reports exit 56, not the "usual" 22 GNU/OpenSSL curl
|
||||
uses for the identical case — so classification checks curl's own error
|
||||
text as a second signal alongside the exit code, not the exit code alone.
|
||||
|
||||
### Status classes
|
||||
|
||||
@ -440,10 +545,37 @@ external secret broker — not a 24-hour token.
|
||||
- `unknown` (a read command failing) is logged/alerted but never acted
|
||||
on, by design.
|
||||
|
||||
### Rollout (nothing below has been run live yet)
|
||||
### Rollout status (2026-09-22)
|
||||
|
||||
Step 1 (baseline) and three attempts at step 2 have been run live.
|
||||
|
||||
- **Attempt 1** failed: `pia-gateway register` correctly refused with
|
||||
`safety invariants failing ... FORWARD jump to kill-switch chain is
|
||||
not first` — a real, pre-existing drift (see the "FORWARD jump's
|
||||
position is actively re-asserted" note above), not a false alarm from
|
||||
this session's work, and not the "session dies" handshake issue
|
||||
either. Root-caused and fixed in `pia-killswitch.sh.j2`.
|
||||
- **Attempt 2**, after that fix, got past the kill-switch check and
|
||||
failed differently: `addKey` rejected every candidate with
|
||||
`kind=tls detail=certificate verification failed`. Root-caused and
|
||||
fixed — see "PIA's CA certificate and why addKey uses curl, not
|
||||
Python's `ssl`" above.
|
||||
- **Attempt 3**, roughly an hour after attempt 1's fix with no reboot,
|
||||
UFW reload, or script change in between, hit the *identical* FORWARD-
|
||||
jump failure again — proving the drift recurs on a timescale of
|
||||
minutes under this cluster's normal CronJob/pod churn, not just at
|
||||
boot-scale events as originally assumed. Unblocked live with a manual
|
||||
`sudo systemctl restart pia-killswitch.service`; the durable fix is
|
||||
`pia-killswitch-reassert.timer` (added, not yet applied or verified
|
||||
live) — see the "FORWARD jump's position" note's "recurs far more
|
||||
often" addendum above.
|
||||
|
||||
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
|
||||
— the original `head -3` suggestion is what hid the first issue; the
|
||||
drift was 4 rules in, not at the top.
|
||||
|
||||
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
|
||||
table pia`, `iptables -S FORWARD | head -3`, `iptables -S PIA-VLAN50`,
|
||||
table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`,
|
||||
`journalctl -u pia-gateway-healthcheck -n 5`, `curl --socks5-hostname
|
||||
10.10.40.20:1080 https://ifconfig.me` from the LAN, `wg show wg0`,
|
||||
`kubectl get nodes`. Confirm console access.
|
||||
@ -454,7 +586,14 @@ external secret broker — not a 24-hour token.
|
||||
verified baseline yet, this run **registers through the shared path**
|
||||
(same-key, explicit restart + handshake verification) — watch the
|
||||
task output; if it fails, `-e pia_registration_backend=legacy`
|
||||
restores the previous behaviour.
|
||||
restores the previous behaviour. If it fails specifically with
|
||||
`FORWARD jump to kill-switch chain is not first`, that's the drift
|
||||
above — the fixed template should correct it via the normal "Restart
|
||||
pia-killswitch" handler on the same run (the template content
|
||||
changing is enough to trigger it); re-running the identical command
|
||||
should then get past registration. Confirm with `iptables -S FORWARD`
|
||||
afterwards that the jump is genuinely first, not just that the retry
|
||||
succeeded.
|
||||
3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy
|
||||
pass makes no changes (`journalctl -u pia-gateway-recover -n 5`).
|
||||
4. Drill the silent death. First get the **live WireGuard UDP endpoint**
|
||||
@ -531,8 +670,10 @@ touching `wg0`, k3s, or the host default route):
|
||||
|
||||
```bash
|
||||
sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown
|
||||
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
|
||||
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-killswitch-reassert.timer pia-gateway-healthcheck.timer
|
||||
sudo rm -f /etc/systemd/system/pia-killswitch.service \
|
||||
/etc/systemd/system/pia-killswitch-reassert.service \
|
||||
/etc/systemd/system/pia-killswitch-reassert.timer \
|
||||
/etc/systemd/system/pia-gateway-healthcheck.service \
|
||||
/etc/systemd/system/pia-gateway-healthcheck.timer \
|
||||
/etc/systemd/system/pia-gateway-recover.service \
|
||||
|
||||
@ -751,33 +751,14 @@ class Classifier:
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
class _PinnedHTTPSConnection(http.client.HTTPSConnection):
|
||||
"""Connect to a fixed IP while verifying TLS against the hostname (SNI)."""
|
||||
|
||||
def __init__(self, host: str, ip: str, port: int, context: ssl.SSLContext, connect_timeout: float, total_timeout: float):
|
||||
super().__init__(host, port, timeout=connect_timeout, context=context)
|
||||
self._pinned_ip = ip
|
||||
self._total_timeout = total_timeout
|
||||
|
||||
def connect(self) -> None: # noqa: D401
|
||||
sock = socket.create_connection((self._pinned_ip, self.port), self.timeout)
|
||||
sock.settimeout(self._total_timeout)
|
||||
self.sock = self._context.wrap_socket(sock, server_hostname=self.host)
|
||||
|
||||
|
||||
class HttpClient:
|
||||
def __init__(self, cfg: Config):
|
||||
def __init__(self, cfg: Config, system: Optional[System] = None):
|
||||
self.cfg = cfg
|
||||
self.system = system or System()
|
||||
|
||||
def _system_ctx(self) -> ssl.SSLContext:
|
||||
return ssl.create_default_context()
|
||||
|
||||
def _pia_ctx(self) -> ssl.SSLContext:
|
||||
ctx = ssl.create_default_context(cafile=self.cfg.ca_file)
|
||||
ctx.check_hostname = True
|
||||
ctx.verify_mode = ssl.CERT_REQUIRED
|
||||
return ctx
|
||||
|
||||
def _request(self, conn: http.client.HTTPSConnection, method: str, path: str, body: Optional[bytes], headers: dict) -> tuple[int, bytes]:
|
||||
try:
|
||||
conn.request(method, path, body=body, headers=headers)
|
||||
@ -785,13 +766,22 @@ class HttpClient:
|
||||
data = resp.read(4_000_000)
|
||||
return resp.status, data
|
||||
except ssl.SSLCertVerificationError as e:
|
||||
raise ApiError("tls", "certificate verification failed") from e
|
||||
# verify_message/reason are standard OpenSSL diagnostic text
|
||||
# (e.g. "unable to get local issuer certificate", "certificate
|
||||
# has expired", a hostname mismatch) — never a secret, and
|
||||
# exactly what's needed to tell "wrong CA", "expired cert",
|
||||
# "clock skew", and "hostname mismatch" apart instead of
|
||||
# collapsing all of them into one opaque string.
|
||||
detail = e.verify_message or e.reason or str(e) or "certificate verification failed"
|
||||
raise ApiError("tls", detail) from e
|
||||
except ssl.SSLError as e:
|
||||
raise ApiError("tls", e.__class__.__name__) from e
|
||||
detail = getattr(e, "reason", None) or str(e) or e.__class__.__name__
|
||||
raise ApiError("tls", f"{e.__class__.__name__}: {detail}") from e
|
||||
except socket.timeout as e:
|
||||
raise ApiError("timeout", "request timed out") from e
|
||||
except (OSError, http.client.HTTPException) as e:
|
||||
raise ApiError("network", e.__class__.__name__) from e
|
||||
detail = str(e) or e.__class__.__name__
|
||||
raise ApiError("network", f"{e.__class__.__name__}: {detail}") from e
|
||||
finally:
|
||||
with contextlib.suppress(Exception):
|
||||
conn.close()
|
||||
@ -816,12 +806,68 @@ class HttpClient:
|
||||
}
|
||||
return self._request(conn, "POST", u.path, body, headers)
|
||||
|
||||
def get_pinned(self, hostname: str, ip: str, port: int, path: str, query: dict[str, str]) -> tuple[int, bytes]:
|
||||
conn = _PinnedHTTPSConnection(
|
||||
hostname, ip, port, self._pia_ctx(), self.cfg.http_connect_timeout_sec, self.cfg.http_total_timeout_sec
|
||||
)
|
||||
full = path + "?" + urllib.parse.urlencode(query)
|
||||
return self._request(conn, "GET", full, None, {"User-Agent": "pia-gateway"})
|
||||
def get_pinned_curl(self, hostname: str, ip: str, port: int, path: str, token: str, pubkey: str) -> tuple[int, bytes]:
|
||||
"""Pinned-IP, SNI-validated, CA-pinned GET — for PIA's addKey only.
|
||||
|
||||
This shells out to curl instead of using the stdlib `ssl` module.
|
||||
Confirmed live (2026-09-22): PIA's own CA certificate (byte-
|
||||
identical to a fresh fetch from PIA's own repo — not a stale
|
||||
vendored copy) has a non-critical Basic Constraints extension on
|
||||
its CA cert, a real RFC 5280 non-conformance in a CA we don't
|
||||
control. Recent OpenSSL (3.2+; confirmed on minisforum's 3.5.7 via
|
||||
a direct repro — SSLCertVerificationError, verify_message
|
||||
"Basic Constraints of CA cert not marked critical") rejects that
|
||||
in its default chain verification, a check promoted from
|
||||
X509_STRICT-only to the default path in recent OpenSSL releases.
|
||||
Python's stdlib `ssl` module has no custom verify-callback hook to
|
||||
selectively tolerate just this one known, third-party-controlled
|
||||
defect while still fully validating everything else (chain,
|
||||
expiry, hostname/SAN) — that gap is long-standing and
|
||||
deliberate upstream, not something to route around by weakening
|
||||
verify_mode. curl's TLS stack on this host does not enforce this
|
||||
specific check and is what PIA's own reference implementation
|
||||
(connect_to_wireguard_with_token.sh) already uses for this exact
|
||||
endpoint — reusing it here with --cacert (never -k/--insecure) is
|
||||
not a security downgrade, just a different, already-proven-correct
|
||||
TLS backend for the one endpoint that needs it. serverlist and
|
||||
token still use the stdlib ssl path above against public WebPKI
|
||||
certs, which do not have this problem.
|
||||
"""
|
||||
argv = [
|
||||
"curl", "--silent", "--show-error", "--fail", "--location", "-G",
|
||||
"--connect-timeout", str(self.cfg.http_connect_timeout_sec),
|
||||
"--max-time", str(self.cfg.http_total_timeout_sec),
|
||||
"--cacert", self.cfg.ca_file,
|
||||
"--connect-to", f"{hostname}::{ip}:",
|
||||
"--data-urlencode", "pt@-",
|
||||
"--data-urlencode", f"pubkey={pubkey}",
|
||||
f"https://{hostname}:{port}{path}",
|
||||
]
|
||||
r = self.system.run(argv, stdin=token, timeout=self.cfg.http_total_timeout_sec + 10)
|
||||
if r.rc == 0:
|
||||
return 200, r.out.encode("utf-8")
|
||||
err = (r.err or "").strip()
|
||||
# Exit codes are the primary signal, but not fully portable across
|
||||
# curl builds/TLS backends — confirmed live: a curl built against
|
||||
# SecureTransport reported exit 56 (CURLE_RECV_ERROR) for the exact
|
||||
# same --fail-triggered HTTP rejection GNU/OpenSSL curl reports as
|
||||
# 22. curl's own error text ("returned error: NNN" for --fail,
|
||||
# "timed out" for CURLE_OPERATION_TIMEDOUT, TLS-related wording for
|
||||
# cert/handshake problems) is a second, more portable signal — used
|
||||
# here as a fallback classifier alongside the exit code, not a
|
||||
# replacement for it.
|
||||
if r.rc == 127:
|
||||
raise ApiError("network", "curl not found on PATH")
|
||||
if r.rc == 28 or "timed out" in err.lower():
|
||||
raise ApiError("timeout", f"curl: connection or transfer timed out: {err}")
|
||||
if r.rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83) or "ssl" in err.lower() or "certificate" in err.lower():
|
||||
raise ApiError("tls", f"curl TLS error (exit {r.rc}): {err}")
|
||||
if r.rc == 22 or "returned error:" in err:
|
||||
# --fail suppresses the body on a non-2xx response, so there is
|
||||
# no status code or body to parse here — same limitation the
|
||||
# legacy curl-based Ansible path already had.
|
||||
raise ApiError("http", f"curl: non-2xx HTTP response (exit {r.rc}): {err}")
|
||||
raise ApiError("network", f"curl exit {r.rc}: {err}")
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
@ -933,7 +979,7 @@ class PiaApi:
|
||||
raise ApiError("http", f"token HTTP {last_status}")
|
||||
|
||||
def add_key(self, cand: Candidate, token: str, pubkey: str) -> PeerInfo:
|
||||
status, data = self.http.get_pinned(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", {"pt": token, "pubkey": pubkey})
|
||||
status, data = self.http.get_pinned_curl(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", token, pubkey)
|
||||
if status != 200:
|
||||
raise ApiError("http", f"addKey HTTP {status}")
|
||||
if not data.strip():
|
||||
@ -1124,7 +1170,7 @@ class Controller:
|
||||
self.cfg = cfg
|
||||
self.sys = system
|
||||
self.log = log
|
||||
self.http = http or HttpClient(cfg)
|
||||
self.http = http or HttpClient(cfg, system)
|
||||
self._creds = creds
|
||||
self.store = StateStore(cfg, system)
|
||||
self.classifier = Classifier(cfg, system)
|
||||
|
||||
@ -21,6 +21,9 @@
|
||||
# killswitch script changed in the same run), pia-wg's own PostUp
|
||||
# needs to run last so its "open" route is the final word, not
|
||||
# killswitch's route-table seeding.
|
||||
# 4. "Restart pia-killswitch reassertion timer" only restarts the
|
||||
# *timer* (a scheduling change, e.g. its interval), never runs the
|
||||
# script itself — no ordering constraint against the others.
|
||||
|
||||
- name: Reload systemd daemon
|
||||
ansible.builtin.systemd:
|
||||
@ -35,6 +38,11 @@
|
||||
name: pia-killswitch.service
|
||||
state: restarted
|
||||
|
||||
- name: Restart pia-killswitch reassertion timer
|
||||
ansible.builtin.systemd:
|
||||
name: pia-killswitch-reassert.timer
|
||||
state: restarted
|
||||
|
||||
- name: Restart pia-wg
|
||||
ansible.builtin.systemd:
|
||||
name: "wg-quick@{{ pia_wg_interface }}"
|
||||
|
||||
@ -27,10 +27,16 @@
|
||||
- wireguard-tools
|
||||
- iproute2
|
||||
- python3
|
||||
- curl
|
||||
state: present
|
||||
update_cache: true
|
||||
# python3-requests is no longer needed: the controller uses only the
|
||||
# standard library.
|
||||
# standard library, EXCEPT for the addKey endpoint specifically, which
|
||||
# shells out to curl — see HttpClient.get_pinned_curl's own docstring
|
||||
# for why (a non-critical Basic Constraints extension on PIA's own CA
|
||||
# cert that recent OpenSSL rejects by default and Python's ssl module
|
||||
# has no way to selectively tolerate). curl is virtually always already
|
||||
# present on Debian, but declared explicitly rather than assumed.
|
||||
|
||||
- name: Pin reverse-path filtering for {{ pia_wg_interface }}
|
||||
ansible.posix.sysctl:
|
||||
|
||||
@ -40,3 +40,35 @@
|
||||
name: pia-killswitch.service
|
||||
enabled: true
|
||||
state: started
|
||||
|
||||
- name: Deploy the kill-switch periodic reassertion systemd service
|
||||
ansible.builtin.template:
|
||||
src: pia-killswitch-reassert.service.j2
|
||||
dest: /etc/systemd/system/pia-killswitch-reassert.service
|
||||
mode: "0644"
|
||||
owner: root
|
||||
group: root
|
||||
notify:
|
||||
- Reload systemd daemon
|
||||
- Restart pia-killswitch reassertion timer
|
||||
|
||||
- name: Deploy the kill-switch periodic reassertion systemd timer
|
||||
ansible.builtin.template:
|
||||
src: pia-killswitch-reassert.timer.j2
|
||||
dest: /etc/systemd/system/pia-killswitch-reassert.timer
|
||||
mode: "0644"
|
||||
owner: root
|
||||
group: root
|
||||
notify:
|
||||
- Reload systemd daemon
|
||||
- Restart pia-killswitch reassertion timer
|
||||
|
||||
- name: Enable and start the kill-switch reassertion timer
|
||||
ansible.builtin.systemd:
|
||||
name: pia-killswitch-reassert.timer
|
||||
enabled: true
|
||||
state: started
|
||||
daemon_reload: true
|
||||
# Unconditional — not gated by pia_auto_recovery_enabled. See the
|
||||
# service unit's own header for why this is scoped differently from
|
||||
# the automatic recovery/registration timers.
|
||||
|
||||
@ -0,0 +1,30 @@
|
||||
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
|
||||
#
|
||||
# Periodically re-runs the kill-switch seed script (pia-killswitch.sh) so
|
||||
# its invariants — source rule, table default, and specifically the
|
||||
# FORWARD jump's position — are re-asserted on a schedule, not only at
|
||||
# the external trigger points that used to be the only times it ran
|
||||
# (boot, a UFW reload, or an Ansible-detected content change to the
|
||||
# script itself). Needed because other host components that also manage
|
||||
# iptables — kube-router and dockerd — reassert their OWN position-1
|
||||
# jumps independently, and confirmed live (2026-09-22) to do so more
|
||||
# often than originally assumed: the FORWARD jump drifted twice within
|
||||
# about an hour under this cluster's ordinary CronJob/pod churn (ca-sync,
|
||||
# porkbun-ddns, gitea-backup, ...), not just at boot-scale events.
|
||||
#
|
||||
# The script itself is already fully idempotent (safe to run repeatedly
|
||||
# — see its own header comment); this unit just gives it more chances to
|
||||
# run. Deliberately separate from pia-gateway-recover.timer and NOT
|
||||
# gated by pia_auto_recovery_enabled: this is the kill switch continuing
|
||||
# to assert its own already-established, narrowly-scoped state — the
|
||||
# same kind of thing it already does unconditionally at boot and after a
|
||||
# UFW reload, just on a schedule too — not the higher-level automatic
|
||||
# recovery/registration logic the plan was deliberately cautious about
|
||||
# running unattended.
|
||||
[Unit]
|
||||
Description=Periodically re-assert the PIA gateway kill switch's FORWARD jump position
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/pia-killswitch.sh
|
||||
LimitCORE=0
|
||||
@ -0,0 +1,11 @@
|
||||
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
|
||||
[Unit]
|
||||
Description=Run the PIA gateway kill-switch reassertion every {{ pia_healthcheck_interval_sec }}s
|
||||
|
||||
[Timer]
|
||||
OnBootSec=30
|
||||
OnUnitActiveSec={{ pia_healthcheck_interval_sec }}
|
||||
AccuracySec=5s
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@ -1,12 +1,16 @@
|
||||
#!/bin/bash
|
||||
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
|
||||
#
|
||||
# Runs once at boot, before wg-quick@{{ pia_wg_interface }}.service
|
||||
# (enforced by the systemd drop-in in pia-killswitch.service's
|
||||
# Before=/wg-quick's Requires=After=), and again any time UFW is
|
||||
# reloaded (see handlers/main.yaml — "Reload ufw" fires before "Restart
|
||||
# pia-killswitch" so a reload can never leave a permanent gap). Seeds the
|
||||
# closed state that PIA gateway's kill switch relies on:
|
||||
# Runs at boot, before wg-quick@{{ pia_wg_interface }}.service (enforced
|
||||
# by the systemd drop-in in pia-killswitch.service's Before=/wg-quick's
|
||||
# Requires=After=); any time UFW is reloaded (see handlers/main.yaml —
|
||||
# "Reload ufw" fires before "Restart pia-killswitch" so a reload can
|
||||
# never leave a permanent gap); and — since 2026-09-22, after the FORWARD
|
||||
# jump was found drifting twice within about an hour under this
|
||||
# cluster's ordinary CronJob/pod churn, far more often than a boot/reload
|
||||
# -only cadence catches — every {{ pia_healthcheck_interval_sec }}s via
|
||||
# pia-killswitch-reassert.timer. Seeds the closed state that PIA
|
||||
# gateway's kill switch relies on:
|
||||
#
|
||||
# - a source rule sending {{ pia_vlan50_subnet }} to the dedicated
|
||||
# "{{ pia_route_table_name }}" table, evaluated after the kernel's
|
||||
@ -34,11 +38,16 @@
|
||||
# source's fate with a broader ACCEPT — the jump is unconditional for
|
||||
# this source and the chain itself ends in an unconditional DROP, not
|
||||
# a RETURN, so nothing after it in FORWARD is ever consulted for
|
||||
# this source either.
|
||||
# this source either. "First" is actively re-asserted on every run,
|
||||
# not just seeded once — see the jump-insertion block below for why
|
||||
# that turned out to matter in practice.
|
||||
#
|
||||
# Idempotent: safe to run more than once (e.g. `systemctl restart
|
||||
# pia-killswitch`, or after this script re-runs following a UFW reload)
|
||||
# without creating duplicate rules or duplicate chains.
|
||||
# without creating duplicate rules or duplicate chains — including
|
||||
# repositioning the FORWARD jump back to position 1 if something else
|
||||
# (kube-router, dockerd) has since pushed it down, rather than treating
|
||||
# its mere existence as proof it's still first.
|
||||
set -euo pipefail
|
||||
|
||||
TABLE="{{ pia_route_table_name }}"
|
||||
@ -68,8 +77,32 @@ fi
|
||||
# there is no direct "does this chain exist" check short of grepping -L).
|
||||
iptables -N "${CHAIN}" 2>/dev/null || true
|
||||
|
||||
# Exactly one jump from FORWARD into it, at absolute position 1.
|
||||
if ! iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
|
||||
# Exactly one jump from FORWARD into it, kept at absolute position 1.
|
||||
# Existence alone ("-C") is not proof of position — confirmed live,
|
||||
# 2026-09-22: kube-router and dockerd each independently reassert their
|
||||
# OWN jump at position 1 whenever THEY resync/restart (kube-router
|
||||
# doesn't even dedupe its own older copy when it does, so duplicates of
|
||||
# ITS rule can appear too), which silently pushes this one down without
|
||||
# removing it. A plain "-C, insert only if absent" check then reports
|
||||
# "already exists" and leaves it stranded wherever it drifted to — this
|
||||
# is exactly what happened here: the jump was 4th (behind kube-router,
|
||||
# DOCKER-USER, DOCKER-FORWARD), not 1st, on a host that had been up 200
|
||||
# days with kube-router/Docker managing FORWARD independently the whole
|
||||
# time. So check the actual first rule in FORWARD, not just whether ours
|
||||
# exists somewhere; if it isn't ours, remove it (by spec — iptables -D
|
||||
# matches wherever it currently sits, not by position) and reinsert at
|
||||
# 1. Safe to do unconditionally on every run: this rule is narrowly
|
||||
# {{ pia_vlan50_subnet }}-scoped, so moving it earlier relative to
|
||||
# kube-router's/Docker's own broader jumps can't affect pod or Docker
|
||||
# bridge traffic either way — confirmed by reading their actual chain
|
||||
# contents (KUBE-ROUTER-FORWARD only matches specific pod /32
|
||||
# addresses, DOCKER-USER was empty, DOCKER-FORWARD's one direct rule is
|
||||
# scoped to -i docker0), not assumed.
|
||||
FORWARD_FIRST=$(iptables -S FORWARD | grep '^-A FORWARD' | head -1 || true)
|
||||
if [[ "${FORWARD_FIRST}" != "-A FORWARD -s ${SUBNET} -j ${CHAIN}" ]]; then
|
||||
if iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
|
||||
iptables -D FORWARD -s "${SUBNET}" -j "${CHAIN}"
|
||||
fi
|
||||
iptables -I FORWARD 1 -s "${SUBNET}" -j "${CHAIN}"
|
||||
fi
|
||||
|
||||
|
||||
@ -255,8 +255,8 @@ class FakeHttp:
|
||||
return 200, val
|
||||
return 200, json.dumps({"token": val}).encode()
|
||||
|
||||
def get_pinned(self, hostname, ip, port, path, query):
|
||||
self.calls.append(("addkey", hostname, ip, port, path, dict(query)))
|
||||
def get_pinned_curl(self, hostname, ip, port, path, token, pubkey):
|
||||
self.calls.append(("addkey", hostname, ip, port, path, {"pt": token, "pubkey": pubkey}))
|
||||
kind, val = self.addkey.get(hostname, ("http", 502))
|
||||
if kind == "err":
|
||||
raise pg.ApiError(val, "fake")
|
||||
@ -1227,6 +1227,106 @@ def dataclass_dict(cfg):
|
||||
return dataclasses.asdict(cfg)
|
||||
|
||||
|
||||
class FakeCurlSystem(pg.System):
|
||||
"""Fakes just enough of System.run to exercise get_pinned_curl's argv
|
||||
construction, stdin handling, and exit-code classification, without a
|
||||
real curl binary or network access."""
|
||||
|
||||
def __init__(self, result=None):
|
||||
self.result = result or pg.CmdResult(0, '{"status":"OK"}', "")
|
||||
self.calls = []
|
||||
self.stdins = []
|
||||
|
||||
def run(self, argv, stdin=None, timeout=60):
|
||||
self.calls.append(list(argv))
|
||||
if stdin is not None:
|
||||
self.stdins.append(stdin)
|
||||
return self.result
|
||||
|
||||
|
||||
class TestPinnedCurlAddKey(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.cfg = pg.Config(ca_file="/etc/wireguard/pia-ca.crt", http_connect_timeout_sec=10, http_total_timeout_sec=30)
|
||||
|
||||
def test_argv_shape_matches_the_proven_legacy_curl_invocation(self):
|
||||
sysm = FakeCurlSystem()
|
||||
http_client = pg.HttpClient(self.cfg, sysm)
|
||||
status, data = http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
|
||||
self.assertEqual(status, 200)
|
||||
self.assertEqual(data, b'{"status":"OK"}')
|
||||
argv = sysm.calls[0]
|
||||
self.assertEqual(argv[0], "curl")
|
||||
self.assertIn("--fail", argv)
|
||||
self.assertIn("-G", argv)
|
||||
self.assertIn("--cacert", argv)
|
||||
self.assertEqual(argv[argv.index("--cacert") + 1], "/etc/wireguard/pia-ca.crt")
|
||||
self.assertIn("--connect-to", argv)
|
||||
self.assertEqual(argv[argv.index("--connect-to") + 1], "Server-11669-10a::69.33.198.97:")
|
||||
self.assertIn("pt@-", argv)
|
||||
self.assertIn(f"pubkey={PUB_A}", argv)
|
||||
self.assertEqual(argv[-1], "https://Server-11669-10a:1337/addKey")
|
||||
# never -k/--insecure — this must still be a real, validating request
|
||||
self.assertNotIn("-k", argv)
|
||||
self.assertNotIn("--insecure", argv)
|
||||
|
||||
def test_token_passed_via_stdin_with_no_trailing_newline_and_never_in_argv(self):
|
||||
sysm = FakeCurlSystem()
|
||||
http_client = pg.HttpClient(self.cfg, sysm)
|
||||
http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
|
||||
self.assertEqual(sysm.stdins, [SECRET_TOKEN]) # exact match: no appended "\n"
|
||||
for arg in sysm.calls[0]:
|
||||
self.assertNotIn(SECRET_TOKEN, arg)
|
||||
|
||||
def test_timeout_exit_code_classified(self):
|
||||
sysm = FakeCurlSystem(pg.CmdResult(28, "", "curl: (28) Connection timed out"))
|
||||
http_client = pg.HttpClient(self.cfg, sysm)
|
||||
with self.assertRaises(pg.ApiError) as cm:
|
||||
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
|
||||
self.assertEqual(cm.exception.kind, "timeout")
|
||||
|
||||
def test_tls_exit_codes_classified_with_stderr_detail(self):
|
||||
for rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83):
|
||||
with self.subTest(rc=rc):
|
||||
sysm = FakeCurlSystem(pg.CmdResult(rc, "", "curl: (60) SSL certificate problem"))
|
||||
http_client = pg.HttpClient(self.cfg, sysm)
|
||||
with self.assertRaises(pg.ApiError) as cm:
|
||||
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
|
||||
self.assertEqual(cm.exception.kind, "tls")
|
||||
self.assertIn("SSL certificate problem", cm.exception.detail)
|
||||
|
||||
def test_http_failure_exit_code_classified(self):
|
||||
sysm = FakeCurlSystem(pg.CmdResult(22, "", "curl: (22) The requested URL returned error: 401"))
|
||||
http_client = pg.HttpClient(self.cfg, sysm)
|
||||
with self.assertRaises(pg.ApiError) as cm:
|
||||
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
|
||||
self.assertEqual(cm.exception.kind, "http")
|
||||
|
||||
def test_nonstandard_exit_code_with_fail_text_still_classified_as_http(self):
|
||||
# Observed live: curl built against SecureTransport reports exit 56
|
||||
# (CURLE_RECV_ERROR), not the "usual" 22, for the same --fail
|
||||
# -triggered HTTP rejection — the error text is the portable signal.
|
||||
sysm = FakeCurlSystem(pg.CmdResult(56, "", "curl: (56) The requested URL returned error: 401"))
|
||||
http_client = pg.HttpClient(self.cfg, sysm)
|
||||
with self.assertRaises(pg.ApiError) as cm:
|
||||
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
|
||||
self.assertEqual(cm.exception.kind, "http")
|
||||
|
||||
def test_curl_missing_classified_as_network_not_a_crash(self):
|
||||
sysm = FakeCurlSystem(pg.CmdResult(127, "", "curl: not found"))
|
||||
http_client = pg.HttpClient(self.cfg, sysm)
|
||||
with self.assertRaises(pg.ApiError) as cm:
|
||||
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
|
||||
self.assertEqual(cm.exception.kind, "network")
|
||||
|
||||
def test_unclassified_exit_code_falls_back_to_network_with_stderr(self):
|
||||
sysm = FakeCurlSystem(pg.CmdResult(7, "", "curl: (7) Couldn't connect to server"))
|
||||
http_client = pg.HttpClient(self.cfg, sysm)
|
||||
with self.assertRaises(pg.ApiError) as cm:
|
||||
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
|
||||
self.assertEqual(cm.exception.kind, "network")
|
||||
self.assertIn("Couldn't connect", cm.exception.detail)
|
||||
|
||||
|
||||
class TestHttpHelpers(unittest.TestCase):
|
||||
def test_key_and_ip_validators(self):
|
||||
self.assertTrue(pg.valid_wg_key(PUB_A))
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user