diff --git a/ansible/roles/pia-gateway/README.md b/ansible/roles/pia-gateway/README.md index 754625d..9c00e99 100644 --- a/ansible/roles/pia-gateway/README.md +++ b/ansible/roles/pia-gateway/README.md @@ -268,10 +268,14 @@ handled automatically — see the next section. ## Automatic recovery `files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway` -(root, `0700`, standard library only — runs on the host's `python3.13`). -It is **one** implementation of PIA registration shared by the unattended -timers and by this role's own registration step, so the two can never -drift. Layout on the host: +(root, `0700`, runs on the host's `python3.13`). Almost entirely standard +library — the one exception is `addKey` specifically, which shells out to +`curl` (now an explicit `apt` dependency of this role); see +`HttpClient.get_pinned_curl`'s own docstring and "PIA's CA certificate and +why addKey uses curl, not Python's `ssl`" below for why. It is **one** +implementation of PIA registration shared by the unattended timers and by +this role's own registration step, so the two can never drift. Layout on +the host: | Path | Purpose | | --- | --- | @@ -285,6 +289,64 @@ drift. Layout on the host: | `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run | | `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` | +### PIA's CA certificate and why addKey uses curl, not Python's `ssl` + +Found live, 2026-09-22, on the very first `pia-gateway register` attempt +after the FORWARD-jump fix above got past the kill-switch check: +`addKey` failed against every candidate with `kind=tls detail=certificate +verification failed`. Root-caused directly on minisforum: + +``` +$ sudo python3 -c '...ctx.wrap_socket(sock, server_hostname="Server-11669-10a")...' +SSLCertVerificationError verify_message= Basic Constraints of CA cert not marked critical +``` + +PIA's own CA certificate (`files/pia-ca.crt`) has a `CA:TRUE` Basic +Constraints extension that isn't flagged `critical` — a real RFC 5280 +non-conformance, confirmed **not** a stale vendored copy: a fresh fetch +from `pia-foss/manual-connections` on GitHub is byte-identical (same +`32e9b1d1...` SHA-256 already recorded in the file's own header) and has +the identical defect. This is PIA's problem, not ours, and not fixable by +re-vendoring. Recent OpenSSL (3.2+; confirmed live against minisforum's +`3.5.7`) rejects this by default — a check that used to be opt-in only +under `X509_V_FLAG_X509_STRICT` and has since been promoted into the +default chain-verification path. `openssl s_client` on this repo's own +dev machine (LibreSSL) and this same host's `curl` both accept the exact +same certificate/chain/CA file without complaint — only Python's `ssl` +module, using this host's system OpenSSL, rejects it. That's also why +this went unnoticed until the auto-recovery work: the legacy Ansible +path's three successful registrations (2026-08-24, 08-25, and this +morning) all went through `curl --cacert` (`addkey-attempt.yaml`), never +through Python's `ssl` module — this was the *first* time that code path +ever ran for real. + +Python's stdlib `ssl` module has no custom-verify-callback hook (a +long-standing, deliberate upstream gap — unlike Go's +`tls.Config.VerifyPeerCertificate` or OpenSSL's own C API), so there is +no clean way to tolerate just this one known, third-party-controlled +defect while still fully validating everything else (chain, expiry, +hostname/SAN). Rather than weakening `verify_mode` for the whole +connection or adding a new dependency (`pyOpenSSL`, for a real verify +callback — considered, rejected as a bigger lift than necessary) to work +around a gap in a library we don't control, `addKey` specifically now +shells out to `curl` (`HttpClient.get_pinned_curl`, `curl` added to this +role's `apt` install list) — the same tool PIA's own reference client +(`connect_to_wireguard_with_token.sh`) already uses for this exact +endpoint, with `--cacert` and never `-k`/`--insecure`, so this is a +different, already-proven-correct TLS backend for the one endpoint that +needs it, not a security downgrade. `serverlist` and the token endpoint +are unaffected (they present public WebPKI certs, which don't have this +problem) and still use the stdlib `ssl` path. Verified: a real subprocess +`curl` call from this repo's dev machine against the actual failing +candidate (`Server-11669-10a` / `69.33.198.97`) gets a genuine HTTP +response from PIA (401, expected — a dummy token was used), proving TLS +validation succeeds via this path where it failed via Python's `ssl`. +curl's own exit codes for a `--fail`-triggered rejection aren't fully +portable either — confirmed live: a curl built against SecureTransport +(this dev machine) reports exit 56, not the "usual" 22 GNU/OpenSSL curl +uses for the identical case — so classification checks curl's own error +text as a second signal alongside the exit code, not the exit code alone. + ### Status classes Every pass classifies the gateway into exactly one of: `healthy`, @@ -468,16 +530,23 @@ external secret broker — not a 24-hour token. ### Rollout status (2026-09-22) -Step 1 (baseline) and the first attempt at step 2 have been run live. -Step 2 failed on the first try: `pia-gateway register` correctly refused -with `safety invariants failing ... FORWARD jump to kill-switch chain is -not first` — a real, pre-existing drift (see the "FORWARD jump's -position is actively re-asserted" note above), not a false alarm from -this session's work, and not the "session dies" handshake issue either. -Root-caused and fixed in `pia-killswitch.sh.j2`; not yet re-applied or -re-verified live. **Use `iptables -S FORWARD` (no `head`) for the -baseline capture below** — the original `head -3` suggestion is what -hid this the first time; the drift was 4 rules in, not at the top. +Step 1 (baseline) and two attempts at step 2 have been run live. + +- **Attempt 1** failed: `pia-gateway register` correctly refused with + `safety invariants failing ... FORWARD jump to kill-switch chain is + not first` — a real, pre-existing drift (see the "FORWARD jump's + position is actively re-asserted" note above), not a false alarm from + this session's work, and not the "session dies" handshake issue + either. Root-caused and fixed in `pia-killswitch.sh.j2`. +- **Attempt 2**, after that fix, got past the kill-switch check and + failed differently: `addKey` rejected every candidate with + `kind=tls detail=certificate verification failed`. Root-caused and + fixed — see "PIA's CA certificate and why addKey uses curl, not + Python's `ssl`" above. Not yet re-applied or re-verified live. + +**Use `iptables -S FORWARD` (no `head`) for the baseline capture below** +— the original `head -3` suggestion is what hid the first issue; the +drift was 4 rules in, not at the top. 1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`, diff --git a/ansible/roles/pia-gateway/files/pia_gateway.py b/ansible/roles/pia-gateway/files/pia_gateway.py index 1c26fa5..edb7f56 100644 --- a/ansible/roles/pia-gateway/files/pia_gateway.py +++ b/ansible/roles/pia-gateway/files/pia_gateway.py @@ -751,33 +751,14 @@ class Classifier: # -------------------------------------------------------------------------- -class _PinnedHTTPSConnection(http.client.HTTPSConnection): - """Connect to a fixed IP while verifying TLS against the hostname (SNI).""" - - def __init__(self, host: str, ip: str, port: int, context: ssl.SSLContext, connect_timeout: float, total_timeout: float): - super().__init__(host, port, timeout=connect_timeout, context=context) - self._pinned_ip = ip - self._total_timeout = total_timeout - - def connect(self) -> None: # noqa: D401 - sock = socket.create_connection((self._pinned_ip, self.port), self.timeout) - sock.settimeout(self._total_timeout) - self.sock = self._context.wrap_socket(sock, server_hostname=self.host) - - class HttpClient: - def __init__(self, cfg: Config): + def __init__(self, cfg: Config, system: Optional[System] = None): self.cfg = cfg + self.system = system or System() def _system_ctx(self) -> ssl.SSLContext: return ssl.create_default_context() - def _pia_ctx(self) -> ssl.SSLContext: - ctx = ssl.create_default_context(cafile=self.cfg.ca_file) - ctx.check_hostname = True - ctx.verify_mode = ssl.CERT_REQUIRED - return ctx - def _request(self, conn: http.client.HTTPSConnection, method: str, path: str, body: Optional[bytes], headers: dict) -> tuple[int, bytes]: try: conn.request(method, path, body=body, headers=headers) @@ -825,12 +806,68 @@ class HttpClient: } return self._request(conn, "POST", u.path, body, headers) - def get_pinned(self, hostname: str, ip: str, port: int, path: str, query: dict[str, str]) -> tuple[int, bytes]: - conn = _PinnedHTTPSConnection( - hostname, ip, port, self._pia_ctx(), self.cfg.http_connect_timeout_sec, self.cfg.http_total_timeout_sec - ) - full = path + "?" + urllib.parse.urlencode(query) - return self._request(conn, "GET", full, None, {"User-Agent": "pia-gateway"}) + def get_pinned_curl(self, hostname: str, ip: str, port: int, path: str, token: str, pubkey: str) -> tuple[int, bytes]: + """Pinned-IP, SNI-validated, CA-pinned GET — for PIA's addKey only. + + This shells out to curl instead of using the stdlib `ssl` module. + Confirmed live (2026-09-22): PIA's own CA certificate (byte- + identical to a fresh fetch from PIA's own repo — not a stale + vendored copy) has a non-critical Basic Constraints extension on + its CA cert, a real RFC 5280 non-conformance in a CA we don't + control. Recent OpenSSL (3.2+; confirmed on minisforum's 3.5.7 via + a direct repro — SSLCertVerificationError, verify_message + "Basic Constraints of CA cert not marked critical") rejects that + in its default chain verification, a check promoted from + X509_STRICT-only to the default path in recent OpenSSL releases. + Python's stdlib `ssl` module has no custom verify-callback hook to + selectively tolerate just this one known, third-party-controlled + defect while still fully validating everything else (chain, + expiry, hostname/SAN) — that gap is long-standing and + deliberate upstream, not something to route around by weakening + verify_mode. curl's TLS stack on this host does not enforce this + specific check and is what PIA's own reference implementation + (connect_to_wireguard_with_token.sh) already uses for this exact + endpoint — reusing it here with --cacert (never -k/--insecure) is + not a security downgrade, just a different, already-proven-correct + TLS backend for the one endpoint that needs it. serverlist and + token still use the stdlib ssl path above against public WebPKI + certs, which do not have this problem. + """ + argv = [ + "curl", "--silent", "--show-error", "--fail", "--location", "-G", + "--connect-timeout", str(self.cfg.http_connect_timeout_sec), + "--max-time", str(self.cfg.http_total_timeout_sec), + "--cacert", self.cfg.ca_file, + "--connect-to", f"{hostname}::{ip}:", + "--data-urlencode", "pt@-", + "--data-urlencode", f"pubkey={pubkey}", + f"https://{hostname}:{port}{path}", + ] + r = self.system.run(argv, stdin=token, timeout=self.cfg.http_total_timeout_sec + 10) + if r.rc == 0: + return 200, r.out.encode("utf-8") + err = (r.err or "").strip() + # Exit codes are the primary signal, but not fully portable across + # curl builds/TLS backends — confirmed live: a curl built against + # SecureTransport reported exit 56 (CURLE_RECV_ERROR) for the exact + # same --fail-triggered HTTP rejection GNU/OpenSSL curl reports as + # 22. curl's own error text ("returned error: NNN" for --fail, + # "timed out" for CURLE_OPERATION_TIMEDOUT, TLS-related wording for + # cert/handshake problems) is a second, more portable signal — used + # here as a fallback classifier alongside the exit code, not a + # replacement for it. + if r.rc == 127: + raise ApiError("network", "curl not found on PATH") + if r.rc == 28 or "timed out" in err.lower(): + raise ApiError("timeout", f"curl: connection or transfer timed out: {err}") + if r.rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83) or "ssl" in err.lower() or "certificate" in err.lower(): + raise ApiError("tls", f"curl TLS error (exit {r.rc}): {err}") + if r.rc == 22 or "returned error:" in err: + # --fail suppresses the body on a non-2xx response, so there is + # no status code or body to parse here — same limitation the + # legacy curl-based Ansible path already had. + raise ApiError("http", f"curl: non-2xx HTTP response (exit {r.rc}): {err}") + raise ApiError("network", f"curl exit {r.rc}: {err}") @dataclasses.dataclass @@ -942,7 +979,7 @@ class PiaApi: raise ApiError("http", f"token HTTP {last_status}") def add_key(self, cand: Candidate, token: str, pubkey: str) -> PeerInfo: - status, data = self.http.get_pinned(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", {"pt": token, "pubkey": pubkey}) + status, data = self.http.get_pinned_curl(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", token, pubkey) if status != 200: raise ApiError("http", f"addKey HTTP {status}") if not data.strip(): @@ -1133,7 +1170,7 @@ class Controller: self.cfg = cfg self.sys = system self.log = log - self.http = http or HttpClient(cfg) + self.http = http or HttpClient(cfg, system) self._creds = creds self.store = StateStore(cfg, system) self.classifier = Classifier(cfg, system) diff --git a/ansible/roles/pia-gateway/tasks/main.yaml b/ansible/roles/pia-gateway/tasks/main.yaml index 386abcb..85afbbc 100644 --- a/ansible/roles/pia-gateway/tasks/main.yaml +++ b/ansible/roles/pia-gateway/tasks/main.yaml @@ -27,10 +27,16 @@ - wireguard-tools - iproute2 - python3 + - curl state: present update_cache: true # python3-requests is no longer needed: the controller uses only the - # standard library. + # standard library, EXCEPT for the addKey endpoint specifically, which + # shells out to curl — see HttpClient.get_pinned_curl's own docstring + # for why (a non-critical Basic Constraints extension on PIA's own CA + # cert that recent OpenSSL rejects by default and Python's ssl module + # has no way to selectively tolerate). curl is virtually always already + # present on Debian, but declared explicitly rather than assumed. - name: Pin reverse-path filtering for {{ pia_wg_interface }} ansible.posix.sysctl: diff --git a/ansible/roles/pia-gateway/tests/test_pia_gateway.py b/ansible/roles/pia-gateway/tests/test_pia_gateway.py index fab44ec..616ac20 100644 --- a/ansible/roles/pia-gateway/tests/test_pia_gateway.py +++ b/ansible/roles/pia-gateway/tests/test_pia_gateway.py @@ -255,8 +255,8 @@ class FakeHttp: return 200, val return 200, json.dumps({"token": val}).encode() - def get_pinned(self, hostname, ip, port, path, query): - self.calls.append(("addkey", hostname, ip, port, path, dict(query))) + def get_pinned_curl(self, hostname, ip, port, path, token, pubkey): + self.calls.append(("addkey", hostname, ip, port, path, {"pt": token, "pubkey": pubkey})) kind, val = self.addkey.get(hostname, ("http", 502)) if kind == "err": raise pg.ApiError(val, "fake") @@ -1227,6 +1227,106 @@ def dataclass_dict(cfg): return dataclasses.asdict(cfg) +class FakeCurlSystem(pg.System): + """Fakes just enough of System.run to exercise get_pinned_curl's argv + construction, stdin handling, and exit-code classification, without a + real curl binary or network access.""" + + def __init__(self, result=None): + self.result = result or pg.CmdResult(0, '{"status":"OK"}', "") + self.calls = [] + self.stdins = [] + + def run(self, argv, stdin=None, timeout=60): + self.calls.append(list(argv)) + if stdin is not None: + self.stdins.append(stdin) + return self.result + + +class TestPinnedCurlAddKey(unittest.TestCase): + def setUp(self): + self.cfg = pg.Config(ca_file="/etc/wireguard/pia-ca.crt", http_connect_timeout_sec=10, http_total_timeout_sec=30) + + def test_argv_shape_matches_the_proven_legacy_curl_invocation(self): + sysm = FakeCurlSystem() + http_client = pg.HttpClient(self.cfg, sysm) + status, data = http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A) + self.assertEqual(status, 200) + self.assertEqual(data, b'{"status":"OK"}') + argv = sysm.calls[0] + self.assertEqual(argv[0], "curl") + self.assertIn("--fail", argv) + self.assertIn("-G", argv) + self.assertIn("--cacert", argv) + self.assertEqual(argv[argv.index("--cacert") + 1], "/etc/wireguard/pia-ca.crt") + self.assertIn("--connect-to", argv) + self.assertEqual(argv[argv.index("--connect-to") + 1], "Server-11669-10a::69.33.198.97:") + self.assertIn("pt@-", argv) + self.assertIn(f"pubkey={PUB_A}", argv) + self.assertEqual(argv[-1], "https://Server-11669-10a:1337/addKey") + # never -k/--insecure — this must still be a real, validating request + self.assertNotIn("-k", argv) + self.assertNotIn("--insecure", argv) + + def test_token_passed_via_stdin_with_no_trailing_newline_and_never_in_argv(self): + sysm = FakeCurlSystem() + http_client = pg.HttpClient(self.cfg, sysm) + http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A) + self.assertEqual(sysm.stdins, [SECRET_TOKEN]) # exact match: no appended "\n" + for arg in sysm.calls[0]: + self.assertNotIn(SECRET_TOKEN, arg) + + def test_timeout_exit_code_classified(self): + sysm = FakeCurlSystem(pg.CmdResult(28, "", "curl: (28) Connection timed out")) + http_client = pg.HttpClient(self.cfg, sysm) + with self.assertRaises(pg.ApiError) as cm: + http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p") + self.assertEqual(cm.exception.kind, "timeout") + + def test_tls_exit_codes_classified_with_stderr_detail(self): + for rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83): + with self.subTest(rc=rc): + sysm = FakeCurlSystem(pg.CmdResult(rc, "", "curl: (60) SSL certificate problem")) + http_client = pg.HttpClient(self.cfg, sysm) + with self.assertRaises(pg.ApiError) as cm: + http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p") + self.assertEqual(cm.exception.kind, "tls") + self.assertIn("SSL certificate problem", cm.exception.detail) + + def test_http_failure_exit_code_classified(self): + sysm = FakeCurlSystem(pg.CmdResult(22, "", "curl: (22) The requested URL returned error: 401")) + http_client = pg.HttpClient(self.cfg, sysm) + with self.assertRaises(pg.ApiError) as cm: + http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p") + self.assertEqual(cm.exception.kind, "http") + + def test_nonstandard_exit_code_with_fail_text_still_classified_as_http(self): + # Observed live: curl built against SecureTransport reports exit 56 + # (CURLE_RECV_ERROR), not the "usual" 22, for the same --fail + # -triggered HTTP rejection — the error text is the portable signal. + sysm = FakeCurlSystem(pg.CmdResult(56, "", "curl: (56) The requested URL returned error: 401")) + http_client = pg.HttpClient(self.cfg, sysm) + with self.assertRaises(pg.ApiError) as cm: + http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p") + self.assertEqual(cm.exception.kind, "http") + + def test_curl_missing_classified_as_network_not_a_crash(self): + sysm = FakeCurlSystem(pg.CmdResult(127, "", "curl: not found")) + http_client = pg.HttpClient(self.cfg, sysm) + with self.assertRaises(pg.ApiError) as cm: + http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p") + self.assertEqual(cm.exception.kind, "network") + + def test_unclassified_exit_code_falls_back_to_network_with_stderr(self): + sysm = FakeCurlSystem(pg.CmdResult(7, "", "curl: (7) Couldn't connect to server")) + http_client = pg.HttpClient(self.cfg, sysm) + with self.assertRaises(pg.ApiError) as cm: + http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p") + self.assertEqual(cm.exception.kind, "network") + self.assertIn("Couldn't connect", cm.exception.detail) + + class TestHttpHelpers(unittest.TestCase): def test_key_and_ip_validators(self): self.assertTrue(pg.valid_wg_key(PUB_A))