fix: use curl for addKey — PIA's CA cert fails modern OpenSSL by default

Found live, 2026-09-22, on the addKey step of the first successful
kill-switch-check pass: every candidate failed with
kind=tls detail="certificate verification failed". Root-caused on
minisforum: PIA's own CA certificate (files/pia-ca.crt) has a CA:TRUE
Basic Constraints extension that isn't marked critical - a real RFC
5280 non-conformance, confirmed not a stale vendored copy (a fresh
fetch from pia-foss/manual-connections is byte-identical, same
sha256 already recorded in the file's own header, same defect).
Recent OpenSSL (3.2+; confirmed via a direct repro against
minisforum's 3.5.7 - SSLCertVerificationError, verify_message
"Basic Constraints of CA cert not marked critical") rejects this by
default, a check promoted from X509_STRICT-only into the baseline
chain-verification path. openssl s_client (LibreSSL, this dev
machine) and curl both accept the identical cert/chain/CA file - only
Python's ssl module, via this host's system OpenSSL, rejects it. This
went unnoticed until now because the three prior successful
registrations all went through curl --cacert (the legacy Ansible
path); this was the first time the Python ssl code path ran for real.

Python's stdlib ssl module has no custom verify-callback hook, so
there's no clean way to tolerate just this one known,
third-party-controlled defect while keeping full validation for
everything else. Rather than weakening verify_mode or adding a new
dependency (pyOpenSSL, for a real callback - considered, rejected as
a bigger lift than necessary) to route around a gap in a library we
don't control, addKey now shells out to curl
(HttpClient.get_pinned_curl, curl added to this role's apt list) -
the same tool PIA's own reference client already uses for this exact
endpoint, with --cacert and never -k/--insecure, so this is a
different, already-proven-correct TLS backend for the one endpoint
that needs it, not a downgrade. serverlist/token are unaffected
(public WebPKI certs) and still use the stdlib ssl path.

Verified: a real subprocess curl call against the actual failing
candidate (Server-11669-10a / 69.33.198.97) gets a genuine HTTP
response from PIA, proving TLS validation succeeds via this path.
Also found and handled: curl's own exit code for a --fail-triggered
rejection isn't fully portable (macOS/SecureTransport curl reports 56
where GNU/OpenSSL curl reports 22 for the same case) - classification
now also matches curl's own error text as a second signal, not just
the exit code.

Added 8 tests (94 total, all passing) covering the curl invocation
shape, stdin token delivery (no trailing newline, never in argv), and
exit-code + text-based error classification including the
cross-platform exit-code case found live.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Nik Afiq 2026-09-22 18:48:48 +09:00
parent 7b78e4a88e
commit 99f6ef84f0
4 changed files with 258 additions and 46 deletions

View File

@ -268,10 +268,14 @@ handled automatically — see the next section.
## Automatic recovery ## Automatic recovery
`files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway` `files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway`
(root, `0700`, standard library only — runs on the host's `python3.13`). (root, `0700`, runs on the host's `python3.13`). Almost entirely standard
It is **one** implementation of PIA registration shared by the unattended library — the one exception is `addKey` specifically, which shells out to
timers and by this role's own registration step, so the two can never `curl` (now an explicit `apt` dependency of this role); see
drift. Layout on the host: `HttpClient.get_pinned_curl`'s own docstring and "PIA's CA certificate and
why addKey uses curl, not Python's `ssl`" below for why. It is **one**
implementation of PIA registration shared by the unattended timers and by
this role's own registration step, so the two can never drift. Layout on
the host:
| Path | Purpose | | Path | Purpose |
| --- | --- | | --- | --- |
@ -285,6 +289,64 @@ drift. Layout on the host:
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run | | `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` | | `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
### PIA's CA certificate and why addKey uses curl, not Python's `ssl`
Found live, 2026-09-22, on the very first `pia-gateway register` attempt
after the FORWARD-jump fix above got past the kill-switch check:
`addKey` failed against every candidate with `kind=tls detail=certificate
verification failed`. Root-caused directly on minisforum:
```
$ sudo python3 -c '...ctx.wrap_socket(sock, server_hostname="Server-11669-10a")...'
SSLCertVerificationError verify_message= Basic Constraints of CA cert not marked critical
```
PIA's own CA certificate (`files/pia-ca.crt`) has a `CA:TRUE` Basic
Constraints extension that isn't flagged `critical` — a real RFC 5280
non-conformance, confirmed **not** a stale vendored copy: a fresh fetch
from `pia-foss/manual-connections` on GitHub is byte-identical (same
`32e9b1d1...` SHA-256 already recorded in the file's own header) and has
the identical defect. This is PIA's problem, not ours, and not fixable by
re-vendoring. Recent OpenSSL (3.2+; confirmed live against minisforum's
`3.5.7`) rejects this by default — a check that used to be opt-in only
under `X509_V_FLAG_X509_STRICT` and has since been promoted into the
default chain-verification path. `openssl s_client` on this repo's own
dev machine (LibreSSL) and this same host's `curl` both accept the exact
same certificate/chain/CA file without complaint — only Python's `ssl`
module, using this host's system OpenSSL, rejects it. That's also why
this went unnoticed until the auto-recovery work: the legacy Ansible
path's three successful registrations (2026-08-24, 08-25, and this
morning) all went through `curl --cacert` (`addkey-attempt.yaml`), never
through Python's `ssl` module — this was the *first* time that code path
ever ran for real.
Python's stdlib `ssl` module has no custom-verify-callback hook (a
long-standing, deliberate upstream gap — unlike Go's
`tls.Config.VerifyPeerCertificate` or OpenSSL's own C API), so there is
no clean way to tolerate just this one known, third-party-controlled
defect while still fully validating everything else (chain, expiry,
hostname/SAN). Rather than weakening `verify_mode` for the whole
connection or adding a new dependency (`pyOpenSSL`, for a real verify
callback — considered, rejected as a bigger lift than necessary) to work
around a gap in a library we don't control, `addKey` specifically now
shells out to `curl` (`HttpClient.get_pinned_curl`, `curl` added to this
role's `apt` install list) — the same tool PIA's own reference client
(`connect_to_wireguard_with_token.sh`) already uses for this exact
endpoint, with `--cacert` and never `-k`/`--insecure`, so this is a
different, already-proven-correct TLS backend for the one endpoint that
needs it, not a security downgrade. `serverlist` and the token endpoint
are unaffected (they present public WebPKI certs, which don't have this
problem) and still use the stdlib `ssl` path. Verified: a real subprocess
`curl` call from this repo's dev machine against the actual failing
candidate (`Server-11669-10a` / `69.33.198.97`) gets a genuine HTTP
response from PIA (401, expected — a dummy token was used), proving TLS
validation succeeds via this path where it failed via Python's `ssl`.
curl's own exit codes for a `--fail`-triggered rejection aren't fully
portable either — confirmed live: a curl built against SecureTransport
(this dev machine) reports exit 56, not the "usual" 22 GNU/OpenSSL curl
uses for the identical case — so classification checks curl's own error
text as a second signal alongside the exit code, not the exit code alone.
### Status classes ### Status classes
Every pass classifies the gateway into exactly one of: `healthy`, Every pass classifies the gateway into exactly one of: `healthy`,
@ -468,16 +530,23 @@ external secret broker — not a 24-hour token.
### Rollout status (2026-09-22) ### Rollout status (2026-09-22)
Step 1 (baseline) and the first attempt at step 2 have been run live. Step 1 (baseline) and two attempts at step 2 have been run live.
Step 2 failed on the first try: `pia-gateway register` correctly refused
with `safety invariants failing ... FORWARD jump to kill-switch chain is - **Attempt 1** failed: `pia-gateway register` correctly refused with
not first` — a real, pre-existing drift (see the "FORWARD jump's `safety invariants failing ... FORWARD jump to kill-switch chain is
position is actively re-asserted" note above), not a false alarm from not first` — a real, pre-existing drift (see the "FORWARD jump's
this session's work, and not the "session dies" handshake issue either. position is actively re-asserted" note above), not a false alarm from
Root-caused and fixed in `pia-killswitch.sh.j2`; not yet re-applied or this session's work, and not the "session dies" handshake issue
re-verified live. **Use `iptables -S FORWARD` (no `head`) for the either. Root-caused and fixed in `pia-killswitch.sh.j2`.
baseline capture below** — the original `head -3` suggestion is what - **Attempt 2**, after that fix, got past the kill-switch check and
hid this the first time; the drift was 4 rules in, not at the top. failed differently: `addKey` rejected every candidate with
`kind=tls detail=certificate verification failed`. Root-caused and
fixed — see "PIA's CA certificate and why addKey uses curl, not
Python's `ssl`" above. Not yet re-applied or re-verified live.
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
— the original `head -3` suggestion is what hid the first issue; the
drift was 4 rules in, not at the top.
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show 1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`, table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`,

View File

@ -751,33 +751,14 @@ class Classifier:
# -------------------------------------------------------------------------- # --------------------------------------------------------------------------
class _PinnedHTTPSConnection(http.client.HTTPSConnection):
"""Connect to a fixed IP while verifying TLS against the hostname (SNI)."""
def __init__(self, host: str, ip: str, port: int, context: ssl.SSLContext, connect_timeout: float, total_timeout: float):
super().__init__(host, port, timeout=connect_timeout, context=context)
self._pinned_ip = ip
self._total_timeout = total_timeout
def connect(self) -> None: # noqa: D401
sock = socket.create_connection((self._pinned_ip, self.port), self.timeout)
sock.settimeout(self._total_timeout)
self.sock = self._context.wrap_socket(sock, server_hostname=self.host)
class HttpClient: class HttpClient:
def __init__(self, cfg: Config): def __init__(self, cfg: Config, system: Optional[System] = None):
self.cfg = cfg self.cfg = cfg
self.system = system or System()
def _system_ctx(self) -> ssl.SSLContext: def _system_ctx(self) -> ssl.SSLContext:
return ssl.create_default_context() return ssl.create_default_context()
def _pia_ctx(self) -> ssl.SSLContext:
ctx = ssl.create_default_context(cafile=self.cfg.ca_file)
ctx.check_hostname = True
ctx.verify_mode = ssl.CERT_REQUIRED
return ctx
def _request(self, conn: http.client.HTTPSConnection, method: str, path: str, body: Optional[bytes], headers: dict) -> tuple[int, bytes]: def _request(self, conn: http.client.HTTPSConnection, method: str, path: str, body: Optional[bytes], headers: dict) -> tuple[int, bytes]:
try: try:
conn.request(method, path, body=body, headers=headers) conn.request(method, path, body=body, headers=headers)
@ -825,12 +806,68 @@ class HttpClient:
} }
return self._request(conn, "POST", u.path, body, headers) return self._request(conn, "POST", u.path, body, headers)
def get_pinned(self, hostname: str, ip: str, port: int, path: str, query: dict[str, str]) -> tuple[int, bytes]: def get_pinned_curl(self, hostname: str, ip: str, port: int, path: str, token: str, pubkey: str) -> tuple[int, bytes]:
conn = _PinnedHTTPSConnection( """Pinned-IP, SNI-validated, CA-pinned GET — for PIA's addKey only.
hostname, ip, port, self._pia_ctx(), self.cfg.http_connect_timeout_sec, self.cfg.http_total_timeout_sec
) This shells out to curl instead of using the stdlib `ssl` module.
full = path + "?" + urllib.parse.urlencode(query) Confirmed live (2026-09-22): PIA's own CA certificate (byte-
return self._request(conn, "GET", full, None, {"User-Agent": "pia-gateway"}) identical to a fresh fetch from PIA's own repo — not a stale
vendored copy) has a non-critical Basic Constraints extension on
its CA cert, a real RFC 5280 non-conformance in a CA we don't
control. Recent OpenSSL (3.2+; confirmed on minisforum's 3.5.7 via
a direct repro — SSLCertVerificationError, verify_message
"Basic Constraints of CA cert not marked critical") rejects that
in its default chain verification, a check promoted from
X509_STRICT-only to the default path in recent OpenSSL releases.
Python's stdlib `ssl` module has no custom verify-callback hook to
selectively tolerate just this one known, third-party-controlled
defect while still fully validating everything else (chain,
expiry, hostname/SAN) — that gap is long-standing and
deliberate upstream, not something to route around by weakening
verify_mode. curl's TLS stack on this host does not enforce this
specific check and is what PIA's own reference implementation
(connect_to_wireguard_with_token.sh) already uses for this exact
endpoint — reusing it here with --cacert (never -k/--insecure) is
not a security downgrade, just a different, already-proven-correct
TLS backend for the one endpoint that needs it. serverlist and
token still use the stdlib ssl path above against public WebPKI
certs, which do not have this problem.
"""
argv = [
"curl", "--silent", "--show-error", "--fail", "--location", "-G",
"--connect-timeout", str(self.cfg.http_connect_timeout_sec),
"--max-time", str(self.cfg.http_total_timeout_sec),
"--cacert", self.cfg.ca_file,
"--connect-to", f"{hostname}::{ip}:",
"--data-urlencode", "pt@-",
"--data-urlencode", f"pubkey={pubkey}",
f"https://{hostname}:{port}{path}",
]
r = self.system.run(argv, stdin=token, timeout=self.cfg.http_total_timeout_sec + 10)
if r.rc == 0:
return 200, r.out.encode("utf-8")
err = (r.err or "").strip()
# Exit codes are the primary signal, but not fully portable across
# curl builds/TLS backends — confirmed live: a curl built against
# SecureTransport reported exit 56 (CURLE_RECV_ERROR) for the exact
# same --fail-triggered HTTP rejection GNU/OpenSSL curl reports as
# 22. curl's own error text ("returned error: NNN" for --fail,
# "timed out" for CURLE_OPERATION_TIMEDOUT, TLS-related wording for
# cert/handshake problems) is a second, more portable signal — used
# here as a fallback classifier alongside the exit code, not a
# replacement for it.
if r.rc == 127:
raise ApiError("network", "curl not found on PATH")
if r.rc == 28 or "timed out" in err.lower():
raise ApiError("timeout", f"curl: connection or transfer timed out: {err}")
if r.rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83) or "ssl" in err.lower() or "certificate" in err.lower():
raise ApiError("tls", f"curl TLS error (exit {r.rc}): {err}")
if r.rc == 22 or "returned error:" in err:
# --fail suppresses the body on a non-2xx response, so there is
# no status code or body to parse here — same limitation the
# legacy curl-based Ansible path already had.
raise ApiError("http", f"curl: non-2xx HTTP response (exit {r.rc}): {err}")
raise ApiError("network", f"curl exit {r.rc}: {err}")
@dataclasses.dataclass @dataclasses.dataclass
@ -942,7 +979,7 @@ class PiaApi:
raise ApiError("http", f"token HTTP {last_status}") raise ApiError("http", f"token HTTP {last_status}")
def add_key(self, cand: Candidate, token: str, pubkey: str) -> PeerInfo: def add_key(self, cand: Candidate, token: str, pubkey: str) -> PeerInfo:
status, data = self.http.get_pinned(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", {"pt": token, "pubkey": pubkey}) status, data = self.http.get_pinned_curl(cand.cn, cand.ip, ADDKEY_PORT, "/addKey", token, pubkey)
if status != 200: if status != 200:
raise ApiError("http", f"addKey HTTP {status}") raise ApiError("http", f"addKey HTTP {status}")
if not data.strip(): if not data.strip():
@ -1133,7 +1170,7 @@ class Controller:
self.cfg = cfg self.cfg = cfg
self.sys = system self.sys = system
self.log = log self.log = log
self.http = http or HttpClient(cfg) self.http = http or HttpClient(cfg, system)
self._creds = creds self._creds = creds
self.store = StateStore(cfg, system) self.store = StateStore(cfg, system)
self.classifier = Classifier(cfg, system) self.classifier = Classifier(cfg, system)

View File

@ -27,10 +27,16 @@
- wireguard-tools - wireguard-tools
- iproute2 - iproute2
- python3 - python3
- curl
state: present state: present
update_cache: true update_cache: true
# python3-requests is no longer needed: the controller uses only the # python3-requests is no longer needed: the controller uses only the
# standard library. # standard library, EXCEPT for the addKey endpoint specifically, which
# shells out to curl — see HttpClient.get_pinned_curl's own docstring
# for why (a non-critical Basic Constraints extension on PIA's own CA
# cert that recent OpenSSL rejects by default and Python's ssl module
# has no way to selectively tolerate). curl is virtually always already
# present on Debian, but declared explicitly rather than assumed.
- name: Pin reverse-path filtering for {{ pia_wg_interface }} - name: Pin reverse-path filtering for {{ pia_wg_interface }}
ansible.posix.sysctl: ansible.posix.sysctl:

View File

@ -255,8 +255,8 @@ class FakeHttp:
return 200, val return 200, val
return 200, json.dumps({"token": val}).encode() return 200, json.dumps({"token": val}).encode()
def get_pinned(self, hostname, ip, port, path, query): def get_pinned_curl(self, hostname, ip, port, path, token, pubkey):
self.calls.append(("addkey", hostname, ip, port, path, dict(query))) self.calls.append(("addkey", hostname, ip, port, path, {"pt": token, "pubkey": pubkey}))
kind, val = self.addkey.get(hostname, ("http", 502)) kind, val = self.addkey.get(hostname, ("http", 502))
if kind == "err": if kind == "err":
raise pg.ApiError(val, "fake") raise pg.ApiError(val, "fake")
@ -1227,6 +1227,106 @@ def dataclass_dict(cfg):
return dataclasses.asdict(cfg) return dataclasses.asdict(cfg)
class FakeCurlSystem(pg.System):
"""Fakes just enough of System.run to exercise get_pinned_curl's argv
construction, stdin handling, and exit-code classification, without a
real curl binary or network access."""
def __init__(self, result=None):
self.result = result or pg.CmdResult(0, '{"status":"OK"}', "")
self.calls = []
self.stdins = []
def run(self, argv, stdin=None, timeout=60):
self.calls.append(list(argv))
if stdin is not None:
self.stdins.append(stdin)
return self.result
class TestPinnedCurlAddKey(unittest.TestCase):
def setUp(self):
self.cfg = pg.Config(ca_file="/etc/wireguard/pia-ca.crt", http_connect_timeout_sec=10, http_total_timeout_sec=30)
def test_argv_shape_matches_the_proven_legacy_curl_invocation(self):
sysm = FakeCurlSystem()
http_client = pg.HttpClient(self.cfg, sysm)
status, data = http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
self.assertEqual(status, 200)
self.assertEqual(data, b'{"status":"OK"}')
argv = sysm.calls[0]
self.assertEqual(argv[0], "curl")
self.assertIn("--fail", argv)
self.assertIn("-G", argv)
self.assertIn("--cacert", argv)
self.assertEqual(argv[argv.index("--cacert") + 1], "/etc/wireguard/pia-ca.crt")
self.assertIn("--connect-to", argv)
self.assertEqual(argv[argv.index("--connect-to") + 1], "Server-11669-10a::69.33.198.97:")
self.assertIn("pt@-", argv)
self.assertIn(f"pubkey={PUB_A}", argv)
self.assertEqual(argv[-1], "https://Server-11669-10a:1337/addKey")
# never -k/--insecure — this must still be a real, validating request
self.assertNotIn("-k", argv)
self.assertNotIn("--insecure", argv)
def test_token_passed_via_stdin_with_no_trailing_newline_and_never_in_argv(self):
sysm = FakeCurlSystem()
http_client = pg.HttpClient(self.cfg, sysm)
http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
self.assertEqual(sysm.stdins, [SECRET_TOKEN]) # exact match: no appended "\n"
for arg in sysm.calls[0]:
self.assertNotIn(SECRET_TOKEN, arg)
def test_timeout_exit_code_classified(self):
sysm = FakeCurlSystem(pg.CmdResult(28, "", "curl: (28) Connection timed out"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "timeout")
def test_tls_exit_codes_classified_with_stderr_detail(self):
for rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83):
with self.subTest(rc=rc):
sysm = FakeCurlSystem(pg.CmdResult(rc, "", "curl: (60) SSL certificate problem"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "tls")
self.assertIn("SSL certificate problem", cm.exception.detail)
def test_http_failure_exit_code_classified(self):
sysm = FakeCurlSystem(pg.CmdResult(22, "", "curl: (22) The requested URL returned error: 401"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "http")
def test_nonstandard_exit_code_with_fail_text_still_classified_as_http(self):
# Observed live: curl built against SecureTransport reports exit 56
# (CURLE_RECV_ERROR), not the "usual" 22, for the same --fail
# -triggered HTTP rejection — the error text is the portable signal.
sysm = FakeCurlSystem(pg.CmdResult(56, "", "curl: (56) The requested URL returned error: 401"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "http")
def test_curl_missing_classified_as_network_not_a_crash(self):
sysm = FakeCurlSystem(pg.CmdResult(127, "", "curl: not found"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "network")
def test_unclassified_exit_code_falls_back_to_network_with_stderr(self):
sysm = FakeCurlSystem(pg.CmdResult(7, "", "curl: (7) Couldn't connect to server"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "network")
self.assertIn("Couldn't connect", cm.exception.detail)
class TestHttpHelpers(unittest.TestCase): class TestHttpHelpers(unittest.TestCase):
def test_key_and_ip_validators(self): def test_key_and_ip_validators(self):
self.assertTrue(pg.valid_wg_key(PUB_A)) self.assertTrue(pg.valid_wg_key(PUB_A))