homelab/ansible/roles/pia-gateway/tests/test_pia_gateway.py
Nik Afiq fb9ecffa9f fix: tolerate content-verified kube-router jumps ahead of the kill switch
Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:

  19:38:58  3rd unhealthy check -> restart
  19:39:09  recovered - route open, handshake fresh
  19:40:03  drift again -> emergency close -> route forced closed,
            counter reset to 0
  19:41:08  drift again -> emergency close again

VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.

This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.

Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.

Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.

pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 19:55:47 +09:00

1520 lines
69 KiB
Python

"""Offline tests for ansible/roles/pia-gateway/files/pia_gateway.py.
Run from the repo root: python3 -m unittest discover -s ansible/roles/pia-gateway/tests -v
Everything that touches the host (ip/wg/iptables/systemctl, clock, HTTP) is
faked; real files are used inside a temporary directory so atomic writes,
modes, staging and backups are exercised for real.
"""
from __future__ import annotations
import importlib.util
import io
import json
import os
import re
import shutil
import stat
import sys
import tempfile
import unittest
from pathlib import Path
from unittest import mock
HERE = Path(__file__).resolve().parent
SRC = HERE.parent / "files" / "pia_gateway.py"
spec = importlib.util.spec_from_file_location("pia_gateway", SRC)
pg = importlib.util.module_from_spec(spec)
sys.modules["pia_gateway"] = pg
spec.loader.exec_module(pg)
SUBNET = "10.10.50.0/24"
CHAIN = "PIA-VLAN50"
IFACE = "pia-wg"
# Sentinels that must never appear in any output.
SECRET_USER = "sentinel-user-9f1c"
SECRET_PASS = "sentinel-pass-77aa"
SECRET_TOKEN = "sentinel-token-13bd"
SECRET_ALERT_KEY = "sentinel-alertkey-55ee"
# Valid-looking 32-byte base64 keys (44 chars, last char from the allowed set).
PRIV_A = "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA="
PRIV_B = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBA="
PUB_A = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaA="
PUB_B = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbA="
SRV_HK1 = "hk1hk1hk1hk1hk1hk1hk1hk1hk1hk1hk1hk1hk1hk1A="
SRV_HK2 = "hk2hk2hk2hk2hk2hk2hk2hk2hk2hk2hk2hk2hk2hk2A="
SRV_TW1 = "tw1tw1tw1tw1tw1tw1tw1tw1tw1tw1tw1tw1tw1tw1A="
SRV_OLD = "oldoldoldoldoldoldoldoldoldoldoldoldoldoldA="
PUBKEYS = {PRIV_A: PUB_A, PRIV_B: PUB_B}
SERVER_KEYS = {"hk1.example": SRV_HK1, "hk2.example": SRV_HK2, "tw1.example": SRV_TW1}
def serverlist_doc():
return {
"regions": [
{"id": "hk", "name": "Hong Kong", "servers": {"wg": [{"cn": "hk1.example", "ip": "10.99.0.1"}, {"cn": "hk2.example", "ip": "10.99.0.2"}]}},
{"id": "taiwan", "name": "Taiwan", "servers": {"wg": [{"cn": "tw1.example", "ip": "10.99.1.1"}]}},
{"id": "japan", "name": "JP Tokyo", "servers": {"wg": []}},
]
}
class FakeSystem(pg.System):
"""Models just enough of the gateway host for the controller."""
def __init__(self, tmp: Path):
self.tmp = tmp
self.t = 1_700_000_000
self.mono = 1000.0
self.boot = "boot-1"
self.iface_exists = True
self.iface_up = True
self.peers: dict[str, int] = {}
self.endpoints: dict[str, str] = {}
self.rule_present = True
self.table_default = "open" # open | closed | missing
self.forward_rules = [f"-A FORWARD -s {SUBNET} -j {CHAIN}", "-A FORWARD -j ufw-before-forward"]
self.chain_rules = self.open_chain()
self.main_default = "default via 10.10.40.1 dev enp1s0 proto static"
self.killswitch_active = True
self.restart_rc = 0
self.restart_count = 0
self.handshake_on_restart = "fresh" # fresh | stale | callable(server_key)->epoch|None
self.commands: list[list[str]] = []
self.stdins: list[str] = []
self.genkey_queue = [PRIV_B]
self.jitter = 0.5
self.extra_chains: dict = {} # name -> list[str] | None ("-A <name> ..." lines, or None = missing)
@staticmethod
def open_chain():
return [
f"-A {CHAIN} -d {SUBNET} -i {IFACE} -m state --state RELATED,ESTABLISHED -j ACCEPT",
f"-A {CHAIN} -s {SUBNET} -o {IFACE} -j ACCEPT",
f"-A {CHAIN} -m limit --limit 5/min --limit-burst 5 -j LOG --log-prefix \"PIA-KILLSWITCH-DROP: \"",
f"-A {CHAIN} -j DROP",
]
@staticmethod
def closed_chain():
return [
f"-A {CHAIN} -m limit --limit 5/min --limit-burst 5 -j LOG --log-prefix \"PIA-KILLSWITCH-DROP: \"",
f"-A {CHAIN} -j DROP",
]
# clock
def now(self):
return self.t
def monotonic(self):
return self.mono
def sleep(self, sec):
self.t += int(sec)
self.mono += sec
def boot_id(self):
return self.boot
def random_jitter(self):
return self.jitter
# helpers
def set_peer(self, key, age, endpoint="10.99.0.1:1337"):
self.peers = {key: self.t - age if age is not None else 0}
self.endpoints = {key: endpoint}
def conf_peer(self, conf_path):
if os.path.exists(conf_path):
return pg.parse_expected_peer(open(conf_path).read())
return None
def _apply_restart(self, conf_path):
self.restart_count += 1
if self.restart_rc != 0:
return pg.CmdResult(self.restart_rc, "", "Job for wg-quick@pia-wg.service failed")
self.t += 1 # a real restart takes measurable wall-clock time; keeps
# any "fresh" handshake strictly after whatever `since_epoch` the
# caller captured just before invoking this restart (see the
# `>` — not `>=` — comparison in wait_for_verified).
key = self.conf_peer(conf_path)
self.iface_exists = True
self.iface_up = True
self.table_default = "open"
self.chain_rules = self.open_chain()
beh = self.handshake_on_restart
if callable(beh):
epoch = beh(key)
elif beh == "fresh":
epoch = self.t
else:
epoch = self.t - 100_000
self.peers = {key: epoch if epoch is not None else 0}
self.endpoints = {key: "10.99.0.9:1337"}
return pg.CmdResult(0, "", "")
# command dispatch
def run(self, argv, stdin=None, timeout=60):
self.commands.append(list(argv))
if stdin is not None:
self.stdins.append(stdin)
a = argv
if a[:2] == ["wg", "genkey"]:
return pg.CmdResult(0, self.genkey_queue.pop(0) + "\n", "")
if a[:2] == ["wg", "pubkey"]:
return pg.CmdResult(0, PUBKEYS[stdin.strip()] + "\n", "")
if a[:2] == ["wg", "show"]:
if not self.iface_exists:
return pg.CmdResult(1, "", "Unable to access interface: No such device")
if a[3] == "latest-handshakes":
return pg.CmdResult(0, "".join(f"{k}\t{v}\n" for k, v in self.peers.items()), "")
if a[3] == "endpoints":
return pg.CmdResult(0, "".join(f"{k}\t{v}\n" for k, v in self.endpoints.items()), "")
if a[:4] == ["ip", "-o", "link", "show"]:
if not self.iface_exists:
return pg.CmdResult(1, "", 'Device "pia-wg" does not exist.')
flags = "POINTOPOINT,NOARP,UP,LOWER_UP" if self.iface_up else "POINTOPOINT,NOARP"
return pg.CmdResult(0, f"5: {IFACE}: <{flags}> mtu 1420 qdisc noqueue state UNKNOWN\n", "")
if a[:3] == ["ip", "rule", "show"]:
lines = ["0:\tfrom all lookup local"]
if self.rule_present:
lines.append(f"100:\tfrom {SUBNET} lookup pia")
lines += ["32766:\tfrom all lookup main", "32767:\tfrom all lookup default"]
return pg.CmdResult(0, "\n".join(lines) + "\n", "")
if a[:3] == ["ip", "route", "show"] and "table" in a:
if self.table_default == "open":
return pg.CmdResult(0, f"default dev {IFACE} scope link\n", "")
if self.table_default == "closed":
return pg.CmdResult(0, "unreachable default\n", "")
return pg.CmdResult(0, "", "")
if a[:4] == ["ip", "-4", "route", "show"]:
return pg.CmdResult(0, self.main_default + "\n", "")
if a[:3] == ["ip", "route", "replace"]:
if "unreachable" in a:
self.table_default = "closed"
return pg.CmdResult(0, "", "")
if a[:4] == ["ip", "link", "delete", "dev"]:
self.iface_exists = False
return pg.CmdResult(0, "", "")
if a[:2] == ["iptables", "-S"]:
if a[2] == "FORWARD":
return pg.CmdResult(0, "-P FORWARD ACCEPT\n" + "\n".join(self.forward_rules) + "\n", "")
if a[2] == CHAIN:
if self.chain_rules is None:
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
return pg.CmdResult(0, f"-N {CHAIN}\n" + "\n".join(self.chain_rules) + "\n", "")
if a[2] in self.extra_chains:
rules = self.extra_chains[a[2]]
if rules is None:
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
return pg.CmdResult(0, f"-N {a[2]}\n" + "\n".join(rules) + "\n", "")
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
if a[:2] == ["iptables", "-C"]:
rule = "-A " + " ".join(a[2:])
return pg.CmdResult(0 if rule in self.forward_rules else 1, "", "")
if a[:2] == ["iptables", "-I"]:
rule = "-A FORWARD " + " ".join(a[4:])
self.forward_rules.insert(0, rule)
return pg.CmdResult(0, "", "")
if a[:2] == ["iptables", "-D"]:
rule = "-A " + " ".join(a[2:])
if rule in self.forward_rules:
self.forward_rules.remove(rule)
return pg.CmdResult(0, "", "")
return pg.CmdResult(1, "", "")
if a[:2] == ["systemctl", "is-active"]:
return pg.CmdResult(0 if self.killswitch_active else 3, "active\n" if self.killswitch_active else "inactive\n", "")
if a[:2] == ["systemctl", "restart"] or a[:2] == ["systemctl", "start"]:
return self._apply_restart(str(self.tmp / "wg" / f"{IFACE}.conf"))
raise AssertionError(f"unexpected command {argv}")
class FakeHttp:
def __init__(self):
self.serverlist = ("ok", serverlist_doc())
self.token = ("ok", SECRET_TOKEN)
self.addkey = {} # cn -> ("ok", {...}) | ("err", kind) | ("http", status) | ("raw", bytes)
self.calls = []
def get(self, url):
self.calls.append(("get", url))
kind, val = self.serverlist
if kind == "err":
raise pg.ApiError(val, "fake")
if kind == "http":
return val, b""
if kind == "raw":
return 200, val
return 200, (json.dumps(val) + "\n-----BEGIN SIG-----\nxxx\n").encode()
def post_multipart(self, url, fields):
self.calls.append(("post", url, dict(fields)))
kind, val = self.token
if kind == "err":
raise pg.ApiError(val, "fake")
if kind == "http":
return val, b"{}"
if kind == "raw":
return 200, val
return 200, json.dumps({"token": val}).encode()
def get_pinned_curl(self, hostname, ip, port, path, token, pubkey):
self.calls.append(("addkey", hostname, ip, port, path, {"pt": token, "pubkey": pubkey}))
kind, val = self.addkey.get(hostname, ("http", 502))
if kind == "err":
raise pg.ApiError(val, "fake")
if kind == "http":
return val, b""
if kind == "raw":
return 200, val
body = {"status": "OK", "server_key": SERVER_KEYS[hostname], "peer_ip": "10.5.5.5", "server_port": 1337, "server_ip": ip}
body.update(val or {})
return 200, json.dumps(body).encode()
class Base(unittest.TestCase):
def setUp(self):
self.tmp = Path(tempfile.mkdtemp(prefix="pia-gateway-test-"))
(self.tmp / "wg").mkdir(mode=0o700)
(self.tmp / "state").mkdir(mode=0o700)
(self.tmp / "etc").mkdir(mode=0o700)
creds = self.tmp / "etc" / "credentials.env"
creds.write_text(f"PIA_USER={SECRET_USER}\nPIA_PASSWORD={SECRET_PASS}\nALERT_BRIDGE_API_KEY={SECRET_ALERT_KEY}\n")
creds.chmod(0o600)
(self.tmp / "wg" / "pia-ca.crt").write_text("dummy")
self.cfg = pg.Config(
config_dir=str(self.tmp / "wg"),
state_dir=str(self.tmp / "state"),
credentials_file=str(creds),
ca_file=str(self.tmp / "wg" / "pia-ca.crt"),
lock_file=str(self.tmp / "lock"),
metrics_file=str(self.tmp / "state" / "metrics.prom"),
status_file=str(self.tmp / "state" / "status.json"),
alert_bridge_url="http://alert-bridge.test/alerts",
)
self.sysm = FakeSystem(self.tmp)
self.http = FakeHttp()
self.out = io.StringIO()
self.log = pg.Log(stream=self.out)
self.notified: list[tuple[str, str]] = []
def fake_send(_self, level, message):
self.notified.append((level, message))
return True
self._send_patch = mock.patch.object(pg.Notifier, "send", fake_send)
self._send_patch.start()
self.write_conf(PRIV_A, SRV_OLD, "10.99.0.1")
(self.tmp / "wg" / "pia-wg.key").write_text(PRIV_A + "\n")
(self.tmp / "wg" / "pia-wg.key").chmod(0o600)
self.sysm.set_peer(SRV_OLD, 30)
self.ctl = pg.Controller(self.cfg, self.sysm, self.log, http=self.http)
# Simulate a controller that has already run once on this boot, so the
# startup grace does not absorb the first checks of every test. The
# fresh-boot behaviour has its own test.
seeded = pg.default_state()
seeded["boot_id"] = self.sysm.boot
self.ctl.store.save(seeded)
def tearDown(self):
self._send_patch.stop()
shutil.rmtree(self.tmp, ignore_errors=True)
def write_conf(self, priv, server_key, ip):
peer = pg.PeerInfo(server_key, "10.5.5.5", 1337, ip, "hk", "Hong Kong", "old.example")
p = self.tmp / "wg" / "pia-wg.conf"
p.write_text(pg.render_wg_conf(self.cfg, priv, peer))
p.chmod(0o600)
def state(self):
return json.loads((self.tmp / "state" / "state.json").read_text())
def run_recover_n(self, n, force=False):
rcs = []
for _ in range(n):
rcs.append(self.ctl.cmd_recover(force=force))
self.sysm.sleep(60)
return rcs
def go_stale(self):
self.sysm.set_peer(SRV_OLD, 5000)
def all_output(self):
s = self.out.getvalue()
for p in (self.tmp / "state" / "status.json", self.tmp / "state" / "metrics.prom", self.tmp / "state" / "state.json"):
if p.exists():
s += p.read_text()
s += json.dumps(self.notified)
s += json.dumps(self.sysm.commands)
return s
def assert_no_secrets(self):
blob = self.all_output()
for sec in (SECRET_USER, SECRET_PASS, SECRET_TOKEN, SECRET_ALERT_KEY, PRIV_A, PRIV_B):
self.assertNotIn(sec, blob, f"secret {sec[:12]}... leaked into output")
class TestClassification(Base):
def test_healthy(self):
st = self.ctl.current_status(pg.default_state())
self.assertEqual(st.cls, pg.CLASS_HEALTHY)
self.assertTrue(st.safety_ok and st.tunnel_ok)
self.assertEqual(st.handshake_age_sec, 30)
def test_stale_handshake(self):
self.go_stale()
st = self.ctl.current_status(pg.default_state())
self.assertEqual(st.cls, pg.CLASS_TUNNEL_STALE)
self.assertIn("handshake 5000s old exceeds 180s", st.reasons)
def test_zero_handshake(self):
self.sysm.set_peer(SRV_OLD, None)
st = self.ctl.current_status(pg.default_state())
self.assertEqual(st.cls, pg.CLASS_TUNNEL_STALE)
self.assertIn("no handshake recorded", st.reasons)
def test_future_handshake_is_not_healthy(self):
self.sysm.set_peer(SRV_OLD, -5000)
st = self.ctl.current_status(pg.default_state())
self.assertEqual(st.cls, pg.CLASS_TUNNEL_STALE)
self.assertTrue(any("future" in r for r in st.reasons))
def test_small_clock_skew_tolerated(self):
self.sysm.set_peer(SRV_OLD, -10)
st = self.ctl.current_status(pg.default_state())
self.assertEqual(st.cls, pg.CLASS_HEALTHY)
def test_malformed_wg_output(self):
self.sysm.peers = {}
orig = self.sysm.run
def run(argv, stdin=None, timeout=60):
if argv[:2] == ["wg", "show"] and argv[3] == "latest-handshakes":
self.sysm.commands.append(argv)
return pg.CmdResult(0, "garbage without tab\n", "")
return orig(argv, stdin, timeout)
self.sysm.run = run
st = self.ctl.current_status(pg.default_state())
self.assertNotEqual(st.cls, pg.CLASS_HEALTHY)
self.assertIn("malformed wg output", st.reasons)
def test_wrong_peer_on_interface(self):
self.sysm.set_peer(SRV_HK1, 10)
st = self.ctl.current_status(pg.default_state())
self.assertEqual(st.cls, pg.CLASS_TUNNEL_ABSENT)
def test_interface_absent(self):
self.sysm.iface_exists = False
st = self.ctl.current_status(pg.default_state())
self.assertEqual(st.cls, pg.CLASS_TUNNEL_ABSENT)
def test_fresh_handshake_but_route_closed_is_not_ok(self):
self.sysm.table_default = "closed"
st = self.ctl.current_status(pg.default_state())
self.assertTrue(st.safety_ok)
self.assertFalse(st.tunnel_ok)
self.assertEqual(st.cls, pg.CLASS_TUNNEL_ABSENT)
def test_boot_grace_class(self):
self.go_stale()
state = pg.default_state()
state["boot_grace_until"] = self.sysm.t + 60
self.assertEqual(self.ctl.current_status(state).cls, pg.CLASS_BOOT_GRACE)
def test_paused_class(self):
self.go_stale()
state = pg.default_state()
state["paused"] = {"since": 1, "reason": "x"}
st = self.ctl.current_status(state)
self.assertEqual(st.cls, pg.CLASS_PAUSED)
self.assertEqual(st.tunnel_class, pg.CLASS_TUNNEL_STALE)
class TestSafetyDrift(Base):
def drift_cases(self):
yield "missing rule", lambda: setattr(self.sysm, "rule_present", False)
yield "missing route", lambda: setattr(self.sysm, "table_default", "missing")
yield "jump below accept", lambda: setattr(self.sysm, "forward_rules", ["-A FORWARD -j ACCEPT", f"-A FORWARD -s {SUBNET} -j {CHAIN}"])
yield "no terminal drop", lambda: setattr(self.sysm, "chain_rules", self.sysm.open_chain()[:-1])
yield "wrong iface accept", lambda: setattr(self.sysm, "chain_rules", [f"-A {CHAIN} -s {SUBNET} -o enp1s0 -j ACCEPT"] + self.sysm.open_chain())
yield "chain missing", lambda: setattr(self.sysm, "chain_rules", None)
yield "host default via tunnel", lambda: setattr(self.sysm, "main_default", "default dev pia-wg scope link")
def test_each_drift_triggers_emergency_close_and_no_restart(self):
for name, apply in self.drift_cases():
with self.subTest(name):
self.setUp()
apply()
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_SAFETY, name)
self.assertEqual(self.sysm.restart_count, 0, name)
self.assertTrue(self.sysm.forward_rules[0].startswith(f"-A FORWARD -s {SUBNET} -m comment --comment pia-gateway-emergency -j DROP"), name)
self.assertEqual(self.sysm.table_default, "closed", name)
self.assertTrue(self.state()["emergency_close_active"], name)
self.assertTrue(any(lvl == "error" and "SAFETY DRIFT" in msg for lvl, msg in self.notified), name)
self.assert_no_secrets()
self.tearDown()
def test_emergency_rule_removed_after_repair(self):
self.sysm.rule_present = False
self.ctl.cmd_recover()
self.assertTrue(self.state()["emergency_close_active"])
# operator repairs the kill switch and the tunnel
self.sysm.rule_present = True
self.sysm.table_default = "open"
self.sysm.set_peer(SRV_OLD, 5)
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_OK)
self.assertFalse(self.state()["emergency_close_active"])
self.assertFalse(any("emergency" in r for r in self.sysm.forward_rules))
def test_emergency_close_is_idempotent(self):
self.sysm.rule_present = False
self.ctl.cmd_recover()
self.ctl.cmd_recover()
self.assertEqual(sum("emergency" in r for r in self.sysm.forward_rules), 1)
def test_emergency_close_repositions_when_pushed_below_a_preceding_accept(self):
self.sysm.rule_present = False
self.ctl.cmd_recover()
self.assertTrue(pg._is_emergency_rule(self.sysm.forward_rules[0]))
# Something outside this role's own handler ordering (e.g. a
# manual `ufw reload`) prepends a broader ACCEPT ahead of our
# rule without removing it — it still exists, but is no longer
# first, so it could be pre-empted by whatever now sits above it.
self.sysm.forward_rules.insert(0, "-A FORWARD -j ACCEPT")
self.assertFalse(pg._is_emergency_rule(self.sysm.forward_rules[0]))
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_SAFETY)
self.assertTrue(pg._is_emergency_rule(self.sysm.forward_rules[0]))
self.assertEqual(sum(1 for r in self.sysm.forward_rules if pg._is_emergency_rule(r)), 1)
self.assertIn("-A FORWARD -j ACCEPT", self.sysm.forward_rules)
def test_register_refuses_on_drift(self):
self.sysm.rule_present = False
rc = self.ctl.cmd_register("same-key", None, "ansible")
self.assertEqual(rc, pg.EXIT_SAFETY)
self.assertEqual(self.http.calls, [])
KUBE_ROUTER_FORWARD_JUMP = '-A FORWARD -m comment --comment "kube-router netpol - TEMCG2JMHZYE7H7T" -j KUBE-ROUTER-FORWARD'
KUBE_ROUTER_SAFE_RULES = [
'-A KUBE-ROUTER-FORWARD -d 10.42.0.39/32 -m comment --comment "pod X" -j KUBE-POD-FW-AAAA',
'-A KUBE-ROUTER-FORWARD -s 10.42.0.39/32 -m comment --comment "pod X" -j KUBE-POD-FW-AAAA',
'-A KUBE-ROUTER-FORWARD -d 10.42.1.5/32 -m physdev --physdev-is-bridged -m comment --comment "pod Y" -j KUBE-POD-FW-BBBB',
]
class TestForwardBypassChains(Base):
"""The content-verified bypass mechanism for kube-router's own FORWARD
jump — found necessary live, 2026-09-22, after the plain reposition
fix alone caused a repeating outage loop: kube-router resyncs (and
re-pushes our jump down) roughly every 60-90s on this cluster, far
more often than assumed, and treating every transient reordering as a
full safety-drift emergency (closing the route each time) kept the
tunnel down far more than it was up.
"""
def classify_with_forward(self, forward_rules, bypass_chains=(), extra_chains=None):
self.cfg.forward_bypass_chains = list(bypass_chains)
self.sysm.forward_rules = list(forward_rules)
if extra_chains:
self.sysm.extra_chains.update(extra_chains)
return self.ctl.classifier.classify(pg.default_state())
def test_disabled_by_default_kube_router_ahead_still_blocks(self):
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=(), # default: nothing configured
)
self.assertFalse(st.safety_ok)
self.assertIn("FORWARD jump to kill-switch chain is not first", " ".join(st.reasons))
def test_configured_bypass_with_verified_safe_content_tolerated(self):
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
)
self.assertTrue(st.safety_ok, st.reasons)
def test_duplicate_bypass_jump_tolerated(self):
# The exact live anomaly: kube-router left two copies of its own
# jump behind after a resync.
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
)
self.assertTrue(st.safety_ok, st.reasons)
def test_bypass_chain_picking_up_a_matching_source_still_blocks(self):
dangerous = KUBE_ROUTER_SAFE_RULES + [f"-A KUBE-ROUTER-FORWARD -s {SUBNET} -j ACCEPT"]
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
)
self.assertFalse(st.safety_ok)
self.assertIn("no longer verified safe", " ".join(st.reasons))
def test_bypass_chain_with_bare_unconditional_rule_blocks(self):
dangerous = KUBE_ROUTER_SAFE_RULES + ["-A KUBE-ROUTER-FORWARD -j ACCEPT"]
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
)
self.assertFalse(st.safety_ok)
def test_bypass_chain_with_negated_match_blocks(self):
dangerous = [f'-A KUBE-ROUTER-FORWARD ! -s {SUBNET} -j ACCEPT']
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
)
self.assertFalse(st.safety_ok)
def test_unrecognized_chain_ahead_still_blocks(self):
# DOCKER-USER is not on the allowlist even when KUBE-ROUTER-FORWARD is.
st = self.classify_with_forward(
["-A FORWARD -j DOCKER-USER", f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"DOCKER-USER": []},
)
self.assertFalse(st.safety_ok)
def test_forward_level_rule_with_real_matcher_not_treated_as_bypass(self):
# Even though KUBE-ROUTER-FORWARD is allowlisted and its content is
# safe, a FORWARD-level rule that itself carries a real matcher
# (not just a cosmetic comment) is never treated as a bypass jump.
st = self.classify_with_forward(
[f"-A FORWARD -s 1.2.3.0/24 -j KUBE-ROUTER-FORWARD", f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
)
self.assertFalse(st.safety_ok)
def test_missing_bypass_chain_blocks(self):
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": None},
)
self.assertFalse(st.safety_ok)
self.assertIn("unreadable", " ".join(st.reasons))
def test_end_to_end_recover_does_not_emergency_close_on_tolerated_drift(self):
# The actual live incident, reproduced end to end: kube-router's
# jump sits ahead of ours, content-verified-safe, and the recovery
# timer must NOT declare safety drift or touch the route.
self.cfg.forward_bypass_chains = ["KUBE-ROUTER-FORWARD"]
self.sysm.forward_rules = [KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"]
self.sysm.extra_chains["KUBE-ROUTER-FORWARD"] = KUBE_ROUTER_SAFE_RULES
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_OK)
self.assertFalse(self.state()["emergency_close_active"])
self.assertEqual(self.sysm.table_default, "open")
self.assertEqual(self.sysm.restart_count, 0)
def test_end_to_end_recover_still_emergency_closes_on_real_drift(self):
# Regression guard: an ACTUALLY unrecognized rule ahead of ours
# must still trigger the full emergency-close path, even with a
# bypass list configured for something else.
self.cfg.forward_bypass_chains = ["KUBE-ROUTER-FORWARD"]
self.sysm.forward_rules = ["-A FORWARD -j DOCKER-USER", f"-A FORWARD -s {SUBNET} -j {CHAIN}"]
self.sysm.extra_chains["DOCKER-USER"] = []
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_SAFETY)
self.assertTrue(self.state()["emergency_close_active"])
class TestRecoveryStateMachine(Base):
def test_healthy_runs_are_idempotent_and_do_not_mutate(self):
self.run_recover_n(3)
mutating = [
c for c in self.sysm.commands
if c[:2] in (["systemctl", "restart"], ["systemctl", "start"], ["wg", "genkey"], ["iptables", "-I"], ["iptables", "-D"], ["ip", "link"])
or c[:3] == ["ip", "route", "replace"]
]
self.assertEqual(mutating, [])
self.assertEqual(self.state()["consecutive_failures"], 0)
def test_threshold_before_action(self):
self.go_stale()
self.run_recover_n(2)
self.assertEqual(self.sysm.restart_count, 0)
self.assertEqual(self.state()["consecutive_failures"], 2)
def test_boot_grace_not_counted(self):
self.go_stale()
self.sysm.boot = "boot-2"
self.run_recover_n(2)
self.assertEqual(self.state()["consecutive_failures"], 0)
self.assertEqual(self.sysm.restart_count, 0)
self.sysm.sleep(120)
self.run_recover_n(3)
self.assertEqual(self.sysm.restart_count, 1)
def test_restart_success_recovers(self):
self.go_stale()
rcs = self.run_recover_n(3)
self.assertEqual(rcs[-1], pg.EXIT_OK)
self.assertEqual(self.sysm.restart_count, 1)
s = self.state()
self.assertEqual(s["last_recovery"]["stage"], "restart")
self.assertEqual(s["last_recovery"]["pre_fix_handshake_age_sec"], 5000 + 120)
self.assertEqual(s["consecutive_failures"], 0)
self.assertFalse(s["recovery"]["in_progress"])
self.assertEqual(self.http.calls, [])
levels = [l for l, _ in self.notified]
self.assertEqual(levels, ["warn", "info"])
self.assert_no_secrets()
def test_restart_failure_escalates_to_same_key_registration(self):
self.go_stale()
self.sysm.handshake_on_restart = lambda key: (self.sysm.t if key == SRV_HK1 else None)
self.http.addkey = {"hk1.example": ("ok", {})}
rcs = self.run_recover_n(3)
self.assertEqual(rcs[-1], pg.EXIT_OK)
s = self.state()
self.assertEqual(s["last_recovery"]["stage"], "reregister")
self.assertEqual(s["last_verified_registration"]["server_cn"], "hk1.example")
self.assertEqual(s["last_verified_registration"]["mode"], "same-key")
self.assertEqual(s["last_verified_registration"]["pubkey"], PUB_A)
conf = (self.tmp / "wg" / "pia-wg.conf").read_text()
self.assertIn(f"PublicKey = {SRV_HK1}", conf)
self.assertIn("Endpoint = 10.99.0.1:1337", conf)
self.assertIn("Table = off", conf)
self.assertEqual(stat.S_IMODE((self.tmp / "wg" / "pia-wg.conf").stat().st_mode), 0o600)
addkeys = [c for c in self.http.calls if c[0] == "addkey"]
self.assertEqual(addkeys[0][5]["pubkey"], PUB_A)
self.assertEqual(addkeys[0][5]["pt"], SECRET_TOKEN)
self.assertTrue(any(n.startswith("pia-wg.conf.bak-") for n in os.listdir(self.tmp / "wg")))
self.assert_no_secrets()
def test_later_candidate_after_activation_failure(self):
self.go_stale()
self.sysm.handshake_on_restart = lambda key: (self.sysm.t if key == SRV_HK2 else None)
self.http.addkey = {"hk1.example": ("ok", {}), "hk2.example": ("ok", {})}
self.run_recover_n(3)
s = self.state()
self.assertEqual(s["last_verified_registration"]["server_cn"], "hk2.example")
order = [c[1] for c in self.http.calls if c[0] == "addkey"]
self.assertEqual(order, ["hk1.example", "hk2.example"])
def test_region_order_hk_then_taiwan(self):
self.go_stale()
self.sysm.handshake_on_restart = lambda key: (self.sysm.t if key == SRV_TW1 else None)
self.http.addkey = {"hk1.example": ("http", 500), "hk2.example": ("err", "timeout"), "tw1.example": ("ok", {})}
self.run_recover_n(3)
order = [c[1] for c in self.http.calls if c[0] == "addkey"]
self.assertEqual(order, ["hk1.example", "hk2.example", "tw1.example"])
self.assertEqual(self.state()["last_verified_registration"]["region"], "taiwan")
def test_rotation_after_same_key_accepted_but_no_handshake(self):
self.go_stale()
def beh(key):
# Only a config carrying the NEW key handshakes.
conf = (self.tmp / "wg" / "pia-wg.conf").read_text()
return self.sysm.t if f"PrivateKey = {PRIV_B}" in conf else None
self.sysm.handshake_on_restart = beh
self.http.addkey = {"hk1.example": ("ok", {}), "hk2.example": ("ok", {}), "tw1.example": ("ok", {})}
rcs = self.run_recover_n(3)
self.assertEqual(rcs[-1], pg.EXIT_OK)
s = self.state()
self.assertEqual(s["last_recovery"]["stage"], "rotate")
self.assertEqual(s["last_verified_registration"]["mode"], "new-key")
self.assertEqual(s["last_verified_registration"]["pubkey"], PUB_B)
self.assertEqual((self.tmp / "wg" / "pia-wg.key").read_text().strip(), PRIV_B)
names = os.listdir(self.tmp / "wg")
self.assertTrue(any(n.startswith("pia-wg.key.bak-") for n in names))
self.assertFalse(any(n.startswith("pia-wg.key.new") for n in names))
self.assertFalse(any(".conf.stage" in n for n in names))
self.assertEqual(stat.S_IMODE((self.tmp / "wg" / "pia-wg.key").stat().st_mode), 0o600)
self.assert_no_secrets()
def test_no_rotation_on_uniform_api_failure(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.addkey = {"hk1.example": ("err", "timeout"), "hk2.example": ("err", "timeout"), "tw1.example": ("err", "timeout")}
self.run_recover_n(3)
s = self.state()
self.assertEqual(s["recovery"]["last_failure_class"], pg.FAIL_HOST_CONNECTIVITY)
self.assertIsNotNone(s["recovery"]["next_retry_epoch"])
self.assertEqual(len(self.sysm.genkey_queue), 1) # no key generated
self.assertEqual((self.tmp / "wg" / "pia-wg.key").read_text().strip(), PRIV_A)
def test_all_candidates_fail_restores_previous_and_backs_off(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.addkey = {"hk1.example": ("ok", {}), "hk2.example": ("http", 500), "tw1.example": ("ok", {})}
rcs = self.run_recover_n(3)
self.assertEqual(rcs[-1], pg.EXIT_UNHEALTHY)
s = self.state()
self.assertTrue(s["recovery"]["in_progress"])
self.assertIsNotNone(s["recovery"]["next_retry_epoch"])
conf = (self.tmp / "wg" / "pia-wg.conf").read_text()
self.assertIn(f"PublicKey = {SRV_OLD}", conf)
self.assertEqual((self.tmp / "wg" / "pia-wg.key").read_text().strip(), PRIV_A)
self.assertFalse(any(n.startswith("pia-wg.key.new") for n in os.listdir(self.tmp / "wg")))
# kill switch untouched: chain still ends in DROP, jump still first
self.assertEqual(self.sysm.forward_rules[0], f"-A FORWARD -s {SUBNET} -j {CHAIN}")
self.assertEqual(self.sysm.chain_rules[-1], f"-A {CHAIN} -j DROP")
self.assertTrue(any("recovery failed" in m for _, m in self.notified))
self.assert_no_secrets()
def test_auth_failure_no_rotation_error_alert(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.token = ("http", 401)
self.run_recover_n(3)
s = self.state()
self.assertEqual(s["recovery"]["last_failure_class"], pg.FAIL_AUTH)
self.assertFalse(any(c[0] == "addkey" for c in self.http.calls))
self.assertTrue(any(l == "error" for l, _ in self.notified))
self.assertEqual(len(self.sysm.genkey_queue), 1)
def test_serverlist_unreachable_is_host_connectivity(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.serverlist = ("err", "network")
self.run_recover_n(3)
self.assertEqual(self.state()["recovery"]["last_failure_class"], pg.FAIL_HOST_CONNECTIVITY)
self.assertFalse(any(c[0] in ("post", "addkey") for c in self.http.calls))
def test_malformed_serverlist(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.serverlist = ("raw", b"not json\n")
self.run_recover_n(3)
self.assertEqual(self.state()["recovery"]["last_failure_class"], pg.FAIL_REGISTRATION)
def test_malformed_and_invalid_addkey_responses_skip_to_next(self):
self.go_stale()
self.sysm.handshake_on_restart = lambda key: (self.sysm.t if key == SRV_TW1 else None)
self.http.addkey = {
"hk1.example": ("raw", b"<html>"),
"hk2.example": ("ok", {"peer_ip": "999.1.1.1"}),
"tw1.example": ("ok", {}),
}
self.run_recover_n(3)
self.assertEqual(self.state()["last_verified_registration"]["server_cn"], "tw1.example")
def test_tls_failure_kind_recorded(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.addkey = {"hk1.example": ("err", "tls"), "hk2.example": ("err", "tls"), "tw1.example": ("err", "tls")}
self.run_recover_n(3)
self.assertEqual(self.state()["recovery"]["last_failure_class"], pg.FAIL_REGISTRATION)
self.assertIn("kind=tls", self.out.getvalue())
def test_backoff_sequence_bounded(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.serverlist = ("err", "network")
delays = []
self.run_recover_n(3)
for _ in range(5):
s = self.state()
delays.append(s["recovery"]["next_retry_epoch"] - s["recovery"]["last_failure_epoch"])
self.sysm.t = s["recovery"]["next_retry_epoch"] + 1
self.ctl.cmd_recover()
self.assertEqual(delays, [300, 900, 1800, 3600, 3600])
def test_backoff_prevents_hammering(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.serverlist = ("err", "network")
self.run_recover_n(3)
before = self.sysm.restart_count
self.run_recover_n(4)
self.assertEqual(self.sysm.restart_count, before)
def test_retry_now_bypasses_backoff(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.serverlist = ("err", "network")
self.run_recover_n(3)
before = self.sysm.restart_count
self.sysm.handshake_on_restart = "fresh"
rc = self.ctl.cmd_retry_now()
self.assertEqual(rc, pg.EXIT_OK)
self.assertEqual(self.sysm.restart_count, before + 1)
def test_credentials_missing_still_backs_off(self):
os.unlink(self.cfg.credentials_file)
self.ctl._creds = None
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.run_recover_n(3)
s = self.state()
self.assertIsNotNone(s["recovery"]["next_retry_epoch"])
before = self.sysm.restart_count
self.run_recover_n(2)
self.assertEqual(self.sysm.restart_count, before)
def test_time_budget_bounds_a_cycle(self):
self.cfg.recovery_max_duration_sec = 100
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.addkey = {"hk1.example": ("ok", {}), "hk2.example": ("ok", {}), "tw1.example": ("ok", {})}
self.run_recover_n(3)
addkeys = [c for c in self.http.calls if c[0] == "addkey"]
self.assertLess(len(addkeys), 3)
self.assertIn("time budget", self.out.getvalue())
def test_recovered_without_our_action_clears_backoff(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.serverlist = ("err", "network")
self.run_recover_n(3)
self.sysm.set_peer(SRV_OLD, 5)
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_OK)
self.assertIsNone(self.state()["recovery"]["next_retry_epoch"])
self.assertFalse(self.state()["recovery"]["in_progress"])
def test_backward_clock_jump_clears_stale_deadline(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.serverlist = ("err", "network")
self.run_recover_n(3)
self.assertIsNotNone(self.state()["recovery"]["next_retry_epoch"])
self.sysm.t -= 6 * 3600 # NTP steps the clock back six hours
self.sysm.handshake_on_restart = "fresh"
self.http.serverlist = ("ok", serverlist_doc())
self.sysm.set_peer(SRV_OLD, 5000)
rc = self.ctl.cmd_recover()
self.assertIn("clock jump detected", self.out.getvalue())
self.assertEqual(rc, pg.EXIT_OK)
self.assertEqual(self.state()["last_recovery"]["stage"], "restart")
def test_normal_backoff_is_not_mistaken_for_clock_jump(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.serverlist = ("err", "network")
self.run_recover_n(3)
self.ctl.cmd_recover()
self.assertNotIn("clock jump detected", self.out.getvalue())
self.assertIsNotNone(self.state()["recovery"]["next_retry_epoch"])
def test_orphaned_interface_recovery_on_restart(self):
self.go_stale()
calls = {"n": 0}
orig = self.sysm._apply_restart
def restart(conf_path):
calls["n"] += 1
if calls["n"] == 1:
self.sysm.restart_count += 1
return pg.CmdResult(1, "", "RTNETLINK answers: File exists\n`pia-wg' already exists")
return orig(conf_path)
self.sysm._apply_restart = restart
self.run_recover_n(3)
self.assertIn(["ip", "link", "delete", "dev", IFACE], self.sysm.commands)
self.assertEqual(self.state()["last_recovery"]["stage"], "restart")
class TestHandshakeVerificationBoundary(Base):
"""wait_for_verified's freshness check, at the exact one-second boundary.
`wg show`'s handshake epoch and `since_epoch` are both whole seconds,
so a handshake reported in the SAME second as `since_epoch` is
indistinguishable from one that predates activation and merely shares
that truncated second. It must never count as proof of a NEW
handshake — that's the whole point of comparing against `since_epoch`
at all (an `addKey` returning `status: OK` was already established
elsewhere as necessary-but-not-sufficient; the same logic applies one
layer down, to the handshake timestamp itself).
"""
def test_handshake_at_exactly_since_epoch_is_rejected(self):
since_epoch = self.sysm.t
self.sysm.set_peer(SRV_OLD, 0) # handshake_epoch == self.sysm.t == since_epoch, exactly
verified = self.ctl.wait_for_verified(pg.default_state(), since_epoch, timeout_sec=1, expected_peer=SRV_OLD)
self.assertIsNone(verified)
def test_handshake_one_second_after_since_epoch_is_accepted(self):
since_epoch = self.sysm.t
self.sysm.sleep(1)
self.sysm.set_peer(SRV_OLD, 0) # handshake_epoch == self.sysm.t == since_epoch + 1
verified = self.ctl.wait_for_verified(pg.default_state(), since_epoch, timeout_sec=1, expected_peer=SRV_OLD)
self.assertIsNotNone(verified)
def test_handshake_one_second_before_since_epoch_is_rejected(self):
self.sysm.sleep(1)
since_epoch = self.sysm.t
self.sysm.set_peer(SRV_OLD, 1) # handshake_epoch == self.sysm.t - 1 == since_epoch - 1
verified = self.ctl.wait_for_verified(pg.default_state(), since_epoch, timeout_sec=1, expected_peer=SRV_OLD)
self.assertIsNone(verified)
class TestPersistenceAndConcurrency(Base):
def test_state_survives_new_controller(self):
self.go_stale()
self.sysm.handshake_on_restart = "stale"
self.http.serverlist = ("err", "network")
self.run_recover_n(3)
s1 = self.state()
ctl2 = pg.Controller(self.cfg, self.sysm, self.log, http=self.http)
self.sysm.boot = "boot-2"
ctl2.cmd_recover()
s2 = self.state()
self.assertEqual(s1["recovery"]["next_retry_epoch"], s2["recovery"]["next_retry_epoch"])
self.assertEqual(s2["boot_id"], "boot-2")
def test_corrupt_state_fails_closed(self):
(self.tmp / "state" / "state.json").write_text("{not json")
self.go_stale()
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_STATE_ERROR)
self.assertEqual(self.sysm.restart_count, 0)
self.assertTrue(any("corrupt" in m for _, m in self.notified))
self.assertEqual(self.ctl.cmd_renew(), pg.EXIT_STATE_ERROR)
self.assertEqual(self.ctl.cmd_register("same-key", None, "operator"), pg.EXIT_STATE_ERROR)
self.assertEqual(self.ctl.cmd_rewrite_config(), pg.EXIT_STATE_ERROR)
self.assertEqual(self.ctl.cmd_pause("x"), pg.EXIT_STATE_ERROR)
self.assertEqual(self.ctl.cmd_resume(), pg.EXIT_STATE_ERROR)
def test_wrong_schema_version_fails_closed(self):
s = pg.default_state()
s["schema_version"] = 99
(self.tmp / "state" / "state.json").write_text(json.dumps(s))
self.assertEqual(self.ctl.cmd_recover(), pg.EXIT_STATE_ERROR)
def test_state_reset_requires_confirm_and_preserves_old(self):
(self.tmp / "state" / "state.json").write_text("{bad")
self.assertEqual(self.ctl.cmd_state_reset(confirm=False), pg.EXIT_USAGE)
self.assertEqual(self.ctl.cmd_state_reset(confirm=True), pg.EXIT_OK)
names = os.listdir(self.tmp / "state")
self.assertTrue(any(n.startswith("state.json.corrupt-") for n in names))
self.assertEqual(self.state()["schema_version"], pg.STATE_SCHEMA_VERSION)
def test_atomic_write_no_temp_left_and_mode(self):
p = self.tmp / "state" / "x.json"
self.sysm.write_atomic(str(p), "abc", 0o600)
self.assertEqual(p.read_text(), "abc")
self.assertEqual(stat.S_IMODE(p.stat().st_mode), 0o600)
self.assertEqual([n for n in os.listdir(self.tmp / "state") if n.endswith(".tmp")], [])
def test_interrupted_write_keeps_original(self):
p = self.tmp / "state" / "state.json"
p.write_text("original")
with mock.patch.object(pg.os, "replace", side_effect=OSError("boom")):
with self.assertRaises(OSError):
self.sysm.write_atomic(str(p), "new", 0o600)
self.assertEqual(p.read_text(), "original")
self.assertEqual([n for n in os.listdir(self.tmp / "state") if n.endswith(".tmp")], [])
def test_lock_contention_skips_without_acting(self):
self.go_stale()
self.run_recover_n(2)
holder = pg.Lock(self.cfg.lock_file, self.sysm)
holder.acquire(0)
try:
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_OK)
self.assertEqual(self.sysm.restart_count, 0)
self.assertIn("another action holds the lock", self.out.getvalue())
self.assertEqual(self.ctl.cmd_register("same-key", None, "operator"), pg.EXIT_LOCKED)
finally:
holder.release()
rc = self.ctl.cmd_recover()
self.assertEqual(self.sysm.restart_count, 1)
def test_lock_released_on_exception(self):
with mock.patch.object(self.ctl, "_decide", side_effect=RuntimeError("x")):
with self.assertRaises(RuntimeError):
self.ctl.cmd_recover()
lk = pg.Lock(self.cfg.lock_file, self.sysm)
lk.acquire(0)
lk.release()
def test_pause_and_resume(self):
self.ctl.cmd_pause("maintenance")
self.go_stale()
self.run_recover_n(5)
self.assertEqual(self.sysm.restart_count, 0)
self.assertTrue(self.state()["paused"])
self.assertEqual(self.ctl.cmd_renew(force_due=True), pg.EXIT_OK)
self.assertEqual(self.http.calls, [])
self.ctl.cmd_resume()
self.run_recover_n(3)
self.assertEqual(self.sysm.restart_count, 1)
def test_pause_does_not_open_gateway(self):
self.sysm.table_default = "closed"
self.sysm.chain_rules = self.sysm.closed_chain()
self.ctl.cmd_pause("x")
self.ctl.cmd_recover()
self.assertEqual(self.sysm.table_default, "closed")
self.assertEqual(self.sysm.restart_count, 0)
def test_pause_and_resume_serialize_with_an_inflight_recovery_pass(self):
# Simulate a recovery pass that is mid-flight: it already holds the
# SAME lock pause/resume now also take, and has not yet saved its
# (unpaused) state back to disk — this is exactly the window in
# which the old, unlocked pause/resume could silently lose the
# race and have their write clobbered by the pass finishing later.
self.cfg.lock_wait_sec = 3
holder = pg.Lock(self.cfg.lock_file, self.sysm)
holder.acquire(0)
try:
self.assertEqual(self.ctl.cmd_pause("maintenance"), pg.EXIT_LOCKED)
self.assertEqual(self.ctl.cmd_resume(), pg.EXIT_LOCKED)
# blocked, not silently skipped-and-lost: state is untouched
self.assertIsNone(self.state().get("paused"))
finally:
# the in-flight pass "finishes": saves its own state (as
# cmd_recover's finally block does) and releases the lock.
self.ctl.store.save(pg.default_state())
holder.release()
# now that the lock is free, pause proceeds against the CURRENT
# on-disk state rather than a copy read before it waited
self.assertEqual(self.ctl.cmd_pause("maintenance"), pg.EXIT_OK)
self.assertTrue(self.state()["paused"])
# a recovery pass started afterwards must also acquire the lock
# and load state fresh before it can act, so it observes the
# pause and never clears or overwrites it
self.go_stale()
self.run_recover_n(5)
self.assertEqual(self.sysm.restart_count, 0)
self.assertTrue(self.state()["paused"])
def test_state_reset_serializes_with_lock(self):
holder = pg.Lock(self.cfg.lock_file, self.sysm)
holder.acquire(0)
try:
self.assertEqual(self.ctl.cmd_state_reset(confirm=True), pg.EXIT_LOCKED)
finally:
holder.release()
self.assertEqual(self.ctl.cmd_state_reset(confirm=True), pg.EXIT_OK)
class TestRenewal(Base):
def seed_registration(self, age_days):
s = pg.default_state()
s["last_verified_registration"] = {"epoch": self.sysm.t - age_days * 86400, "region": "hk", "server_cn": "old.example", "pubkey": PUB_A, "mode": "same-key", "via": "ansible"}
self.ctl.store.save(s)
def test_not_due(self):
self.seed_registration(10)
self.assertEqual(self.ctl.cmd_renew(), pg.EXIT_OK)
self.assertEqual(self.http.calls, [])
def test_due_and_succeeds_advances_timestamp(self):
self.seed_registration(22)
self.sysm.handshake_on_restart = "fresh"
self.http.addkey = {"hk1.example": ("ok", {})}
self.assertEqual(self.ctl.cmd_renew(), pg.EXIT_OK)
s = self.state()
self.assertEqual(s["last_verified_registration"]["via"], "renew")
self.assertEqual(s["last_verified_registration"]["mode"], "same-key")
self.assertGreaterEqual(s["last_verified_registration"]["epoch"], self.sysm.t - 60)
self.assertEqual(len(self.sysm.genkey_queue), 1)
def test_due_and_fails_keeps_timestamp_and_backs_off(self):
self.seed_registration(22)
old = self.state()["last_verified_registration"]["epoch"]
self.sysm.handshake_on_restart = lambda key: (self.sysm.t if key == SRV_OLD else None)
self.http.addkey = {"hk1.example": ("ok", {}), "hk2.example": ("ok", {}), "tw1.example": ("ok", {})}
self.assertEqual(self.ctl.cmd_renew(), pg.EXIT_UNHEALTHY)
s = self.state()
self.assertEqual(s["last_verified_registration"]["epoch"], old)
self.assertIsNotNone(s["renewal"]["next_retry_epoch"])
self.assertIn(f"PublicKey = {SRV_OLD}", (self.tmp / "wg" / "pia-wg.conf").read_text())
self.assertEqual(self.ctl.cmd_renew(), pg.EXIT_OK) # in backoff, no call
self.assertTrue(any("renewal failed" in m for _, m in self.notified))
def test_force_due(self):
self.seed_registration(1)
self.sysm.handshake_on_restart = "fresh"
self.http.addkey = {"hk1.example": ("ok", {})}
self.assertEqual(self.ctl.cmd_renew(force_due=True), pg.EXIT_OK)
self.assertEqual(self.state()["last_verified_registration"]["via"], "renew")
def test_defers_to_recovery_when_unhealthy(self):
self.seed_registration(30)
self.go_stale()
self.assertEqual(self.ctl.cmd_renew(), pg.EXIT_OK)
self.assertEqual(self.http.calls, [])
class TestRegisterCommand(Base):
def test_fresh_install_generates_key_and_registers(self):
os.unlink(self.tmp / "wg" / "pia-wg.conf")
os.unlink(self.tmp / "wg" / "pia-wg.key")
self.sysm.iface_exists = False
self.sysm.table_default = "closed"
self.sysm.chain_rules = self.sysm.closed_chain()
self.sysm.genkey_queue = [PRIV_A]
self.sysm.handshake_on_restart = "fresh"
self.http.addkey = {"hk1.example": ("ok", {})}
rc = self.ctl.cmd_register("same-key", None, "ansible")
self.assertEqual(rc, pg.EXIT_OK)
self.assertEqual((self.tmp / "wg" / "pia-wg.key").read_text().strip(), PRIV_A)
self.assertEqual(stat.S_IMODE((self.tmp / "wg" / "pia-wg.key").stat().st_mode), 0o600)
s = self.state()
self.assertEqual(s["last_verified_registration"]["via"], "ansible")
self.assertEqual(s["active_registration"]["server_key"], SRV_HK1)
self.assert_no_secrets()
def test_register_refuses_when_killswitch_inactive(self):
self.sysm.killswitch_active = False
self.sysm.handshake_on_restart = "fresh"
self.http.addkey = {"hk1.example": ("ok", {})}
rc = self.ctl.cmd_register("same-key", None, "ansible")
self.assertEqual(rc, pg.EXIT_UNHEALTHY)
self.assertEqual(self.sysm.restart_count, 0)
self.assertIn(f"PublicKey = {SRV_OLD}", (self.tmp / "wg" / "pia-wg.conf").read_text())
def test_region_override(self):
self.sysm.handshake_on_restart = "fresh"
self.http.addkey = {"tw1.example": ("ok", {})}
rc = self.ctl.cmd_register("same-key", "taiwan", "operator")
self.assertEqual(rc, pg.EXIT_OK)
self.assertEqual([c[1] for c in self.http.calls if c[0] == "addkey"], ["tw1.example"])
def test_new_key_mode_with_abandoned_stage_cleanup(self):
(self.tmp / "wg" / "pia-wg.key.new").write_text("abandoned\n")
(self.tmp / "wg" / "pia-wg.key.new.20260101T000000Z.1").write_text("abandoned2\n")
self.sysm.handshake_on_restart = "fresh"
self.http.addkey = {"hk1.example": ("ok", {})}
rc = self.ctl.cmd_register("new-key", None, "operator")
self.assertEqual(rc, pg.EXIT_OK)
names = os.listdir(self.tmp / "wg")
self.assertFalse(any(n.startswith("pia-wg.key.new") for n in names))
self.assertEqual((self.tmp / "wg" / "pia-wg.key").read_text().strip(), PRIV_B)
self.assertIn(f"pubkey", json.dumps(self.state()["last_verified_registration"]))
self.assertEqual(self.state()["last_verified_registration"]["pubkey"], PUB_B)
def test_same_key_does_not_rotate(self):
self.sysm.handshake_on_restart = "fresh"
self.http.addkey = {"hk1.example": ("ok", {})}
self.ctl.cmd_register("same-key", None, "operator")
self.assertEqual((self.tmp / "wg" / "pia-wg.key").read_text().strip(), PRIV_A)
self.assertEqual(len(self.sysm.genkey_queue), 1)
def test_backups_bounded(self):
self.cfg.backup_keep = 2
self.sysm.handshake_on_restart = "fresh"
self.http.addkey = {"hk1.example": ("ok", {})}
for i in range(5):
self.sysm.sleep(1)
self.ctl.cmd_register("same-key", None, "operator")
baks = [n for n in os.listdir(self.tmp / "wg") if n.startswith("pia-wg.conf.bak-")]
self.assertEqual(len(baks), 2)
def test_rewrite_config_uses_recorded_registration(self):
self.sysm.handshake_on_restart = "fresh"
self.http.addkey = {"hk1.example": ("ok", {})}
self.ctl.cmd_register("same-key", None, "operator")
self.cfg.mss_clamp_enabled = False
calls_before = len(self.http.calls)
rc = self.ctl.cmd_rewrite_config()
self.assertEqual(rc, pg.EXIT_OK)
self.assertEqual(len(self.http.calls), calls_before)
conf = (self.tmp / "wg" / "pia-wg.conf").read_text()
self.assertNotIn("TCPMSS", conf)
self.assertIn(f"PublicKey = {SRV_HK1}", conf)
class TestConfigValidation(Base):
def render(self, **kw):
peer = pg.PeerInfo(SRV_HK1, "10.5.5.5", 1337, "10.99.0.1", "hk", "Hong Kong", "hk1.example")
return pg.render_wg_conf(self.cfg, PRIV_A, peer)
def test_rendered_config_is_valid(self):
self.assertEqual(pg.validate_wg_conf(self.cfg, self.render()), [])
def test_rejects_table_not_off(self):
text = self.render().replace("Table = off", "Table = auto")
self.assertIn("Table must be off", pg.validate_wg_conf(self.cfg, text))
def test_rejects_bad_keys_and_endpoint(self):
text = self.render().replace(f"PublicKey = {SRV_HK1}", "PublicKey = nope").replace("Endpoint = 10.99.0.1:1337", "Endpoint = bad")
problems = pg.validate_wg_conf(self.cfg, text)
self.assertIn("peer PublicKey invalid", problems)
self.assertIn("Endpoint invalid", problems)
def test_rejects_unscoped_hook_and_policy_change(self):
text = self.render().replace(
f"PostUp = ip route replace default dev %i table pia",
"PostUp = ip route replace default dev %i\nPostUp = iptables -P FORWARD ACCEPT",
)
problems = pg.validate_wg_conf(self.cfg, text)
self.assertTrue(any("without a table" in p for p in problems))
self.assertIn("hook changes FORWARD policy", problems)
self.assertIn("PostUp lacks table route open", problems)
def test_rejects_wide_allowed_ips_change(self):
text = self.render().replace("AllowedIPs = 0.0.0.0/0", "AllowedIPs = ::/0")
self.assertIn("AllowedIPs must be 0.0.0.0/0", pg.validate_wg_conf(self.cfg, text))
def test_expected_peer_parse(self):
self.assertEqual(pg.parse_expected_peer(self.render()), SRV_HK1)
self.assertIn("Hong Kong (hk), hk1.example", pg.parse_configured_peer_line(self.render()))
class TestSecretsAndModes(Base):
def test_credentials_mode_enforced(self):
os.chmod(self.cfg.credentials_file, 0o644)
with self.assertRaises(pg.GatewayError):
pg.load_credentials(self.cfg.credentials_file)
def test_credentials_repr_redacted(self):
c = pg.load_credentials(self.cfg.credentials_file)
self.assertNotIn(SECRET_PASS, repr(c) + str(c))
def test_no_secret_in_argv_stdin_logs_or_files_across_full_cycle(self):
self.go_stale()
self.sysm.handshake_on_restart = lambda key: (self.sysm.t if key == SRV_HK2 else None)
self.http.addkey = {"hk1.example": ("ok", {}), "hk2.example": ("ok", {})}
self.run_recover_n(3)
self.ctl.cmd_check()
self.ctl.cmd_status()
self.assert_no_secrets()
for argv in self.sysm.commands:
for tok in argv:
self.assertNotIn(SECRET_TOKEN, tok)
self.assertNotIn(PRIV_A, tok)
# the private key goes to `wg pubkey` via stdin only (never argv) and nowhere else
for s in self.sysm.stdins:
self.assertIn(s.strip(), (PRIV_A, PRIV_B))
def test_notification_payloads_are_secret_free_and_failures_do_not_block(self):
self._send_patch.stop()
sent = []
def flaky(_self, level, message):
sent.append((level, message))
return False
with mock.patch.object(pg.Notifier, "send", flaky):
self.go_stale()
self.run_recover_n(3)
self._send_patch.start()
self.assertEqual(self.state()["last_recovery"]["stage"], "restart")
self.assertGreaterEqual(len(self.state()["pending_notifications"]), 1)
for _, m in sent:
for sec in (SECRET_PASS, SECRET_TOKEN, PRIV_A):
self.assertNotIn(sec, m)
def test_pending_notifications_flushed_later_and_bounded(self):
s = pg.default_state()
s["pending_notifications"] = [{"level": "info", "message": f"m{i}", "epoch": 1} for i in range(30)]
self.ctl.store.save(s)
self.ctl.cmd_recover()
self.assertEqual(self.state()["pending_notifications"], [])
self.assertEqual(len([m for _, m in self.notified if m.startswith("m")]), 30)
def test_state_and_status_file_modes(self):
self.ctl.cmd_recover()
self.assertEqual(stat.S_IMODE((self.tmp / "state" / "state.json").stat().st_mode), 0o600)
self.assertTrue((self.tmp / "state" / "metrics.prom").exists())
m = (self.tmp / "state" / "metrics.prom").read_text()
self.assertIn("pia_gateway_healthy 1", m)
self.assertIn("pia_gateway_check_timestamp_seconds", m)
def test_check_is_read_only(self):
self.go_stale()
before = (self.tmp / "state" / "state.json").read_text()
rc = self.ctl.cmd_check()
self.assertEqual(rc, pg.EXIT_UNHEALTHY)
self.assertEqual(self.sysm.restart_count, 0)
self.assertEqual((self.tmp / "state" / "state.json").read_text(), before)
self.assertIn("class=tunnel_stale", self.out.getvalue())
def test_main_never_prints_traceback(self):
cfgp = self.tmp / "config.json"
cfgp.write_text(json.dumps(dataclass_dict(self.cfg)))
sysm = self.sysm
class Boom(FakeSystem):
def run(self, argv, stdin=None, timeout=60):
raise RuntimeError("kaboom " + SECRET_PASS)
out = io.StringIO()
rc = pg.main(["--config", str(cfgp), "recover"], system=Boom(self.tmp), http=self.http, stream=out)
self.assertEqual(rc, pg.EXIT_UNHEALTHY)
self.assertNotIn("Traceback", out.getvalue())
self.assertNotIn(SECRET_PASS, out.getvalue())
self.assertIn("RuntimeError", out.getvalue())
def test_config_rejects_unknown_keys(self):
cfgp = self.tmp / "config.json"
d = dataclass_dict(self.cfg)
d["typo_key"] = 1
cfgp.write_text(json.dumps(d))
out = io.StringIO()
rc = pg.main(["--config", str(cfgp), "status"], system=self.sysm, http=self.http, stream=out)
self.assertEqual(rc, pg.EXIT_USAGE)
def dataclass_dict(cfg):
import dataclasses
return dataclasses.asdict(cfg)
class FakeCurlSystem(pg.System):
"""Fakes just enough of System.run to exercise get_pinned_curl's argv
construction, stdin handling, and exit-code classification, without a
real curl binary or network access."""
def __init__(self, result=None):
self.result = result or pg.CmdResult(0, '{"status":"OK"}', "")
self.calls = []
self.stdins = []
def run(self, argv, stdin=None, timeout=60):
self.calls.append(list(argv))
if stdin is not None:
self.stdins.append(stdin)
return self.result
class TestPinnedCurlAddKey(unittest.TestCase):
def setUp(self):
self.cfg = pg.Config(ca_file="/etc/wireguard/pia-ca.crt", http_connect_timeout_sec=10, http_total_timeout_sec=30)
def test_argv_shape_matches_the_proven_legacy_curl_invocation(self):
sysm = FakeCurlSystem()
http_client = pg.HttpClient(self.cfg, sysm)
status, data = http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
self.assertEqual(status, 200)
self.assertEqual(data, b'{"status":"OK"}')
argv = sysm.calls[0]
self.assertEqual(argv[0], "curl")
self.assertIn("--fail", argv)
self.assertIn("-G", argv)
self.assertIn("--cacert", argv)
self.assertEqual(argv[argv.index("--cacert") + 1], "/etc/wireguard/pia-ca.crt")
self.assertIn("--connect-to", argv)
self.assertEqual(argv[argv.index("--connect-to") + 1], "Server-11669-10a::69.33.198.97:")
self.assertIn("pt@-", argv)
self.assertIn(f"pubkey={PUB_A}", argv)
self.assertEqual(argv[-1], "https://Server-11669-10a:1337/addKey")
# never -k/--insecure — this must still be a real, validating request
self.assertNotIn("-k", argv)
self.assertNotIn("--insecure", argv)
def test_token_passed_via_stdin_with_no_trailing_newline_and_never_in_argv(self):
sysm = FakeCurlSystem()
http_client = pg.HttpClient(self.cfg, sysm)
http_client.get_pinned_curl("Server-11669-10a", "69.33.198.97", 1337, "/addKey", SECRET_TOKEN, PUB_A)
self.assertEqual(sysm.stdins, [SECRET_TOKEN]) # exact match: no appended "\n"
for arg in sysm.calls[0]:
self.assertNotIn(SECRET_TOKEN, arg)
def test_timeout_exit_code_classified(self):
sysm = FakeCurlSystem(pg.CmdResult(28, "", "curl: (28) Connection timed out"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "timeout")
def test_tls_exit_codes_classified_with_stderr_detail(self):
for rc in (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83):
with self.subTest(rc=rc):
sysm = FakeCurlSystem(pg.CmdResult(rc, "", "curl: (60) SSL certificate problem"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "tls")
self.assertIn("SSL certificate problem", cm.exception.detail)
def test_http_failure_exit_code_classified(self):
sysm = FakeCurlSystem(pg.CmdResult(22, "", "curl: (22) The requested URL returned error: 401"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "http")
def test_nonstandard_exit_code_with_fail_text_still_classified_as_http(self):
# Observed live: curl built against SecureTransport reports exit 56
# (CURLE_RECV_ERROR), not the "usual" 22, for the same --fail
# -triggered HTTP rejection — the error text is the portable signal.
sysm = FakeCurlSystem(pg.CmdResult(56, "", "curl: (56) The requested URL returned error: 401"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "http")
def test_curl_missing_classified_as_network_not_a_crash(self):
sysm = FakeCurlSystem(pg.CmdResult(127, "", "curl: not found"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "network")
def test_unclassified_exit_code_falls_back_to_network_with_stderr(self):
sysm = FakeCurlSystem(pg.CmdResult(7, "", "curl: (7) Couldn't connect to server"))
http_client = pg.HttpClient(self.cfg, sysm)
with self.assertRaises(pg.ApiError) as cm:
http_client.get_pinned_curl("h", "1.2.3.4", 1337, "/addKey", "t", "p")
self.assertEqual(cm.exception.kind, "network")
self.assertIn("Couldn't connect", cm.exception.detail)
class TestHttpHelpers(unittest.TestCase):
def test_key_and_ip_validators(self):
self.assertTrue(pg.valid_wg_key(PUB_A))
self.assertFalse(pg.valid_wg_key("short"))
self.assertFalse(pg.valid_wg_key(PUB_A[:-1] + "B")) # bad final char
self.assertTrue(pg.valid_ipv4("10.0.0.1"))
self.assertFalse(pg.valid_ipv4("300.0.0.1"))
self.assertFalse(pg.valid_ipv4("::1"))
def test_candidates_order_and_missing_region(self):
api = pg.PiaApi(pg.Config(), None, pg.Log(stream=io.StringIO()))
cands = api.build_candidates(serverlist_doc(), ["hk", "taiwan"])
self.assertEqual([c.cn for c in cands], ["hk1.example", "hk2.example", "tw1.example"])
with self.assertRaises(pg.ApiError):
api.build_candidates(serverlist_doc(), ["hk", "nowhere"])
with self.assertRaises(pg.ApiError):
api.build_candidates(serverlist_doc(), ["japan"]) # exists but no wg servers -> nothing usable
# an empty fallback region is skipped, not fatal
self.assertEqual([c.cn for c in api.build_candidates(serverlist_doc(), ["japan", "hk"])], ["hk1.example", "hk2.example"])
def test_token_retries_then_auth(self):
http = FakeHttp()
api = pg.PiaApi(pg.Config(), http, pg.Log(stream=io.StringIO()))
http.token = ("http", 401)
with self.assertRaises(pg.ApiError) as cm:
api.get_token(pg.Credentials("u", "p"))
self.assertEqual(cm.exception.kind, "auth")
http.token = ("err", "network")
with self.assertRaises(pg.ApiError):
api.get_token(pg.Credentials("u", "p"))
self.assertEqual(len([c for c in http.calls if c[0] == "post"]), 4)
def test_multipart_body_contains_fields_once(self):
captured = {}
class C(pg.HttpClient):
def _request(self, conn, method, path, body, headers):
captured["body"] = body
captured["headers"] = headers
return 200, b'{"token":"t"}'
c = C(pg.Config())
c.post_multipart(pg.TOKEN_URL, {"username": "u1", "password": "p1"})
self.assertIn(b'name="username"\r\n\r\nu1\r\n', captured["body"])
self.assertIn(b'name="password"\r\n\r\np1\r\n', captured["body"])
self.assertTrue(captured["headers"]["Content-Type"].startswith("multipart/form-data; boundary="))
if __name__ == "__main__":
unittest.main()