diff --git a/ansible/roles/pia-gateway/README.md b/ansible/roles/pia-gateway/README.md index 060c606..d1ae8de 100644 --- a/ansible/roles/pia-gateway/README.md +++ b/ansible/roles/pia-gateway/README.md @@ -205,6 +205,60 @@ real. gated by `pia_auto_recovery_enabled`) exists specifically to close this gap by giving the already-correct, already-idempotent script regular chances to run instead of only sporadic ones. + + **The reassert timer alone wasn't the full fix either — it created a + new failure mode.** Confirmed live, 2026-09-22, minutes after enabling + `pia-gateway-recover.timer` for the first time (rollout step 3): + `pia-gateway-recover` and `pia-killswitch-reassert` are two + independently-scheduled 60s timers with no coordination between them. + kube-router turned out to resync (and re-push our jump down) roughly + every 60-90s on this cluster — far more often than "rare, boot-scale" + — so a `recover` pass would periodically sample `iptables -S FORWARD` + in the window between a kube-router push-down and the next reassert + fix, correctly detect real (if momentary) drift, and call + `emergency_close()` — which unconditionally forces table `pia`'s route + to `unreachable` and resets `consecutive_failures` to 0 in the same + step. The result was a self-reinforcing outage loop, observed directly + in the journal: + + ``` + 19:38:58 3rd consecutive unhealthy check → recovery_cycle → restart + 19:39:09 recovered via restart (10s) — route open, handshake fresh + 19:40:03 drift detected again ("FORWARD jump ... not first") + → emergency close → route forced closed, counter reset to 0 + 19:41:08 drift again → emergency close again → route stays closed + ``` + + VLAN 50 was open for roughly 10 seconds out of every 60-90 second + cycle. Three consecutive real failures were needed to trigger the + restart that reopens the route, but a fresh drift-and-emergency-close + kept arriving before three could accumulate — the counter reset every + time. This is not a bug in the detection (the drift was real, if + transient) — it's that treating *every* momentary reordering as a full + safety emergency was stricter than the actual invariant required, + given kube-router's own resync cadence on this specific host. + + **Fix: `pia_forward_bypass_chains` (`Classifier + ._forward_jump_effectively_first` in `files/pia_gateway.py`).** The + safety check now tolerates a configured, named FORWARD chain sitting + ahead of our jump — but only for as long as that chain's *own current + content* re-verifies, live, on every single check, as structurally + unable to match `10.10.50.0/24` (every `-s`/`-d` value on every rule + disjoint from our subnet; a bare/unconditional rule, a negated match, + or any construct the checker doesn't explicitly recognize fails + closed, not open). This is not a one-time trust of kube-router — if + its rules ever changed to include something that could actually + intercept VLAN 50 traffic, or an unrecognized chain/rule appeared + ahead of ours instead, it still fails closed exactly as before. Scoped + to `KUBE-ROUTER-FORWARD` only (not `DOCKER-USER`/`DOCKER-FORWARD` — + see `pia_forward_bypass_chains`'s own comment in `defaults/main.yaml` + for why only kube-router earned the exception). An emergency close and + a hard stop remain the outcome for anything not on this narrow, + content-reverified allowlist. Immediate live mitigation while this fix + was being built: `sudo systemctl stop pia-gateway-recover.timer`, then + a manual `sudo systemctl restart wg-quick@pia-wg` to reopen the route + (`pia-killswitch-reassert.timer` was left running throughout — it was + never the problem). - `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on top of that closed baseline: default route via `pia-wg` in the `pia` table, ACCEPT + established/related return inserted into the @@ -547,11 +601,13 @@ external secret broker — not a 24-hour token. ### Rollout status (2026-09-22) -Step 1 (baseline) and three attempts at step 2 have been run live. +Step 1 (baseline) and step 2 (deploy with timers off) are done and +verified live. Step 3 (enable the recovery timer) was attempted, caused +a real outage, and has been fixed but not yet re-verified live. -- **Attempt 1** failed: `pia-gateway register` correctly refused with - `safety invariants failing ... FORWARD jump to kill-switch chain is - not first` — a real, pre-existing drift (see the "FORWARD jump's +- **Step 2, attempt 1** failed: `pia-gateway register` correctly refused + with `safety invariants failing ... FORWARD jump to kill-switch chain + is not first` — a real, pre-existing drift (see the "FORWARD jump's position is actively re-asserted" note above), not a false alarm from this session's work, and not the "session dies" handshake issue either. Root-caused and fixed in `pia-killswitch.sh.j2`. @@ -565,14 +621,41 @@ Step 1 (baseline) and three attempts at step 2 have been run live. jump failure again — proving the drift recurs on a timescale of minutes under this cluster's normal CronJob/pod churn, not just at boot-scale events as originally assumed. Unblocked live with a manual - `sudo systemctl restart pia-killswitch.service`; the durable fix is - `pia-killswitch-reassert.timer` (added, not yet applied or verified - live) — see the "FORWARD jump's position" note's "recurs far more - often" addendum above. + `sudo systemctl restart pia-killswitch.service`; `pia-killswitch + -reassert.timer` added as the durable fix. +- **Attempt 4** succeeded cleanly end to end: kill-switch check passed, + `addKey` accepted (`hk`/`Server-11669-10a`), and a **fresh handshake + was verified** (age 10s) — the first-ever successful shared-path + registration. `pia-gateway status` confirmed `healthy`/`safety_ok: + True`, `iptables -S FORWARD` confirmed the jump first, and + `pia-killswitch-reassert.timer` confirmed cycling correctly (7 + successful runs, ~60s apart, in the journal). **Step 2 is done.** +- **Step 3** (`ansible-playbook ... -K -J`, no override — enables + `pia-gateway-recover.timer` for the first time) deployed cleanly + (`Run the shared PIA registration transaction` correctly skipped, a + verified baseline already existed) but caused a **real live outage** + within minutes: `pia-gateway-recover` and `pia-killswitch-reassert`, + two independent 60s timers, raced — kube-router's actual resync + cadence (~60-90s) turned out to be far more frequent than assumed, and + every momentary drift-then-self-correct was treated as a full safety + emergency, forcibly closing the route and resetting the 3-strikes + counter before it could accumulate. VLAN 50 was open for ~10s out of + every 60-90s cycle for several minutes, confirmed via Discord alerts + from `alert-bridge` (working end to end, first real trigger) and the + full `pia-gateway-recover`/`pia-killswitch-reassert` journals. + Mitigated live: `sudo systemctl stop pia-gateway-recover.timer` + + manual `sudo systemctl restart wg-quick@pia-wg`. Root-caused and fixed + — see the "reassert timer alone wasn't the full fix" addendum above + (`pia_forward_bypass_chains`, content-verified live on every check). + **Not yet re-applied or re-verified live** — `pia-gateway-recover + .timer` is currently stopped on minisforum pending that. **Use `iptables -S FORWARD` (no `head`) for the baseline capture below** — the original `head -3` suggestion is what hid the first issue; the -drift was 4 rules in, not at the top. +drift was 4 rules in, not at the top. **Use `head -2`, not `head -1`**, +for any quick re-check — `-S FORWARD`'s first line is always the `-P` +policy line, never a real rule (this cost a round-trip during the +investigation above). 1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`, @@ -594,8 +677,20 @@ drift was 4 rules in, not at the top. should then get past registration. Confirm with `iptables -S FORWARD` afterwards that the jump is genuinely first, not just that the retry succeeded. -3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy - pass makes no changes (`journalctl -u pia-gateway-recover -n 5`). +3. Enable: `ansible-playbook ... -K -J` (recovery on; also redeploys + `pia_forward_bypass_chains` into `config.json` if not already applied + — confirm with `sudo pia-gateway --config /etc/pia-gateway/config.json + status` or just `cat /etc/pia-gateway/config.json`). **Watch longer + than one pass** — the live incident here took 2-3 cycles (several + minutes) to manifest, not the first check. `journalctl -u + pia-gateway-recover -f` for at least 5-10 minutes; expect `class= + healthy` throughout, with **zero** `SAFETY DRIFT` lines, even while + `pia-killswitch-reassert` keeps cycling in parallel + (`journalctl -u pia-killswitch-reassert -f` alongside it). If a + `bypass chain KUBE-ROUTER-FORWARD content no longer verified safe` + reason ever appears, treat it as real — it means kube-router's rules + changed to include something that could actually match VLAN 50 + traffic, not a false positive to relax further. 4. Drill the silent death. First get the **live WireGuard UDP endpoint** from the interface itself — **not** port 1337, which is only the `addKey` registration API port and has nothing to do with the tunnel's diff --git a/ansible/roles/pia-gateway/defaults/main.yaml b/ansible/roles/pia-gateway/defaults/main.yaml index 9902d19..8f30871 100644 --- a/ansible/roles/pia-gateway/defaults/main.yaml +++ b/ansible/roles/pia-gateway/defaults/main.yaml @@ -94,6 +94,35 @@ pia_iptables_chain: PIA-VLAN50 # {{ pia_wg_interface }} rather than enumerating the normal interface. pia_lan_interface: enp1s0 +# FORWARD chains the controller may tolerate sitting AHEAD of our own +# kill-switch jump, on the condition that their OWN current content is +# re-verified — live, on every check, never a one-time trust — to be +# structurally unable to match {{ pia_vlan50_subnet }} traffic. Empty (the +# safe default for any other host) means the literal, original "must be +# absolutely first" check. Set to ["KUBE-ROUTER-FORWARD"] here because +# k3s's kube-router network-policy controller and dockerd both also +# insert unconditional jumps near the top of FORWARD, independently of +# this role, and — confirmed live, 2026-09-22 — kube-router resyncs (and +# re-asserts its own top-of-chain position) roughly every 60-90s on this +# cluster's normal CronJob/pod churn, not just at boot/restart scale as +# first assumed. Treating every one of those transient reorderings as a +# full safety-drift emergency (which unconditionally closes table pia's +# route) caused a real live outage: a repeating cycle of recover-succeeds +# -> drift-detected-within-a-minute -> emergency-close -> counter reset +# -> repeat, which kept VLAN 50 open for only seconds at a time. Only +# kube-router is listed — NOT dockerd's DOCKER-USER/DOCKER-FORWARD — +# because kube-router was the only one observed reasserting position +# repeatedly (dockerd sets its hooks up once at daemon start, not on a +# recurring cycle), and DOCKER-FORWARD itself jumps to three further +# nested chains (DOCKER-CT, DOCKER-INTERNAL, DOCKER-BRIDGE) never fully +# content-verified — safer to leave those as a hard drift trigger than +# assume they're equally inert. See +# Classifier._forward_jump_effectively_first's own docstring in +# files/pia_gateway.py for exactly what "re-verified safe" requires of a +# bypassed chain's content. +pia_forward_bypass_chains: + - KUBE-ROUTER-FORWARD + # Reverse-path filtering, applied to {{ pia_wg_interface }} only. The # kernel default on minisforum (conf.default.rp_filter) is already loose # (2) at time of writing, so new interfaces inherit loose mode — this task diff --git a/ansible/roles/pia-gateway/files/pia_gateway.py b/ansible/roles/pia-gateway/files/pia_gateway.py index edb7f56..f546712 100644 --- a/ansible/roles/pia-gateway/files/pia_gateway.py +++ b/ansible/roles/pia-gateway/files/pia_gateway.py @@ -35,6 +35,7 @@ import datetime as _dt import fcntl import http.client import io +import ipaddress import json import os import random @@ -119,6 +120,7 @@ class Config: vlan50_subnet: str = "10.10.50.0/24" iptables_chain: str = "PIA-VLAN50" lan_interface: str = "enp1s0" + forward_bypass_chains: list = dataclasses.field(default_factory=list) mss_clamp_enabled: bool = True region_candidates: list = dataclasses.field(default_factory=lambda: ["hk", "taiwan", "japan"]) handshake_max_age_sec: int = 180 @@ -509,6 +511,59 @@ def _is_emergency_rule(rule: str) -> bool: return f"--comment {EMERGENCY_COMMENT}" in rule or f'--comment "{EMERGENCY_COMMENT}"' in rule +_COMMENT_ONLY_RE = re.compile(r'-m comment --comment "[^"]*"') + + +def _forward_bypass_target(rule: str) -> Optional[str]: + """If `rule` is an UNCONDITIONAL FORWARD-level jump to another chain + (optionally with only a cosmetic -m comment match, nothing that + actually narrows which packets it applies to), return that chain's + name; otherwise None. A rule with any real matcher (-s/-d/-p/-i/-o/ + any -m other than comment) is deliberately never treated as a bypass + candidate here, however harmless it might look — that judgment is + made once, by a human, for the caller's own forward_bypass_chains + list, not inferred from the FORWARD-level rule shape. + """ + stripped = _COMMENT_ONLY_RE.sub("", rule).split() + if len(stripped) == 4 and stripped[0] == "-A" and stripped[1] == "FORWARD" and stripped[2] == "-j": + return stripped[3] + return None + + +def _cidr_overlaps(value: str, subnet: str) -> bool: + """True if `value` could overlap `subnet`; conservative (True) on + anything unparseable rather than silently treating it as disjoint.""" + try: + return ipaddress.ip_network(value, strict=False).overlaps(ipaddress.ip_network(subnet)) + except ValueError: + return True + + +def _bypass_chain_rule_is_safe(rule: str, vlan50_subnet: str) -> bool: + """Conservative check for one rule inside a configured bypass chain: + True only if this rule is PROVABLY unable to match traffic sourced + from vlan50_subnet — every -s/-d value on it is a CIDR disjoint from + vlan50_subnet, there is at least one such value (a bare/unconditional + rule is never safe), and there is no negation or other construct this + function doesn't explicitly recognize (fails closed, not open, on + anything it can't prove). + """ + if "!" in rule.split(): + return False + tokens = rule.split() + addrs: list[str] = [] + i = 0 + while i < len(tokens): + if tokens[i] in ("-s", "-d") and i + 1 < len(tokens): + addrs.append(tokens[i + 1]) + i += 2 + continue + i += 1 + if not addrs: + return False + return all(not _cidr_overlaps(a, vlan50_subnet) for a in addrs) + + class Classifier: def __init__(self, cfg: Config, system: System): self.cfg = cfg @@ -517,6 +572,34 @@ class Classifier: def _rc(self, argv: list[str]) -> CmdResult: return self.sys.run(argv, timeout=20) + def _forward_jump_effectively_first(self, effective_rules: list[str]) -> tuple[bool, list[str]]: + """Walk FORWARD from the top. Tolerate a configured bypass chain + (cfg.forward_bypass_chains) sitting ahead of our own jump ONLY + while that chain's OWN current content is re-verified, live, on + THIS call to be unable to match VLAN 50 traffic — never a one-time + trust. Anything else ahead of our jump (an unrecognized rule, or a + bypass-listed chain whose content no longer verifies safe) fails + closed. Returns (ok, extra_reasons). + """ + cfg = self.cfg + expected_jump = f"-A FORWARD -s {cfg.vlan50_subnet} -j {cfg.iptables_chain}" + extra: list[str] = [] + for r in effective_rules: + if r.strip() == expected_jump: + return True, extra + target = _forward_bypass_target(r) + if target is None or target not in cfg.forward_bypass_chains: + return False, extra + chain = self._rc(["iptables", "-S", target]) + if chain.rc != 0: + extra.append(f"bypass chain {target} unreadable") + return False, extra + chain_rules = [l for l in chain.out.splitlines() if l.startswith(f"-A {target}")] + if not chain_rules or not all(_bypass_chain_rule_is_safe(cr, cfg.vlan50_subnet) for cr in chain_rules): + extra.append(f"bypass chain {target} content no longer verified safe") + return False, extra + return False, extra + def classify(self, state: dict) -> Status: cfg = self.cfg now = self.sys.now() @@ -636,6 +719,7 @@ class Classifier: fwd = self._rc(["iptables", "-S", "FORWARD"]) jump_first = False emergency_present = False + jump_bypass_reasons: list[str] = [] if fwd.rc == 0: rules = [l for l in fwd.out.splitlines() if l.startswith("-A FORWARD")] effective = [] @@ -644,15 +728,15 @@ class Classifier: emergency_present = True continue effective.append(r) - expected_jump = f"-A FORWARD -s {cfg.vlan50_subnet} -j {cfg.iptables_chain}" - jump_first = bool(effective) and effective[0].strip() == expected_jump + jump_first, jump_bypass_reasons = self._forward_jump_effectively_first(effective) else: unknown = True checks["forward_jump_first"] = jump_first checks["emergency_rule_present"] = emergency_present if not jump_first: safety_ok = False - reasons.append("FORWARD jump to kill-switch chain is not first") + detail = f" ({'; '.join(jump_bypass_reasons)})" if jump_bypass_reasons else "" + reasons.append(f"FORWARD jump to kill-switch chain is not first{detail}") # -- chain contents chain = self._rc(["iptables", "-S", cfg.iptables_chain]) diff --git a/ansible/roles/pia-gateway/templates/pia-gateway-config.json.j2 b/ansible/roles/pia-gateway/templates/pia-gateway-config.json.j2 index 7e769b4..3ba9b5d 100644 --- a/ansible/roles/pia-gateway/templates/pia-gateway-config.json.j2 +++ b/ansible/roles/pia-gateway/templates/pia-gateway-config.json.j2 @@ -10,6 +10,7 @@ "vlan50_subnet": {{ pia_vlan50_subnet | to_json }}, "iptables_chain": {{ pia_iptables_chain | to_json }}, "lan_interface": {{ pia_lan_interface | to_json }}, + "forward_bypass_chains": {{ pia_forward_bypass_chains | to_json }}, "mss_clamp_enabled": {{ pia_mss_clamp_enabled | bool | to_json }}, "region_candidates": {{ pia_region_candidates | to_json }}, "handshake_max_age_sec": {{ pia_healthcheck_handshake_max_age_sec | int }}, diff --git a/ansible/roles/pia-gateway/tests/test_pia_gateway.py b/ansible/roles/pia-gateway/tests/test_pia_gateway.py index 616ac20..2dc7e30 100644 --- a/ansible/roles/pia-gateway/tests/test_pia_gateway.py +++ b/ansible/roles/pia-gateway/tests/test_pia_gateway.py @@ -88,6 +88,7 @@ class FakeSystem(pg.System): self.stdins: list[str] = [] self.genkey_queue = [PRIV_B] self.jitter = 0.5 + self.extra_chains: dict = {} # name -> list[str] | None ("-A ..." lines, or None = missing) @staticmethod def open_chain(): @@ -206,6 +207,12 @@ class FakeSystem(pg.System): if self.chain_rules is None: return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.") return pg.CmdResult(0, f"-N {CHAIN}\n" + "\n".join(self.chain_rules) + "\n", "") + if a[2] in self.extra_chains: + rules = self.extra_chains[a[2]] + if rules is None: + return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.") + return pg.CmdResult(0, f"-N {a[2]}\n" + "\n".join(rules) + "\n", "") + return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.") if a[:2] == ["iptables", "-C"]: rule = "-A " + " ".join(a[2:]) return pg.CmdResult(0 if rule in self.forward_rules else 1, "", "") @@ -495,6 +502,139 @@ class TestSafetyDrift(Base): self.assertEqual(self.http.calls, []) +KUBE_ROUTER_FORWARD_JUMP = '-A FORWARD -m comment --comment "kube-router netpol - TEMCG2JMHZYE7H7T" -j KUBE-ROUTER-FORWARD' +KUBE_ROUTER_SAFE_RULES = [ + '-A KUBE-ROUTER-FORWARD -d 10.42.0.39/32 -m comment --comment "pod X" -j KUBE-POD-FW-AAAA', + '-A KUBE-ROUTER-FORWARD -s 10.42.0.39/32 -m comment --comment "pod X" -j KUBE-POD-FW-AAAA', + '-A KUBE-ROUTER-FORWARD -d 10.42.1.5/32 -m physdev --physdev-is-bridged -m comment --comment "pod Y" -j KUBE-POD-FW-BBBB', +] + + +class TestForwardBypassChains(Base): + """The content-verified bypass mechanism for kube-router's own FORWARD + jump — found necessary live, 2026-09-22, after the plain reposition + fix alone caused a repeating outage loop: kube-router resyncs (and + re-pushes our jump down) roughly every 60-90s on this cluster, far + more often than assumed, and treating every transient reordering as a + full safety-drift emergency (closing the route each time) kept the + tunnel down far more than it was up. + """ + + def classify_with_forward(self, forward_rules, bypass_chains=(), extra_chains=None): + self.cfg.forward_bypass_chains = list(bypass_chains) + self.sysm.forward_rules = list(forward_rules) + if extra_chains: + self.sysm.extra_chains.update(extra_chains) + return self.ctl.classifier.classify(pg.default_state()) + + def test_disabled_by_default_kube_router_ahead_still_blocks(self): + st = self.classify_with_forward( + [KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"], + bypass_chains=(), # default: nothing configured + ) + self.assertFalse(st.safety_ok) + self.assertIn("FORWARD jump to kill-switch chain is not first", " ".join(st.reasons)) + + def test_configured_bypass_with_verified_safe_content_tolerated(self): + st = self.classify_with_forward( + [KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"], + bypass_chains=["KUBE-ROUTER-FORWARD"], + extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES}, + ) + self.assertTrue(st.safety_ok, st.reasons) + + def test_duplicate_bypass_jump_tolerated(self): + # The exact live anomaly: kube-router left two copies of its own + # jump behind after a resync. + st = self.classify_with_forward( + [KUBE_ROUTER_FORWARD_JUMP, KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"], + bypass_chains=["KUBE-ROUTER-FORWARD"], + extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES}, + ) + self.assertTrue(st.safety_ok, st.reasons) + + def test_bypass_chain_picking_up_a_matching_source_still_blocks(self): + dangerous = KUBE_ROUTER_SAFE_RULES + [f"-A KUBE-ROUTER-FORWARD -s {SUBNET} -j ACCEPT"] + st = self.classify_with_forward( + [KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"], + bypass_chains=["KUBE-ROUTER-FORWARD"], + extra_chains={"KUBE-ROUTER-FORWARD": dangerous}, + ) + self.assertFalse(st.safety_ok) + self.assertIn("no longer verified safe", " ".join(st.reasons)) + + def test_bypass_chain_with_bare_unconditional_rule_blocks(self): + dangerous = KUBE_ROUTER_SAFE_RULES + ["-A KUBE-ROUTER-FORWARD -j ACCEPT"] + st = self.classify_with_forward( + [KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"], + bypass_chains=["KUBE-ROUTER-FORWARD"], + extra_chains={"KUBE-ROUTER-FORWARD": dangerous}, + ) + self.assertFalse(st.safety_ok) + + def test_bypass_chain_with_negated_match_blocks(self): + dangerous = [f'-A KUBE-ROUTER-FORWARD ! -s {SUBNET} -j ACCEPT'] + st = self.classify_with_forward( + [KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"], + bypass_chains=["KUBE-ROUTER-FORWARD"], + extra_chains={"KUBE-ROUTER-FORWARD": dangerous}, + ) + self.assertFalse(st.safety_ok) + + def test_unrecognized_chain_ahead_still_blocks(self): + # DOCKER-USER is not on the allowlist even when KUBE-ROUTER-FORWARD is. + st = self.classify_with_forward( + ["-A FORWARD -j DOCKER-USER", f"-A FORWARD -s {SUBNET} -j {CHAIN}"], + bypass_chains=["KUBE-ROUTER-FORWARD"], + extra_chains={"DOCKER-USER": []}, + ) + self.assertFalse(st.safety_ok) + + def test_forward_level_rule_with_real_matcher_not_treated_as_bypass(self): + # Even though KUBE-ROUTER-FORWARD is allowlisted and its content is + # safe, a FORWARD-level rule that itself carries a real matcher + # (not just a cosmetic comment) is never treated as a bypass jump. + st = self.classify_with_forward( + [f"-A FORWARD -s 1.2.3.0/24 -j KUBE-ROUTER-FORWARD", f"-A FORWARD -s {SUBNET} -j {CHAIN}"], + bypass_chains=["KUBE-ROUTER-FORWARD"], + extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES}, + ) + self.assertFalse(st.safety_ok) + + def test_missing_bypass_chain_blocks(self): + st = self.classify_with_forward( + [KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"], + bypass_chains=["KUBE-ROUTER-FORWARD"], + extra_chains={"KUBE-ROUTER-FORWARD": None}, + ) + self.assertFalse(st.safety_ok) + self.assertIn("unreadable", " ".join(st.reasons)) + + def test_end_to_end_recover_does_not_emergency_close_on_tolerated_drift(self): + # The actual live incident, reproduced end to end: kube-router's + # jump sits ahead of ours, content-verified-safe, and the recovery + # timer must NOT declare safety drift or touch the route. + self.cfg.forward_bypass_chains = ["KUBE-ROUTER-FORWARD"] + self.sysm.forward_rules = [KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"] + self.sysm.extra_chains["KUBE-ROUTER-FORWARD"] = KUBE_ROUTER_SAFE_RULES + rc = self.ctl.cmd_recover() + self.assertEqual(rc, pg.EXIT_OK) + self.assertFalse(self.state()["emergency_close_active"]) + self.assertEqual(self.sysm.table_default, "open") + self.assertEqual(self.sysm.restart_count, 0) + + def test_end_to_end_recover_still_emergency_closes_on_real_drift(self): + # Regression guard: an ACTUALLY unrecognized rule ahead of ours + # must still trigger the full emergency-close path, even with a + # bypass list configured for something else. + self.cfg.forward_bypass_chains = ["KUBE-ROUTER-FORWARD"] + self.sysm.forward_rules = ["-A FORWARD -j DOCKER-USER", f"-A FORWARD -s {SUBNET} -j {CHAIN}"] + self.sysm.extra_chains["DOCKER-USER"] = [] + rc = self.ctl.cmd_recover() + self.assertEqual(rc, pg.EXIT_SAFETY) + self.assertTrue(self.state()["emergency_close_active"]) + + class TestRecoveryStateMachine(Base): def test_healthy_runs_are_idempotent_and_do_not_mutate(self): self.run_recover_n(3)