fix: tolerate content-verified kube-router jumps ahead of the kill switch

Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:

  19:38:58  3rd unhealthy check -> restart
  19:39:09  recovered - route open, handshake fresh
  19:40:03  drift again -> emergency close -> route forced closed,
            counter reset to 0
  19:41:08  drift again -> emergency close again

VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.

This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.

Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.

Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.

pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Nik Afiq 2026-09-22 19:55:47 +09:00
parent 89d289addf
commit fb9ecffa9f
5 changed files with 363 additions and 14 deletions

View File

@ -205,6 +205,60 @@ real.
gated by `pia_auto_recovery_enabled`) exists specifically to close this
gap by giving the already-correct, already-idempotent script regular
chances to run instead of only sporadic ones.
**The reassert timer alone wasn't the full fix either — it created a
new failure mode.** Confirmed live, 2026-09-22, minutes after enabling
`pia-gateway-recover.timer` for the first time (rollout step 3):
`pia-gateway-recover` and `pia-killswitch-reassert` are two
independently-scheduled 60s timers with no coordination between them.
kube-router turned out to resync (and re-push our jump down) roughly
every 60-90s on this cluster — far more often than "rare, boot-scale"
— so a `recover` pass would periodically sample `iptables -S FORWARD`
in the window between a kube-router push-down and the next reassert
fix, correctly detect real (if momentary) drift, and call
`emergency_close()` — which unconditionally forces table `pia`'s route
to `unreachable` and resets `consecutive_failures` to 0 in the same
step. The result was a self-reinforcing outage loop, observed directly
in the journal:
```
19:38:58 3rd consecutive unhealthy check → recovery_cycle → restart
19:39:09 recovered via restart (10s) — route open, handshake fresh
19:40:03 drift detected again ("FORWARD jump ... not first")
→ emergency close → route forced closed, counter reset to 0
19:41:08 drift again → emergency close again → route stays closed
```
VLAN 50 was open for roughly 10 seconds out of every 60-90 second
cycle. Three consecutive real failures were needed to trigger the
restart that reopens the route, but a fresh drift-and-emergency-close
kept arriving before three could accumulate — the counter reset every
time. This is not a bug in the detection (the drift was real, if
transient) — it's that treating *every* momentary reordering as a full
safety emergency was stricter than the actual invariant required,
given kube-router's own resync cadence on this specific host.
**Fix: `pia_forward_bypass_chains` (`Classifier
._forward_jump_effectively_first` in `files/pia_gateway.py`).** The
safety check now tolerates a configured, named FORWARD chain sitting
ahead of our jump — but only for as long as that chain's *own current
content* re-verifies, live, on every single check, as structurally
unable to match `10.10.50.0/24` (every `-s`/`-d` value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct the checker doesn't explicitly recognize fails
closed, not open). This is not a one-time trust of kube-router — if
its rules ever changed to include something that could actually
intercept VLAN 50 traffic, or an unrecognized chain/rule appeared
ahead of ours instead, it still fails closed exactly as before. Scoped
to `KUBE-ROUTER-FORWARD` only (not `DOCKER-USER`/`DOCKER-FORWARD` —
see `pia_forward_bypass_chains`'s own comment in `defaults/main.yaml`
for why only kube-router earned the exception). An emergency close and
a hard stop remain the outcome for anything not on this narrow,
content-reverified allowlist. Immediate live mitigation while this fix
was being built: `sudo systemctl stop pia-gateway-recover.timer`, then
a manual `sudo systemctl restart wg-quick@pia-wg` to reopen the route
(`pia-killswitch-reassert.timer` was left running throughout — it was
never the problem).
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
top of that closed baseline: default route via `pia-wg` in the `pia`
table, ACCEPT + established/related return inserted into the
@ -547,11 +601,13 @@ external secret broker — not a 24-hour token.
### Rollout status (2026-09-22)
Step 1 (baseline) and three attempts at step 2 have been run live.
Step 1 (baseline) and step 2 (deploy with timers off) are done and
verified live. Step 3 (enable the recovery timer) was attempted, caused
a real outage, and has been fixed but not yet re-verified live.
- **Attempt 1** failed: `pia-gateway register` correctly refused with
`safety invariants failing ... FORWARD jump to kill-switch chain is
not first` — a real, pre-existing drift (see the "FORWARD jump's
- **Step 2, attempt 1** failed: `pia-gateway register` correctly refused
with `safety invariants failing ... FORWARD jump to kill-switch chain
is not first` — a real, pre-existing drift (see the "FORWARD jump's
position is actively re-asserted" note above), not a false alarm from
this session's work, and not the "session dies" handshake issue
either. Root-caused and fixed in `pia-killswitch.sh.j2`.
@ -565,14 +621,41 @@ Step 1 (baseline) and three attempts at step 2 have been run live.
jump failure again — proving the drift recurs on a timescale of
minutes under this cluster's normal CronJob/pod churn, not just at
boot-scale events as originally assumed. Unblocked live with a manual
`sudo systemctl restart pia-killswitch.service`; the durable fix is
`pia-killswitch-reassert.timer` (added, not yet applied or verified
live) — see the "FORWARD jump's position" note's "recurs far more
often" addendum above.
`sudo systemctl restart pia-killswitch.service`; `pia-killswitch
-reassert.timer` added as the durable fix.
- **Attempt 4** succeeded cleanly end to end: kill-switch check passed,
`addKey` accepted (`hk`/`Server-11669-10a`), and a **fresh handshake
was verified** (age 10s) — the first-ever successful shared-path
registration. `pia-gateway status` confirmed `healthy`/`safety_ok:
True`, `iptables -S FORWARD` confirmed the jump first, and
`pia-killswitch-reassert.timer` confirmed cycling correctly (7
successful runs, ~60s apart, in the journal). **Step 2 is done.**
- **Step 3** (`ansible-playbook ... -K -J`, no override — enables
`pia-gateway-recover.timer` for the first time) deployed cleanly
(`Run the shared PIA registration transaction` correctly skipped, a
verified baseline already existed) but caused a **real live outage**
within minutes: `pia-gateway-recover` and `pia-killswitch-reassert`,
two independent 60s timers, raced — kube-router's actual resync
cadence (~60-90s) turned out to be far more frequent than assumed, and
every momentary drift-then-self-correct was treated as a full safety
emergency, forcibly closing the route and resetting the 3-strikes
counter before it could accumulate. VLAN 50 was open for ~10s out of
every 60-90s cycle for several minutes, confirmed via Discord alerts
from `alert-bridge` (working end to end, first real trigger) and the
full `pia-gateway-recover`/`pia-killswitch-reassert` journals.
Mitigated live: `sudo systemctl stop pia-gateway-recover.timer` +
manual `sudo systemctl restart wg-quick@pia-wg`. Root-caused and fixed
— see the "reassert timer alone wasn't the full fix" addendum above
(`pia_forward_bypass_chains`, content-verified live on every check).
**Not yet re-applied or re-verified live** — `pia-gateway-recover
.timer` is currently stopped on minisforum pending that.
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
— the original `head -3` suggestion is what hid the first issue; the
drift was 4 rules in, not at the top.
drift was 4 rules in, not at the top. **Use `head -2`, not `head -1`**,
for any quick re-check — `-S FORWARD`'s first line is always the `-P`
policy line, never a real rule (this cost a round-trip during the
investigation above).
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`,
@ -594,8 +677,20 @@ drift was 4 rules in, not at the top.
should then get past registration. Confirm with `iptables -S FORWARD`
afterwards that the jump is genuinely first, not just that the retry
succeeded.
3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy
pass makes no changes (`journalctl -u pia-gateway-recover -n 5`).
3. Enable: `ansible-playbook ... -K -J` (recovery on; also redeploys
`pia_forward_bypass_chains` into `config.json` if not already applied
— confirm with `sudo pia-gateway --config /etc/pia-gateway/config.json
status` or just `cat /etc/pia-gateway/config.json`). **Watch longer
than one pass** — the live incident here took 2-3 cycles (several
minutes) to manifest, not the first check. `journalctl -u
pia-gateway-recover -f` for at least 5-10 minutes; expect `class=
healthy` throughout, with **zero** `SAFETY DRIFT` lines, even while
`pia-killswitch-reassert` keeps cycling in parallel
(`journalctl -u pia-killswitch-reassert -f` alongside it). If a
`bypass chain KUBE-ROUTER-FORWARD content no longer verified safe`
reason ever appears, treat it as real — it means kube-router's rules
changed to include something that could actually match VLAN 50
traffic, not a false positive to relax further.
4. Drill the silent death. First get the **live WireGuard UDP endpoint**
from the interface itself — **not** port 1337, which is only the
`addKey` registration API port and has nothing to do with the tunnel's

View File

@ -94,6 +94,35 @@ pia_iptables_chain: PIA-VLAN50
# {{ pia_wg_interface }} rather than enumerating the normal interface.
pia_lan_interface: enp1s0
# FORWARD chains the controller may tolerate sitting AHEAD of our own
# kill-switch jump, on the condition that their OWN current content is
# re-verified — live, on every check, never a one-time trust — to be
# structurally unable to match {{ pia_vlan50_subnet }} traffic. Empty (the
# safe default for any other host) means the literal, original "must be
# absolutely first" check. Set to ["KUBE-ROUTER-FORWARD"] here because
# k3s's kube-router network-policy controller and dockerd both also
# insert unconditional jumps near the top of FORWARD, independently of
# this role, and — confirmed live, 2026-09-22 — kube-router resyncs (and
# re-asserts its own top-of-chain position) roughly every 60-90s on this
# cluster's normal CronJob/pod churn, not just at boot/restart scale as
# first assumed. Treating every one of those transient reorderings as a
# full safety-drift emergency (which unconditionally closes table pia's
# route) caused a real live outage: a repeating cycle of recover-succeeds
# -> drift-detected-within-a-minute -> emergency-close -> counter reset
# -> repeat, which kept VLAN 50 open for only seconds at a time. Only
# kube-router is listed — NOT dockerd's DOCKER-USER/DOCKER-FORWARD —
# because kube-router was the only one observed reasserting position
# repeatedly (dockerd sets its hooks up once at daemon start, not on a
# recurring cycle), and DOCKER-FORWARD itself jumps to three further
# nested chains (DOCKER-CT, DOCKER-INTERNAL, DOCKER-BRIDGE) never fully
# content-verified — safer to leave those as a hard drift trigger than
# assume they're equally inert. See
# Classifier._forward_jump_effectively_first's own docstring in
# files/pia_gateway.py for exactly what "re-verified safe" requires of a
# bypassed chain's content.
pia_forward_bypass_chains:
- KUBE-ROUTER-FORWARD
# Reverse-path filtering, applied to {{ pia_wg_interface }} only. The
# kernel default on minisforum (conf.default.rp_filter) is already loose
# (2) at time of writing, so new interfaces inherit loose mode — this task

View File

@ -35,6 +35,7 @@ import datetime as _dt
import fcntl
import http.client
import io
import ipaddress
import json
import os
import random
@ -119,6 +120,7 @@ class Config:
vlan50_subnet: str = "10.10.50.0/24"
iptables_chain: str = "PIA-VLAN50"
lan_interface: str = "enp1s0"
forward_bypass_chains: list = dataclasses.field(default_factory=list)
mss_clamp_enabled: bool = True
region_candidates: list = dataclasses.field(default_factory=lambda: ["hk", "taiwan", "japan"])
handshake_max_age_sec: int = 180
@ -509,6 +511,59 @@ def _is_emergency_rule(rule: str) -> bool:
return f"--comment {EMERGENCY_COMMENT}" in rule or f'--comment "{EMERGENCY_COMMENT}"' in rule
_COMMENT_ONLY_RE = re.compile(r'-m comment --comment "[^"]*"')
def _forward_bypass_target(rule: str) -> Optional[str]:
"""If `rule` is an UNCONDITIONAL FORWARD-level jump to another chain
(optionally with only a cosmetic -m comment match, nothing that
actually narrows which packets it applies to), return that chain's
name; otherwise None. A rule with any real matcher (-s/-d/-p/-i/-o/
any -m other than comment) is deliberately never treated as a bypass
candidate here, however harmless it might look — that judgment is
made once, by a human, for the caller's own forward_bypass_chains
list, not inferred from the FORWARD-level rule shape.
"""
stripped = _COMMENT_ONLY_RE.sub("", rule).split()
if len(stripped) == 4 and stripped[0] == "-A" and stripped[1] == "FORWARD" and stripped[2] == "-j":
return stripped[3]
return None
def _cidr_overlaps(value: str, subnet: str) -> bool:
"""True if `value` could overlap `subnet`; conservative (True) on
anything unparseable rather than silently treating it as disjoint."""
try:
return ipaddress.ip_network(value, strict=False).overlaps(ipaddress.ip_network(subnet))
except ValueError:
return True
def _bypass_chain_rule_is_safe(rule: str, vlan50_subnet: str) -> bool:
"""Conservative check for one rule inside a configured bypass chain:
True only if this rule is PROVABLY unable to match traffic sourced
from vlan50_subnet — every -s/-d value on it is a CIDR disjoint from
vlan50_subnet, there is at least one such value (a bare/unconditional
rule is never safe), and there is no negation or other construct this
function doesn't explicitly recognize (fails closed, not open, on
anything it can't prove).
"""
if "!" in rule.split():
return False
tokens = rule.split()
addrs: list[str] = []
i = 0
while i < len(tokens):
if tokens[i] in ("-s", "-d") and i + 1 < len(tokens):
addrs.append(tokens[i + 1])
i += 2
continue
i += 1
if not addrs:
return False
return all(not _cidr_overlaps(a, vlan50_subnet) for a in addrs)
class Classifier:
def __init__(self, cfg: Config, system: System):
self.cfg = cfg
@ -517,6 +572,34 @@ class Classifier:
def _rc(self, argv: list[str]) -> CmdResult:
return self.sys.run(argv, timeout=20)
def _forward_jump_effectively_first(self, effective_rules: list[str]) -> tuple[bool, list[str]]:
"""Walk FORWARD from the top. Tolerate a configured bypass chain
(cfg.forward_bypass_chains) sitting ahead of our own jump ONLY
while that chain's OWN current content is re-verified, live, on
THIS call to be unable to match VLAN 50 traffic — never a one-time
trust. Anything else ahead of our jump (an unrecognized rule, or a
bypass-listed chain whose content no longer verifies safe) fails
closed. Returns (ok, extra_reasons).
"""
cfg = self.cfg
expected_jump = f"-A FORWARD -s {cfg.vlan50_subnet} -j {cfg.iptables_chain}"
extra: list[str] = []
for r in effective_rules:
if r.strip() == expected_jump:
return True, extra
target = _forward_bypass_target(r)
if target is None or target not in cfg.forward_bypass_chains:
return False, extra
chain = self._rc(["iptables", "-S", target])
if chain.rc != 0:
extra.append(f"bypass chain {target} unreadable")
return False, extra
chain_rules = [l for l in chain.out.splitlines() if l.startswith(f"-A {target}")]
if not chain_rules or not all(_bypass_chain_rule_is_safe(cr, cfg.vlan50_subnet) for cr in chain_rules):
extra.append(f"bypass chain {target} content no longer verified safe")
return False, extra
return False, extra
def classify(self, state: dict) -> Status:
cfg = self.cfg
now = self.sys.now()
@ -636,6 +719,7 @@ class Classifier:
fwd = self._rc(["iptables", "-S", "FORWARD"])
jump_first = False
emergency_present = False
jump_bypass_reasons: list[str] = []
if fwd.rc == 0:
rules = [l for l in fwd.out.splitlines() if l.startswith("-A FORWARD")]
effective = []
@ -644,15 +728,15 @@ class Classifier:
emergency_present = True
continue
effective.append(r)
expected_jump = f"-A FORWARD -s {cfg.vlan50_subnet} -j {cfg.iptables_chain}"
jump_first = bool(effective) and effective[0].strip() == expected_jump
jump_first, jump_bypass_reasons = self._forward_jump_effectively_first(effective)
else:
unknown = True
checks["forward_jump_first"] = jump_first
checks["emergency_rule_present"] = emergency_present
if not jump_first:
safety_ok = False
reasons.append("FORWARD jump to kill-switch chain is not first")
detail = f" ({'; '.join(jump_bypass_reasons)})" if jump_bypass_reasons else ""
reasons.append(f"FORWARD jump to kill-switch chain is not first{detail}")
# -- chain contents
chain = self._rc(["iptables", "-S", cfg.iptables_chain])

View File

@ -10,6 +10,7 @@
"vlan50_subnet": {{ pia_vlan50_subnet | to_json }},
"iptables_chain": {{ pia_iptables_chain | to_json }},
"lan_interface": {{ pia_lan_interface | to_json }},
"forward_bypass_chains": {{ pia_forward_bypass_chains | to_json }},
"mss_clamp_enabled": {{ pia_mss_clamp_enabled | bool | to_json }},
"region_candidates": {{ pia_region_candidates | to_json }},
"handshake_max_age_sec": {{ pia_healthcheck_handshake_max_age_sec | int }},

View File

@ -88,6 +88,7 @@ class FakeSystem(pg.System):
self.stdins: list[str] = []
self.genkey_queue = [PRIV_B]
self.jitter = 0.5
self.extra_chains: dict = {} # name -> list[str] | None ("-A <name> ..." lines, or None = missing)
@staticmethod
def open_chain():
@ -206,6 +207,12 @@ class FakeSystem(pg.System):
if self.chain_rules is None:
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
return pg.CmdResult(0, f"-N {CHAIN}\n" + "\n".join(self.chain_rules) + "\n", "")
if a[2] in self.extra_chains:
rules = self.extra_chains[a[2]]
if rules is None:
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
return pg.CmdResult(0, f"-N {a[2]}\n" + "\n".join(rules) + "\n", "")
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
if a[:2] == ["iptables", "-C"]:
rule = "-A " + " ".join(a[2:])
return pg.CmdResult(0 if rule in self.forward_rules else 1, "", "")
@ -495,6 +502,139 @@ class TestSafetyDrift(Base):
self.assertEqual(self.http.calls, [])
KUBE_ROUTER_FORWARD_JUMP = '-A FORWARD -m comment --comment "kube-router netpol - TEMCG2JMHZYE7H7T" -j KUBE-ROUTER-FORWARD'
KUBE_ROUTER_SAFE_RULES = [
'-A KUBE-ROUTER-FORWARD -d 10.42.0.39/32 -m comment --comment "pod X" -j KUBE-POD-FW-AAAA',
'-A KUBE-ROUTER-FORWARD -s 10.42.0.39/32 -m comment --comment "pod X" -j KUBE-POD-FW-AAAA',
'-A KUBE-ROUTER-FORWARD -d 10.42.1.5/32 -m physdev --physdev-is-bridged -m comment --comment "pod Y" -j KUBE-POD-FW-BBBB',
]
class TestForwardBypassChains(Base):
"""The content-verified bypass mechanism for kube-router's own FORWARD
jump — found necessary live, 2026-09-22, after the plain reposition
fix alone caused a repeating outage loop: kube-router resyncs (and
re-pushes our jump down) roughly every 60-90s on this cluster, far
more often than assumed, and treating every transient reordering as a
full safety-drift emergency (closing the route each time) kept the
tunnel down far more than it was up.
"""
def classify_with_forward(self, forward_rules, bypass_chains=(), extra_chains=None):
self.cfg.forward_bypass_chains = list(bypass_chains)
self.sysm.forward_rules = list(forward_rules)
if extra_chains:
self.sysm.extra_chains.update(extra_chains)
return self.ctl.classifier.classify(pg.default_state())
def test_disabled_by_default_kube_router_ahead_still_blocks(self):
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=(), # default: nothing configured
)
self.assertFalse(st.safety_ok)
self.assertIn("FORWARD jump to kill-switch chain is not first", " ".join(st.reasons))
def test_configured_bypass_with_verified_safe_content_tolerated(self):
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
)
self.assertTrue(st.safety_ok, st.reasons)
def test_duplicate_bypass_jump_tolerated(self):
# The exact live anomaly: kube-router left two copies of its own
# jump behind after a resync.
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
)
self.assertTrue(st.safety_ok, st.reasons)
def test_bypass_chain_picking_up_a_matching_source_still_blocks(self):
dangerous = KUBE_ROUTER_SAFE_RULES + [f"-A KUBE-ROUTER-FORWARD -s {SUBNET} -j ACCEPT"]
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
)
self.assertFalse(st.safety_ok)
self.assertIn("no longer verified safe", " ".join(st.reasons))
def test_bypass_chain_with_bare_unconditional_rule_blocks(self):
dangerous = KUBE_ROUTER_SAFE_RULES + ["-A KUBE-ROUTER-FORWARD -j ACCEPT"]
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
)
self.assertFalse(st.safety_ok)
def test_bypass_chain_with_negated_match_blocks(self):
dangerous = [f'-A KUBE-ROUTER-FORWARD ! -s {SUBNET} -j ACCEPT']
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
)
self.assertFalse(st.safety_ok)
def test_unrecognized_chain_ahead_still_blocks(self):
# DOCKER-USER is not on the allowlist even when KUBE-ROUTER-FORWARD is.
st = self.classify_with_forward(
["-A FORWARD -j DOCKER-USER", f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"DOCKER-USER": []},
)
self.assertFalse(st.safety_ok)
def test_forward_level_rule_with_real_matcher_not_treated_as_bypass(self):
# Even though KUBE-ROUTER-FORWARD is allowlisted and its content is
# safe, a FORWARD-level rule that itself carries a real matcher
# (not just a cosmetic comment) is never treated as a bypass jump.
st = self.classify_with_forward(
[f"-A FORWARD -s 1.2.3.0/24 -j KUBE-ROUTER-FORWARD", f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
)
self.assertFalse(st.safety_ok)
def test_missing_bypass_chain_blocks(self):
st = self.classify_with_forward(
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
bypass_chains=["KUBE-ROUTER-FORWARD"],
extra_chains={"KUBE-ROUTER-FORWARD": None},
)
self.assertFalse(st.safety_ok)
self.assertIn("unreadable", " ".join(st.reasons))
def test_end_to_end_recover_does_not_emergency_close_on_tolerated_drift(self):
# The actual live incident, reproduced end to end: kube-router's
# jump sits ahead of ours, content-verified-safe, and the recovery
# timer must NOT declare safety drift or touch the route.
self.cfg.forward_bypass_chains = ["KUBE-ROUTER-FORWARD"]
self.sysm.forward_rules = [KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"]
self.sysm.extra_chains["KUBE-ROUTER-FORWARD"] = KUBE_ROUTER_SAFE_RULES
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_OK)
self.assertFalse(self.state()["emergency_close_active"])
self.assertEqual(self.sysm.table_default, "open")
self.assertEqual(self.sysm.restart_count, 0)
def test_end_to_end_recover_still_emergency_closes_on_real_drift(self):
# Regression guard: an ACTUALLY unrecognized rule ahead of ours
# must still trigger the full emergency-close path, even with a
# bypass list configured for something else.
self.cfg.forward_bypass_chains = ["KUBE-ROUTER-FORWARD"]
self.sysm.forward_rules = ["-A FORWARD -j DOCKER-USER", f"-A FORWARD -s {SUBNET} -j {CHAIN}"]
self.sysm.extra_chains["DOCKER-USER"] = []
rc = self.ctl.cmd_recover()
self.assertEqual(rc, pg.EXIT_SAFETY)
self.assertTrue(self.state()["emergency_close_active"])
class TestRecoveryStateMachine(Base):
def test_healthy_runs_are_idempotent_and_do_not_mutate(self):
self.run_recover_n(3)