fix: tolerate content-verified kube-router jumps ahead of the kill switch
Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:
19:38:58 3rd unhealthy check -> restart
19:39:09 recovered - route open, handshake fresh
19:40:03 drift again -> emergency close -> route forced closed,
counter reset to 0
19:41:08 drift again -> emergency close again
VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.
This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.
Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.
Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.
pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
89d289addf
commit
fb9ecffa9f
@ -205,6 +205,60 @@ real.
|
||||
gated by `pia_auto_recovery_enabled`) exists specifically to close this
|
||||
gap by giving the already-correct, already-idempotent script regular
|
||||
chances to run instead of only sporadic ones.
|
||||
|
||||
**The reassert timer alone wasn't the full fix either — it created a
|
||||
new failure mode.** Confirmed live, 2026-09-22, minutes after enabling
|
||||
`pia-gateway-recover.timer` for the first time (rollout step 3):
|
||||
`pia-gateway-recover` and `pia-killswitch-reassert` are two
|
||||
independently-scheduled 60s timers with no coordination between them.
|
||||
kube-router turned out to resync (and re-push our jump down) roughly
|
||||
every 60-90s on this cluster — far more often than "rare, boot-scale"
|
||||
— so a `recover` pass would periodically sample `iptables -S FORWARD`
|
||||
in the window between a kube-router push-down and the next reassert
|
||||
fix, correctly detect real (if momentary) drift, and call
|
||||
`emergency_close()` — which unconditionally forces table `pia`'s route
|
||||
to `unreachable` and resets `consecutive_failures` to 0 in the same
|
||||
step. The result was a self-reinforcing outage loop, observed directly
|
||||
in the journal:
|
||||
|
||||
```
|
||||
19:38:58 3rd consecutive unhealthy check → recovery_cycle → restart
|
||||
19:39:09 recovered via restart (10s) — route open, handshake fresh
|
||||
19:40:03 drift detected again ("FORWARD jump ... not first")
|
||||
→ emergency close → route forced closed, counter reset to 0
|
||||
19:41:08 drift again → emergency close again → route stays closed
|
||||
```
|
||||
|
||||
VLAN 50 was open for roughly 10 seconds out of every 60-90 second
|
||||
cycle. Three consecutive real failures were needed to trigger the
|
||||
restart that reopens the route, but a fresh drift-and-emergency-close
|
||||
kept arriving before three could accumulate — the counter reset every
|
||||
time. This is not a bug in the detection (the drift was real, if
|
||||
transient) — it's that treating *every* momentary reordering as a full
|
||||
safety emergency was stricter than the actual invariant required,
|
||||
given kube-router's own resync cadence on this specific host.
|
||||
|
||||
**Fix: `pia_forward_bypass_chains` (`Classifier
|
||||
._forward_jump_effectively_first` in `files/pia_gateway.py`).** The
|
||||
safety check now tolerates a configured, named FORWARD chain sitting
|
||||
ahead of our jump — but only for as long as that chain's *own current
|
||||
content* re-verifies, live, on every single check, as structurally
|
||||
unable to match `10.10.50.0/24` (every `-s`/`-d` value on every rule
|
||||
disjoint from our subnet; a bare/unconditional rule, a negated match,
|
||||
or any construct the checker doesn't explicitly recognize fails
|
||||
closed, not open). This is not a one-time trust of kube-router — if
|
||||
its rules ever changed to include something that could actually
|
||||
intercept VLAN 50 traffic, or an unrecognized chain/rule appeared
|
||||
ahead of ours instead, it still fails closed exactly as before. Scoped
|
||||
to `KUBE-ROUTER-FORWARD` only (not `DOCKER-USER`/`DOCKER-FORWARD` —
|
||||
see `pia_forward_bypass_chains`'s own comment in `defaults/main.yaml`
|
||||
for why only kube-router earned the exception). An emergency close and
|
||||
a hard stop remain the outcome for anything not on this narrow,
|
||||
content-reverified allowlist. Immediate live mitigation while this fix
|
||||
was being built: `sudo systemctl stop pia-gateway-recover.timer`, then
|
||||
a manual `sudo systemctl restart wg-quick@pia-wg` to reopen the route
|
||||
(`pia-killswitch-reassert.timer` was left running throughout — it was
|
||||
never the problem).
|
||||
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
|
||||
top of that closed baseline: default route via `pia-wg` in the `pia`
|
||||
table, ACCEPT + established/related return inserted into the
|
||||
@ -547,11 +601,13 @@ external secret broker — not a 24-hour token.
|
||||
|
||||
### Rollout status (2026-09-22)
|
||||
|
||||
Step 1 (baseline) and three attempts at step 2 have been run live.
|
||||
Step 1 (baseline) and step 2 (deploy with timers off) are done and
|
||||
verified live. Step 3 (enable the recovery timer) was attempted, caused
|
||||
a real outage, and has been fixed but not yet re-verified live.
|
||||
|
||||
- **Attempt 1** failed: `pia-gateway register` correctly refused with
|
||||
`safety invariants failing ... FORWARD jump to kill-switch chain is
|
||||
not first` — a real, pre-existing drift (see the "FORWARD jump's
|
||||
- **Step 2, attempt 1** failed: `pia-gateway register` correctly refused
|
||||
with `safety invariants failing ... FORWARD jump to kill-switch chain
|
||||
is not first` — a real, pre-existing drift (see the "FORWARD jump's
|
||||
position is actively re-asserted" note above), not a false alarm from
|
||||
this session's work, and not the "session dies" handshake issue
|
||||
either. Root-caused and fixed in `pia-killswitch.sh.j2`.
|
||||
@ -565,14 +621,41 @@ Step 1 (baseline) and three attempts at step 2 have been run live.
|
||||
jump failure again — proving the drift recurs on a timescale of
|
||||
minutes under this cluster's normal CronJob/pod churn, not just at
|
||||
boot-scale events as originally assumed. Unblocked live with a manual
|
||||
`sudo systemctl restart pia-killswitch.service`; the durable fix is
|
||||
`pia-killswitch-reassert.timer` (added, not yet applied or verified
|
||||
live) — see the "FORWARD jump's position" note's "recurs far more
|
||||
often" addendum above.
|
||||
`sudo systemctl restart pia-killswitch.service`; `pia-killswitch
|
||||
-reassert.timer` added as the durable fix.
|
||||
- **Attempt 4** succeeded cleanly end to end: kill-switch check passed,
|
||||
`addKey` accepted (`hk`/`Server-11669-10a`), and a **fresh handshake
|
||||
was verified** (age 10s) — the first-ever successful shared-path
|
||||
registration. `pia-gateway status` confirmed `healthy`/`safety_ok:
|
||||
True`, `iptables -S FORWARD` confirmed the jump first, and
|
||||
`pia-killswitch-reassert.timer` confirmed cycling correctly (7
|
||||
successful runs, ~60s apart, in the journal). **Step 2 is done.**
|
||||
- **Step 3** (`ansible-playbook ... -K -J`, no override — enables
|
||||
`pia-gateway-recover.timer` for the first time) deployed cleanly
|
||||
(`Run the shared PIA registration transaction` correctly skipped, a
|
||||
verified baseline already existed) but caused a **real live outage**
|
||||
within minutes: `pia-gateway-recover` and `pia-killswitch-reassert`,
|
||||
two independent 60s timers, raced — kube-router's actual resync
|
||||
cadence (~60-90s) turned out to be far more frequent than assumed, and
|
||||
every momentary drift-then-self-correct was treated as a full safety
|
||||
emergency, forcibly closing the route and resetting the 3-strikes
|
||||
counter before it could accumulate. VLAN 50 was open for ~10s out of
|
||||
every 60-90s cycle for several minutes, confirmed via Discord alerts
|
||||
from `alert-bridge` (working end to end, first real trigger) and the
|
||||
full `pia-gateway-recover`/`pia-killswitch-reassert` journals.
|
||||
Mitigated live: `sudo systemctl stop pia-gateway-recover.timer` +
|
||||
manual `sudo systemctl restart wg-quick@pia-wg`. Root-caused and fixed
|
||||
— see the "reassert timer alone wasn't the full fix" addendum above
|
||||
(`pia_forward_bypass_chains`, content-verified live on every check).
|
||||
**Not yet re-applied or re-verified live** — `pia-gateway-recover
|
||||
.timer` is currently stopped on minisforum pending that.
|
||||
|
||||
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
|
||||
— the original `head -3` suggestion is what hid the first issue; the
|
||||
drift was 4 rules in, not at the top.
|
||||
drift was 4 rules in, not at the top. **Use `head -2`, not `head -1`**,
|
||||
for any quick re-check — `-S FORWARD`'s first line is always the `-P`
|
||||
policy line, never a real rule (this cost a round-trip during the
|
||||
investigation above).
|
||||
|
||||
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
|
||||
table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`,
|
||||
@ -594,8 +677,20 @@ drift was 4 rules in, not at the top.
|
||||
should then get past registration. Confirm with `iptables -S FORWARD`
|
||||
afterwards that the jump is genuinely first, not just that the retry
|
||||
succeeded.
|
||||
3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy
|
||||
pass makes no changes (`journalctl -u pia-gateway-recover -n 5`).
|
||||
3. Enable: `ansible-playbook ... -K -J` (recovery on; also redeploys
|
||||
`pia_forward_bypass_chains` into `config.json` if not already applied
|
||||
— confirm with `sudo pia-gateway --config /etc/pia-gateway/config.json
|
||||
status` or just `cat /etc/pia-gateway/config.json`). **Watch longer
|
||||
than one pass** — the live incident here took 2-3 cycles (several
|
||||
minutes) to manifest, not the first check. `journalctl -u
|
||||
pia-gateway-recover -f` for at least 5-10 minutes; expect `class=
|
||||
healthy` throughout, with **zero** `SAFETY DRIFT` lines, even while
|
||||
`pia-killswitch-reassert` keeps cycling in parallel
|
||||
(`journalctl -u pia-killswitch-reassert -f` alongside it). If a
|
||||
`bypass chain KUBE-ROUTER-FORWARD content no longer verified safe`
|
||||
reason ever appears, treat it as real — it means kube-router's rules
|
||||
changed to include something that could actually match VLAN 50
|
||||
traffic, not a false positive to relax further.
|
||||
4. Drill the silent death. First get the **live WireGuard UDP endpoint**
|
||||
from the interface itself — **not** port 1337, which is only the
|
||||
`addKey` registration API port and has nothing to do with the tunnel's
|
||||
|
||||
@ -94,6 +94,35 @@ pia_iptables_chain: PIA-VLAN50
|
||||
# {{ pia_wg_interface }} rather than enumerating the normal interface.
|
||||
pia_lan_interface: enp1s0
|
||||
|
||||
# FORWARD chains the controller may tolerate sitting AHEAD of our own
|
||||
# kill-switch jump, on the condition that their OWN current content is
|
||||
# re-verified — live, on every check, never a one-time trust — to be
|
||||
# structurally unable to match {{ pia_vlan50_subnet }} traffic. Empty (the
|
||||
# safe default for any other host) means the literal, original "must be
|
||||
# absolutely first" check. Set to ["KUBE-ROUTER-FORWARD"] here because
|
||||
# k3s's kube-router network-policy controller and dockerd both also
|
||||
# insert unconditional jumps near the top of FORWARD, independently of
|
||||
# this role, and — confirmed live, 2026-09-22 — kube-router resyncs (and
|
||||
# re-asserts its own top-of-chain position) roughly every 60-90s on this
|
||||
# cluster's normal CronJob/pod churn, not just at boot/restart scale as
|
||||
# first assumed. Treating every one of those transient reorderings as a
|
||||
# full safety-drift emergency (which unconditionally closes table pia's
|
||||
# route) caused a real live outage: a repeating cycle of recover-succeeds
|
||||
# -> drift-detected-within-a-minute -> emergency-close -> counter reset
|
||||
# -> repeat, which kept VLAN 50 open for only seconds at a time. Only
|
||||
# kube-router is listed — NOT dockerd's DOCKER-USER/DOCKER-FORWARD —
|
||||
# because kube-router was the only one observed reasserting position
|
||||
# repeatedly (dockerd sets its hooks up once at daemon start, not on a
|
||||
# recurring cycle), and DOCKER-FORWARD itself jumps to three further
|
||||
# nested chains (DOCKER-CT, DOCKER-INTERNAL, DOCKER-BRIDGE) never fully
|
||||
# content-verified — safer to leave those as a hard drift trigger than
|
||||
# assume they're equally inert. See
|
||||
# Classifier._forward_jump_effectively_first's own docstring in
|
||||
# files/pia_gateway.py for exactly what "re-verified safe" requires of a
|
||||
# bypassed chain's content.
|
||||
pia_forward_bypass_chains:
|
||||
- KUBE-ROUTER-FORWARD
|
||||
|
||||
# Reverse-path filtering, applied to {{ pia_wg_interface }} only. The
|
||||
# kernel default on minisforum (conf.default.rp_filter) is already loose
|
||||
# (2) at time of writing, so new interfaces inherit loose mode — this task
|
||||
|
||||
@ -35,6 +35,7 @@ import datetime as _dt
|
||||
import fcntl
|
||||
import http.client
|
||||
import io
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
@ -119,6 +120,7 @@ class Config:
|
||||
vlan50_subnet: str = "10.10.50.0/24"
|
||||
iptables_chain: str = "PIA-VLAN50"
|
||||
lan_interface: str = "enp1s0"
|
||||
forward_bypass_chains: list = dataclasses.field(default_factory=list)
|
||||
mss_clamp_enabled: bool = True
|
||||
region_candidates: list = dataclasses.field(default_factory=lambda: ["hk", "taiwan", "japan"])
|
||||
handshake_max_age_sec: int = 180
|
||||
@ -509,6 +511,59 @@ def _is_emergency_rule(rule: str) -> bool:
|
||||
return f"--comment {EMERGENCY_COMMENT}" in rule or f'--comment "{EMERGENCY_COMMENT}"' in rule
|
||||
|
||||
|
||||
_COMMENT_ONLY_RE = re.compile(r'-m comment --comment "[^"]*"')
|
||||
|
||||
|
||||
def _forward_bypass_target(rule: str) -> Optional[str]:
|
||||
"""If `rule` is an UNCONDITIONAL FORWARD-level jump to another chain
|
||||
(optionally with only a cosmetic -m comment match, nothing that
|
||||
actually narrows which packets it applies to), return that chain's
|
||||
name; otherwise None. A rule with any real matcher (-s/-d/-p/-i/-o/
|
||||
any -m other than comment) is deliberately never treated as a bypass
|
||||
candidate here, however harmless it might look — that judgment is
|
||||
made once, by a human, for the caller's own forward_bypass_chains
|
||||
list, not inferred from the FORWARD-level rule shape.
|
||||
"""
|
||||
stripped = _COMMENT_ONLY_RE.sub("", rule).split()
|
||||
if len(stripped) == 4 and stripped[0] == "-A" and stripped[1] == "FORWARD" and stripped[2] == "-j":
|
||||
return stripped[3]
|
||||
return None
|
||||
|
||||
|
||||
def _cidr_overlaps(value: str, subnet: str) -> bool:
|
||||
"""True if `value` could overlap `subnet`; conservative (True) on
|
||||
anything unparseable rather than silently treating it as disjoint."""
|
||||
try:
|
||||
return ipaddress.ip_network(value, strict=False).overlaps(ipaddress.ip_network(subnet))
|
||||
except ValueError:
|
||||
return True
|
||||
|
||||
|
||||
def _bypass_chain_rule_is_safe(rule: str, vlan50_subnet: str) -> bool:
|
||||
"""Conservative check for one rule inside a configured bypass chain:
|
||||
True only if this rule is PROVABLY unable to match traffic sourced
|
||||
from vlan50_subnet — every -s/-d value on it is a CIDR disjoint from
|
||||
vlan50_subnet, there is at least one such value (a bare/unconditional
|
||||
rule is never safe), and there is no negation or other construct this
|
||||
function doesn't explicitly recognize (fails closed, not open, on
|
||||
anything it can't prove).
|
||||
"""
|
||||
if "!" in rule.split():
|
||||
return False
|
||||
tokens = rule.split()
|
||||
addrs: list[str] = []
|
||||
i = 0
|
||||
while i < len(tokens):
|
||||
if tokens[i] in ("-s", "-d") and i + 1 < len(tokens):
|
||||
addrs.append(tokens[i + 1])
|
||||
i += 2
|
||||
continue
|
||||
i += 1
|
||||
if not addrs:
|
||||
return False
|
||||
return all(not _cidr_overlaps(a, vlan50_subnet) for a in addrs)
|
||||
|
||||
|
||||
class Classifier:
|
||||
def __init__(self, cfg: Config, system: System):
|
||||
self.cfg = cfg
|
||||
@ -517,6 +572,34 @@ class Classifier:
|
||||
def _rc(self, argv: list[str]) -> CmdResult:
|
||||
return self.sys.run(argv, timeout=20)
|
||||
|
||||
def _forward_jump_effectively_first(self, effective_rules: list[str]) -> tuple[bool, list[str]]:
|
||||
"""Walk FORWARD from the top. Tolerate a configured bypass chain
|
||||
(cfg.forward_bypass_chains) sitting ahead of our own jump ONLY
|
||||
while that chain's OWN current content is re-verified, live, on
|
||||
THIS call to be unable to match VLAN 50 traffic — never a one-time
|
||||
trust. Anything else ahead of our jump (an unrecognized rule, or a
|
||||
bypass-listed chain whose content no longer verifies safe) fails
|
||||
closed. Returns (ok, extra_reasons).
|
||||
"""
|
||||
cfg = self.cfg
|
||||
expected_jump = f"-A FORWARD -s {cfg.vlan50_subnet} -j {cfg.iptables_chain}"
|
||||
extra: list[str] = []
|
||||
for r in effective_rules:
|
||||
if r.strip() == expected_jump:
|
||||
return True, extra
|
||||
target = _forward_bypass_target(r)
|
||||
if target is None or target not in cfg.forward_bypass_chains:
|
||||
return False, extra
|
||||
chain = self._rc(["iptables", "-S", target])
|
||||
if chain.rc != 0:
|
||||
extra.append(f"bypass chain {target} unreadable")
|
||||
return False, extra
|
||||
chain_rules = [l for l in chain.out.splitlines() if l.startswith(f"-A {target}")]
|
||||
if not chain_rules or not all(_bypass_chain_rule_is_safe(cr, cfg.vlan50_subnet) for cr in chain_rules):
|
||||
extra.append(f"bypass chain {target} content no longer verified safe")
|
||||
return False, extra
|
||||
return False, extra
|
||||
|
||||
def classify(self, state: dict) -> Status:
|
||||
cfg = self.cfg
|
||||
now = self.sys.now()
|
||||
@ -636,6 +719,7 @@ class Classifier:
|
||||
fwd = self._rc(["iptables", "-S", "FORWARD"])
|
||||
jump_first = False
|
||||
emergency_present = False
|
||||
jump_bypass_reasons: list[str] = []
|
||||
if fwd.rc == 0:
|
||||
rules = [l for l in fwd.out.splitlines() if l.startswith("-A FORWARD")]
|
||||
effective = []
|
||||
@ -644,15 +728,15 @@ class Classifier:
|
||||
emergency_present = True
|
||||
continue
|
||||
effective.append(r)
|
||||
expected_jump = f"-A FORWARD -s {cfg.vlan50_subnet} -j {cfg.iptables_chain}"
|
||||
jump_first = bool(effective) and effective[0].strip() == expected_jump
|
||||
jump_first, jump_bypass_reasons = self._forward_jump_effectively_first(effective)
|
||||
else:
|
||||
unknown = True
|
||||
checks["forward_jump_first"] = jump_first
|
||||
checks["emergency_rule_present"] = emergency_present
|
||||
if not jump_first:
|
||||
safety_ok = False
|
||||
reasons.append("FORWARD jump to kill-switch chain is not first")
|
||||
detail = f" ({'; '.join(jump_bypass_reasons)})" if jump_bypass_reasons else ""
|
||||
reasons.append(f"FORWARD jump to kill-switch chain is not first{detail}")
|
||||
|
||||
# -- chain contents
|
||||
chain = self._rc(["iptables", "-S", cfg.iptables_chain])
|
||||
|
||||
@ -10,6 +10,7 @@
|
||||
"vlan50_subnet": {{ pia_vlan50_subnet | to_json }},
|
||||
"iptables_chain": {{ pia_iptables_chain | to_json }},
|
||||
"lan_interface": {{ pia_lan_interface | to_json }},
|
||||
"forward_bypass_chains": {{ pia_forward_bypass_chains | to_json }},
|
||||
"mss_clamp_enabled": {{ pia_mss_clamp_enabled | bool | to_json }},
|
||||
"region_candidates": {{ pia_region_candidates | to_json }},
|
||||
"handshake_max_age_sec": {{ pia_healthcheck_handshake_max_age_sec | int }},
|
||||
|
||||
@ -88,6 +88,7 @@ class FakeSystem(pg.System):
|
||||
self.stdins: list[str] = []
|
||||
self.genkey_queue = [PRIV_B]
|
||||
self.jitter = 0.5
|
||||
self.extra_chains: dict = {} # name -> list[str] | None ("-A <name> ..." lines, or None = missing)
|
||||
|
||||
@staticmethod
|
||||
def open_chain():
|
||||
@ -206,6 +207,12 @@ class FakeSystem(pg.System):
|
||||
if self.chain_rules is None:
|
||||
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
|
||||
return pg.CmdResult(0, f"-N {CHAIN}\n" + "\n".join(self.chain_rules) + "\n", "")
|
||||
if a[2] in self.extra_chains:
|
||||
rules = self.extra_chains[a[2]]
|
||||
if rules is None:
|
||||
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
|
||||
return pg.CmdResult(0, f"-N {a[2]}\n" + "\n".join(rules) + "\n", "")
|
||||
return pg.CmdResult(1, "", "iptables: No chain/target/match by that name.")
|
||||
if a[:2] == ["iptables", "-C"]:
|
||||
rule = "-A " + " ".join(a[2:])
|
||||
return pg.CmdResult(0 if rule in self.forward_rules else 1, "", "")
|
||||
@ -495,6 +502,139 @@ class TestSafetyDrift(Base):
|
||||
self.assertEqual(self.http.calls, [])
|
||||
|
||||
|
||||
KUBE_ROUTER_FORWARD_JUMP = '-A FORWARD -m comment --comment "kube-router netpol - TEMCG2JMHZYE7H7T" -j KUBE-ROUTER-FORWARD'
|
||||
KUBE_ROUTER_SAFE_RULES = [
|
||||
'-A KUBE-ROUTER-FORWARD -d 10.42.0.39/32 -m comment --comment "pod X" -j KUBE-POD-FW-AAAA',
|
||||
'-A KUBE-ROUTER-FORWARD -s 10.42.0.39/32 -m comment --comment "pod X" -j KUBE-POD-FW-AAAA',
|
||||
'-A KUBE-ROUTER-FORWARD -d 10.42.1.5/32 -m physdev --physdev-is-bridged -m comment --comment "pod Y" -j KUBE-POD-FW-BBBB',
|
||||
]
|
||||
|
||||
|
||||
class TestForwardBypassChains(Base):
|
||||
"""The content-verified bypass mechanism for kube-router's own FORWARD
|
||||
jump — found necessary live, 2026-09-22, after the plain reposition
|
||||
fix alone caused a repeating outage loop: kube-router resyncs (and
|
||||
re-pushes our jump down) roughly every 60-90s on this cluster, far
|
||||
more often than assumed, and treating every transient reordering as a
|
||||
full safety-drift emergency (closing the route each time) kept the
|
||||
tunnel down far more than it was up.
|
||||
"""
|
||||
|
||||
def classify_with_forward(self, forward_rules, bypass_chains=(), extra_chains=None):
|
||||
self.cfg.forward_bypass_chains = list(bypass_chains)
|
||||
self.sysm.forward_rules = list(forward_rules)
|
||||
if extra_chains:
|
||||
self.sysm.extra_chains.update(extra_chains)
|
||||
return self.ctl.classifier.classify(pg.default_state())
|
||||
|
||||
def test_disabled_by_default_kube_router_ahead_still_blocks(self):
|
||||
st = self.classify_with_forward(
|
||||
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
|
||||
bypass_chains=(), # default: nothing configured
|
||||
)
|
||||
self.assertFalse(st.safety_ok)
|
||||
self.assertIn("FORWARD jump to kill-switch chain is not first", " ".join(st.reasons))
|
||||
|
||||
def test_configured_bypass_with_verified_safe_content_tolerated(self):
|
||||
st = self.classify_with_forward(
|
||||
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
|
||||
bypass_chains=["KUBE-ROUTER-FORWARD"],
|
||||
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
|
||||
)
|
||||
self.assertTrue(st.safety_ok, st.reasons)
|
||||
|
||||
def test_duplicate_bypass_jump_tolerated(self):
|
||||
# The exact live anomaly: kube-router left two copies of its own
|
||||
# jump behind after a resync.
|
||||
st = self.classify_with_forward(
|
||||
[KUBE_ROUTER_FORWARD_JUMP, KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
|
||||
bypass_chains=["KUBE-ROUTER-FORWARD"],
|
||||
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
|
||||
)
|
||||
self.assertTrue(st.safety_ok, st.reasons)
|
||||
|
||||
def test_bypass_chain_picking_up_a_matching_source_still_blocks(self):
|
||||
dangerous = KUBE_ROUTER_SAFE_RULES + [f"-A KUBE-ROUTER-FORWARD -s {SUBNET} -j ACCEPT"]
|
||||
st = self.classify_with_forward(
|
||||
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
|
||||
bypass_chains=["KUBE-ROUTER-FORWARD"],
|
||||
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
|
||||
)
|
||||
self.assertFalse(st.safety_ok)
|
||||
self.assertIn("no longer verified safe", " ".join(st.reasons))
|
||||
|
||||
def test_bypass_chain_with_bare_unconditional_rule_blocks(self):
|
||||
dangerous = KUBE_ROUTER_SAFE_RULES + ["-A KUBE-ROUTER-FORWARD -j ACCEPT"]
|
||||
st = self.classify_with_forward(
|
||||
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
|
||||
bypass_chains=["KUBE-ROUTER-FORWARD"],
|
||||
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
|
||||
)
|
||||
self.assertFalse(st.safety_ok)
|
||||
|
||||
def test_bypass_chain_with_negated_match_blocks(self):
|
||||
dangerous = [f'-A KUBE-ROUTER-FORWARD ! -s {SUBNET} -j ACCEPT']
|
||||
st = self.classify_with_forward(
|
||||
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
|
||||
bypass_chains=["KUBE-ROUTER-FORWARD"],
|
||||
extra_chains={"KUBE-ROUTER-FORWARD": dangerous},
|
||||
)
|
||||
self.assertFalse(st.safety_ok)
|
||||
|
||||
def test_unrecognized_chain_ahead_still_blocks(self):
|
||||
# DOCKER-USER is not on the allowlist even when KUBE-ROUTER-FORWARD is.
|
||||
st = self.classify_with_forward(
|
||||
["-A FORWARD -j DOCKER-USER", f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
|
||||
bypass_chains=["KUBE-ROUTER-FORWARD"],
|
||||
extra_chains={"DOCKER-USER": []},
|
||||
)
|
||||
self.assertFalse(st.safety_ok)
|
||||
|
||||
def test_forward_level_rule_with_real_matcher_not_treated_as_bypass(self):
|
||||
# Even though KUBE-ROUTER-FORWARD is allowlisted and its content is
|
||||
# safe, a FORWARD-level rule that itself carries a real matcher
|
||||
# (not just a cosmetic comment) is never treated as a bypass jump.
|
||||
st = self.classify_with_forward(
|
||||
[f"-A FORWARD -s 1.2.3.0/24 -j KUBE-ROUTER-FORWARD", f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
|
||||
bypass_chains=["KUBE-ROUTER-FORWARD"],
|
||||
extra_chains={"KUBE-ROUTER-FORWARD": KUBE_ROUTER_SAFE_RULES},
|
||||
)
|
||||
self.assertFalse(st.safety_ok)
|
||||
|
||||
def test_missing_bypass_chain_blocks(self):
|
||||
st = self.classify_with_forward(
|
||||
[KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"],
|
||||
bypass_chains=["KUBE-ROUTER-FORWARD"],
|
||||
extra_chains={"KUBE-ROUTER-FORWARD": None},
|
||||
)
|
||||
self.assertFalse(st.safety_ok)
|
||||
self.assertIn("unreadable", " ".join(st.reasons))
|
||||
|
||||
def test_end_to_end_recover_does_not_emergency_close_on_tolerated_drift(self):
|
||||
# The actual live incident, reproduced end to end: kube-router's
|
||||
# jump sits ahead of ours, content-verified-safe, and the recovery
|
||||
# timer must NOT declare safety drift or touch the route.
|
||||
self.cfg.forward_bypass_chains = ["KUBE-ROUTER-FORWARD"]
|
||||
self.sysm.forward_rules = [KUBE_ROUTER_FORWARD_JUMP, f"-A FORWARD -s {SUBNET} -j {CHAIN}"]
|
||||
self.sysm.extra_chains["KUBE-ROUTER-FORWARD"] = KUBE_ROUTER_SAFE_RULES
|
||||
rc = self.ctl.cmd_recover()
|
||||
self.assertEqual(rc, pg.EXIT_OK)
|
||||
self.assertFalse(self.state()["emergency_close_active"])
|
||||
self.assertEqual(self.sysm.table_default, "open")
|
||||
self.assertEqual(self.sysm.restart_count, 0)
|
||||
|
||||
def test_end_to_end_recover_still_emergency_closes_on_real_drift(self):
|
||||
# Regression guard: an ACTUALLY unrecognized rule ahead of ours
|
||||
# must still trigger the full emergency-close path, even with a
|
||||
# bypass list configured for something else.
|
||||
self.cfg.forward_bypass_chains = ["KUBE-ROUTER-FORWARD"]
|
||||
self.sysm.forward_rules = ["-A FORWARD -j DOCKER-USER", f"-A FORWARD -s {SUBNET} -j {CHAIN}"]
|
||||
self.sysm.extra_chains["DOCKER-USER"] = []
|
||||
rc = self.ctl.cmd_recover()
|
||||
self.assertEqual(rc, pg.EXIT_SAFETY)
|
||||
self.assertTrue(self.state()["emergency_close_active"])
|
||||
|
||||
|
||||
class TestRecoveryStateMachine(Base):
|
||||
def test_healthy_runs_are_idempotent_and_do_not_mutate(self):
|
||||
self.run_recover_n(3)
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user