fix: reposition pia-killswitch's FORWARD jump, not just check it exists

Found live on the first deployment of the auto-recovery controller
(2026-09-22): pia-gateway register correctly refused with "FORWARD
jump to kill-switch chain is not first". Root cause confirmed on
minisforum via iptables -S FORWARD/-S KUBE-ROUTER-FORWARD/-S
DOCKER-USER/-S DOCKER-FORWARD: the PIA-VLAN50 jump was 4th, behind
kube-router's and dockerd's own jumps, which each independently
reassert position 1 on their own resyncs/restarts (kube-router had
even left a duplicate of its own rule behind). pia-killswitch.sh.j2
only checked whether its jump existed (iptables -C) before deciding
not to touch anything — existence isn't proof of position, so once
pushed down it stayed down.

Confirmed safe to reposition unconditionally by reading the actual
content of what's ahead of it rather than assuming: KUBE-ROUTER-FORWARD
only matches specific pod /32 addresses, DOCKER-USER was empty, and
DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50
arrives via enp1s0) - none can intercept VLAN 50 traffic regardless of
order. Fixed the script itself (delete + reinsert at position 1 if the
current first rule isn't ours) rather than loosening the Python
controller's forward_jump_first safety check, which was working
correctly when it refused to register against the drifted state.

Verified offline: rendered template passes bash -n, and the exact
live topology (rule present but 4th) reproduced against a bash mock
of iptables, confirming reposition-when-drifted, idempotency when
already first, and insert-when-absent all behave correctly. No real
iptables/wg/ip on this dev machine to test further; not yet
re-applied or re-verified against minisforum.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Nik Afiq 2026-09-22 16:44:28 +09:00
parent 87a9303cc3
commit de5ce7bf52
2 changed files with 80 additions and 7 deletions

View File

@ -163,6 +163,32 @@ real.
Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW
**input** allow for 53/tcp+udp — it's locally-terminated traffic, not
routed — see `tasks/firewall.yaml`.
**The FORWARD jump's position is actively re-asserted on every run, not
just seeded once** — found live, 2026-09-22, on the very first
deployment of the auto-recovery controller. minisforum's `FORWARD`
chain default policy is `DROP` (kube-router's doing, not this role's —
correcting an assumption an earlier version of this doc made), and
kube-router and `dockerd` each independently reassert their **own**
jump at position 1 whenever they resync or restart, silently pushing
this rule down without removing it (kube-router doesn't even dedupe
its own older copy when it does — a duplicate of *its* rule was found
live too). The script used to check only whether its rule *existed*
(`iptables -C`) before deciding not to touch anything; existence isn't
proof of position, so once pushed down it stayed down until something
else fixed it. It now checks the actual first rule and repositions
(delete + reinsert at 1) if it isn't this one. Confirmed safe to do
unconditionally by reading kube-router's and Docker's actual chain
contents rather than assuming: `KUBE-ROUTER-FORWARD` only matches
specific pod `/32` addresses (disjoint from `10.10.50.0/24`),
`DOCKER-USER` was empty, and `DOCKER-FORWARD`'s one direct rule is
scoped to `-i docker0` (VLAN 50 traffic arrives via `enp1s0`) — none
of them can intercept VLAN 50 traffic regardless of order, so moving
this rule back to position 1 relative to them changes nothing for pod
or Docker-bridge traffic. The controller's own safety check
(`forward_jump_first`) was working as intended when it refused to
register against the drifted state — this is a fix to the thing it
checks, not a loosening of the check itself.
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
top of that closed baseline: default route via `pia-wg` in the `pia`
table, ACCEPT + established/related return inserted into the
@ -440,10 +466,21 @@ external secret broker — not a 24-hour token.
- `unknown` (a read command failing) is logged/alerted but never acted
on, by design.
### Rollout (nothing below has been run live yet)
### Rollout status (2026-09-22)
Step 1 (baseline) and the first attempt at step 2 have been run live.
Step 2 failed on the first try: `pia-gateway register` correctly refused
with `safety invariants failing ... FORWARD jump to kill-switch chain is
not first` — a real, pre-existing drift (see the "FORWARD jump's
position is actively re-asserted" note above), not a false alarm from
this session's work, and not the "session dies" handshake issue either.
Root-caused and fixed in `pia-killswitch.sh.j2`; not yet re-applied or
re-verified live. **Use `iptables -S FORWARD` (no `head`) for the
baseline capture below** — the original `head -3` suggestion is what
hid this the first time; the drift was 4 rules in, not at the top.
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
table pia`, `iptables -S FORWARD | head -3`, `iptables -S PIA-VLAN50`,
table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`,
`journalctl -u pia-gateway-healthcheck -n 5`, `curl --socks5-hostname
10.10.40.20:1080 https://ifconfig.me` from the LAN, `wg show wg0`,
`kubectl get nodes`. Confirm console access.
@ -454,7 +491,14 @@ external secret broker — not a 24-hour token.
verified baseline yet, this run **registers through the shared path**
(same-key, explicit restart + handshake verification) — watch the
task output; if it fails, `-e pia_registration_backend=legacy`
restores the previous behaviour.
restores the previous behaviour. If it fails specifically with
`FORWARD jump to kill-switch chain is not first`, that's the drift
above — the fixed template should correct it via the normal "Restart
pia-killswitch" handler on the same run (the template content
changing is enough to trigger it); re-running the identical command
should then get past registration. Confirm with `iptables -S FORWARD`
afterwards that the jump is genuinely first, not just that the retry
succeeded.
3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy
pass makes no changes (`journalctl -u pia-gateway-recover -n 5`).
4. Drill the silent death. First get the **live WireGuard UDP endpoint**

View File

@ -34,11 +34,16 @@
# source's fate with a broader ACCEPT — the jump is unconditional for
# this source and the chain itself ends in an unconditional DROP, not
# a RETURN, so nothing after it in FORWARD is ever consulted for
# this source either.
# this source either. "First" is actively re-asserted on every run,
# not just seeded once — see the jump-insertion block below for why
# that turned out to matter in practice.
#
# Idempotent: safe to run more than once (e.g. `systemctl restart
# pia-killswitch`, or after this script re-runs following a UFW reload)
# without creating duplicate rules or duplicate chains.
# without creating duplicate rules or duplicate chains — including
# repositioning the FORWARD jump back to position 1 if something else
# (kube-router, dockerd) has since pushed it down, rather than treating
# its mere existence as proof it's still first.
set -euo pipefail
TABLE="{{ pia_route_table_name }}"
@ -68,8 +73,32 @@ fi
# there is no direct "does this chain exist" check short of grepping -L).
iptables -N "${CHAIN}" 2>/dev/null || true
# Exactly one jump from FORWARD into it, at absolute position 1.
if ! iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
# Exactly one jump from FORWARD into it, kept at absolute position 1.
# Existence alone ("-C") is not proof of position — confirmed live,
# 2026-09-22: kube-router and dockerd each independently reassert their
# OWN jump at position 1 whenever THEY resync/restart (kube-router
# doesn't even dedupe its own older copy when it does, so duplicates of
# ITS rule can appear too), which silently pushes this one down without
# removing it. A plain "-C, insert only if absent" check then reports
# "already exists" and leaves it stranded wherever it drifted to — this
# is exactly what happened here: the jump was 4th (behind kube-router,
# DOCKER-USER, DOCKER-FORWARD), not 1st, on a host that had been up 200
# days with kube-router/Docker managing FORWARD independently the whole
# time. So check the actual first rule in FORWARD, not just whether ours
# exists somewhere; if it isn't ours, remove it (by spec — iptables -D
# matches wherever it currently sits, not by position) and reinsert at
# 1. Safe to do unconditionally on every run: this rule is narrowly
# {{ pia_vlan50_subnet }}-scoped, so moving it earlier relative to
# kube-router's/Docker's own broader jumps can't affect pod or Docker
# bridge traffic either way — confirmed by reading their actual chain
# contents (KUBE-ROUTER-FORWARD only matches specific pod /32
# addresses, DOCKER-USER was empty, DOCKER-FORWARD's one direct rule is
# scoped to -i docker0), not assumed.
FORWARD_FIRST=$(iptables -S FORWARD | grep '^-A FORWARD' | head -1 || true)
if [[ "${FORWARD_FIRST}" != "-A FORWARD -s ${SUBNET} -j ${CHAIN}" ]]; then
if iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
iptables -D FORWARD -s "${SUBNET}" -j "${CHAIN}"
fi
iptables -I FORWARD 1 -s "${SUBNET}" -j "${CHAIN}"
fi