From de5ce7bf5276dd83fa659626e12a70e6ceac6e72 Mon Sep 17 00:00:00 2001 From: Nik Afiq Date: Tue, 22 Sep 2026 16:44:28 +0900 Subject: [PATCH] fix: reposition pia-killswitch's FORWARD jump, not just check it exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found live on the first deployment of the auto-recovery controller (2026-09-22): pia-gateway register correctly refused with "FORWARD jump to kill-switch chain is not first". Root cause confirmed on minisforum via iptables -S FORWARD/-S KUBE-ROUTER-FORWARD/-S DOCKER-USER/-S DOCKER-FORWARD: the PIA-VLAN50 jump was 4th, behind kube-router's and dockerd's own jumps, which each independently reassert position 1 on their own resyncs/restarts (kube-router had even left a duplicate of its own rule behind). pia-killswitch.sh.j2 only checked whether its jump existed (iptables -C) before deciding not to touch anything — existence isn't proof of position, so once pushed down it stayed down. Confirmed safe to reposition unconditionally by reading the actual content of what's ahead of it rather than assuming: KUBE-ROUTER-FORWARD only matches specific pod /32 addresses, DOCKER-USER was empty, and DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50 arrives via enp1s0) - none can intercept VLAN 50 traffic regardless of order. Fixed the script itself (delete + reinsert at position 1 if the current first rule isn't ours) rather than loosening the Python controller's forward_jump_first safety check, which was working correctly when it refused to register against the drifted state. Verified offline: rendered template passes bash -n, and the exact live topology (rule present but 4th) reproduced against a bash mock of iptables, confirming reposition-when-drifted, idempotency when already first, and insert-when-absent all behave correctly. No real iptables/wg/ip on this dev machine to test further; not yet re-applied or re-verified against minisforum. Co-Authored-By: Claude Sonnet 5 --- ansible/roles/pia-gateway/README.md | 50 +++++++++++++++++-- .../templates/pia-killswitch.sh.j2 | 37 ++++++++++++-- 2 files changed, 80 insertions(+), 7 deletions(-) diff --git a/ansible/roles/pia-gateway/README.md b/ansible/roles/pia-gateway/README.md index 436492e..754625d 100644 --- a/ansible/roles/pia-gateway/README.md +++ b/ansible/roles/pia-gateway/README.md @@ -163,6 +163,32 @@ real. Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW **input** allow for 53/tcp+udp — it's locally-terminated traffic, not routed — see `tasks/firewall.yaml`. + + **The FORWARD jump's position is actively re-asserted on every run, not + just seeded once** — found live, 2026-09-22, on the very first + deployment of the auto-recovery controller. minisforum's `FORWARD` + chain default policy is `DROP` (kube-router's doing, not this role's — + correcting an assumption an earlier version of this doc made), and + kube-router and `dockerd` each independently reassert their **own** + jump at position 1 whenever they resync or restart, silently pushing + this rule down without removing it (kube-router doesn't even dedupe + its own older copy when it does — a duplicate of *its* rule was found + live too). The script used to check only whether its rule *existed* + (`iptables -C`) before deciding not to touch anything; existence isn't + proof of position, so once pushed down it stayed down until something + else fixed it. It now checks the actual first rule and repositions + (delete + reinsert at 1) if it isn't this one. Confirmed safe to do + unconditionally by reading kube-router's and Docker's actual chain + contents rather than assuming: `KUBE-ROUTER-FORWARD` only matches + specific pod `/32` addresses (disjoint from `10.10.50.0/24`), + `DOCKER-USER` was empty, and `DOCKER-FORWARD`'s one direct rule is + scoped to `-i docker0` (VLAN 50 traffic arrives via `enp1s0`) — none + of them can intercept VLAN 50 traffic regardless of order, so moving + this rule back to position 1 relative to them changes nothing for pod + or Docker-bridge traffic. The controller's own safety check + (`forward_jump_first`) was working as intended when it refused to + register against the drifted state — this is a fix to the thing it + checks, not a loosening of the check itself. - `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on top of that closed baseline: default route via `pia-wg` in the `pia` table, ACCEPT + established/related return inserted into the @@ -440,10 +466,21 @@ external secret broker — not a 24-hour token. - `unknown` (a read command failing) is logged/alerted but never acted on, by design. -### Rollout (nothing below has been run live yet) +### Rollout status (2026-09-22) + +Step 1 (baseline) and the first attempt at step 2 have been run live. +Step 2 failed on the first try: `pia-gateway register` correctly refused +with `safety invariants failing ... FORWARD jump to kill-switch chain is +not first` — a real, pre-existing drift (see the "FORWARD jump's +position is actively re-asserted" note above), not a false alarm from +this session's work, and not the "session dies" handshake issue either. +Root-caused and fixed in `pia-killswitch.sh.j2`; not yet re-applied or +re-verified live. **Use `iptables -S FORWARD` (no `head`) for the +baseline capture below** — the original `head -3` suggestion is what +hid this the first time; the drift was 4 rules in, not at the top. 1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show - table pia`, `iptables -S FORWARD | head -3`, `iptables -S PIA-VLAN50`, + table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`, `journalctl -u pia-gateway-healthcheck -n 5`, `curl --socks5-hostname 10.10.40.20:1080 https://ifconfig.me` from the LAN, `wg show wg0`, `kubectl get nodes`. Confirm console access. @@ -454,7 +491,14 @@ external secret broker — not a 24-hour token. verified baseline yet, this run **registers through the shared path** (same-key, explicit restart + handshake verification) — watch the task output; if it fails, `-e pia_registration_backend=legacy` - restores the previous behaviour. + restores the previous behaviour. If it fails specifically with + `FORWARD jump to kill-switch chain is not first`, that's the drift + above — the fixed template should correct it via the normal "Restart + pia-killswitch" handler on the same run (the template content + changing is enough to trigger it); re-running the identical command + should then get past registration. Confirm with `iptables -S FORWARD` + afterwards that the jump is genuinely first, not just that the retry + succeeded. 3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy pass makes no changes (`journalctl -u pia-gateway-recover -n 5`). 4. Drill the silent death. First get the **live WireGuard UDP endpoint** diff --git a/ansible/roles/pia-gateway/templates/pia-killswitch.sh.j2 b/ansible/roles/pia-gateway/templates/pia-killswitch.sh.j2 index ce60e1b..a63d079 100644 --- a/ansible/roles/pia-gateway/templates/pia-killswitch.sh.j2 +++ b/ansible/roles/pia-gateway/templates/pia-killswitch.sh.j2 @@ -34,11 +34,16 @@ # source's fate with a broader ACCEPT — the jump is unconditional for # this source and the chain itself ends in an unconditional DROP, not # a RETURN, so nothing after it in FORWARD is ever consulted for -# this source either. +# this source either. "First" is actively re-asserted on every run, +# not just seeded once — see the jump-insertion block below for why +# that turned out to matter in practice. # # Idempotent: safe to run more than once (e.g. `systemctl restart # pia-killswitch`, or after this script re-runs following a UFW reload) -# without creating duplicate rules or duplicate chains. +# without creating duplicate rules or duplicate chains — including +# repositioning the FORWARD jump back to position 1 if something else +# (kube-router, dockerd) has since pushed it down, rather than treating +# its mere existence as proof it's still first. set -euo pipefail TABLE="{{ pia_route_table_name }}" @@ -68,8 +73,32 @@ fi # there is no direct "does this chain exist" check short of grepping -L). iptables -N "${CHAIN}" 2>/dev/null || true -# Exactly one jump from FORWARD into it, at absolute position 1. -if ! iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then +# Exactly one jump from FORWARD into it, kept at absolute position 1. +# Existence alone ("-C") is not proof of position — confirmed live, +# 2026-09-22: kube-router and dockerd each independently reassert their +# OWN jump at position 1 whenever THEY resync/restart (kube-router +# doesn't even dedupe its own older copy when it does, so duplicates of +# ITS rule can appear too), which silently pushes this one down without +# removing it. A plain "-C, insert only if absent" check then reports +# "already exists" and leaves it stranded wherever it drifted to — this +# is exactly what happened here: the jump was 4th (behind kube-router, +# DOCKER-USER, DOCKER-FORWARD), not 1st, on a host that had been up 200 +# days with kube-router/Docker managing FORWARD independently the whole +# time. So check the actual first rule in FORWARD, not just whether ours +# exists somewhere; if it isn't ours, remove it (by spec — iptables -D +# matches wherever it currently sits, not by position) and reinsert at +# 1. Safe to do unconditionally on every run: this rule is narrowly +# {{ pia_vlan50_subnet }}-scoped, so moving it earlier relative to +# kube-router's/Docker's own broader jumps can't affect pod or Docker +# bridge traffic either way — confirmed by reading their actual chain +# contents (KUBE-ROUTER-FORWARD only matches specific pod /32 +# addresses, DOCKER-USER was empty, DOCKER-FORWARD's one direct rule is +# scoped to -i docker0), not assumed. +FORWARD_FIRST=$(iptables -S FORWARD | grep '^-A FORWARD' | head -1 || true) +if [[ "${FORWARD_FIRST}" != "-A FORWARD -s ${SUBNET} -j ${CHAIN}" ]]; then + if iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then + iptables -D FORWARD -s "${SUBNET}" -j "${CHAIN}" + fi iptables -I FORWARD 1 -s "${SUBNET}" -j "${CHAIN}" fi