Found live, 2026-09-22: after fixing pia-killswitch.sh's FORWARD-jump repositioning and getting one pia-gateway register attempt past the safety check, the very next attempt - roughly an hour later, no reboot, no UFW reload, no script content change in between - hit the identical "FORWARD jump to kill-switch chain is not first" failure again. This disproves the earlier assumption that kube-router/dockerd reassert their own position-1 jumps only at boot/restart scale: kube-router resyncs on ordinary Kubernetes API events, and this cluster's CronJobs (ca-sync, porkbun-ddns, gitea-backup) each create/delete a Pod every few minutes - plausible triggers for kube-router's own reconciliation loop, and therefore plausible triggers for this drift, recurring on a timescale of minutes. The reposition logic itself (previous commit) is correct - it just only ran on the existing triggers (boot, UFW reload, an Ansible-detected script content change), none of which fired between the two attempts. Adds pia-killswitch-reassert.timer/.service: re-runs the already fully idempotent pia-killswitch.sh every pia_healthcheck_interval_sec (same cadence as the health check), giving it regular chances to run instead of only sporadic ones. Deliberately separate from pia-gateway-recover.timer and NOT gated by pia_auto_recovery_enabled - this is the kill switch continuing to assert its own already- established, narrowly-scoped state (source rule, table default, FORWARD jump position), the same kind of thing it already does unconditionally at boot and after a UFW reload, just on a schedule too - not the higher-level automatic recovery/registration logic the plan was deliberately cautious about running unattended. Unblocked live in the meantime with a manual `sudo systemctl restart pia-killswitch.service`. Offline checks: yamllint, ansible-playbook --syntax-check, rendered templates (Jinja2 + bash -n on the updated script), ansible-lint (diffed against main - no new finding categories), full pia_gateway.py test suite (94/94, unaffected by this change) all pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
126 lines
7.0 KiB
Django/Jinja
126 lines
7.0 KiB
Django/Jinja
#!/bin/bash
|
|
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
|
|
#
|
|
# Runs at boot, before wg-quick@{{ pia_wg_interface }}.service (enforced
|
|
# by the systemd drop-in in pia-killswitch.service's Before=/wg-quick's
|
|
# Requires=After=); any time UFW is reloaded (see handlers/main.yaml —
|
|
# "Reload ufw" fires before "Restart pia-killswitch" so a reload can
|
|
# never leave a permanent gap); and — since 2026-09-22, after the FORWARD
|
|
# jump was found drifting twice within about an hour under this
|
|
# cluster's ordinary CronJob/pod churn, far more often than a boot/reload
|
|
# -only cadence catches — every {{ pia_healthcheck_interval_sec }}s via
|
|
# pia-killswitch-reassert.timer. Seeds the closed state that PIA
|
|
# gateway's kill switch relies on:
|
|
#
|
|
# - a source rule sending {{ pia_vlan50_subnet }} to the dedicated
|
|
# "{{ pia_route_table_name }}" table, evaluated after the kernel's
|
|
# built-in "local" rule (priority 0) and before "main" (32766) — this
|
|
# is what keeps Technitium (10.10.40.53, a local address) delivered
|
|
# via the local table rather than pulled into policy routing.
|
|
# - a terminal "unreachable" default route in that table — but only
|
|
# when {{ pia_wg_interface }} isn't already up — so VLAN 50 traffic
|
|
# fails closed instead of falling through to main if the interface is
|
|
# down, absent, or its PostUp hasn't run yet, without this script
|
|
# stomping an already-open, healthy route on a restart unrelated to
|
|
# the tunnel's own state (see the check below).
|
|
# - a DEDICATED "{{ pia_iptables_chain }}" chain, jumped into by exactly
|
|
# one rule at the very top of FORWARD (position 1), matching only
|
|
# source {{ pia_vlan50_subnet }}. This is deliberately NOT a global
|
|
# FORWARD default-policy change: minisforum is the k3s server and
|
|
# runs Flannel, which depends on its own broad FORWARD-chain
|
|
# ACCEPTs for pod-to-pod and pod-to-internet traffic (source
|
|
# 10.42.0.0/16, entirely disjoint from 10.10.50.0/24) — changing the
|
|
# chain-wide default policy was never actually proven safe against
|
|
# that, so this scopes the kill switch to a chain that Flannet/k3s
|
|
# traffic can never enter in the first place, rather than risking it.
|
|
# Being the FIRST rule in FORWARD also means no other, later rule
|
|
# (ufw's own, Docker's, anything) can accidentally pre-empt this
|
|
# source's fate with a broader ACCEPT — the jump is unconditional for
|
|
# this source and the chain itself ends in an unconditional DROP, not
|
|
# a RETURN, so nothing after it in FORWARD is ever consulted for
|
|
# this source either. "First" is actively re-asserted on every run,
|
|
# not just seeded once — see the jump-insertion block below for why
|
|
# that turned out to matter in practice.
|
|
#
|
|
# Idempotent: safe to run more than once (e.g. `systemctl restart
|
|
# pia-killswitch`, or after this script re-runs following a UFW reload)
|
|
# without creating duplicate rules or duplicate chains — including
|
|
# repositioning the FORWARD jump back to position 1 if something else
|
|
# (kube-router, dockerd) has since pushed it down, rather than treating
|
|
# its mere existence as proof it's still first.
|
|
set -euo pipefail
|
|
|
|
TABLE="{{ pia_route_table_name }}"
|
|
SUBNET="{{ pia_vlan50_subnet }}"
|
|
WG_IF="{{ pia_wg_interface }}"
|
|
PRIORITY="{{ pia_rule_priority }}"
|
|
CHAIN="{{ pia_iptables_chain }}"
|
|
|
|
if ! ip rule show | grep -qE "^${PRIORITY}:[[:space:]]*from ${SUBNET} lookup ${TABLE}\$"; then
|
|
ip rule add from "${SUBNET}" table "${TABLE}" priority "${PRIORITY}"
|
|
fi
|
|
|
|
# Only seed the terminal-unreachable fallback if {{ pia_wg_interface }}
|
|
# isn't already up. This script re-runs any time it's restarted for
|
|
# reasons unrelated to the tunnel's own state (a UFW reload, a content
|
|
# fix to this script itself) — if the interface is already up and
|
|
# healthy, its own PostUp already set the real "default dev %i" route in
|
|
# this table, and blindly overwriting that back to "unreachable" here
|
|
# would break a working tunnel's routing for no reason until something
|
|
# re-runs PostUp. Only actual absence of the interface should close it.
|
|
if ! ip link show "${WG_IF}" up &>/dev/null; then
|
|
ip route replace unreachable default table "${TABLE}"
|
|
fi
|
|
|
|
# Dedicated chain: create if missing (iptables -N fails harmlessly if it
|
|
# already exists, hence || true rather than a -C-style existence check —
|
|
# there is no direct "does this chain exist" check short of grepping -L).
|
|
iptables -N "${CHAIN}" 2>/dev/null || true
|
|
|
|
# Exactly one jump from FORWARD into it, kept at absolute position 1.
|
|
# Existence alone ("-C") is not proof of position — confirmed live,
|
|
# 2026-09-22: kube-router and dockerd each independently reassert their
|
|
# OWN jump at position 1 whenever THEY resync/restart (kube-router
|
|
# doesn't even dedupe its own older copy when it does, so duplicates of
|
|
# ITS rule can appear too), which silently pushes this one down without
|
|
# removing it. A plain "-C, insert only if absent" check then reports
|
|
# "already exists" and leaves it stranded wherever it drifted to — this
|
|
# is exactly what happened here: the jump was 4th (behind kube-router,
|
|
# DOCKER-USER, DOCKER-FORWARD), not 1st, on a host that had been up 200
|
|
# days with kube-router/Docker managing FORWARD independently the whole
|
|
# time. So check the actual first rule in FORWARD, not just whether ours
|
|
# exists somewhere; if it isn't ours, remove it (by spec — iptables -D
|
|
# matches wherever it currently sits, not by position) and reinsert at
|
|
# 1. Safe to do unconditionally on every run: this rule is narrowly
|
|
# {{ pia_vlan50_subnet }}-scoped, so moving it earlier relative to
|
|
# kube-router's/Docker's own broader jumps can't affect pod or Docker
|
|
# bridge traffic either way — confirmed by reading their actual chain
|
|
# contents (KUBE-ROUTER-FORWARD only matches specific pod /32
|
|
# addresses, DOCKER-USER was empty, DOCKER-FORWARD's one direct rule is
|
|
# scoped to -i docker0), not assumed.
|
|
FORWARD_FIRST=$(iptables -S FORWARD | grep '^-A FORWARD' | head -1 || true)
|
|
if [[ "${FORWARD_FIRST}" != "-A FORWARD -s ${SUBNET} -j ${CHAIN}" ]]; then
|
|
if iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
|
|
iptables -D FORWARD -s "${SUBNET}" -j "${CHAIN}"
|
|
fi
|
|
iptables -I FORWARD 1 -s "${SUBNET}" -j "${CHAIN}"
|
|
fi
|
|
|
|
# Static catch-all inside the chain: anything that reaches this point
|
|
# (i.e., wasn't already ACCEPTed by a more specific rule that
|
|
# pia-wg.conf.j2's PostUp inserts ahead of these when the interface is
|
|
# up) is logged and dropped. --log-limit is not a real iptables option
|
|
# (there is no such flag — rate-limiting a LOG target is done via a
|
|
# separate -m limit match ahead of it, as below); the -C check must
|
|
# match the rule as actually inserted, term for term, or it will never
|
|
# find it and silently re-append a duplicate on every run.
|
|
if ! iptables -C "${CHAIN}" -m limit --limit 5/minute -j LOG \
|
|
--log-prefix "PIA-KILLSWITCH-DROP: " 2>/dev/null; then
|
|
iptables -A "${CHAIN}" -m limit --limit 5/minute -j LOG \
|
|
--log-prefix "PIA-KILLSWITCH-DROP: "
|
|
fi
|
|
|
|
if ! iptables -C "${CHAIN}" -j DROP 2>/dev/null; then
|
|
iptables -A "${CHAIN}" -j DROP
|
|
fi
|