Nik Afiq 89d289addf feat: add periodic kill-switch reassertion timer
Found live, 2026-09-22: after fixing pia-killswitch.sh's FORWARD-jump
repositioning and getting one pia-gateway register attempt past the
safety check, the very next attempt - roughly an hour later, no
reboot, no UFW reload, no script content change in between - hit the
identical "FORWARD jump to kill-switch chain is not first" failure
again. This disproves the earlier assumption that kube-router/dockerd
reassert their own position-1 jumps only at boot/restart scale:
kube-router resyncs on ordinary Kubernetes API events, and this
cluster's CronJobs (ca-sync, porkbun-ddns, gitea-backup) each
create/delete a Pod every few minutes - plausible triggers for
kube-router's own reconciliation loop, and therefore plausible
triggers for this drift, recurring on a timescale of minutes.

The reposition logic itself (previous commit) is correct - it just
only ran on the existing triggers (boot, UFW reload, an
Ansible-detected script content change), none of which fired between
the two attempts.

Adds pia-killswitch-reassert.timer/.service: re-runs the already fully
idempotent pia-killswitch.sh every pia_healthcheck_interval_sec (same
cadence as the health check), giving it regular chances to run instead
of only sporadic ones. Deliberately separate from
pia-gateway-recover.timer and NOT gated by pia_auto_recovery_enabled -
this is the kill switch continuing to assert its own already-
established, narrowly-scoped state (source rule, table default,
FORWARD jump position), the same kind of thing it already does
unconditionally at boot and after a UFW reload, just on a schedule
too - not the higher-level automatic recovery/registration logic the
plan was deliberately cautious about running unattended.

Unblocked live in the meantime with a manual
`sudo systemctl restart pia-killswitch.service`.

Offline checks: yamllint, ansible-playbook --syntax-check, rendered
templates (Jinja2 + bash -n on the updated script), ansible-lint
(diffed against main - no new finding categories), full pia_gateway.py
test suite (94/94, unaffected by this change) all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 19:08:28 +09:00

126 lines
7.0 KiB
Django/Jinja

#!/bin/bash
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Runs at boot, before wg-quick@{{ pia_wg_interface }}.service (enforced
# by the systemd drop-in in pia-killswitch.service's Before=/wg-quick's
# Requires=After=); any time UFW is reloaded (see handlers/main.yaml —
# "Reload ufw" fires before "Restart pia-killswitch" so a reload can
# never leave a permanent gap); and — since 2026-09-22, after the FORWARD
# jump was found drifting twice within about an hour under this
# cluster's ordinary CronJob/pod churn, far more often than a boot/reload
# -only cadence catches — every {{ pia_healthcheck_interval_sec }}s via
# pia-killswitch-reassert.timer. Seeds the closed state that PIA
# gateway's kill switch relies on:
#
# - a source rule sending {{ pia_vlan50_subnet }} to the dedicated
# "{{ pia_route_table_name }}" table, evaluated after the kernel's
# built-in "local" rule (priority 0) and before "main" (32766) — this
# is what keeps Technitium (10.10.40.53, a local address) delivered
# via the local table rather than pulled into policy routing.
# - a terminal "unreachable" default route in that table — but only
# when {{ pia_wg_interface }} isn't already up — so VLAN 50 traffic
# fails closed instead of falling through to main if the interface is
# down, absent, or its PostUp hasn't run yet, without this script
# stomping an already-open, healthy route on a restart unrelated to
# the tunnel's own state (see the check below).
# - a DEDICATED "{{ pia_iptables_chain }}" chain, jumped into by exactly
# one rule at the very top of FORWARD (position 1), matching only
# source {{ pia_vlan50_subnet }}. This is deliberately NOT a global
# FORWARD default-policy change: minisforum is the k3s server and
# runs Flannel, which depends on its own broad FORWARD-chain
# ACCEPTs for pod-to-pod and pod-to-internet traffic (source
# 10.42.0.0/16, entirely disjoint from 10.10.50.0/24) — changing the
# chain-wide default policy was never actually proven safe against
# that, so this scopes the kill switch to a chain that Flannet/k3s
# traffic can never enter in the first place, rather than risking it.
# Being the FIRST rule in FORWARD also means no other, later rule
# (ufw's own, Docker's, anything) can accidentally pre-empt this
# source's fate with a broader ACCEPT — the jump is unconditional for
# this source and the chain itself ends in an unconditional DROP, not
# a RETURN, so nothing after it in FORWARD is ever consulted for
# this source either. "First" is actively re-asserted on every run,
# not just seeded once — see the jump-insertion block below for why
# that turned out to matter in practice.
#
# Idempotent: safe to run more than once (e.g. `systemctl restart
# pia-killswitch`, or after this script re-runs following a UFW reload)
# without creating duplicate rules or duplicate chains — including
# repositioning the FORWARD jump back to position 1 if something else
# (kube-router, dockerd) has since pushed it down, rather than treating
# its mere existence as proof it's still first.
set -euo pipefail
TABLE="{{ pia_route_table_name }}"
SUBNET="{{ pia_vlan50_subnet }}"
WG_IF="{{ pia_wg_interface }}"
PRIORITY="{{ pia_rule_priority }}"
CHAIN="{{ pia_iptables_chain }}"
if ! ip rule show | grep -qE "^${PRIORITY}:[[:space:]]*from ${SUBNET} lookup ${TABLE}\$"; then
ip rule add from "${SUBNET}" table "${TABLE}" priority "${PRIORITY}"
fi
# Only seed the terminal-unreachable fallback if {{ pia_wg_interface }}
# isn't already up. This script re-runs any time it's restarted for
# reasons unrelated to the tunnel's own state (a UFW reload, a content
# fix to this script itself) — if the interface is already up and
# healthy, its own PostUp already set the real "default dev %i" route in
# this table, and blindly overwriting that back to "unreachable" here
# would break a working tunnel's routing for no reason until something
# re-runs PostUp. Only actual absence of the interface should close it.
if ! ip link show "${WG_IF}" up &>/dev/null; then
ip route replace unreachable default table "${TABLE}"
fi
# Dedicated chain: create if missing (iptables -N fails harmlessly if it
# already exists, hence || true rather than a -C-style existence check —
# there is no direct "does this chain exist" check short of grepping -L).
iptables -N "${CHAIN}" 2>/dev/null || true
# Exactly one jump from FORWARD into it, kept at absolute position 1.
# Existence alone ("-C") is not proof of position — confirmed live,
# 2026-09-22: kube-router and dockerd each independently reassert their
# OWN jump at position 1 whenever THEY resync/restart (kube-router
# doesn't even dedupe its own older copy when it does, so duplicates of
# ITS rule can appear too), which silently pushes this one down without
# removing it. A plain "-C, insert only if absent" check then reports
# "already exists" and leaves it stranded wherever it drifted to — this
# is exactly what happened here: the jump was 4th (behind kube-router,
# DOCKER-USER, DOCKER-FORWARD), not 1st, on a host that had been up 200
# days with kube-router/Docker managing FORWARD independently the whole
# time. So check the actual first rule in FORWARD, not just whether ours
# exists somewhere; if it isn't ours, remove it (by spec — iptables -D
# matches wherever it currently sits, not by position) and reinsert at
# 1. Safe to do unconditionally on every run: this rule is narrowly
# {{ pia_vlan50_subnet }}-scoped, so moving it earlier relative to
# kube-router's/Docker's own broader jumps can't affect pod or Docker
# bridge traffic either way — confirmed by reading their actual chain
# contents (KUBE-ROUTER-FORWARD only matches specific pod /32
# addresses, DOCKER-USER was empty, DOCKER-FORWARD's one direct rule is
# scoped to -i docker0), not assumed.
FORWARD_FIRST=$(iptables -S FORWARD | grep '^-A FORWARD' | head -1 || true)
if [[ "${FORWARD_FIRST}" != "-A FORWARD -s ${SUBNET} -j ${CHAIN}" ]]; then
if iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
iptables -D FORWARD -s "${SUBNET}" -j "${CHAIN}"
fi
iptables -I FORWARD 1 -s "${SUBNET}" -j "${CHAIN}"
fi
# Static catch-all inside the chain: anything that reaches this point
# (i.e., wasn't already ACCEPTed by a more specific rule that
# pia-wg.conf.j2's PostUp inserts ahead of these when the interface is
# up) is logged and dropped. --log-limit is not a real iptables option
# (there is no such flag — rate-limiting a LOG target is done via a
# separate -m limit match ahead of it, as below); the -C check must
# match the rule as actually inserted, term for term, or it will never
# find it and silently re-append a duplicate on every run.
if ! iptables -C "${CHAIN}" -m limit --limit 5/minute -j LOG \
--log-prefix "PIA-KILLSWITCH-DROP: " 2>/dev/null; then
iptables -A "${CHAIN}" -m limit --limit 5/minute -j LOG \
--log-prefix "PIA-KILLSWITCH-DROP: "
fi
if ! iptables -C "${CHAIN}" -j DROP 2>/dev/null; then
iptables -A "${CHAIN}" -j DROP
fi