Nik Afiq fb9ecffa9f fix: tolerate content-verified kube-router jumps ahead of the kill switch
Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:

  19:38:58  3rd unhealthy check -> restart
  19:39:09  recovered - route open, handshake fresh
  19:40:03  drift again -> emergency close -> route forced closed,
            counter reset to 0
  19:41:08  drift again -> emergency close again

VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.

This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.

Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.

Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.

pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 19:55:47 +09:00

216 lines
11 KiB
YAML

---
# Part of role: pia-gateway
# Called by: ansible/playbooks/pia-gateway.yaml
# Description: Default variables for minisforum's PIA WireGuard egress
# gateway for VPN VLAN 50 — policy routing, kill switch, and health
# checks. See ~/repo/homelab/plan.md for the cross-repo contract this
# role implements (fixed values below match that contract).
# PIA regions to try, in order, for WireGuard addKey registration —
# first server (in the first region that has any) to return
# `status: OK` wins. These are API region **ids** from PIA's serverlist
# (https://serverlist.piaservers.net/vpninfo/servers/v6), e.g.
# {"id": "japan", "name": "JP Tokyo", ...} — NOT Gluetun's display names
# ("JP Tokyo", "Hong Kong", "Taiwan") and not the slug you might guess
# from one (confirmed live 2026-08-24: Tokyo's id is "japan", not
# "jp_tokyo" — that matched zero regions on a first pass). Order matches
# the region set already used by the Gluetun sidecars, Hong Kong/Taiwan
# moved ahead of Tokyo after all three Tokyo servers were observed
# failing in practice — see tasks/register.yaml's header for what that
# means and doesn't mean.
pia_region_candidates:
- hk
- taiwan
- japan
# Force a SINGLE region for troubleshooting, bypassing
# pia_region_candidates entirely — e.g. `-e pia_region=hk`. This is also
# an API id, not a display name; same rule as above. Leave null (the
# default) to use the ordered candidate list. tasks/register.yaml
# computes the actual list to try (pia_effective_regions) from whichever
# of these two is set — see that file's own comment for the exact logic,
# not duplicated here to avoid the two drifting out of sync.
pia_region: null
# WireGuard interface identity. Deliberately not "wg0" — that name is
# already used by ansible/roles/wireguard, minisforum's separate *inbound*
# home-VPN server role. Colliding with it would tear down remote access.
pia_wg_interface: pia-wg
pia_wg_config_dir: /etc/wireguard
# Re-registration behaviour — two independent, unambiguous knobs (an
# earlier single pia_force_reregister conflated "re-run the API calls"
# with "get a new keypair", which isn't the same thing and isn't safe to
# conflate — the key-generation task uses `creates:`, so a naive
# "force" only re-ran the PIA API calls with the SAME existing key,
# silently, unless you also knew to delete the key file yourself):
#
# - pia_force_reregister: re-run PIA registration (serverlist lookup,
# token, addKey) even if {{ pia_wg_interface }}.conf already exists.
# Reuses whatever keypair is already on disk (generates one only if
# none exists yet). Use this to pick up a new PIA-assigned peer/server
# without changing minisforum's own WireGuard identity.
# - pia_force_key_rotation: additionally generate a NEW keypair and
# register THAT with PIA (implies pia_force_reregister). Rotation is
# safe-by-construction: the new key is generated to a *.key.new path
# and only promoted (with the old key backed up alongside a timestamp)
# after PIA has actually accepted the new key and the new config has
# been written — see tasks/register.yaml. If registration fails
# partway, the previous key/config are left completely untouched
# rather than ending up in a broken half-state.
pia_force_reregister: false
pia_force_key_rotation: false
# Dedicated policy-routing table for VLAN 50 egress. 0/253/254/255 are
# kernel-reserved (unspec/default/main/local) — this must stay clear of
# those and of any other custom table already registered on the host.
pia_route_table_id: 52
pia_route_table_name: pia
# Rule priority for "from 10.10.50.0/24 lookup pia". Must sort after the
# kernel's built-in local table (priority 0, always evaluated first, keeps
# traffic addressed to the host's own local/Technitium address delivered
# locally per plan.md) and before main (priority 32766).
pia_rule_priority: 100
# Fixed cross-repo contract (see plan.md "Fixed cross-repository
# contract") — VLAN 50 subnet and the Flint gateway that owns the return
# route back to VLAN 50. Not derived from live discovery; change only if
# the contract itself changes.
pia_vlan50_subnet: 10.10.50.0/24
pia_flint_gateway: 10.10.50.1
# Dedicated iptables chain the kill switch governs, jumped into by a
# single rule at FORWARD position 1 matching only
# {{ pia_vlan50_subnet }} — deliberately not a global FORWARD default-
# policy change, so k3s/Flannel's own FORWARD-chain traffic (source
# 10.42.0.0/16, disjoint from VLAN 50) is never affected by this role at
# all. See templates/pia-killswitch.sh.j2 for the full reasoning.
pia_iptables_chain: PIA-VLAN50
# Real parent LAN interface on minisforum — verified live 2026-08-23 via
# `ip -brief addr`; it is enp1s0, not "eth0". Used for MASQUERADE-exclusion
# bookkeeping and documentation only, since NAT/forward rules below key off
# {{ pia_wg_interface }} rather than enumerating the normal interface.
pia_lan_interface: enp1s0
# FORWARD chains the controller may tolerate sitting AHEAD of our own
# kill-switch jump, on the condition that their OWN current content is
# re-verified — live, on every check, never a one-time trust — to be
# structurally unable to match {{ pia_vlan50_subnet }} traffic. Empty (the
# safe default for any other host) means the literal, original "must be
# absolutely first" check. Set to ["KUBE-ROUTER-FORWARD"] here because
# k3s's kube-router network-policy controller and dockerd both also
# insert unconditional jumps near the top of FORWARD, independently of
# this role, and — confirmed live, 2026-09-22 — kube-router resyncs (and
# re-asserts its own top-of-chain position) roughly every 60-90s on this
# cluster's normal CronJob/pod churn, not just at boot/restart scale as
# first assumed. Treating every one of those transient reorderings as a
# full safety-drift emergency (which unconditionally closes table pia's
# route) caused a real live outage: a repeating cycle of recover-succeeds
# -> drift-detected-within-a-minute -> emergency-close -> counter reset
# -> repeat, which kept VLAN 50 open for only seconds at a time. Only
# kube-router is listed — NOT dockerd's DOCKER-USER/DOCKER-FORWARD —
# because kube-router was the only one observed reasserting position
# repeatedly (dockerd sets its hooks up once at daemon start, not on a
# recurring cycle), and DOCKER-FORWARD itself jumps to three further
# nested chains (DOCKER-CT, DOCKER-INTERNAL, DOCKER-BRIDGE) never fully
# content-verified — safer to leave those as a hard drift trigger than
# assume they're equally inert. See
# Classifier._forward_jump_effectively_first's own docstring in
# files/pia_gateway.py for exactly what "re-verified safe" requires of a
# bypassed chain's content.
pia_forward_bypass_chains:
- KUBE-ROUTER-FORWARD
# Reverse-path filtering, applied to {{ pia_wg_interface }} only. The
# kernel default on minisforum (conf.default.rp_filter) is already loose
# (2) at time of writing, so new interfaces inherit loose mode — this task
# pins it explicitly rather than relying on that inherited default holding
# across future changes. Loose mode is required because PIA's return
# traffic can legitimately arrive with routing asymmetry relative to how
# minisforum would route toward that source itself.
pia_rp_filter_mode: 2
# MSS clamping on the pia-wg forward path — ON. Confirmed needed live
# (2026-08-25), not guessed: a real JDownloader download to
# sbs237.sbsf.tech hung indefinitely (TLS read timeout waiting for the
# server's handshake response — small requests like a CDN version check
# worked fine, only the larger TLS response never arrived). Verified
# with the exact test plan.md called for before flipping this: `ping -M
# do -s 1450 -c 3 1.1.1.1` from a VLAN 50 pod returned `From 10.10.40.53
# Frag needed and DF set (mtu = 1420)` — our own PMTU discovery between
# the pod and minisforum works correctly, which means the black hole is
# further out (PIA's network or the remote server's own path), not on
# our side — exactly the class of problem MSS clamping fixes by
# avoiding the need for that ICMP round-trip at all, clamping the MSS
# at the TCP handshake instead of relying on mid-connection PMTU
# discovery that something further along the path may not relay back.
pia_mss_clamp_enabled: true
# Health check cadence and thresholds. The check only observes and logs —
# see tasks/healthcheck.yaml — it must never itself open the kill switch.
pia_healthcheck_interval_sec: 60
pia_healthcheck_handshake_max_age_sec: 180
# ── Automatic recovery (files/pia_gateway.py, tasks/controller.yaml) ────────
#
# Which implementation performs PIA registration when this role runs:
# shared — `pia-gateway register` on the host (the same code the unattended
# recovery/renewal timers use; activates and VERIFIES a handshake
# before reporting success).
# legacy — the original in-Ansible token/addKey flow (tasks/register.yaml).
# Kept only until the shared path has passed a live manual
# re-registration; then it is to be deleted (see README).
pia_registration_backend: shared
# Master switch for the reactive recovery timer. false stops+disables the
# timer but leaves the program, credentials, kill switch, health check and
# pia-wg in place (controlled rollback). Renewal has its own switch below.
pia_auto_recovery_enabled: true
# Proactive same-key re-registration when the last VERIFIED registration is
# older than pia_renewal_after_days. Off until the reactive path has passed
# its live drills (README "Rollout"). Needs a verified baseline in state —
# established by the shared registration path, never inferred from file
# mtimes.
pia_renewal_enabled: false
pia_renewal_after_days: 21
pia_renewal_calendar: "*-*-* 03:30:00"
pia_renewal_jitter_sec: 1800
# Recovery state machine tunables (seconds unless noted).
pia_recovery_failure_threshold: 3
pia_recovery_startup_grace_sec: 120
pia_recovery_restart_wait_sec: 90
pia_recovery_activation_wait_sec: 90
pia_recovery_max_duration_sec: 600
pia_recovery_backoff_schedule_sec: [300, 900, 1800, 3600]
pia_recovery_max_candidates_per_cycle: 6
pia_recovery_max_rotation_candidates: 3
pia_recovery_http_connect_timeout_sec: 10
pia_recovery_http_total_timeout_sec: 30
pia_recovery_backup_keep: 5
pia_recovery_lock_wait_sec: 600
# Where the on-host controller keeps things.
pia_gateway_program: /usr/local/sbin/pia-gateway
pia_gateway_etc_dir: /etc/pia-gateway
pia_gateway_state_dir: /var/lib/pia-gateway
# alert-bridge (manifests/home-services/alert-bridge*.yaml) delivery for
# state-transition events. Its API has ONE shared bearer key (no per-caller
# tokens — confirmed against the alert-bridge source), so the same key the
# cluster uses is provisioned to minisforum's root-only credentials file
# when pia_alert_bridge_api_key is set (host_vars reads it from .env). Set
# the URL to null to disable delivery; events still go to the journal.
pia_alert_bridge_url: "http://alert-bridge.home.arpa/alerts"
pia_alert_source: "pia-gateway@minisforum"
pia_alert_bridge_api_key: ""
# Deliberate credential removal. Disabling automation never deletes the
# on-host PIA credentials on its own; set this true (one run) to remove
# {{ pia_gateway_etc_dir }}/credentials.env, e.g. when decommissioning or
# after rotating the PIA password elsewhere.
pia_remove_host_credentials: false