Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:
19:38:58 3rd unhealthy check -> restart
19:39:09 recovered - route open, handshake fresh
19:40:03 drift again -> emergency close -> route forced closed,
counter reset to 0
19:41:08 drift again -> emergency close again
VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.
This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.
Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.
Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.
pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
216 lines
11 KiB
YAML
216 lines
11 KiB
YAML
---
|
|
# Part of role: pia-gateway
|
|
# Called by: ansible/playbooks/pia-gateway.yaml
|
|
# Description: Default variables for minisforum's PIA WireGuard egress
|
|
# gateway for VPN VLAN 50 — policy routing, kill switch, and health
|
|
# checks. See ~/repo/homelab/plan.md for the cross-repo contract this
|
|
# role implements (fixed values below match that contract).
|
|
|
|
# PIA regions to try, in order, for WireGuard addKey registration —
|
|
# first server (in the first region that has any) to return
|
|
# `status: OK` wins. These are API region **ids** from PIA's serverlist
|
|
# (https://serverlist.piaservers.net/vpninfo/servers/v6), e.g.
|
|
# {"id": "japan", "name": "JP Tokyo", ...} — NOT Gluetun's display names
|
|
# ("JP Tokyo", "Hong Kong", "Taiwan") and not the slug you might guess
|
|
# from one (confirmed live 2026-08-24: Tokyo's id is "japan", not
|
|
# "jp_tokyo" — that matched zero regions on a first pass). Order matches
|
|
# the region set already used by the Gluetun sidecars, Hong Kong/Taiwan
|
|
# moved ahead of Tokyo after all three Tokyo servers were observed
|
|
# failing in practice — see tasks/register.yaml's header for what that
|
|
# means and doesn't mean.
|
|
pia_region_candidates:
|
|
- hk
|
|
- taiwan
|
|
- japan
|
|
|
|
# Force a SINGLE region for troubleshooting, bypassing
|
|
# pia_region_candidates entirely — e.g. `-e pia_region=hk`. This is also
|
|
# an API id, not a display name; same rule as above. Leave null (the
|
|
# default) to use the ordered candidate list. tasks/register.yaml
|
|
# computes the actual list to try (pia_effective_regions) from whichever
|
|
# of these two is set — see that file's own comment for the exact logic,
|
|
# not duplicated here to avoid the two drifting out of sync.
|
|
pia_region: null
|
|
|
|
# WireGuard interface identity. Deliberately not "wg0" — that name is
|
|
# already used by ansible/roles/wireguard, minisforum's separate *inbound*
|
|
# home-VPN server role. Colliding with it would tear down remote access.
|
|
pia_wg_interface: pia-wg
|
|
pia_wg_config_dir: /etc/wireguard
|
|
|
|
# Re-registration behaviour — two independent, unambiguous knobs (an
|
|
# earlier single pia_force_reregister conflated "re-run the API calls"
|
|
# with "get a new keypair", which isn't the same thing and isn't safe to
|
|
# conflate — the key-generation task uses `creates:`, so a naive
|
|
# "force" only re-ran the PIA API calls with the SAME existing key,
|
|
# silently, unless you also knew to delete the key file yourself):
|
|
#
|
|
# - pia_force_reregister: re-run PIA registration (serverlist lookup,
|
|
# token, addKey) even if {{ pia_wg_interface }}.conf already exists.
|
|
# Reuses whatever keypair is already on disk (generates one only if
|
|
# none exists yet). Use this to pick up a new PIA-assigned peer/server
|
|
# without changing minisforum's own WireGuard identity.
|
|
# - pia_force_key_rotation: additionally generate a NEW keypair and
|
|
# register THAT with PIA (implies pia_force_reregister). Rotation is
|
|
# safe-by-construction: the new key is generated to a *.key.new path
|
|
# and only promoted (with the old key backed up alongside a timestamp)
|
|
# after PIA has actually accepted the new key and the new config has
|
|
# been written — see tasks/register.yaml. If registration fails
|
|
# partway, the previous key/config are left completely untouched
|
|
# rather than ending up in a broken half-state.
|
|
pia_force_reregister: false
|
|
pia_force_key_rotation: false
|
|
|
|
# Dedicated policy-routing table for VLAN 50 egress. 0/253/254/255 are
|
|
# kernel-reserved (unspec/default/main/local) — this must stay clear of
|
|
# those and of any other custom table already registered on the host.
|
|
pia_route_table_id: 52
|
|
pia_route_table_name: pia
|
|
|
|
# Rule priority for "from 10.10.50.0/24 lookup pia". Must sort after the
|
|
# kernel's built-in local table (priority 0, always evaluated first, keeps
|
|
# traffic addressed to the host's own local/Technitium address delivered
|
|
# locally per plan.md) and before main (priority 32766).
|
|
pia_rule_priority: 100
|
|
|
|
# Fixed cross-repo contract (see plan.md "Fixed cross-repository
|
|
# contract") — VLAN 50 subnet and the Flint gateway that owns the return
|
|
# route back to VLAN 50. Not derived from live discovery; change only if
|
|
# the contract itself changes.
|
|
pia_vlan50_subnet: 10.10.50.0/24
|
|
pia_flint_gateway: 10.10.50.1
|
|
|
|
# Dedicated iptables chain the kill switch governs, jumped into by a
|
|
# single rule at FORWARD position 1 matching only
|
|
# {{ pia_vlan50_subnet }} — deliberately not a global FORWARD default-
|
|
# policy change, so k3s/Flannel's own FORWARD-chain traffic (source
|
|
# 10.42.0.0/16, disjoint from VLAN 50) is never affected by this role at
|
|
# all. See templates/pia-killswitch.sh.j2 for the full reasoning.
|
|
pia_iptables_chain: PIA-VLAN50
|
|
|
|
# Real parent LAN interface on minisforum — verified live 2026-08-23 via
|
|
# `ip -brief addr`; it is enp1s0, not "eth0". Used for MASQUERADE-exclusion
|
|
# bookkeeping and documentation only, since NAT/forward rules below key off
|
|
# {{ pia_wg_interface }} rather than enumerating the normal interface.
|
|
pia_lan_interface: enp1s0
|
|
|
|
# FORWARD chains the controller may tolerate sitting AHEAD of our own
|
|
# kill-switch jump, on the condition that their OWN current content is
|
|
# re-verified — live, on every check, never a one-time trust — to be
|
|
# structurally unable to match {{ pia_vlan50_subnet }} traffic. Empty (the
|
|
# safe default for any other host) means the literal, original "must be
|
|
# absolutely first" check. Set to ["KUBE-ROUTER-FORWARD"] here because
|
|
# k3s's kube-router network-policy controller and dockerd both also
|
|
# insert unconditional jumps near the top of FORWARD, independently of
|
|
# this role, and — confirmed live, 2026-09-22 — kube-router resyncs (and
|
|
# re-asserts its own top-of-chain position) roughly every 60-90s on this
|
|
# cluster's normal CronJob/pod churn, not just at boot/restart scale as
|
|
# first assumed. Treating every one of those transient reorderings as a
|
|
# full safety-drift emergency (which unconditionally closes table pia's
|
|
# route) caused a real live outage: a repeating cycle of recover-succeeds
|
|
# -> drift-detected-within-a-minute -> emergency-close -> counter reset
|
|
# -> repeat, which kept VLAN 50 open for only seconds at a time. Only
|
|
# kube-router is listed — NOT dockerd's DOCKER-USER/DOCKER-FORWARD —
|
|
# because kube-router was the only one observed reasserting position
|
|
# repeatedly (dockerd sets its hooks up once at daemon start, not on a
|
|
# recurring cycle), and DOCKER-FORWARD itself jumps to three further
|
|
# nested chains (DOCKER-CT, DOCKER-INTERNAL, DOCKER-BRIDGE) never fully
|
|
# content-verified — safer to leave those as a hard drift trigger than
|
|
# assume they're equally inert. See
|
|
# Classifier._forward_jump_effectively_first's own docstring in
|
|
# files/pia_gateway.py for exactly what "re-verified safe" requires of a
|
|
# bypassed chain's content.
|
|
pia_forward_bypass_chains:
|
|
- KUBE-ROUTER-FORWARD
|
|
|
|
# Reverse-path filtering, applied to {{ pia_wg_interface }} only. The
|
|
# kernel default on minisforum (conf.default.rp_filter) is already loose
|
|
# (2) at time of writing, so new interfaces inherit loose mode — this task
|
|
# pins it explicitly rather than relying on that inherited default holding
|
|
# across future changes. Loose mode is required because PIA's return
|
|
# traffic can legitimately arrive with routing asymmetry relative to how
|
|
# minisforum would route toward that source itself.
|
|
pia_rp_filter_mode: 2
|
|
|
|
# MSS clamping on the pia-wg forward path — ON. Confirmed needed live
|
|
# (2026-08-25), not guessed: a real JDownloader download to
|
|
# sbs237.sbsf.tech hung indefinitely (TLS read timeout waiting for the
|
|
# server's handshake response — small requests like a CDN version check
|
|
# worked fine, only the larger TLS response never arrived). Verified
|
|
# with the exact test plan.md called for before flipping this: `ping -M
|
|
# do -s 1450 -c 3 1.1.1.1` from a VLAN 50 pod returned `From 10.10.40.53
|
|
# Frag needed and DF set (mtu = 1420)` — our own PMTU discovery between
|
|
# the pod and minisforum works correctly, which means the black hole is
|
|
# further out (PIA's network or the remote server's own path), not on
|
|
# our side — exactly the class of problem MSS clamping fixes by
|
|
# avoiding the need for that ICMP round-trip at all, clamping the MSS
|
|
# at the TCP handshake instead of relying on mid-connection PMTU
|
|
# discovery that something further along the path may not relay back.
|
|
pia_mss_clamp_enabled: true
|
|
|
|
# Health check cadence and thresholds. The check only observes and logs —
|
|
# see tasks/healthcheck.yaml — it must never itself open the kill switch.
|
|
pia_healthcheck_interval_sec: 60
|
|
pia_healthcheck_handshake_max_age_sec: 180
|
|
|
|
# ── Automatic recovery (files/pia_gateway.py, tasks/controller.yaml) ────────
|
|
#
|
|
# Which implementation performs PIA registration when this role runs:
|
|
# shared — `pia-gateway register` on the host (the same code the unattended
|
|
# recovery/renewal timers use; activates and VERIFIES a handshake
|
|
# before reporting success).
|
|
# legacy — the original in-Ansible token/addKey flow (tasks/register.yaml).
|
|
# Kept only until the shared path has passed a live manual
|
|
# re-registration; then it is to be deleted (see README).
|
|
pia_registration_backend: shared
|
|
|
|
# Master switch for the reactive recovery timer. false stops+disables the
|
|
# timer but leaves the program, credentials, kill switch, health check and
|
|
# pia-wg in place (controlled rollback). Renewal has its own switch below.
|
|
pia_auto_recovery_enabled: true
|
|
|
|
# Proactive same-key re-registration when the last VERIFIED registration is
|
|
# older than pia_renewal_after_days. Off until the reactive path has passed
|
|
# its live drills (README "Rollout"). Needs a verified baseline in state —
|
|
# established by the shared registration path, never inferred from file
|
|
# mtimes.
|
|
pia_renewal_enabled: false
|
|
pia_renewal_after_days: 21
|
|
pia_renewal_calendar: "*-*-* 03:30:00"
|
|
pia_renewal_jitter_sec: 1800
|
|
|
|
# Recovery state machine tunables (seconds unless noted).
|
|
pia_recovery_failure_threshold: 3
|
|
pia_recovery_startup_grace_sec: 120
|
|
pia_recovery_restart_wait_sec: 90
|
|
pia_recovery_activation_wait_sec: 90
|
|
pia_recovery_max_duration_sec: 600
|
|
pia_recovery_backoff_schedule_sec: [300, 900, 1800, 3600]
|
|
pia_recovery_max_candidates_per_cycle: 6
|
|
pia_recovery_max_rotation_candidates: 3
|
|
pia_recovery_http_connect_timeout_sec: 10
|
|
pia_recovery_http_total_timeout_sec: 30
|
|
pia_recovery_backup_keep: 5
|
|
pia_recovery_lock_wait_sec: 600
|
|
|
|
# Where the on-host controller keeps things.
|
|
pia_gateway_program: /usr/local/sbin/pia-gateway
|
|
pia_gateway_etc_dir: /etc/pia-gateway
|
|
pia_gateway_state_dir: /var/lib/pia-gateway
|
|
|
|
# alert-bridge (manifests/home-services/alert-bridge*.yaml) delivery for
|
|
# state-transition events. Its API has ONE shared bearer key (no per-caller
|
|
# tokens — confirmed against the alert-bridge source), so the same key the
|
|
# cluster uses is provisioned to minisforum's root-only credentials file
|
|
# when pia_alert_bridge_api_key is set (host_vars reads it from .env). Set
|
|
# the URL to null to disable delivery; events still go to the journal.
|
|
pia_alert_bridge_url: "http://alert-bridge.home.arpa/alerts"
|
|
pia_alert_source: "pia-gateway@minisforum"
|
|
pia_alert_bridge_api_key: ""
|
|
|
|
# Deliberate credential removal. Disabling automation never deletes the
|
|
# on-host PIA credentials on its own; set this true (one run) to remove
|
|
# {{ pia_gateway_etc_dir }}/credentials.env, e.g. when decommissioning or
|
|
# after rotating the PIA password elsewhere.
|
|
pia_remove_host_credentials: false
|