Nik Afiq 80bf24cf0b docs: record step 3 as verified live over a full 25.5-hour window
pia_forward_bypass_chains held up against real production conditions,
not just a short recheck: 1528 healthy passes, 0 SAFETY DRIFT events,
0 safety_drift classifications since the fix was deployed
(2026-09-22 20:37 -> 2026-09-23 22:02). kube-router kept resyncing
throughout; the bypass tolerance handled it silently every time.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 22:09:13 +09:00
..

pia-gateway

Makes minisforum a PIA WireGuard egress gateway for VPN VLAN 50 (10.10.50.0/24), without changing its own default route. Implements Phase 2 of ~/repo/homelab/plan.md.

Before the first real run

  1. The PIA API calls have gone through three rounds of correction against real live testing (2026-08-24). They now live in files/pia_gateway.py (PiaApi), with the original in-Ansible flow kept as tasks/register-legacy.yaml until the shared path is proven live — see "Automatic recovery" below:
    • serverlist.piaservers.net/vpninfo/servers/v6: works. Region values everywhere in this role are the API's id field (e.g. japan, hk, taiwan), never a display name ("JP Tokyo", "Hong Kong", "Taiwan" — those are Gluetun's labels, not something this role or PIA's API accepts).
    • Token acquisition originally guessed a regional-meta-server endpoint (/authv3/generateToken) that turned out not to be the real flow at all — it was rebuilt from PIA's actual pia-foss/manual-connections get_token.sh (fetched and read in full): a single fixed POST https://www.privateinternetaccess.com/api/client/v2/token, multipart username/password, normal system CA validation (no --cacert, no validate_certs: false), independent of region entirely — only WireGuard server selection is regional. Not yet exercised with real credentials end to end.
    • {wg}:1337/addKey (WireGuard key registration) — confirmed against connect_to_wireguard_with_token.sh (fetched and read in full): this one genuinely does need PIA's own CA bundle (files/pia-ca.crt, --cacert) and --connect-to. Originally failed over only across servers within a single fixed region (pia_region: japan) — in practice all three JP Tokyo servers failed the same way, one after another, so failover now also crosses regions: every server in Hong Kong, then every server in Taiwan, then every server in JP Tokyo (pia_region_candidates in defaults/main.yaml), stopping at the first status: OK. Token passed via stdin rather than argv. See tasks/addkey-attempt.yaml and "Region fallback" below.
  2. Make sure the repo-root .env has PIA_USER/PIA_PASSWORD set (same keys manifests/media/pia-secret.sh already reads) — ansible/ host_vars/minisforum.yaml reads them directly from there via a lookup('ansible.builtin.file', ...) + regex, deliberately not ansible-vault. The lookup runs on the control node; .env itself is never copied and never committed (gitignored). Since the auto-recovery work, only PIA_USER/PIA_PASSWORD (and, if present, ALERT_BRIDGE_API_KEY) are provisioned to minisforum's root-only /etc/pia-gateway/credentials.env so the host can re-register unattended — see "Credentials on the host" below. The playbook still needs -J because group_vars/all/vault.yaml is vault-encrypted and loaded for every host, not because the PIA credentials are in it.
  3. Confirm console/recovery access to minisforum (IPMI/physical/other out-of-band path) before applying — this role changes host routing and firewall policy. If SSH becomes unreachable, use that path.
  4. ufw status verbose and wg show (existing wg0) were not inspected live before writing this role (no passwordless sudo in discovery) — spot-check that wg0 (the separate home-VPN role) is unaffected after applying.

Region fallback

Registration tries every WireGuard server in every configured region, in order, stopping at the first status: OK:

hk (Hong Kong):
  wg server 1
  wg server 2
taiwan (Taiwan):
  wg server 1
  wg server 2
japan (JP Tokyo):
  wg server 1
  wg server 2
  wg server 3

The actual number and order of servers within each region always comes from the live serverlist response — never hardcoded here.

Automatic fallback (default — tries pia_region_candidates in order):

ansible-playbook -i ansible/inventory.yaml \
  ansible/playbooks/pia-gateway.yaml -K -J

Forced single region, for troubleshooting one region in isolation — bypasses pia_region_candidates entirely, does not fall back to the others:

ansible-playbook -i ansible/inventory.yaml \
  ansible/playbooks/pia-gateway.yaml -K -J \
  -e pia_region=hk

Both pia_region and every entry in pia_region_candidates are PIA API ids — hk, taiwan, japan — not the display names Gluetun's SERVER_REGIONS config uses ("Hong Kong", "Taiwan", "JP Tokyo"). Passing a display name here matches nothing and fails the "region exists" assertion before any network call is even made.

Which region/server actually ended up registered is recorded as pia_region_used/pia_region_name_used/pia_wg_server_used, written into pia-wg.conf's own [Peer] comment, and surfaced by the health check (INFO configured-peer: ...) — it is whichever one answered first, not necessarily pia_region_candidates[0].

If every region fails: journalctl (or the play's own output) has one debug line per failed attempt, each showing region name+id, server hostname+IP, curl exit code, a classified reason (connection timeout / TLS-SSL failure / HTTP failure / empty response / malformed JSON / parsed-but-not-OK status, never a generic catch-all), and sanitized stderr — never the token, credentials, the full curl invocation, request stdin, or a complete response body. If Hong Kong and Taiwan fail exactly the same way Tokyo did, that's a strong signal the problem is shared across all three (the request shape, the account credentials, or minisforum's own network path) rather than one region being down — see tasks/register-legacy.yaml's header comment (the controller logs the same per-candidate kind=/detail= classification).

Worked example (2026-08-24): exactly that happened — all 7 servers across all 3 regions came back HTTP failure (non-2xx response), curl exit 22, The requested URL returned error: 401, uniformly. The cause was in addkey-attempt.yaml's own curl task (legacy path), not PIA or any region: ansible.builtin.command's stdin argument appends a trailing newline by default (stdin_add_newline defaults to true), and --data-urlencode pt@- does not strip it — every server was receiving "<real token>\n" (URL-encoded, so a trailing %0A) as pt, a different and invalid value, and correctly rejecting it. Fixed with stdin_add_newline: false on that task. connect_to_wireguard_with_token .sh never hits this because it passes the token as a literal shell variable, not via stdin — the stdin delivery here is this repo's own addition (to keep the token out of ps), so it needed the flag PIA's reference script never had to think about. Left as a worked example because "every candidate failed identically" pointing at one shared bug in this code, not PIA, is exactly the diagnostic story this section promises — and it happened to be true the first time it was tested for real.

What it does

  • Registers minisforum as a PIA WireGuard peer — see "Region fallback" above — and writes /etc/wireguard/pia-wg.conf (Table = off — this role owns all routing for the interface, not wg-quick).

  • Installs pia-killswitch.service, ordered Before= wg-quick@pia-wg.service, that seeds a closed state at every boot (and after any UFW reload — see handlers/main.yaml): the from 10.10.50.0/24 lookup pia rule, a terminal unreachable default route in the pia table (only when pia-wg isn't already up — see the script's own comment for why), and a dedicated PIA-VLAN50 iptables chain jumped into by a single rule at the very top of FORWARD, matching only source 10.10.50.0/24, ending in an unconditional rate-limited log+drop. This is deliberately not a global FORWARD default-policy change — minisforum runs Flannel, which needs its own broad FORWARD ACCEPTs for pod traffic (source 10.42.0.0/16, disjoint from VLAN 50) — so the kill switch is scoped to a chain Flannel/k3s traffic can never enter, rather than risking a chain-wide policy that was never actually proven safe against it. Technitium (10.10.40.53, minisforum's own address) gets a normal UFW input allow for 53/tcp+udp — it's locally-terminated traffic, not routed — see tasks/firewall.yaml.

    The FORWARD jump's position is actively re-asserted on every run, not just seeded once — found live, 2026-09-22, on the very first deployment of the auto-recovery controller. minisforum's FORWARD chain default policy is DROP (kube-router's doing, not this role's — correcting an assumption an earlier version of this doc made), and kube-router and dockerd each independently reassert their own jump at position 1 whenever they resync or restart, silently pushing this rule down without removing it (kube-router doesn't even dedupe its own older copy when it does — a duplicate of its rule was found live too). The script used to check only whether its rule existed (iptables -C) before deciding not to touch anything; existence isn't proof of position, so once pushed down it stayed down until something else fixed it. It now checks the actual first rule and repositions (delete + reinsert at 1) if it isn't this one. Confirmed safe to do unconditionally by reading kube-router's and Docker's actual chain contents rather than assuming: KUBE-ROUTER-FORWARD only matches specific pod /32 addresses (disjoint from 10.10.50.0/24), DOCKER-USER was empty, and DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50 traffic arrives via enp1s0) — none of them can intercept VLAN 50 traffic regardless of order, so moving this rule back to position 1 relative to them changes nothing for pod or Docker-bridge traffic. The controller's own safety check (forward_jump_first) was working as intended when it refused to register against the drifted state — this is a fix to the thing it checks, not a loosening of the check itself.

    This recurs far more often than "boot-scale," and the fix above only helped once. The reposition logic is correct, but it only ran on the existing triggers (boot, a UFW reload, an Ansible-detected content change to the script). Confirmed live: after that fix landed and got one pia-gateway register attempt past this check, the very next attempt — roughly an hour later, no reboot, no UFW reload, no script change in between — hit the identical failure again. kube-router resyncs on ordinary Kubernetes API events (this cluster's CronJobs — ca-sync, porkbun-ddns, gitea-backup — each create and delete a Pod every few minutes, each a plausible trigger), not just at boot/restart scale as originally assumed. pia-killswitch-reassert .timer (every pia_healthcheck_interval_sec, unconditional, not gated by pia_auto_recovery_enabled) exists specifically to close this gap by giving the already-correct, already-idempotent script regular chances to run instead of only sporadic ones.

    The reassert timer alone wasn't the full fix either — it created a new failure mode. Confirmed live, 2026-09-22, minutes after enabling pia-gateway-recover.timer for the first time (rollout step 3): pia-gateway-recover and pia-killswitch-reassert are two independently-scheduled 60s timers with no coordination between them. kube-router turned out to resync (and re-push our jump down) roughly every 60-90s on this cluster — far more often than "rare, boot-scale" — so a recover pass would periodically sample iptables -S FORWARD in the window between a kube-router push-down and the next reassert fix, correctly detect real (if momentary) drift, and call emergency_close() — which unconditionally forces table pia's route to unreachable and resets consecutive_failures to 0 in the same step. The result was a self-reinforcing outage loop, observed directly in the journal:

    19:38:58  3rd consecutive unhealthy check → recovery_cycle → restart
    19:39:09  recovered via restart (10s) — route open, handshake fresh
    19:40:03  drift detected again ("FORWARD jump ... not first")
              → emergency close → route forced closed, counter reset to 0
    19:41:08  drift again → emergency close again → route stays closed
    

    VLAN 50 was open for roughly 10 seconds out of every 60-90 second cycle. Three consecutive real failures were needed to trigger the restart that reopens the route, but a fresh drift-and-emergency-close kept arriving before three could accumulate — the counter reset every time. This is not a bug in the detection (the drift was real, if transient) — it's that treating every momentary reordering as a full safety emergency was stricter than the actual invariant required, given kube-router's own resync cadence on this specific host.

    Fix: pia_forward_bypass_chains (Classifier ._forward_jump_effectively_first in files/pia_gateway.py). The safety check now tolerates a configured, named FORWARD chain sitting ahead of our jump — but only for as long as that chain's own current content re-verifies, live, on every single check, as structurally unable to match 10.10.50.0/24 (every -s/-d value on every rule disjoint from our subnet; a bare/unconditional rule, a negated match, or any construct the checker doesn't explicitly recognize fails closed, not open). This is not a one-time trust of kube-router — if its rules ever changed to include something that could actually intercept VLAN 50 traffic, or an unrecognized chain/rule appeared ahead of ours instead, it still fails closed exactly as before. Scoped to KUBE-ROUTER-FORWARD only (not DOCKER-USER/DOCKER-FORWARD — see pia_forward_bypass_chains's own comment in defaults/main.yaml for why only kube-router earned the exception). An emergency close and a hard stop remain the outcome for anything not on this narrow, content-reverified allowlist. Immediate live mitigation while this fix was being built: sudo systemctl stop pia-gateway-recover.timer, then a manual sudo systemctl restart wg-quick@pia-wg to reopen the route (pia-killswitch-reassert.timer was left running throughout — it was never the problem).

  • pia-wg.conf's own PostUp/PreDown open/close the narrower path on top of that closed baseline: default route via pia-wg in the pia table, ACCEPT + established/related return inserted into the PIA-VLAN50 chain (not FORWARD directly), and source-NAT/MASQUERADE scoped to 10.10.50.0/24 on pia-wg only.

  • Clamps TCP MSS on the pia-wg forward path (pia_mss_clamp_enabled: true, iptables -t mangle ... TCPMSS --clamp-mss-to-pmtu, scoped to SYN packets sourced from 10.10.50.0/24 outbound on pia-wg only, in PostUp/PreDown alongside the rules above). Confirmed needed live (2026-08-25), not enabled speculatively: a real download hung with a TLS read timeout on the larger handshake response while small requests worked fine, and a direct ping -M do -s 1450 test from a VLAN 50 pod confirmed the real path MTU is pia-wg's 1420 (the ICMP "Frag needed" reply arrives correctly from minisforum itself — our own side relays PMTU discovery fine, so the black hole is further out, on PIA's network or the remote server's own path, where clamping the MSS up front avoids needing that ICMP round-trip at all).

  • pia-wg.conf's PreDown lines are deliberately tolerant of the rule they're removing not existing (iptables -D ... 2>/dev/null || true) — enabling MSS clamping live (above) exposed why this matters: systemctl restart wg-quick@pia-wg tears down the currently running interface using whatever PreDown lines are on disk right now — if Ansible already rewrote the config with a new/changed rule before the restart handler fires, the live interface (brought up under the old rules) won't have whatever the new PreDown line is trying to delete. wg-quick's own execute_hooks() aborts the entire down/up sequence on the first failing hook (confirmed against its real source, not assumed), so one non-idempotent -D used to turn any future PostUp/PreDown content change into a broken restart and an orphaned interface (the never-reached built-in ip link delete step left pia-wg existing but unconfigured, which then made the following wg-quick up fail too with `pia-wg' already exists) — requiring manual recovery (ip link delete dev pia-wg before a fresh systemctl start). Every -D line now tolerates this by design, so a config change that alters PostUp/PreDown content can never break a restart this way again — confirmed live, 2026-08-27.

  • Installs an observability-only health check (pia-gateway-healthcheck .timer, every pia_healthcheck_interval_sec) that runs pia-gateway check — the same status classifier the recovery controller uses — and logs interface, handshake age, rule/route ordering and firewall state to the journal plus /var/lib/pia-gateway/status.json / metrics.prom. It never remediates. Neither it nor the controller sends a packet through the tunnel: a "healthy" class is a fresh handshake plus intact kill-switch invariants, not proof of egress identity.

  • Installs the automatic recovery controller and its timers — see "Automatic recovery" below.

What it deliberately does not do

  • Does not touch minisforum's own default route (asserted at the start of every run — tasks/assert-baseline.yaml fails loudly if that's already wrong).
  • Does not enable IPv6 forwarding or any IPv6 handling for VLAN 50.
  • Does not modify ansible/roles/wireguard (the separate wg0 home-VPN server role) or its interface.

Known failure mode: the session dies while the interface looks fine

Seen twice so far, ~28 days apart — 2026-08-25 (root-caused in INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md, Incident 1) and 2026-09-22. pia-wg's handshake with PIA's server stops renewing; the interface, wg-quick@pia-wg (active (exited) is normal for wg-quick), the pia routing table and all firewall rules stay exactly as PostUp left them. The kill switch's "interface down → unreachable route" trigger never fires because the interface never goes down, so VLAN 50 fails closed — correctly, but silently.

On 2026-08-25 a plain systemctl restart wg-quick@pia-wg did not produce a new handshake and only a fresh addKey registration did — strongly suggesting PIA had dropped the registered key server-side. The 2026-09-22 occurrence went straight to re-registration (bare restart untested). The underlying trigger is still not confirmed; the ~4-week gap is consistent with a fixed registration lifetime on PIA's side, which is what the proactive renewal timer below is for. This failure mode is now handled automatically — see the next section.

Automatic recovery

files/pia_gateway.py is installed as /usr/local/sbin/pia-gateway (root, 0700, runs on the host's python3.13). Almost entirely standard library — the one exception is addKey specifically, which shells out to curl (now an explicit apt dependency of this role); see HttpClient.get_pinned_curl's own docstring and "PIA's CA certificate and why addKey uses curl, not Python's ssl" below for why. It is one implementation of PIA registration shared by the unattended timers and by this role's own registration step, so the two can never drift. Layout on the host:

Path Purpose
/etc/pia-gateway/config.json (0600) rendered from role defaults (templates/pia-gateway-config.json.j2)
/etc/pia-gateway/credentials.env (0600) PIA_USER, PIA_PASSWORD, optional ALERT_BRIDGE_API_KEY — see "Credentials on the host"
/var/lib/pia-gateway/state.json (0600) versioned controller state (atomic tmp+fsync+rename); no secrets
/var/lib/pia-gateway/status.json, metrics.prom (0644) last classification, for humans/monitoring
/run/lock/pia-gateway.lock one flock shared by every mutating action (timers, register, rewrite-config); released by the kernel if a process dies
pia-gateway-recover.timer → .service every 60 s, one bounded recovery pass (recover)
pia-gateway-renew.timer → .service daily 03:30 ±30 min, proactive same-key renewal (renew); disabled by default (pia_renewal_enabled)
pia-gateway-onfailure.service OnFailure= of the two above: posts an error alert if a unit crashes or refuses to run
pia-gateway-healthcheck.timer unchanged cadence, now runs pia-gateway check
pia-killswitch-reassert.timer → .service every pia_healthcheck_interval_sec, re-runs pia-killswitch.sh (idempotent). Not gated by pia_auto_recovery_enabled — see the "FORWARD jump's position" note above and the service unit's own header for why

PIA's CA certificate and why addKey uses curl, not Python's ssl

Found live, 2026-09-22, on the very first pia-gateway register attempt after the FORWARD-jump fix above got past the kill-switch check: addKey failed against every candidate with kind=tls detail=certificate verification failed. Root-caused directly on minisforum:

$ sudo python3 -c '...ctx.wrap_socket(sock, server_hostname="Server-11669-10a")...'
SSLCertVerificationError verify_message= Basic Constraints of CA cert not marked critical

PIA's own CA certificate (files/pia-ca.crt) has a CA:TRUE Basic Constraints extension that isn't flagged critical — a real RFC 5280 non-conformance, confirmed not a stale vendored copy: a fresh fetch from pia-foss/manual-connections on GitHub is byte-identical (same 32e9b1d1... SHA-256 already recorded in the file's own header) and has the identical defect. This is PIA's problem, not ours, and not fixable by re-vendoring. Recent OpenSSL (3.2+; confirmed live against minisforum's 3.5.7) rejects this by default — a check that used to be opt-in only under X509_V_FLAG_X509_STRICT and has since been promoted into the default chain-verification path. openssl s_client on this repo's own dev machine (LibreSSL) and this same host's curl both accept the exact same certificate/chain/CA file without complaint — only Python's ssl module, using this host's system OpenSSL, rejects it. That's also why this went unnoticed until the auto-recovery work: the legacy Ansible path's three successful registrations (2026-08-24, 08-25, and this morning) all went through curl --cacert (addkey-attempt.yaml), never through Python's ssl module — this was the first time that code path ever ran for real.

Python's stdlib ssl module has no custom-verify-callback hook (a long-standing, deliberate upstream gap — unlike Go's tls.Config.VerifyPeerCertificate or OpenSSL's own C API), so there is no clean way to tolerate just this one known, third-party-controlled defect while still fully validating everything else (chain, expiry, hostname/SAN). Rather than weakening verify_mode for the whole connection or adding a new dependency (pyOpenSSL, for a real verify callback — considered, rejected as a bigger lift than necessary) to work around a gap in a library we don't control, addKey specifically now shells out to curl (HttpClient.get_pinned_curl, curl added to this role's apt install list) — the same tool PIA's own reference client (connect_to_wireguard_with_token.sh) already uses for this exact endpoint, with --cacert and never -k/--insecure, so this is a different, already-proven-correct TLS backend for the one endpoint that needs it, not a security downgrade. serverlist and the token endpoint are unaffected (they present public WebPKI certs, which don't have this problem) and still use the stdlib ssl path. Verified: a real subprocess curl call from this repo's dev machine against the actual failing candidate (Server-11669-10a / 69.33.198.97) gets a genuine HTTP response from PIA (401, expected — a dummy token was used), proving TLS validation succeeds via this path where it failed via Python's ssl. curl's own exit codes for a --fail-triggered rejection aren't fully portable either — confirmed live: a curl built against SecureTransport (this dev machine) reports exit 56, not the "usual" 22 GNU/OpenSSL curl uses for the identical case — so classification checks curl's own error text as a second signal alongside the exit code, not the exit code alone.

Status classes

Every pass classifies the gateway into exactly one of: healthy, boot_grace (stale within 120 s of a boot or of an intentional restart — not counted), paused, tunnel_stale (handshake 0 / older than pia_healthcheck_handshake_max_age_sec / in the future beyond 60 s of skew), tunnel_absent (interface down/missing, expected peer missing, fresh handshake but table route closed), registration_or_endpoint_failure / host_connectivity_failure (stale and in backoff after a cycle failed that way), safety_drift, unknown (a read command failed — no action taken, alert after 3). status.json carries both the decision class and the raw tunnel_class, so "paused" never hides a dead tunnel.

Safety invariants checked on every pass, all of which must hold before any restart/registration is attempted: the priority-100 source rule; a default in table pia that is either dev pia-wg (open) or unreachable (closed) — no default at all is a failure, not "closed"; the FORWARD jump to PIA-VLAN50 is the first effective rule (an emergency rule of our own is the only thing allowed above it); the chain ends in an unconditional DROP and every ACCEPT before it is scoped to pia-wg; the host default route is still via enp1s0.

Recovery state machine (recover, every 60 s)

State Action
healthy reset counters; if a backoff was pending, log/notify "healthy again"
1–2 consecutive stale/absent count only
3 consecutive (pia_recovery_failure_threshold) save pre-fix evidence (age, endpoint, reasons) → stage 1: systemctl restart wg-quick@pia-wg once, wait up to 90 s for two consecutive healthy classifications with a handshake newer than the attempt start
stage 1 fails stage 2: serverlist → token → addKey with the existing key against candidates in pia_region_candidates order (max pia_recovery_max_candidates_per_cycle), each accepted response staged as a validated config, activated, and verified the same way; a candidate that answers OK but never handshakes is skipped for the next one
stage 2: API accepted the key but nothing handshook stage 3: exactly one staged fresh-key attempt (pia-wg.key.new.<ts>.<pid>, promoted only after verification, previous key backed up). Skipped when the failure was uniform (auth error, host connectivity, every API call failing the same way)
any failure previous config restored (best effort) and re-activated; gateway stays closed; backoff 5 → 15 → 30 → 60 min (±10 % jitter, capped) persisted in state; classified failure recorded; alert
verified recovery stage, region/server, elapsed time and pre-fix age recorded in last_recovery; last_verified_registration updated when a registration happened; one "recovered" alert
safety drift no restart/registration. Insert -I FORWARD 1 -s 10.10.50.0/24 -m comment --comment pia-gateway-emergency -j DROP, then ip route replace unreachable default table pia, verify the DROP is first, alert error. If the rule already exists but isn't first any more (something else prepended a rule ahead of it, e.g. an out-of-band ufw reload), it's deleted and reinserted at position 1 rather than left where it drifted to — a mere existence check (iptables -C) says nothing about position. Left in place until a later pass sees the invariants intact, then removed under the lock. Automatic repair of the kill switch itself is deliberately not attempted (see "Residual risk")
paused (pia-gateway pause) checks and notifications continue; never restart/register/renew. Safety drift is still contained by the emergency close while paused — pausing suspends recovery, never the fail-closed guarantee
corrupt/unknown-schema state file refuse to act (exit 2, unit failure → alert). pia-gateway state-reset --confirm moves the file aside and starts fresh

One pass is bounded by pia_recovery_max_duration_sec (600 s) inside the program and TimeoutStartSec (900 s) in the unit; systemd never starts a second instance while one runs, and the lock makes a manual register wait (up to pia_recovery_lock_wait_sec) rather than interrupt an activation. Reboots: the boot id is tracked, a new boot gets pia_recovery_startup_grace_sec before stale counts; retry/backoff state survives because it is on disk. A backward wall-clock step (NTP) that leaves a persisted retry deadline further out than the backoff cap is treated as a clock jump and the deadline is cleared rather than stalling recovery.

restart returning 0, active (exited), or addKey returning status: OK are never treated as success on their own — only an observed fresh handshake on the activated config is, and specifically one reported strictly after the moment activation started: a handshake in the same whole second as that moment doesn't count either, since wg show only has one-second resolution and there'd be no way to tell it apart from a stale, pre-activation handshake that happens to share the second.

Every mutating command — recover, renew, register, rewrite-config, pause, resume, state-reset — takes the same /run/lock/pia-gateway.lock before reading state, not after: reading state first and only locking around the write would let a slower command silently overwrite whatever a faster one wrote in between (in particular, it's how a pause could otherwise be lost to an in-flight recovery pass that read state before the pause ran and only saves its own copy back once it finishes). pause/resume/register/rewrite-config/ state-reset wait up to pia_recovery_lock_wait_sec for the lock and exit 4 if it's still busy; recover/renew (the timers) never wait — they just skip that tick if something else currently holds it.

Operator commands (all on minisforum, as root)

pia-gateway status                      # human summary incl. backoff, last verified registration, pause
pia-gateway check                       # what the health-check timer runs (read-only, exit 1 = unhealthy)
pia-gateway check --json                # machine-readable
pia-gateway pause --reason "maintenance"   # persists across reboot; suppresses recover AND renew; kill switch untouched
pia-gateway resume
pia-gateway retry-now                   # one recovery pass ignoring threshold and backoff
pia-gateway register [--mode same-key|new-key] [--region hk]   # explicit transaction, verified, under the lock
pia-gateway rewrite-config              # re-render pia-wg.conf from the recorded registration (hook changes), no API call
pia-gateway renew --force-due           # exercise renewal without waiting 21 days
pia-gateway state-reset --confirm       # only after reading the corrupt file it moves aside
journalctl -u pia-gateway-recover -u pia-gateway-renew -u pia-gateway-healthcheck -n 100
cat /var/lib/pia-gateway/status.json

Through Ansible (repo root, -K -J):

ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml -K -J   # idempotent; registers only if no config or no verified baseline
ansible-playbook ... -e pia_force_reregister=true        # re-register with the EXISTING key (same-key)
ansible-playbook ... -e pia_force_key_rotation=true      # stage a NEW key, register it, promote only after verification
ansible-playbook ... -e pia_region=hk                    # single-region override (either of the above)
ansible-playbook ... -e pia_auto_recovery_enabled=false  # stop+disable both timers, keep everything else (rollback)
ansible-playbook ... -e pia_renewal_enabled=true         # after the live drills — enables the daily renewal timer
ansible-playbook ... -e pia_remove_host_credentials=true # deliberate removal of /etc/pia-gateway/credentials.env

pia_force_reregister reuses the on-disk key; only pia_force_key_rotation generates a new one (the root README used to say otherwise — corrected). A normal run also quiesces both timers, waits for any in-flight pass, and re-enables them after handlers, so an Ansible restart never races an automatic one. A run whose only change is config.json (e.g. flipping pia_mss_clamp_enabled) re-renders the WireGuard config via rewrite-config without touching PIA — the "template-only change needs a full re-registration" gap from the 2026-08-25 incident is closed.

Proactive renewal (renew)

Off until enabled. When on, once a day it re-registers with the existing key if last_verified_registration is older than pia_renewal_after_days (21) — or missing, which is why the baseline must come from a verified run of the shared path, never from a file mtime. Same transaction, same lock, same verification. A failed renewal leaves the baseline untouched and enters the same bounded backoff (it does not silently wait another 21 days); if the tunnel is already unhealthy it defers to the recovery path. Renewal restarts pia-wg, so qBittorrent/JDownloader connections drop for a few seconds at ~03:30 — verify they reconnect during the drill. Routine key rotation is not done: rotation happens only in recovery stage 3, on evidence.

Alerts and monitoring

Events (not every 60 s): recovery started, recovered, recovery failed + backoff (warn; error for auth failure), safety drift (error), emergency rule removed, renewal failed, status unknown ×3, state corrupt, unit crashed (via OnFailure). Delivery is POST to pia_alert_bridge_url with the shared alert-bridge bearer key ({"source": "pia-gateway@minisforum", "message": ..., "level": ...} — schema read from ~/repo/home-service/alert-bridge, which has no per-caller tokens, so the cluster's key is what gets provisioned; accepted tradeoff). 5 s timeout; failures are queued in state (bounded, 20) and flushed on the next pass; they never block recovery.

metrics.prom is node-exporter textfile format (pia_gateway_healthy, _tunnel_ok, _safety_ok, _handshake_age_seconds, _consecutive_failures, _paused, _recovery_in_backoff, _last_verified_registration_timestamp_seconds, _check_timestamp_seconds, _emergency_close_active). Not yet scraped: values/kube-prometheus-stack.yaml has no textfile collector and node-exporter runs as a pod, so wiring it needs a hostPath mount + --collector.textfile.directory plus a PrometheusRule including a stale-check rule (time() - pia_gateway_check_timestamp_seconds > 300) so a stopped timer alerts even when the last value was healthy. Until that exists, a dead controller is only visible as silence — check systemctl list-timers 'pia-gateway-*' and status.json's timestamp.

Credentials on the host

/etc/pia-gateway/credentials.env holds the PIA account username and password in plaintext, root-only. This is the explicit tradeoff the design requires: recovery must work while the laptop is off, and PIA's token is only valid for 24 h, so an unattended host needs a renewable credential. The program reads the file at runtime and never places values in argv, environment, logs, notifications, metrics or state; core dumps are disabled (LimitCORE=0) and unexpected exceptions are logged by class name only. Rotation: change PIA_PASSWORD in .env, run the playbook (the file is re-rendered with no_log), then pia-gateway register (or wait for the next cycle). Revocation: -e pia_remove_host_credentials=true removes the file (recovery then fails closed with auth/credentials errors and alerts); change the PIA password; disabling pia_auto_recovery_enabled alone never deletes it. If an on-host secret is unacceptable, the alternative is an always-on external secret broker — not a 24-hour token.

Residual risk (read before trusting it unattended)

  • Safety drift is contained by the emergency DROP + unreachable route, not repaired: the kill switch's own rule/jump can be deleted by something outside this role (a manual ufw reload flushes FORWARD, removing both the jump and the emergency rule). The controller re-inserts the emergency rule on its next pass (≤ 60 s) and every boot re-seeds the kill switch; that 60 s window after an out-of-band flush is the residual leak risk, and it predates this work.
  • Verification is handshake-based. It proves the tunnel is talking to PIA, not that a VLAN 50 packet leaves via PIA; the end-to-end check is the SOCKS-proxy/egress-IP step in the live drill.
  • The alert-bridge key on minisforum is the cluster's shared key.
  • unknown (a read command failing) is logged/alerted but never acted on, by design.

Rollout status (2026-09-23)

Steps 1-3 are done and verified live. Step 3 required two attempts — the first caused a real (safely fail-closed) outage, root-caused and fixed same-day; the second was verified against a full day of real production conditions, not just a short window.

  • Step 2, attempt 1 failed: pia-gateway register correctly refused with safety invariants failing ... FORWARD jump to kill-switch chain is not first — a real, pre-existing drift (see the "FORWARD jump's position is actively re-asserted" note above), not a false alarm from this session's work, and not the "session dies" handshake issue either. Root-caused and fixed in pia-killswitch.sh.j2.
  • Attempt 2, after that fix, got past the kill-switch check and failed differently: addKey rejected every candidate with kind=tls detail=certificate verification failed. Root-caused and fixed — see "PIA's CA certificate and why addKey uses curl, not Python's ssl" above.
  • Attempt 3, roughly an hour after attempt 1's fix with no reboot, UFW reload, or script change in between, hit the identical FORWARD- jump failure again — proving the drift recurs on a timescale of minutes under this cluster's normal CronJob/pod churn, not just at boot-scale events as originally assumed. Unblocked live with a manual sudo systemctl restart pia-killswitch.service; pia-killswitch -reassert.timer added as the durable fix.
  • Attempt 4 succeeded cleanly end to end: kill-switch check passed, addKey accepted (hk/Server-11669-10a), and a fresh handshake was verified (age 10s) — the first-ever successful shared-path registration. pia-gateway status confirmed healthy/safety_ok: True, iptables -S FORWARD confirmed the jump first, and pia-killswitch-reassert.timer confirmed cycling correctly (7 successful runs, ~60s apart, in the journal). Step 2 is done.
  • Step 3 (ansible-playbook ... -K -J, no override — enables pia-gateway-recover.timer for the first time) deployed cleanly (Run the shared PIA registration transaction correctly skipped, a verified baseline already existed) but caused a real live outage within minutes: pia-gateway-recover and pia-killswitch-reassert, two independent 60s timers, raced — kube-router's actual resync cadence (~60-90s) turned out to be far more frequent than assumed, and every momentary drift-then-self-correct was treated as a full safety emergency, forcibly closing the route and resetting the 3-strikes counter before it could accumulate. VLAN 50 was open for ~10s out of every 60-90s cycle for several minutes, confirmed via Discord alerts from alert-bridge (working end to end, first real trigger) and the full pia-gateway-recover/pia-killswitch-reassert journals. Mitigated live: sudo systemctl stop pia-gateway-recover.timer + manual sudo systemctl restart wg-quick@pia-wg. Root-caused and fixed same-day — see the "reassert timer alone wasn't the full fix" addendum above (pia_forward_bypass_chains, content-verified live on every check).
  • Step 3, re-attempt, with the bypass fix deployed (ansible-playbook ... -K -J; config re-rendered with forward_bypass_chains: ["KUBE-ROUTER-FORWARD"], rewrite-config correctly re-rendered pia-wg.conf from the config-only change without a PIA call since a baseline already existed, recovery timer re-enabled). Left running unattended for a full ~25.5 hours before re-checking, deliberately — the original incident took several minutes and multiple cycles to manifest, so a short recheck window would have proven nothing. Confirmed via exact counts, not inference: journalctl -u pia-gateway-recover --since "2026-09-22 20:37:00" showed 1528 class=healthy passes (matches the expected ~1530 at 60s cadence — the timer ran continuously the whole window), 0 SAFETY DRIFT lines, 0 class=safety_drift classifications. pia-gateway status at recheck time: healthy, safety_ok: True, emergency_close_active: False, consecutive_failures: 0, last recovery: 1d ago restart — the last restart was the one from before this fix; none has been needed since. Step 3 is done. kube-router did keep resyncing during this window (that's the whole premise of the fix); the bypass tolerance handled it silently every time, with no observable effect on the tunnel.

Use iptables -S FORWARD (no head) for the baseline capture below — the original head -3 suggestion is what hid the first issue; the drift was 4 rules in, not at the top. Use head -2, not head -1, for any quick re-check — -S FORWARD's first line is always the -P policy line, never a real rule (this cost a round-trip during the investigation above).

  1. Baseline on minisforum: wg show pia-wg, ip rule, ip route show table pia, iptables -S FORWARD, iptables -S PIA-VLAN50, journalctl -u pia-gateway-healthcheck -n 5, curl --socks5-hostname 10.10.40.20:1080 https://ifconfig.me from the LAN, wg show wg0, kubectl get nodes. Confirm console access.

  2. Deploy with timers off: ansible-playbook ... -K -J -e pia_auto_recovery_enabled=false. Check ls -l /etc/pia-gateway /var/lib/pia-gateway, pia-gateway status, systemctl list-timers 'pia-gateway-*' (recover/renew inactive). Because the host has no verified baseline yet, this run registers through the shared path (same-key, explicit restart + handshake verification) — watch the task output; if it fails, -e pia_registration_backend=legacy restores the previous behaviour. If it fails specifically with FORWARD jump to kill-switch chain is not first, that's the drift above — the fixed template should correct it via the normal "Restart pia-killswitch" handler on the same run (the template content changing is enough to trigger it); re-running the identical command should then get past registration. Confirm with iptables -S FORWARD afterwards that the jump is genuinely first, not just that the retry succeeded.

  3. Enable: ansible-playbook ... -K -J (recovery on; also redeploys pia_forward_bypass_chains into config.json if not already applied — confirm with sudo pia-gateway --config /etc/pia-gateway/config.json status or just cat /etc/pia-gateway/config.json). Watch longer than one pass — the live incident here took 2-3 cycles (several minutes) to manifest, not the first check. journalctl -u pia-gateway-recover -f for at least 5-10 minutes; expect class= healthy throughout, with zero SAFETY DRIFT lines, even while pia-killswitch-reassert keeps cycling in parallel (journalctl -u pia-killswitch-reassert -f alongside it). If a bypass chain KUBE-ROUTER-FORWARD content no longer verified safe reason ever appears, treat it as real — it means kube-router's rules changed to include something that could actually match VLAN 50 traffic, not a false positive to relax further.

  4. Drill the silent death. First get the live WireGuard UDP endpoint from the interface itself — not port 1337, which is only the addKey registration API port and has nothing to do with the tunnel's own data-plane port (PIA WireGuard servers commonly use a different port per server):

    sudo wg show pia-wg endpoints   # <peer-pubkey>\t<ip>:<port> — this ip:port is the block target
    

    Then, on minisforum, leave pia-wg administratively up and drop only that path:

    sudo iptables -I OUTPUT -d <endpoint ip> -p udp --dport <endpoint port> -j DROP
    

    Expect: 3 stale passes → restart → (still blocked) registration to a different server → recovery; record times from the journal. To test backoff, block every candidate server before recovery can reach it — since each candidate's actual WireGuard port isn't known until its own addKey response arrives (only the registration port, 1337, is fixed and shared), block by IP alone rather than guessing a port:

    sudo iptables -I OUTPUT -d <candidate ip> -p udp -j DROP   # repeat per candidate IP, or -e pia_region=hk to narrow the set first
    

    Remove the rule(s) and confirm autonomous recovery. iptables -D every injected rule afterwards — check iptables -S OUTPUT is back to its pre-drill state, not just that the tunnel recovered.

  5. Exercise: a failed first server (block one IP only), invalid credentials (temporarily wrong PIA_PASSWORD → expect auth_failure, no rotation, error alert), alert-bridge down, pia-gateway register while a pass runs (expect it to wait), reboot with a pending backoff, pause/resume.

  6. During every transition, capture on enp1s0 with an explicit egress-only direction filter:

    sudo tcpdump -ni enp1s0 -Q out src net 10.10.50.0/24
    

    -Q out matters: VLAN 50 traffic arriving at minisforum on enp1s0 (from the Flint router, for policy routing into table pia) is normal and expected — a plain src net 10.10.50.0/24 filter with no direction would match that inbound traffic too and could hide a real leak in the noise, or be misread as one. Only a hit in the out direction — a VLAN 50-sourced packet actually being transmitted out enp1s0 toward the LAN/Internet instead of via pia-wg — is a leak, and this must stay completely silent through every failure and recovery transition. Also capture on pia-wg itself to confirm traffic is flowing there while open. Verify a request through 10.10.40.20:1080 and qBittorrent/JDownloader afterwards; check IPv6 separately; confirm wg0, the host default route, Technitium and k3s.

  7. Only then -e pia_renewal_enabled=true; pia-gateway renew --force-due; confirm last_verified_registration.epoch advances only on verification. Wire the Prometheus stale-check before calling the feature done. Record actual times and captures in the incident file.

Rollback

Partial rollback (keep the gateway, drop only the automation): ansible-playbook ... -e pia_auto_recovery_enabled=false stops and disables both timers and leaves the program, credentials, health check, kill switch and pia-wg in place. Add -e pia_remove_host_credentials=true to also delete the on-host PIA credentials; nothing else deletes them.

No tag-based full rollback is defined — this role doesn't have an "absent" mode. Roll back manually on minisforum (reverses this role without touching wg0, k3s, or the host default route):

sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer   # controller first, so it cannot react to the teardown
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-killswitch-reassert.timer pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \
           /etc/systemd/system/pia-killswitch-reassert.service \
           /etc/systemd/system/pia-killswitch-reassert.timer \
           /etc/systemd/system/pia-gateway-healthcheck.service \
           /etc/systemd/system/pia-gateway-healthcheck.timer \
           /etc/systemd/system/pia-gateway-recover.service \
           /etc/systemd/system/pia-gateway-recover.timer \
           /etc/systemd/system/pia-gateway-renew.service \
           /etc/systemd/system/pia-gateway-renew.timer \
           /etc/systemd/system/pia-gateway-onfailure.service
sudo systemctl daemon-reload
sudo iptables -D FORWARD -s 10.10.50.0/24 -m comment --comment pia-gateway-emergency -j DROP 2>/dev/null || true
sudo rm -rf /etc/pia-gateway /var/lib/pia-gateway /usr/local/sbin/pia-gateway /run/lock/pia-gateway.lock   # includes the on-host PIA credentials
sudo ip rule del from 10.10.50.0/24 table pia priority 100
sudo ip route flush table pia
sudo sed -i '/^[0-9]\+\s\+pia$/d' /etc/iproute2/rt_tables
sudo iptables -D FORWARD -s 10.10.50.0/24 -j PIA-VLAN50
sudo iptables -F PIA-VLAN50
sudo iptables -X PIA-VLAN50
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp
sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.conf.bak-* /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.bak-* /etc/wireguard/pia-wg.key.new* /etc/wireguard/pia-ca.crt

Inspection commands to confirm rollback actually took (read-only):

sudo iptables -S FORWARD | grep PIA-VLAN50   # expect: no output
sudo iptables -L PIA-VLAN50                  # expect: "iptables: No chain/target/match by that name"
sudo ufw status verbose | grep 10.10.50      # expect: no output
ip rule show | grep pia                      # expect: no output
ip route show table pia                      # expect: empty/error (table gone)

Does not remove pia-credentials/pia-credentials-sealed.yaml (the K8s Secret used by the Gluetun sidecars) — that's a separate, unrelated secret and rollback path. Does not touch ansible/roles/common's own UFW rules (Flannel pod-to-pod, pod-to-Technitium) — this role never modified those.