Nik Afiq 80bf24cf0b docs: record step 3 as verified live over a full 25.5-hour window
pia_forward_bypass_chains held up against real production conditions,
not just a short recheck: 1528 healthy passes, 0 SAFETY DRIFT events,
0 safety_drift classifications since the fix was deployed
(2026-09-22 20:37 -> 2026-09-23 22:02). kube-router kept resyncing
throughout; the bypass tolerance handled it silently every time.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 22:09:13 +09:00

828 lines
50 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# pia-gateway
Makes `minisforum` a PIA WireGuard egress gateway for VPN VLAN 50
(`10.10.50.0/24`), without changing its own default route. Implements
Phase 2 of `~/repo/homelab/plan.md`.
## Before the first real run
1. **The PIA API calls have gone through three rounds of correction
against real live testing (2026-08-24). They now live in
`files/pia_gateway.py` (`PiaApi`), with the original in-Ansible flow
kept as `tasks/register-legacy.yaml` until the shared path is proven
live — see "Automatic recovery" below:**
- `serverlist.piaservers.net/vpninfo/servers/v6`: works. Region values
everywhere in this role are the API's **id** field (e.g. `japan`,
`hk`, `taiwan`), never a display name ("JP Tokyo", "Hong Kong",
"Taiwan" — those are Gluetun's labels, not something this role or
PIA's API accepts).
- Token acquisition originally guessed a regional-meta-server
endpoint (`/authv3/generateToken`) that turned out not to be the
real flow at all — it was rebuilt from PIA's actual
`pia-foss/manual-connections` `get_token.sh` (fetched and read in
full): a single fixed `POST
https://www.privateinternetaccess.com/api/client/v2/token`,
multipart `username`/`password`, **normal system CA validation**
(no `--cacert`, no `validate_certs: false`), independent of region
entirely — only WireGuard server *selection* is regional. Not yet
exercised with real credentials end to end.
- `{wg}:1337/addKey` (WireGuard key registration) — confirmed against
`connect_to_wireguard_with_token.sh` (fetched and read in full):
this one genuinely does need PIA's own CA bundle
(`files/pia-ca.crt`, `--cacert`) and `--connect-to`. Originally
failed over only across servers *within* a single fixed region
(`pia_region: japan`) — in practice all three JP Tokyo servers
failed the same way, one after another, so failover now also
crosses regions: every server in Hong Kong, then every server in
Taiwan, then every server in JP Tokyo (`pia_region_candidates` in
`defaults/main.yaml`), stopping at the first `status: OK`. Token
passed via stdin rather than argv. See `tasks/addkey-attempt.yaml`
and "Region fallback" below.
2. Make sure the repo-root `.env` has `PIA_USER`/`PIA_PASSWORD` set (same
keys `manifests/media/pia-secret.sh` already reads) — `ansible/
host_vars/minisforum.yaml` reads them directly from there via a
`lookup('ansible.builtin.file', ...)` + regex, deliberately not
ansible-vault. The lookup runs on the control node; `.env` itself is
never copied and never committed (gitignored). **Since the
auto-recovery work, only `PIA_USER`/`PIA_PASSWORD` (and, if present,
`ALERT_BRIDGE_API_KEY`) are provisioned to minisforum's root-only
`/etc/pia-gateway/credentials.env`** so the host can re-register
unattended — see "Credentials on the host" below. The playbook still
needs `-J` because `group_vars/all/vault.yaml` is vault-encrypted and
loaded for every host, not because the PIA credentials are in it.
3. Confirm console/recovery access to `minisforum` (IPMI/physical/other
out-of-band path) before applying — this role changes host routing and
firewall policy. If SSH becomes unreachable, use that path.
4. `ufw status verbose` and `wg show` (existing `wg0`) were not inspected
live before writing this role (no passwordless sudo in discovery) —
spot-check that `wg0` (the separate home-VPN role) is unaffected after
applying.
## Region fallback
Registration tries every WireGuard server in every configured region, in
order, stopping at the first `status: OK`:
```text
hk (Hong Kong):
wg server 1
wg server 2
taiwan (Taiwan):
wg server 1
wg server 2
japan (JP Tokyo):
wg server 1
wg server 2
wg server 3
```
The actual number and order of servers within each region always comes
from the live serverlist response — never hardcoded here.
**Automatic fallback** (default — tries `pia_region_candidates` in
order):
```bash
ansible-playbook -i ansible/inventory.yaml \
ansible/playbooks/pia-gateway.yaml -K -J
```
**Forced single region**, for troubleshooting one region in isolation —
bypasses `pia_region_candidates` entirely, does not fall back to the
others:
```bash
ansible-playbook -i ansible/inventory.yaml \
ansible/playbooks/pia-gateway.yaml -K -J \
-e pia_region=hk
```
Both `pia_region` and every entry in `pia_region_candidates` are PIA API
**ids** — `hk`, `taiwan`, `japan` — not the display names Gluetun's
`SERVER_REGIONS` config uses ("Hong Kong", "Taiwan", "JP Tokyo"). Passing
a display name here matches nothing and fails the "region exists"
assertion before any network call is even made.
Which region/server actually ended up registered is recorded as
`pia_region_used`/`pia_region_name_used`/`pia_wg_server_used`, written
into `pia-wg.conf`'s own `[Peer]` comment, and surfaced by the health
check (`INFO configured-peer: ...`) — it is whichever one answered first,
not necessarily `pia_region_candidates[0]`.
If every region fails: `journalctl` (or the play's own output) has one
`debug` line per failed attempt, each showing region name+id, server
hostname+IP, curl exit code, a classified reason (connection timeout /
TLS-SSL failure / HTTP failure / empty response / malformed JSON /
parsed-but-not-OK status, never a generic catch-all), and sanitized
stderr — never the token, credentials, the full curl invocation, request
stdin, or a complete response body. If Hong Kong and Taiwan fail exactly
the same way Tokyo did, that's a strong signal the problem is shared
across all three (the request shape, the account credentials, or
minisforum's own network path) rather than one region being down — see
`tasks/register-legacy.yaml`'s header comment (the controller logs the
same per-candidate `kind=`/`detail=` classification).
**Worked example (2026-08-24):** exactly that happened — all 7 servers
across all 3 regions came back `HTTP failure (non-2xx response)`, curl
exit 22, `The requested URL returned error: 401`, uniformly. The cause
was in `addkey-attempt.yaml`'s own curl task (legacy path), not PIA or any region:
`ansible.builtin.command`'s `stdin` argument appends a trailing newline
by default (`stdin_add_newline` defaults to `true`), and
`--data-urlencode pt@-` does not strip it — every server was receiving
`"<real token>\n"` (URL-encoded, so a trailing `%0A`) as `pt`, a
different and invalid value, and correctly rejecting it. Fixed with
`stdin_add_newline: false` on that task. `connect_to_wireguard_with_token
.sh` never hits this because it passes the token as a literal shell
variable, not via stdin — the stdin delivery here is this repo's own
addition (to keep the token out of `ps`), so it needed the flag PIA's
reference script never had to think about. Left as a worked example
because "every candidate failed identically" pointing at one shared bug
in *this* code, not PIA, is exactly the diagnostic story this section
promises — and it happened to be true the first time it was tested for
real.
## What it does
- Registers minisforum as a PIA WireGuard peer — see "Region fallback"
above — and writes `/etc/wireguard/pia-wg.conf` (`Table = off` — this
role owns all routing for the interface, not wg-quick).
- Installs `pia-killswitch.service`, ordered `Before=
wg-quick@pia-wg.service`, that seeds a closed state at every boot (and
after any UFW reload — see handlers/main.yaml): the
`from 10.10.50.0/24 lookup pia` rule, a terminal `unreachable default`
route in the `pia` table (only when `pia-wg` isn't already up — see
the script's own comment for why), and a dedicated `PIA-VLAN50`
iptables chain jumped into by a single rule at the very top of
`FORWARD`, matching only source `10.10.50.0/24`, ending in an
unconditional rate-limited log+drop. This is deliberately **not** a
global `FORWARD` default-policy change — minisforum runs Flannel,
which needs its own broad `FORWARD` ACCEPTs for pod traffic (source
`10.42.0.0/16`, disjoint from VLAN 50) — so the kill switch is scoped
to a chain Flannel/k3s traffic can never enter, rather than risking a
chain-wide policy that was never actually proven safe against it.
Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW
**input** allow for 53/tcp+udp — it's locally-terminated traffic, not
routed — see `tasks/firewall.yaml`.
**The FORWARD jump's position is actively re-asserted on every run, not
just seeded once** — found live, 2026-09-22, on the very first
deployment of the auto-recovery controller. minisforum's `FORWARD`
chain default policy is `DROP` (kube-router's doing, not this role's —
correcting an assumption an earlier version of this doc made), and
kube-router and `dockerd` each independently reassert their **own**
jump at position 1 whenever they resync or restart, silently pushing
this rule down without removing it (kube-router doesn't even dedupe
its own older copy when it does — a duplicate of *its* rule was found
live too). The script used to check only whether its rule *existed*
(`iptables -C`) before deciding not to touch anything; existence isn't
proof of position, so once pushed down it stayed down until something
else fixed it. It now checks the actual first rule and repositions
(delete + reinsert at 1) if it isn't this one. Confirmed safe to do
unconditionally by reading kube-router's and Docker's actual chain
contents rather than assuming: `KUBE-ROUTER-FORWARD` only matches
specific pod `/32` addresses (disjoint from `10.10.50.0/24`),
`DOCKER-USER` was empty, and `DOCKER-FORWARD`'s one direct rule is
scoped to `-i docker0` (VLAN 50 traffic arrives via `enp1s0`) — none
of them can intercept VLAN 50 traffic regardless of order, so moving
this rule back to position 1 relative to them changes nothing for pod
or Docker-bridge traffic. The controller's own safety check
(`forward_jump_first`) was working as intended when it refused to
register against the drifted state — this is a fix to the thing it
checks, not a loosening of the check itself.
**This recurs far more often than "boot-scale," and the fix above only
helped once.** The reposition logic is correct, but it only *ran* on
the existing triggers (boot, a UFW reload, an Ansible-detected content
change to the script). Confirmed live: after that fix landed and got
one `pia-gateway register` attempt past this check, the very next
attempt — roughly an hour later, no reboot, no UFW reload, no script
change in between — hit the identical failure again. kube-router
resyncs on ordinary Kubernetes API events (this cluster's CronJobs —
`ca-sync`, `porkbun-ddns`, `gitea-backup` — each create and delete a
Pod every few minutes, each a plausible trigger), not just at
boot/restart scale as originally assumed. `pia-killswitch-reassert
.timer` (every `pia_healthcheck_interval_sec`, unconditional, not
gated by `pia_auto_recovery_enabled`) exists specifically to close this
gap by giving the already-correct, already-idempotent script regular
chances to run instead of only sporadic ones.
**The reassert timer alone wasn't the full fix either — it created a
new failure mode.** Confirmed live, 2026-09-22, minutes after enabling
`pia-gateway-recover.timer` for the first time (rollout step 3):
`pia-gateway-recover` and `pia-killswitch-reassert` are two
independently-scheduled 60s timers with no coordination between them.
kube-router turned out to resync (and re-push our jump down) roughly
every 60-90s on this cluster — far more often than "rare, boot-scale"
— so a `recover` pass would periodically sample `iptables -S FORWARD`
in the window between a kube-router push-down and the next reassert
fix, correctly detect real (if momentary) drift, and call
`emergency_close()` — which unconditionally forces table `pia`'s route
to `unreachable` and resets `consecutive_failures` to 0 in the same
step. The result was a self-reinforcing outage loop, observed directly
in the journal:
```
19:38:58 3rd consecutive unhealthy check → recovery_cycle → restart
19:39:09 recovered via restart (10s) — route open, handshake fresh
19:40:03 drift detected again ("FORWARD jump ... not first")
→ emergency close → route forced closed, counter reset to 0
19:41:08 drift again → emergency close again → route stays closed
```
VLAN 50 was open for roughly 10 seconds out of every 60-90 second
cycle. Three consecutive real failures were needed to trigger the
restart that reopens the route, but a fresh drift-and-emergency-close
kept arriving before three could accumulate — the counter reset every
time. This is not a bug in the detection (the drift was real, if
transient) — it's that treating *every* momentary reordering as a full
safety emergency was stricter than the actual invariant required,
given kube-router's own resync cadence on this specific host.
**Fix: `pia_forward_bypass_chains` (`Classifier
._forward_jump_effectively_first` in `files/pia_gateway.py`).** The
safety check now tolerates a configured, named FORWARD chain sitting
ahead of our jump — but only for as long as that chain's *own current
content* re-verifies, live, on every single check, as structurally
unable to match `10.10.50.0/24` (every `-s`/`-d` value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct the checker doesn't explicitly recognize fails
closed, not open). This is not a one-time trust of kube-router — if
its rules ever changed to include something that could actually
intercept VLAN 50 traffic, or an unrecognized chain/rule appeared
ahead of ours instead, it still fails closed exactly as before. Scoped
to `KUBE-ROUTER-FORWARD` only (not `DOCKER-USER`/`DOCKER-FORWARD` —
see `pia_forward_bypass_chains`'s own comment in `defaults/main.yaml`
for why only kube-router earned the exception). An emergency close and
a hard stop remain the outcome for anything not on this narrow,
content-reverified allowlist. Immediate live mitigation while this fix
was being built: `sudo systemctl stop pia-gateway-recover.timer`, then
a manual `sudo systemctl restart wg-quick@pia-wg` to reopen the route
(`pia-killswitch-reassert.timer` was left running throughout — it was
never the problem).
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
top of that closed baseline: default route via `pia-wg` in the `pia`
table, ACCEPT + established/related return inserted into the
`PIA-VLAN50` chain (not `FORWARD` directly), and source-NAT/MASQUERADE
scoped to `10.10.50.0/24` on `pia-wg` only.
- Clamps TCP MSS on the `pia-wg` forward path (`pia_mss_clamp_enabled:
true`, `iptables -t mangle ... TCPMSS --clamp-mss-to-pmtu`, scoped to
SYN packets sourced from `10.10.50.0/24` outbound on `pia-wg` only, in
`PostUp`/`PreDown` alongside the rules above). Confirmed needed live
(2026-08-25), not enabled speculatively: a real download hung with a
TLS read timeout on the larger handshake response while small
requests worked fine, and a direct `ping -M do -s 1450` test from a
VLAN 50 pod confirmed the real path MTU is `pia-wg`'s `1420` (the ICMP
"Frag needed" reply arrives correctly from minisforum itself — our own
side relays PMTU discovery fine, so the black hole is further out, on
PIA's network or the remote server's own path, where clamping the MSS
up front avoids needing that ICMP round-trip at all).
- `pia-wg.conf`'s `PreDown` lines are deliberately tolerant of the rule
they're removing not existing (`iptables -D ... 2>/dev/null || true`)
— enabling MSS clamping live (above) exposed why this matters:
`systemctl restart wg-quick@pia-wg` tears down the *currently running*
interface using whatever `PreDown` lines are on disk *right now* — if
Ansible already rewrote the config with a new/changed rule before the
restart handler fires, the live interface (brought up under the *old*
rules) won't have whatever the new `PreDown` line is trying to delete.
`wg-quick`'s own `execute_hooks()` aborts the entire down/up sequence
on the first failing hook (confirmed against its real source, not
assumed), so one non-idempotent `-D` used to turn any future
`PostUp`/`PreDown` content change into a broken restart *and* an
orphaned interface (the never-reached built-in `ip link delete` step
left `pia-wg` existing but unconfigured, which then made the
following `wg-quick up` fail too with `` `pia-wg' already exists``) —
requiring manual recovery (`ip link delete dev pia-wg` before a fresh
`systemctl start`). Every `-D` line now tolerates this by design, so a
config change that alters `PostUp`/`PreDown` content can never break
a restart this way again — confirmed live, 2026-08-27.
- Installs an observability-only health check (`pia-gateway-healthcheck
.timer`, every `pia_healthcheck_interval_sec`) that runs `pia-gateway
check` — the same status classifier the recovery controller uses — and
logs interface, handshake age, rule/route ordering and firewall state to
the journal plus `/var/lib/pia-gateway/status.json` / `metrics.prom`.
It never remediates. Neither it nor the controller sends a packet
through the tunnel: a "healthy" class is a fresh handshake plus intact
kill-switch invariants, not proof of egress identity.
- Installs the automatic recovery controller and its timers — see
"Automatic recovery" below.
## What it deliberately does not do
- Does not touch minisforum's own default route (asserted at the start of
every run — `tasks/assert-baseline.yaml` fails loudly if that's already
wrong).
- Does not enable IPv6 forwarding or any IPv6 handling for VLAN 50.
- Does not modify `ansible/roles/wireguard` (the separate `wg0` home-VPN
server role) or its interface.
## Known failure mode: the session dies while the interface looks fine
Seen twice so far, ~28 days apart — 2026-08-25 (root-caused in
`INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md`, Incident 1) and
2026-09-22. `pia-wg`'s handshake with PIA's server stops renewing;
the interface, `wg-quick@pia-wg` (`active (exited)` is normal for
`wg-quick`), the `pia` routing table and all firewall rules stay exactly
as `PostUp` left them. The kill switch's "interface down → unreachable
route" trigger never fires because the interface never goes down, so
VLAN 50 fails closed — correctly, but silently.
On 2026-08-25 a plain `systemctl restart wg-quick@pia-wg` did not produce
a new handshake and only a fresh `addKey` registration did — strongly
suggesting PIA had dropped the registered key server-side. The
2026-09-22 occurrence went straight to re-registration (bare restart
untested). The underlying trigger is still not confirmed; the ~4-week gap
is consistent with a fixed registration lifetime on PIA's side, which is
what the proactive renewal timer below is for. This failure mode is now
handled automatically — see the next section.
## Automatic recovery
`files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway`
(root, `0700`, runs on the host's `python3.13`). Almost entirely standard
library — the one exception is `addKey` specifically, which shells out to
`curl` (now an explicit `apt` dependency of this role); see
`HttpClient.get_pinned_curl`'s own docstring and "PIA's CA certificate and
why addKey uses curl, not Python's `ssl`" below for why. It is **one**
implementation of PIA registration shared by the unattended timers and by
this role's own registration step, so the two can never drift. Layout on
the host:
| Path | Purpose |
| --- | --- |
| `/etc/pia-gateway/config.json` (`0600`) | rendered from role defaults (`templates/pia-gateway-config.json.j2`) |
| `/etc/pia-gateway/credentials.env` (`0600`) | `PIA_USER`, `PIA_PASSWORD`, optional `ALERT_BRIDGE_API_KEY` — see "Credentials on the host" |
| `/var/lib/pia-gateway/state.json` (`0600`) | versioned controller state (atomic tmp+fsync+rename); **no secrets** |
| `/var/lib/pia-gateway/status.json`, `metrics.prom` (`0644`) | last classification, for humans/monitoring |
| `/run/lock/pia-gateway.lock` | one `flock` shared by every mutating action (timers, `register`, `rewrite-config`); released by the kernel if a process dies |
| `pia-gateway-recover.timer` → `.service` | every 60 s, one bounded recovery pass (`recover`) |
| `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) |
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
| `pia-killswitch-reassert.timer` → `.service` | every `pia_healthcheck_interval_sec`, re-runs `pia-killswitch.sh` (idempotent). **Not** gated by `pia_auto_recovery_enabled` — see the "FORWARD jump's position" note above and the service unit's own header for why |
### PIA's CA certificate and why addKey uses curl, not Python's `ssl`
Found live, 2026-09-22, on the very first `pia-gateway register` attempt
after the FORWARD-jump fix above got past the kill-switch check:
`addKey` failed against every candidate with `kind=tls detail=certificate
verification failed`. Root-caused directly on minisforum:
```
$ sudo python3 -c '...ctx.wrap_socket(sock, server_hostname="Server-11669-10a")...'
SSLCertVerificationError verify_message= Basic Constraints of CA cert not marked critical
```
PIA's own CA certificate (`files/pia-ca.crt`) has a `CA:TRUE` Basic
Constraints extension that isn't flagged `critical` — a real RFC 5280
non-conformance, confirmed **not** a stale vendored copy: a fresh fetch
from `pia-foss/manual-connections` on GitHub is byte-identical (same
`32e9b1d1...` SHA-256 already recorded in the file's own header) and has
the identical defect. This is PIA's problem, not ours, and not fixable by
re-vendoring. Recent OpenSSL (3.2+; confirmed live against minisforum's
`3.5.7`) rejects this by default — a check that used to be opt-in only
under `X509_V_FLAG_X509_STRICT` and has since been promoted into the
default chain-verification path. `openssl s_client` on this repo's own
dev machine (LibreSSL) and this same host's `curl` both accept the exact
same certificate/chain/CA file without complaint — only Python's `ssl`
module, using this host's system OpenSSL, rejects it. That's also why
this went unnoticed until the auto-recovery work: the legacy Ansible
path's three successful registrations (2026-08-24, 08-25, and this
morning) all went through `curl --cacert` (`addkey-attempt.yaml`), never
through Python's `ssl` module — this was the *first* time that code path
ever ran for real.
Python's stdlib `ssl` module has no custom-verify-callback hook (a
long-standing, deliberate upstream gap — unlike Go's
`tls.Config.VerifyPeerCertificate` or OpenSSL's own C API), so there is
no clean way to tolerate just this one known, third-party-controlled
defect while still fully validating everything else (chain, expiry,
hostname/SAN). Rather than weakening `verify_mode` for the whole
connection or adding a new dependency (`pyOpenSSL`, for a real verify
callback — considered, rejected as a bigger lift than necessary) to work
around a gap in a library we don't control, `addKey` specifically now
shells out to `curl` (`HttpClient.get_pinned_curl`, `curl` added to this
role's `apt` install list) — the same tool PIA's own reference client
(`connect_to_wireguard_with_token.sh`) already uses for this exact
endpoint, with `--cacert` and never `-k`/`--insecure`, so this is a
different, already-proven-correct TLS backend for the one endpoint that
needs it, not a security downgrade. `serverlist` and the token endpoint
are unaffected (they present public WebPKI certs, which don't have this
problem) and still use the stdlib `ssl` path. Verified: a real subprocess
`curl` call from this repo's dev machine against the actual failing
candidate (`Server-11669-10a` / `69.33.198.97`) gets a genuine HTTP
response from PIA (401, expected — a dummy token was used), proving TLS
validation succeeds via this path where it failed via Python's `ssl`.
curl's own exit codes for a `--fail`-triggered rejection aren't fully
portable either — confirmed live: a curl built against SecureTransport
(this dev machine) reports exit 56, not the "usual" 22 GNU/OpenSSL curl
uses for the identical case — so classification checks curl's own error
text as a second signal alongside the exit code, not the exit code alone.
### Status classes
Every pass classifies the gateway into exactly one of: `healthy`,
`boot_grace` (stale within 120 s of a boot or of an intentional restart —
not counted), `paused`, `tunnel_stale` (handshake 0 / older than
`pia_healthcheck_handshake_max_age_sec` / in the future beyond 60 s of
skew), `tunnel_absent` (interface down/missing, expected peer missing,
fresh handshake but table route closed), `registration_or_endpoint_failure`
/ `host_connectivity_failure` (stale **and** in backoff after a cycle
failed that way), `safety_drift`, `unknown` (a read command failed — no
action taken, alert after 3). `status.json` carries both the decision
class and the raw `tunnel_class`, so "paused" never hides a dead tunnel.
Safety invariants checked on every pass, all of which must hold before
any restart/registration is attempted: the priority-100 source rule; a
default in table `pia` that is either `dev pia-wg` (open) or
`unreachable` (closed) — *no* default at all is a failure, not "closed";
the FORWARD jump to `PIA-VLAN50` is the **first effective** rule (an
emergency rule of our own is the only thing allowed above it); the chain
ends in an unconditional `DROP` and every `ACCEPT` before it is scoped to
`pia-wg`; the host default route is still via `enp1s0`.
### Recovery state machine (`recover`, every 60 s)
| State | Action |
| --- | --- |
| healthy | reset counters; if a backoff was pending, log/notify "healthy again" |
| 1–2 consecutive stale/absent | count only |
| 3 consecutive (`pia_recovery_failure_threshold`) | save pre-fix evidence (age, endpoint, reasons) → **stage 1**: `systemctl restart wg-quick@pia-wg` once, wait up to 90 s for two consecutive healthy classifications with a handshake newer than the attempt start |
| stage 1 fails | **stage 2**: serverlist → token → `addKey` with the *existing* key against candidates in `pia_region_candidates` order (max `pia_recovery_max_candidates_per_cycle`), each accepted response staged as a validated config, activated, and verified the same way; a candidate that answers `OK` but never handshakes is skipped for the next one |
| stage 2: API accepted the key but nothing handshook | **stage 3**: exactly one staged fresh-key attempt (`pia-wg.key.new.<ts>.<pid>`, promoted only after verification, previous key backed up). Skipped when the failure was uniform (auth error, host connectivity, every API call failing the same way) |
| any failure | previous config restored (best effort) and re-activated; gateway stays closed; backoff 5 → 15 → 30 → 60 min (±10 % jitter, capped) persisted in state; classified failure recorded; alert |
| verified recovery | stage, region/server, elapsed time and pre-fix age recorded in `last_recovery`; `last_verified_registration` updated when a registration happened; one "recovered" alert |
| safety drift | **no** restart/registration. Insert `-I FORWARD 1 -s 10.10.50.0/24 -m comment --comment pia-gateway-emergency -j DROP`, then `ip route replace unreachable default table pia`, verify the DROP is first, alert `error`. If the rule already exists but isn't first any more (something else prepended a rule ahead of it, e.g. an out-of-band `ufw reload`), it's deleted and reinserted at position 1 rather than left where it drifted to — a mere existence check (`iptables -C`) says nothing about position. Left in place until a later pass sees the invariants intact, then removed under the lock. Automatic repair of the kill switch itself is deliberately **not** attempted (see "Residual risk") |
| paused (`pia-gateway pause`) | checks and notifications continue; never restart/register/renew. Safety drift is **still** contained by the emergency close while paused — pausing suspends recovery, never the fail-closed guarantee |
| corrupt/unknown-schema state file | refuse to act (exit 2, unit failure → alert). `pia-gateway state-reset --confirm` moves the file aside and starts fresh |
One pass is bounded by `pia_recovery_max_duration_sec` (600 s) inside
the program and `TimeoutStartSec` (900 s) in the unit; systemd never
starts a second instance while one runs, and the lock makes a manual
`register` wait (up to `pia_recovery_lock_wait_sec`) rather than
interrupt an activation. Reboots: the boot id is tracked, a new boot gets
`pia_recovery_startup_grace_sec` before stale counts; retry/backoff state
survives because it is on disk. A backward wall-clock step (NTP) that
leaves a persisted retry deadline further out than the backoff cap is
treated as a clock jump and the deadline is cleared rather than stalling
recovery.
`restart` returning 0, `active (exited)`, or `addKey` returning `status:
OK` are never treated as success on their own — only an observed fresh
handshake on the activated config is, and specifically one reported
**strictly after** the moment activation started: a handshake in the
same whole second as that moment doesn't count either, since `wg show`
only has one-second resolution and there'd be no way to tell it apart
from a stale, pre-activation handshake that happens to share the second.
Every mutating command — `recover`, `renew`, `register`,
`rewrite-config`, `pause`, `resume`, `state-reset` — takes the same
`/run/lock/pia-gateway.lock` **before** reading state, not after: reading
state first and only locking around the write would let a slower command
silently overwrite whatever a faster one wrote in between (in particular,
it's how a `pause` could otherwise be lost to an in-flight recovery pass
that read state before the pause ran and only saves its own copy back
once it finishes). `pause`/`resume`/`register`/`rewrite-config`/
`state-reset` wait up to `pia_recovery_lock_wait_sec` for the lock and
exit 4 if it's still busy; `recover`/`renew` (the timers) never wait —
they just skip that tick if something else currently holds it.
### Operator commands (all on minisforum, as root)
```bash
pia-gateway status # human summary incl. backoff, last verified registration, pause
pia-gateway check # what the health-check timer runs (read-only, exit 1 = unhealthy)
pia-gateway check --json # machine-readable
pia-gateway pause --reason "maintenance" # persists across reboot; suppresses recover AND renew; kill switch untouched
pia-gateway resume
pia-gateway retry-now # one recovery pass ignoring threshold and backoff
pia-gateway register [--mode same-key|new-key] [--region hk] # explicit transaction, verified, under the lock
pia-gateway rewrite-config # re-render pia-wg.conf from the recorded registration (hook changes), no API call
pia-gateway renew --force-due # exercise renewal without waiting 21 days
pia-gateway state-reset --confirm # only after reading the corrupt file it moves aside
journalctl -u pia-gateway-recover -u pia-gateway-renew -u pia-gateway-healthcheck -n 100
cat /var/lib/pia-gateway/status.json
```
Through Ansible (repo root, `-K -J`):
```bash
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml -K -J # idempotent; registers only if no config or no verified baseline
ansible-playbook ... -e pia_force_reregister=true # re-register with the EXISTING key (same-key)
ansible-playbook ... -e pia_force_key_rotation=true # stage a NEW key, register it, promote only after verification
ansible-playbook ... -e pia_region=hk # single-region override (either of the above)
ansible-playbook ... -e pia_auto_recovery_enabled=false # stop+disable both timers, keep everything else (rollback)
ansible-playbook ... -e pia_renewal_enabled=true # after the live drills — enables the daily renewal timer
ansible-playbook ... -e pia_remove_host_credentials=true # deliberate removal of /etc/pia-gateway/credentials.env
```
`pia_force_reregister` **reuses** the on-disk key; only
`pia_force_key_rotation` generates a new one (the root README used to say
otherwise — corrected). A normal run also quiesces both timers, waits for
any in-flight pass, and re-enables them after handlers, so an Ansible
restart never races an automatic one. A run whose only change is
`config.json` (e.g. flipping `pia_mss_clamp_enabled`) re-renders the
WireGuard config via `rewrite-config` without touching PIA — the
"template-only change needs a full re-registration" gap from the
2026-08-25 incident is closed.
### Proactive renewal (`renew`)
Off until enabled. When on, once a day it re-registers with the
**existing** key if `last_verified_registration` is older than
`pia_renewal_after_days` (21) — or missing, which is why the baseline
must come from a *verified* run of the shared path, never from a file
mtime. Same transaction, same lock, same verification. A failed renewal
leaves the baseline untouched and enters the same bounded backoff (it
does not silently wait another 21 days); if the tunnel is already
unhealthy it defers to the recovery path. Renewal restarts `pia-wg`, so
qBittorrent/JDownloader connections drop for a few seconds at ~03:30 —
verify they reconnect during the drill. Routine key rotation is not
done: rotation happens only in recovery stage 3, on evidence.
### Alerts and monitoring
Events (not every 60 s): recovery started, recovered, recovery failed +
backoff (`warn`; `error` for auth failure), safety drift (`error`),
emergency rule removed, renewal failed, status unknown ×3, state corrupt,
unit crashed (via `OnFailure`). Delivery is `POST` to
`pia_alert_bridge_url` with the shared alert-bridge bearer key
(`{"source": "pia-gateway@minisforum", "message": ..., "level": ...}` —
schema read from `~/repo/home-service/alert-bridge`, which has **no**
per-caller tokens, so the cluster's key is what gets provisioned;
accepted tradeoff). 5 s timeout; failures are queued in state (bounded,
20) and flushed on the next pass; they never block recovery.
`metrics.prom` is node-exporter textfile format
(`pia_gateway_healthy`, `_tunnel_ok`, `_safety_ok`,
`_handshake_age_seconds`, `_consecutive_failures`, `_paused`,
`_recovery_in_backoff`, `_last_verified_registration_timestamp_seconds`,
`_check_timestamp_seconds`, `_emergency_close_active`). **Not yet
scraped**: `values/kube-prometheus-stack.yaml` has no textfile collector
and node-exporter runs as a pod, so wiring it needs a hostPath mount +
`--collector.textfile.directory` plus a `PrometheusRule` including a
stale-check rule (`time() - pia_gateway_check_timestamp_seconds > 300`)
so a stopped timer alerts even when the last value was healthy. Until
that exists, a dead controller is only visible as silence — check
`systemctl list-timers 'pia-gateway-*'` and `status.json`'s timestamp.
### Credentials on the host
`/etc/pia-gateway/credentials.env` holds the PIA account username and
password in plaintext, root-only. This is the explicit tradeoff the
design requires: recovery must work while the laptop is off, and PIA's
token is only valid for 24 h, so an unattended host needs a renewable
credential. The program reads the file at runtime and never places
values in argv, environment, logs, notifications, metrics or state; core
dumps are disabled (`LimitCORE=0`) and unexpected exceptions are logged
by class name only. **Rotation:** change `PIA_PASSWORD` in `.env`, run
the playbook (the file is re-rendered with `no_log`), then `pia-gateway
register` (or wait for the next cycle). **Revocation:** `-e
pia_remove_host_credentials=true` removes the file (recovery then fails
closed with `auth`/credentials errors and alerts); change the PIA
password; disabling `pia_auto_recovery_enabled` alone never deletes it.
If an on-host secret is unacceptable, the alternative is an always-on
external secret broker — not a 24-hour token.
### Residual risk (read before trusting it unattended)
- Safety drift is contained by the emergency DROP + unreachable route,
not repaired: the kill switch's own rule/jump can be deleted by
something outside this role (a manual `ufw reload` flushes `FORWARD`,
removing both the jump **and** the emergency rule). The controller
re-inserts the emergency rule on its next pass (≤ 60 s) and every boot
re-seeds the kill switch; that 60 s window after an out-of-band flush
is the residual leak risk, and it predates this work.
- Verification is handshake-based. It proves the tunnel is talking to
PIA, not that a VLAN 50 packet leaves via PIA; the end-to-end check is
the SOCKS-proxy/egress-IP step in the live drill.
- The alert-bridge key on minisforum is the cluster's shared key.
- `unknown` (a read command failing) is logged/alerted but never acted
on, by design.
### Rollout status (2026-09-23)
Steps 1-3 are done and verified live. Step 3 required two attempts —
the first caused a real (safely fail-closed) outage, root-caused and
fixed same-day; the second was verified against a full day of real
production conditions, not just a short window.
- **Step 2, attempt 1** failed: `pia-gateway register` correctly refused
with `safety invariants failing ... FORWARD jump to kill-switch chain
is not first` — a real, pre-existing drift (see the "FORWARD jump's
position is actively re-asserted" note above), not a false alarm from
this session's work, and not the "session dies" handshake issue
either. Root-caused and fixed in `pia-killswitch.sh.j2`.
- **Attempt 2**, after that fix, got past the kill-switch check and
failed differently: `addKey` rejected every candidate with
`kind=tls detail=certificate verification failed`. Root-caused and
fixed — see "PIA's CA certificate and why addKey uses curl, not
Python's `ssl`" above.
- **Attempt 3**, roughly an hour after attempt 1's fix with no reboot,
UFW reload, or script change in between, hit the *identical* FORWARD-
jump failure again — proving the drift recurs on a timescale of
minutes under this cluster's normal CronJob/pod churn, not just at
boot-scale events as originally assumed. Unblocked live with a manual
`sudo systemctl restart pia-killswitch.service`; `pia-killswitch
-reassert.timer` added as the durable fix.
- **Attempt 4** succeeded cleanly end to end: kill-switch check passed,
`addKey` accepted (`hk`/`Server-11669-10a`), and a **fresh handshake
was verified** (age 10s) — the first-ever successful shared-path
registration. `pia-gateway status` confirmed `healthy`/`safety_ok:
True`, `iptables -S FORWARD` confirmed the jump first, and
`pia-killswitch-reassert.timer` confirmed cycling correctly (7
successful runs, ~60s apart, in the journal). **Step 2 is done.**
- **Step 3** (`ansible-playbook ... -K -J`, no override — enables
`pia-gateway-recover.timer` for the first time) deployed cleanly
(`Run the shared PIA registration transaction` correctly skipped, a
verified baseline already existed) but caused a **real live outage**
within minutes: `pia-gateway-recover` and `pia-killswitch-reassert`,
two independent 60s timers, raced — kube-router's actual resync
cadence (~60-90s) turned out to be far more frequent than assumed, and
every momentary drift-then-self-correct was treated as a full safety
emergency, forcibly closing the route and resetting the 3-strikes
counter before it could accumulate. VLAN 50 was open for ~10s out of
every 60-90s cycle for several minutes, confirmed via Discord alerts
from `alert-bridge` (working end to end, first real trigger) and the
full `pia-gateway-recover`/`pia-killswitch-reassert` journals.
Mitigated live: `sudo systemctl stop pia-gateway-recover.timer` +
manual `sudo systemctl restart wg-quick@pia-wg`. Root-caused and fixed
same-day — see the "reassert timer alone wasn't the full fix" addendum
above (`pia_forward_bypass_chains`, content-verified live on every
check).
- **Step 3, re-attempt**, with the bypass fix deployed (`ansible-playbook
... -K -J`; config re-rendered with `forward_bypass_chains:
["KUBE-ROUTER-FORWARD"]`, `rewrite-config` correctly re-rendered
`pia-wg.conf` from the config-only change without a PIA call since a
baseline already existed, recovery timer re-enabled). Left running
unattended for a full **~25.5 hours** before re-checking, deliberately
— the original incident took several minutes and multiple cycles to
manifest, so a short recheck window would have proven nothing.
Confirmed via exact counts, not inference: `journalctl -u
pia-gateway-recover --since "2026-09-22 20:37:00"` showed **1528**
`class=healthy` passes (matches the expected ~1530 at 60s cadence —
the timer ran continuously the whole window), **0** `SAFETY DRIFT`
lines, **0** `class=safety_drift` classifications. `pia-gateway
status` at recheck time: `healthy`, `safety_ok: True`,
`emergency_close_active: False`, `consecutive_failures: 0`,
`last recovery: 1d ago restart` — the last restart was the one from
*before* this fix; none has been needed since. **Step 3 is done.**
kube-router did keep resyncing during this window (that's the whole
premise of the fix); the bypass tolerance handled it silently every
time, with no observable effect on the tunnel.
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
— the original `head -3` suggestion is what hid the first issue; the
drift was 4 rules in, not at the top. **Use `head -2`, not `head -1`**,
for any quick re-check — `-S FORWARD`'s first line is always the `-P`
policy line, never a real rule (this cost a round-trip during the
investigation above).
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`,
`journalctl -u pia-gateway-healthcheck -n 5`, `curl --socks5-hostname
10.10.40.20:1080 https://ifconfig.me` from the LAN, `wg show wg0`,
`kubectl get nodes`. Confirm console access.
2. Deploy with timers off: `ansible-playbook ... -K -J -e
pia_auto_recovery_enabled=false`. Check `ls -l /etc/pia-gateway
/var/lib/pia-gateway`, `pia-gateway status`, `systemctl list-timers
'pia-gateway-*'` (recover/renew inactive). Because the host has no
verified baseline yet, this run **registers through the shared path**
(same-key, explicit restart + handshake verification) — watch the
task output; if it fails, `-e pia_registration_backend=legacy`
restores the previous behaviour. If it fails specifically with
`FORWARD jump to kill-switch chain is not first`, that's the drift
above — the fixed template should correct it via the normal "Restart
pia-killswitch" handler on the same run (the template content
changing is enough to trigger it); re-running the identical command
should then get past registration. Confirm with `iptables -S FORWARD`
afterwards that the jump is genuinely first, not just that the retry
succeeded.
3. Enable: `ansible-playbook ... -K -J` (recovery on; also redeploys
`pia_forward_bypass_chains` into `config.json` if not already applied
— confirm with `sudo pia-gateway --config /etc/pia-gateway/config.json
status` or just `cat /etc/pia-gateway/config.json`). **Watch longer
than one pass** — the live incident here took 2-3 cycles (several
minutes) to manifest, not the first check. `journalctl -u
pia-gateway-recover -f` for at least 5-10 minutes; expect `class=
healthy` throughout, with **zero** `SAFETY DRIFT` lines, even while
`pia-killswitch-reassert` keeps cycling in parallel
(`journalctl -u pia-killswitch-reassert -f` alongside it). If a
`bypass chain KUBE-ROUTER-FORWARD content no longer verified safe`
reason ever appears, treat it as real — it means kube-router's rules
changed to include something that could actually match VLAN 50
traffic, not a false positive to relax further.
4. Drill the silent death. First get the **live WireGuard UDP endpoint**
from the interface itself — **not** port 1337, which is only the
`addKey` registration API port and has nothing to do with the tunnel's
own data-plane port (PIA WireGuard servers commonly use a different
port per server):
```bash
sudo wg show pia-wg endpoints # <peer-pubkey>\t<ip>:<port> — this ip:port is the block target
```
Then, on minisforum, leave `pia-wg` administratively up and drop only
that path:
```bash
sudo iptables -I OUTPUT -d <endpoint ip> -p udp --dport <endpoint port> -j DROP
```
Expect: 3 stale passes → restart → (still blocked) registration to a
different server → recovery; record times from the journal. To test
backoff, block every candidate server before recovery can reach it —
since each candidate's actual WireGuard port isn't known until its own
`addKey` response arrives (only the registration port, 1337, is fixed
and shared), block by IP alone rather than guessing a port:
```bash
sudo iptables -I OUTPUT -d <candidate ip> -p udp -j DROP # repeat per candidate IP, or -e pia_region=hk to narrow the set first
```
Remove the rule(s) and confirm autonomous recovery. `iptables -D`
every injected rule afterwards — check `iptables -S OUTPUT` is back to
its pre-drill state, not just that the tunnel recovered.
5. Exercise: a failed first server (block one IP only), invalid
credentials (temporarily wrong `PIA_PASSWORD` → expect `auth_failure`,
no rotation, error alert), alert-bridge down, `pia-gateway register`
while a pass runs (expect it to wait), reboot with a pending backoff,
`pause`/`resume`.
6. During every transition, capture on `enp1s0` with an explicit
**egress-only** direction filter:
```bash
sudo tcpdump -ni enp1s0 -Q out src net 10.10.50.0/24
```
`-Q out` matters: VLAN 50 traffic **arriving** at minisforum on
`enp1s0` (from the Flint router, for policy routing into table `pia`)
is normal and expected — a plain `src net 10.10.50.0/24` filter with
no direction would match that inbound traffic too and could hide a
real leak in the noise, or be misread as one. Only a hit in the `out`
direction — a VLAN 50-sourced packet actually being transmitted out
`enp1s0` toward the LAN/Internet instead of via `pia-wg` — is a leak,
and this must stay completely silent through every failure and
recovery transition. Also capture on `pia-wg` itself to confirm
traffic **is** flowing there while open. Verify a request through
`10.10.40.20:1080` and qBittorrent/JDownloader afterwards; check IPv6
separately; confirm `wg0`, the host default route, Technitium and k3s.
7. Only then `-e pia_renewal_enabled=true`; `pia-gateway renew
--force-due`; confirm `last_verified_registration.epoch` advances only
on verification. Wire the Prometheus stale-check before calling the
feature done. Record actual times and captures in the incident file.
## Rollback
**Partial rollback (keep the gateway, drop only the automation):**
`ansible-playbook ... -e pia_auto_recovery_enabled=false` stops and
disables both timers and leaves the program, credentials, health check,
kill switch and `pia-wg` in place. Add `-e pia_remove_host_credentials=true`
to also delete the on-host PIA credentials; nothing else deletes them.
No tag-based full rollback is defined — this role doesn't have an
"absent" mode. Roll back manually on minisforum (reverses this role without
touching `wg0`, k3s, or the host default route):
```bash
sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-killswitch-reassert.timer pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \
/etc/systemd/system/pia-killswitch-reassert.service \
/etc/systemd/system/pia-killswitch-reassert.timer \
/etc/systemd/system/pia-gateway-healthcheck.service \
/etc/systemd/system/pia-gateway-healthcheck.timer \
/etc/systemd/system/pia-gateway-recover.service \
/etc/systemd/system/pia-gateway-recover.timer \
/etc/systemd/system/pia-gateway-renew.service \
/etc/systemd/system/pia-gateway-renew.timer \
/etc/systemd/system/pia-gateway-onfailure.service
sudo systemctl daemon-reload
sudo iptables -D FORWARD -s 10.10.50.0/24 -m comment --comment pia-gateway-emergency -j DROP 2>/dev/null || true
sudo rm -rf /etc/pia-gateway /var/lib/pia-gateway /usr/local/sbin/pia-gateway /run/lock/pia-gateway.lock # includes the on-host PIA credentials
sudo ip rule del from 10.10.50.0/24 table pia priority 100
sudo ip route flush table pia
sudo sed -i '/^[0-9]\+\s\+pia$/d' /etc/iproute2/rt_tables
sudo iptables -D FORWARD -s 10.10.50.0/24 -j PIA-VLAN50
sudo iptables -F PIA-VLAN50
sudo iptables -X PIA-VLAN50
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp
sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.conf.bak-* /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.bak-* /etc/wireguard/pia-wg.key.new* /etc/wireguard/pia-ca.crt
```
Inspection commands to confirm rollback actually took (read-only):
```bash
sudo iptables -S FORWARD | grep PIA-VLAN50 # expect: no output
sudo iptables -L PIA-VLAN50 # expect: "iptables: No chain/target/match by that name"
sudo ufw status verbose | grep 10.10.50 # expect: no output
ip rule show | grep pia # expect: no output
ip route show table pia # expect: empty/error (table gone)
```
Does not remove `pia-credentials`/`pia-credentials-sealed.yaml` (the K8s
Secret used by the Gluetun sidecars) — that's a separate, unrelated
secret and rollback path. Does not touch `ansible/roles/common`'s own
UFW rules (Flannel pod-to-pod, pod-to-Technitium) — this role never
modified those.