fix: address code review on pia-gateway auto-recovery controller
Some checks failed
validate / lint (push) Failing after 2s

- Serialize every mutating command (recover, renew, register,
  rewrite-config, pause, resume, state-reset) on the shared lock,
  acquired BEFORE state is read rather than after: reading first and
  locking only around the write let a slower command silently clobber
  whatever a faster one wrote in between — in particular, an operator
  pause could be lost to an in-flight recovery pass that read state
  before the pause ran and only saved its own copy back afterwards.
- Fix emergency_close() to reposition its VLAN 50 DROP to the first
  effective FORWARD position when it exists but has been pushed down
  by something else (e.g. an out-of-band ufw reload) — a bare `-C`
  existence check said nothing about position.
- Tighten wait_for_verified()'s handshake freshness check from >= to >
  since_epoch: wg show and since_epoch are both whole seconds, so a
  handshake at exactly the same second as since_epoch is
  indistinguishable from a stale, pre-activation one and must not
  count as proof of a new one.
- Fix the Ansible config-validation task to fail on any exit code
  outside status's three documented health exits (0/1/2), not just on
  5 — verified live against the real program (no wg/ip/iptables on
  this dev machine reproduces the "nothing configured yet" first-run
  case; status correctly returns 1, and broken/missing config.json
  correctly returns 5 and is now caught).
- Correct the live fault-injection drill in the role README: port
  1337 is only the addKey registration API port, not the tunnel's own
  WireGuard UDP endpoint (derive that from `wg show pia-wg endpoints`
  instead), and make the packet-capture check direction-aware
  (`tcpdump -Q out`) so it can't conflate a real egress leak with
  VLAN 50's normal inbound arrival on enp1s0.

Added 6 tests (86 total, all passing): emergency-close repositioning
past a preceding ACCEPT, pause/resume blocking on an in-flight
lock holder and surviving into a subsequent recovery pass, and the
handshake boundary at exactly/one-second-either-side of since_epoch.

yamllint, ansible-playbook --syntax-check, and ansible-lint (diffed
against main — no new finding categories) all pass. No live changes;
nothing pushed to main.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Nik Afiq 2026-09-22 15:51:02 +09:00
parent f76b1e6f13
commit 87a9303cc3
22 changed files with 4298 additions and 159 deletions

1
.gitignore vendored
View File

@ -9,3 +9,4 @@ kubeconfig*
*.swp
*.swo
*~
__pycache__/

View File

@ -2,7 +2,7 @@
**Date:** 2026-08-25 to 2026-08-27 (discovered/resolved across this window; underlying triggers occurred 2026-08-24/25)
**Systems:** `minisforum` (PIA WireGuard gateway), `nik-debian` (VLAN 50 workloads: `browser-vpn-proxy`, `qbittorrent`, `jdownloader`)
**Status:** All four issues resolved live, with code fixes committed for three of them (qBittorrent, MSS clamping, `wg-quick` restart idempotency). The remaining gap (PIA tunnel handshake-death alerting/remediation) is documented with prevention options, deliberately not yet implemented — see "Open follow-up" below.
**Status:** All four issues resolved live, with code fixes committed for three of them (qBittorrent, MSS clamping, `wg-quick` restart idempotency). The remaining gap (PIA tunnel handshake-death alerting/remediation) is documented with prevention options, deliberately not yet implemented — see "Open follow-up" below. **Incident 1 has since recurred (2026-09-22) — see its "Recurrences" subsection.**
## Summary
@ -89,6 +89,15 @@ ansible-playbook -i ansible/inventory.yaml \
This hit a real, separate, now-fixed bug on the way: `ansible/roles/pia-gateway/tasks/register.yaml`'s `when:` conditions used `pia_force_reregister`/`pia_force_key_rotation` directly in boolean expressions. Passed via the CLI's plain `key=value` form (`-e pia_force_reregister=true`, exactly what the role's own README documents), Ansible sets the variable as the **string** `"true"`, not a real boolean — and recent `ansible-core` rejects using a string directly in a `when:` boolean expression (`Conditional result (True) was derived from value of type 'str'`). Fixed by wrapping every use in `| bool` (commit `e757850`), which correctly coerces both real booleans (the `defaults/main.yaml` case) and `"true"`/`"false"` strings (the CLI-override case). Verified with an isolated local test playbook exercising both cases before trusting it against the live host.
### Recurrences
| Date | Discovered by | Pre-fix evidence | Fix | Notes |
| --- | --- | --- | --- | --- |
| 2026-08-25 | User: proxy not working after nik-debian reboot | Handshake 7h29m old; healthcheck logging `FAIL handshake` every 60s; bare `wg-quick` restart did **not** recover | `pia_force_reregister=true` | This report |
| 2026-09-22 (~09:50 JST) | User: browser can't use the SOCKS5 proxy | **Not captured** — the re-registration was run before `wg show` was checked, so handshake age and healthcheck journal were not recorded; bare restart not attempted | `pia_force_reregister=true` (~10:06 JST) | Kubernetes side verified healthy first (`browser-vpn-proxy` pod, Service, both `svclb` pods all Running). Post-fix `latest-handshakes` = `1790039182` (10:06:22 JST), ~47s before the check — fresh. Two doc gaps hit on the way: root README's command omitted `-J` (fails with "no vault secrets found") and nothing said the paths are repo-root-relative (fails from `ansible/`). Both fixed in the README. |
~28 days between the two. Not enough to claim a fixed PIA-side registration lifetime, but enough that the next occurrence should be checked against that hypothesis first — if it lands around 2026-10-20, a scheduled proactive `pia_force_reregister` run (every ~3 weeks) becomes the cheapest prevention, independent of whichever alerting option below is chosen. Next time: **run the diagnostic commands before the fix** (root README, "Tunnel silently dead") so the age and journal are on record.
### Known gap / prevention (not yet implemented — deferred by request)
`ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2` (run via `pia-gateway-healthcheck.timer`, every 60s) **already correctly detected this failure the entire time** — every single run logged `FAIL handshake: ...s old, exceeds max 180s`. The script is explicitly designed as "observability only, no remediation" (its own header comment says so), which is a deliberate prior design choice, not an oversight. The actual gap is that nothing consumes that detection — no alert, no dashboard, nothing — so a real, correctly-detected outage sat unnoticed until a user happened to test the affected service directly.
@ -238,7 +247,7 @@ None identified for this specific failure mode going forward — the fix address
## Open follow-up
- **Decide and implement PIA tunnel remediation/alerting** (Incident 1's "Known gap" section above) — explicitly deferred by the user to a later date, options documented above.
- **Consider splitting "rewrite config from template" from "re-register with PIA" in the role** (Incident 4's root cause #1) — currently there's no way to pick up a template-only default change without forcing a full re-registration.
- **Decide and implement PIA tunnel remediation/alerting** (Incident 1's "Known gap" section above) — explicitly deferred by the user to a later date, options documented above. **Recurred 2026-09-22** (see Incident 1's "Recurrences" table). **Implemented 2026-09-22 (option 3 + proactive renewal)** on branch `pia-gateway-auto-recovery`, per `PLAN-pia-gateway-auto-recovery.md`: an on-host controller (`ansible/roles/pia-gateway/files/pia_gateway.py`, `pia-gateway-recover.timer`) that restarts, then re-registers with the existing key, then makes one evidence-gated key rotation, verifying a fresh handshake at each step, with persisted bounded backoff, pause/resume, emergency close on kill-switch drift, and alert-bridge notifications; plus a 21-day renewal timer (off by default). 78 offline tests pass. **Not yet deployed or drilled live** — the controlled rollout is in the role README ("Rollout"); the legacy in-Ansible registration path remains selectable (`pia_registration_backend=legacy`) until the shared path passes a live re-registration. Still open after that: node-exporter textfile scrape + a Prometheus stale-check rule so a stopped timer is distinguishable from a healthy gateway.
- **Consider splitting "rewrite config from template" from "re-register with PIA" in the role** (Incident 4's root cause #1) — closed by the same change: `pia-gateway rewrite-config` re-renders `pia-wg.conf` from the recorded registration without a PIA call, and the role runs it automatically when only the controller config changed.
- Confirm whether a newer `lscr.io/linuxserver/qbittorrent` tag already includes the upstream fix for #24164, as a potential alternative/complement to the init-container workaround.
- If it recurs, try to catch what actually triggers qBittorrent's unclean shutdown in the first place (Incident 2's root cause section) — nothing in this investigation identified it.

View File

@ -305,14 +305,79 @@ piece's own README for exact commands and rollback:
routing/firewall state and is ordered `Before=` `wg-quick@pia-wg.service`;
both are `WantedBy=multi-user.target` so a normal boot orders them
correctly with no manual step. `pia-gateway-healthcheck.timer` runs
independently and only observes — see
`ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2`.
independently and only observes (`pia-gateway check`).
**PIA endpoint/key rotation**: re-run `ansible/playbooks/pia-gateway.yaml
-e pia_force_reregister=true -K` — this regenerates the local keypair and
re-registers with PIA, then rewrites `pia-wg.conf` (handler-triggered
restart). Rotating the PIA account password itself just needs
`ansible-vault edit ansible/group_vars/all/vault.yaml` first.
**Automatic recovery** (`ansible/roles/pia-gateway/README.md` →
"Automatic recovery" — read that for the state machine, alerts,
residual risk and the live-drill rollout): `pia-gateway-recover.timer`
on minisforum runs a bounded pass every 60 s — after 3 stale checks it
restarts `pia-wg`, escalates to a same-key PIA re-registration, then to
one staged key rotation on evidence, verifies a fresh handshake before
calling anything fixed, backs off 5→15→30→60 min on failure, and keeps
VLAN 50 fail-closed throughout. Operator commands on minisforum:
```bash
sudo pia-gateway status # class, handshake age, backoff, last verified registration, pause
sudo pia-gateway pause --reason "..." # stop recovery+renewal (persists across reboot); kill switch untouched
sudo pia-gateway resume
sudo pia-gateway retry-now # one pass ignoring threshold/backoff
sudo journalctl -u pia-gateway-recover -n 50
```
Status as of 2026-09-22: implemented and tested offline (78 unit tests),
**not yet deployed or drilled live** — see the role README "Rollout".
Renewal (`pia_renewal_enabled`) stays off until the drills pass.
**PIA re-registration / key rotation** — from the repo root (the paths
are repo-root-relative — running from `ansible/` fails with "playbook
could not be found"):
```bash
# re-register with the EXISTING key (what fixed both incidents):
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml \
-e pia_force_reregister=true -K -J
# stage a NEW keypair, register it, promote it only after a verified handshake:
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml \
-e pia_force_key_rotation=true -K -J
```
`pia_force_reregister` **reuses** the on-disk key (an earlier version of
this paragraph said it regenerated the keypair — it never did; that is
`pia_force_key_rotation`). Both run the shared on-host transaction
(`pia-gateway register`), which restarts and **verifies a handshake**
even when the rendered config is byte-identical. `-J` is required
because `ansible/group_vars/all/vault.yaml` is vault-encrypted and
loaded for every host; the PIA credentials themselves are **not** in
the vault — `ansible/host_vars/minisforum.yaml` reads `PIA_USER`/
`PIA_PASSWORD` from the repo-root `.env`, and the role now also
provisions those two values to minisforum's root-only
`/etc/pia-gateway/credentials.env` so recovery works unattended.
Rotating the PIA password: edit `.env`, re-run the playbook, re-register.
**Tunnel silently dead — recurring, known, now auto-recovered**: the
symptom is "the proxy doesn't work" (browser through `10.10.40.20:1080`
fails, qBittorrent/JDownloader have no egress) while everything in
Kubernetes looks healthy and `pia-wg` on minisforum also looks up
(interface present, `wg-quick@pia-wg` `active (exited)`, routes in
place). The kill switch is doing its job — VLAN 50 fails closed rather
than leaking. With the controller deployed, expect an alert and a
recovery within ~10 minutes; check `sudo pia-gateway status` first. If
it is paused, in backoff, or the controller is not deployed, diagnose
before fixing so the age gets recorded:
```bash
ssh minisforum "sudo wg show pia-wg latest-handshakes" # epoch seconds; >180s old = dead
ssh minisforum "sudo journalctl -u pia-gateway-healthcheck -n 5"
```
then `sudo pia-gateway retry-now` on minisforum, or the re-registration
command above. A bare `systemctl restart wg-quick@pia-wg` was **not**
enough the first time (PIA had stopped recognising the registered key).
Occurrences so far — 2026-08-25 (full write-up in
`INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md`, Incident 1) and
2026-09-22 (~28 days later; pre-fix handshake age not captured, fix
confirmed by a fresh handshake immediately after). The ~28-day cadence
is the hypothesis behind the 21-day proactive renewal timer.
**Maintenance / taking the gateway down deliberately**: `sudo systemctl
stop wg-quick@pia-wg` on minisforum closes VLAN 50 Internet egress

View File

@ -12,3 +12,12 @@ ansible_python_interpreter: /usr/bin/python3.13
# playbook triggers evaluation of these two vars.
pia_user: "{{ lookup('ansible.builtin.file', playbook_dir + '/../../.env') | regex_search('^PIA_USER=(.*)$', '\\1', multiline=True) | first }}"
pia_password: "{{ lookup('ansible.builtin.file', playbook_dir + '/../../.env') | regex_search('^PIA_PASSWORD=(.*)$', '\\1', multiline=True) | first }}"
# Since the auto-recovery work these two ARE copied to minisforum, into the
# root-only /etc/pia-gateway/credentials.env (tasks/controller.yaml,
# no_log) — the host must be able to re-register with PIA unattended. See
# the role README "Credentials on the host" for the tradeoff and rotation.
#
# Optional: alert-bridge's shared bearer key so the controller can post
# state transitions to Discord. Empty (no ALERT_BRIDGE_API_KEY in .env)
# disables delivery; events still go to the journal.
pia_alert_bridge_api_key: "{{ (lookup('ansible.builtin.file', playbook_dir + '/../../.env') | regex_search('^ALERT_BRIDGE_API_KEY=(.*)$', '\\1', multiline=True) | default([''], true)) | first }}"

View File

@ -6,8 +6,11 @@ Phase 2 of `~/repo/homelab/plan.md`.
## Before the first real run
1. **The PIA API calls in `tasks/register.yaml` have gone through three
rounds of correction against real live testing (2026-08-24):**
1. **The PIA API calls have gone through three rounds of correction
against real live testing (2026-08-24). They now live in
`files/pia_gateway.py` (`PiaApi`), with the original in-Ansible flow
kept as `tasks/register-legacy.yaml` until the shared path is proven
live — see "Automatic recovery" below:**
- `serverlist.piaservers.net/vpninfo/servers/v6`: works. Region values
everywhere in this role are the API's **id** field (e.g. `japan`,
`hk`, `taiwan`), never a display name ("JP Tokyo", "Hong Kong",
@ -39,9 +42,14 @@ Phase 2 of `~/repo/homelab/plan.md`.
keys `manifests/media/pia-secret.sh` already reads) — `ansible/
host_vars/minisforum.yaml` reads them directly from there via a
`lookup('ansible.builtin.file', ...)` + regex, deliberately not
ansible-vault, so this needs no vault password at all. The lookup runs
on the control node, so `.env` never touches minisforum and is never
committed (already gitignored).
ansible-vault. The lookup runs on the control node; `.env` itself is
never copied and never committed (gitignored). **Since the
auto-recovery work, only `PIA_USER`/`PIA_PASSWORD` (and, if present,
`ALERT_BRIDGE_API_KEY`) are provisioned to minisforum's root-only
`/etc/pia-gateway/credentials.env`** so the host can re-register
unattended — see "Credentials on the host" below. The playbook still
needs `-J` because `group_vars/all/vault.yaml` is vault-encrypted and
loaded for every host, not because the PIA credentials are in it.
3. Confirm console/recovery access to `minisforum` (IPMI/physical/other
out-of-band path) before applying — this role changes host routing and
firewall policy. If SSH becomes unreachable, use that path.
@ -111,12 +119,13 @@ stdin, or a complete response body. If Hong Kong and Taiwan fail exactly
the same way Tokyo did, that's a strong signal the problem is shared
across all three (the request shape, the account credentials, or
minisforum's own network path) rather than one region being down — see
`tasks/register.yaml`'s header comment.
`tasks/register-legacy.yaml`'s header comment (the controller logs the
same per-candidate `kind=`/`detail=` classification).
**Worked example (2026-08-24):** exactly that happened — all 7 servers
across all 3 regions came back `HTTP failure (non-2xx response)`, curl
exit 22, `The requested URL returned error: 401`, uniformly. The cause
was in `addkey-attempt.yaml`'s own curl task, not PIA or any region:
was in `addkey-attempt.yaml`'s own curl task (legacy path), not PIA or any region:
`ansible.builtin.command`'s `stdin` argument appends a trailing newline
by default (`stdin_add_newline` defaults to `true`), and
`--data-urlencode pt@-` does not strip it — every server was receiving
@ -191,10 +200,15 @@ real.
config change that alters `PostUp`/`PreDown` content can never break
a restart this way again — confirmed live, 2026-08-27.
- Installs an observability-only health check (`pia-gateway-healthcheck
.timer`, every `pia_healthcheck_interval_sec`) that logs interface,
handshake age, rule/route, and firewall-policy state to the journal. It
never remediates — see the script's header comment for why, and for what
its "route-decision" check does and does not prove.
.timer`, every `pia_healthcheck_interval_sec`) that runs `pia-gateway
check` — the same status classifier the recovery controller uses — and
logs interface, handshake age, rule/route ordering and firewall state to
the journal plus `/var/lib/pia-gateway/status.json` / `metrics.prom`.
It never remediates. Neither it nor the controller sends a packet
through the tunnel: a "healthy" class is a fresh handshake plus intact
kill-switch invariants, not proof of egress identity.
- Installs the automatic recovery controller and its timers — see
"Automatic recovery" below.
## What it deliberately does not do
@ -205,18 +219,330 @@ real.
- Does not modify `ansible/roles/wireguard` (the separate `wg0` home-VPN
server role) or its interface.
## Known failure mode: the session dies while the interface looks fine
Seen twice so far, ~28 days apart — 2026-08-25 (root-caused in
`INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md`, Incident 1) and
2026-09-22. `pia-wg`'s handshake with PIA's server stops renewing;
the interface, `wg-quick@pia-wg` (`active (exited)` is normal for
`wg-quick`), the `pia` routing table and all firewall rules stay exactly
as `PostUp` left them. The kill switch's "interface down → unreachable
route" trigger never fires because the interface never goes down, so
VLAN 50 fails closed — correctly, but silently.
On 2026-08-25 a plain `systemctl restart wg-quick@pia-wg` did not produce
a new handshake and only a fresh `addKey` registration did — strongly
suggesting PIA had dropped the registered key server-side. The
2026-09-22 occurrence went straight to re-registration (bare restart
untested). The underlying trigger is still not confirmed; the ~4-week gap
is consistent with a fixed registration lifetime on PIA's side, which is
what the proactive renewal timer below is for. This failure mode is now
handled automatically — see the next section.
## Automatic recovery
`files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway`
(root, `0700`, standard library only — runs on the host's `python3.13`).
It is **one** implementation of PIA registration shared by the unattended
timers and by this role's own registration step, so the two can never
drift. Layout on the host:
| Path | Purpose |
| --- | --- |
| `/etc/pia-gateway/config.json` (`0600`) | rendered from role defaults (`templates/pia-gateway-config.json.j2`) |
| `/etc/pia-gateway/credentials.env` (`0600`) | `PIA_USER`, `PIA_PASSWORD`, optional `ALERT_BRIDGE_API_KEY` — see "Credentials on the host" |
| `/var/lib/pia-gateway/state.json` (`0600`) | versioned controller state (atomic tmp+fsync+rename); **no secrets** |
| `/var/lib/pia-gateway/status.json`, `metrics.prom` (`0644`) | last classification, for humans/monitoring |
| `/run/lock/pia-gateway.lock` | one `flock` shared by every mutating action (timers, `register`, `rewrite-config`); released by the kernel if a process dies |
| `pia-gateway-recover.timer` → `.service` | every 60 s, one bounded recovery pass (`recover`) |
| `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) |
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
### Status classes
Every pass classifies the gateway into exactly one of: `healthy`,
`boot_grace` (stale within 120 s of a boot or of an intentional restart —
not counted), `paused`, `tunnel_stale` (handshake 0 / older than
`pia_healthcheck_handshake_max_age_sec` / in the future beyond 60 s of
skew), `tunnel_absent` (interface down/missing, expected peer missing,
fresh handshake but table route closed), `registration_or_endpoint_failure`
/ `host_connectivity_failure` (stale **and** in backoff after a cycle
failed that way), `safety_drift`, `unknown` (a read command failed — no
action taken, alert after 3). `status.json` carries both the decision
class and the raw `tunnel_class`, so "paused" never hides a dead tunnel.
Safety invariants checked on every pass, all of which must hold before
any restart/registration is attempted: the priority-100 source rule; a
default in table `pia` that is either `dev pia-wg` (open) or
`unreachable` (closed) — *no* default at all is a failure, not "closed";
the FORWARD jump to `PIA-VLAN50` is the **first effective** rule (an
emergency rule of our own is the only thing allowed above it); the chain
ends in an unconditional `DROP` and every `ACCEPT` before it is scoped to
`pia-wg`; the host default route is still via `enp1s0`.
### Recovery state machine (`recover`, every 60 s)
| State | Action |
| --- | --- |
| healthy | reset counters; if a backoff was pending, log/notify "healthy again" |
| 1–2 consecutive stale/absent | count only |
| 3 consecutive (`pia_recovery_failure_threshold`) | save pre-fix evidence (age, endpoint, reasons) → **stage 1**: `systemctl restart wg-quick@pia-wg` once, wait up to 90 s for two consecutive healthy classifications with a handshake newer than the attempt start |
| stage 1 fails | **stage 2**: serverlist → token → `addKey` with the *existing* key against candidates in `pia_region_candidates` order (max `pia_recovery_max_candidates_per_cycle`), each accepted response staged as a validated config, activated, and verified the same way; a candidate that answers `OK` but never handshakes is skipped for the next one |
| stage 2: API accepted the key but nothing handshook | **stage 3**: exactly one staged fresh-key attempt (`pia-wg.key.new.<ts>.<pid>`, promoted only after verification, previous key backed up). Skipped when the failure was uniform (auth error, host connectivity, every API call failing the same way) |
| any failure | previous config restored (best effort) and re-activated; gateway stays closed; backoff 5 → 15 → 30 → 60 min (±10 % jitter, capped) persisted in state; classified failure recorded; alert |
| verified recovery | stage, region/server, elapsed time and pre-fix age recorded in `last_recovery`; `last_verified_registration` updated when a registration happened; one "recovered" alert |
| safety drift | **no** restart/registration. Insert `-I FORWARD 1 -s 10.10.50.0/24 -m comment --comment pia-gateway-emergency -j DROP`, then `ip route replace unreachable default table pia`, verify the DROP is first, alert `error`. If the rule already exists but isn't first any more (something else prepended a rule ahead of it, e.g. an out-of-band `ufw reload`), it's deleted and reinserted at position 1 rather than left where it drifted to — a mere existence check (`iptables -C`) says nothing about position. Left in place until a later pass sees the invariants intact, then removed under the lock. Automatic repair of the kill switch itself is deliberately **not** attempted (see "Residual risk") |
| paused (`pia-gateway pause`) | checks and notifications continue; never restart/register/renew. Safety drift is **still** contained by the emergency close while paused — pausing suspends recovery, never the fail-closed guarantee |
| corrupt/unknown-schema state file | refuse to act (exit 2, unit failure → alert). `pia-gateway state-reset --confirm` moves the file aside and starts fresh |
One pass is bounded by `pia_recovery_max_duration_sec` (600 s) inside
the program and `TimeoutStartSec` (900 s) in the unit; systemd never
starts a second instance while one runs, and the lock makes a manual
`register` wait (up to `pia_recovery_lock_wait_sec`) rather than
interrupt an activation. Reboots: the boot id is tracked, a new boot gets
`pia_recovery_startup_grace_sec` before stale counts; retry/backoff state
survives because it is on disk. A backward wall-clock step (NTP) that
leaves a persisted retry deadline further out than the backoff cap is
treated as a clock jump and the deadline is cleared rather than stalling
recovery.
`restart` returning 0, `active (exited)`, or `addKey` returning `status:
OK` are never treated as success on their own — only an observed fresh
handshake on the activated config is, and specifically one reported
**strictly after** the moment activation started: a handshake in the
same whole second as that moment doesn't count either, since `wg show`
only has one-second resolution and there'd be no way to tell it apart
from a stale, pre-activation handshake that happens to share the second.
Every mutating command — `recover`, `renew`, `register`,
`rewrite-config`, `pause`, `resume`, `state-reset` — takes the same
`/run/lock/pia-gateway.lock` **before** reading state, not after: reading
state first and only locking around the write would let a slower command
silently overwrite whatever a faster one wrote in between (in particular,
it's how a `pause` could otherwise be lost to an in-flight recovery pass
that read state before the pause ran and only saves its own copy back
once it finishes). `pause`/`resume`/`register`/`rewrite-config`/
`state-reset` wait up to `pia_recovery_lock_wait_sec` for the lock and
exit 4 if it's still busy; `recover`/`renew` (the timers) never wait —
they just skip that tick if something else currently holds it.
### Operator commands (all on minisforum, as root)
```bash
pia-gateway status # human summary incl. backoff, last verified registration, pause
pia-gateway check # what the health-check timer runs (read-only, exit 1 = unhealthy)
pia-gateway check --json # machine-readable
pia-gateway pause --reason "maintenance" # persists across reboot; suppresses recover AND renew; kill switch untouched
pia-gateway resume
pia-gateway retry-now # one recovery pass ignoring threshold and backoff
pia-gateway register [--mode same-key|new-key] [--region hk] # explicit transaction, verified, under the lock
pia-gateway rewrite-config # re-render pia-wg.conf from the recorded registration (hook changes), no API call
pia-gateway renew --force-due # exercise renewal without waiting 21 days
pia-gateway state-reset --confirm # only after reading the corrupt file it moves aside
journalctl -u pia-gateway-recover -u pia-gateway-renew -u pia-gateway-healthcheck -n 100
cat /var/lib/pia-gateway/status.json
```
Through Ansible (repo root, `-K -J`):
```bash
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml -K -J # idempotent; registers only if no config or no verified baseline
ansible-playbook ... -e pia_force_reregister=true # re-register with the EXISTING key (same-key)
ansible-playbook ... -e pia_force_key_rotation=true # stage a NEW key, register it, promote only after verification
ansible-playbook ... -e pia_region=hk # single-region override (either of the above)
ansible-playbook ... -e pia_auto_recovery_enabled=false # stop+disable both timers, keep everything else (rollback)
ansible-playbook ... -e pia_renewal_enabled=true # after the live drills — enables the daily renewal timer
ansible-playbook ... -e pia_remove_host_credentials=true # deliberate removal of /etc/pia-gateway/credentials.env
```
`pia_force_reregister` **reuses** the on-disk key; only
`pia_force_key_rotation` generates a new one (the root README used to say
otherwise — corrected). A normal run also quiesces both timers, waits for
any in-flight pass, and re-enables them after handlers, so an Ansible
restart never races an automatic one. A run whose only change is
`config.json` (e.g. flipping `pia_mss_clamp_enabled`) re-renders the
WireGuard config via `rewrite-config` without touching PIA — the
"template-only change needs a full re-registration" gap from the
2026-08-25 incident is closed.
### Proactive renewal (`renew`)
Off until enabled. When on, once a day it re-registers with the
**existing** key if `last_verified_registration` is older than
`pia_renewal_after_days` (21) — or missing, which is why the baseline
must come from a *verified* run of the shared path, never from a file
mtime. Same transaction, same lock, same verification. A failed renewal
leaves the baseline untouched and enters the same bounded backoff (it
does not silently wait another 21 days); if the tunnel is already
unhealthy it defers to the recovery path. Renewal restarts `pia-wg`, so
qBittorrent/JDownloader connections drop for a few seconds at ~03:30 —
verify they reconnect during the drill. Routine key rotation is not
done: rotation happens only in recovery stage 3, on evidence.
### Alerts and monitoring
Events (not every 60 s): recovery started, recovered, recovery failed +
backoff (`warn`; `error` for auth failure), safety drift (`error`),
emergency rule removed, renewal failed, status unknown ×3, state corrupt,
unit crashed (via `OnFailure`). Delivery is `POST` to
`pia_alert_bridge_url` with the shared alert-bridge bearer key
(`{"source": "pia-gateway@minisforum", "message": ..., "level": ...}` —
schema read from `~/repo/home-service/alert-bridge`, which has **no**
per-caller tokens, so the cluster's key is what gets provisioned;
accepted tradeoff). 5 s timeout; failures are queued in state (bounded,
20) and flushed on the next pass; they never block recovery.
`metrics.prom` is node-exporter textfile format
(`pia_gateway_healthy`, `_tunnel_ok`, `_safety_ok`,
`_handshake_age_seconds`, `_consecutive_failures`, `_paused`,
`_recovery_in_backoff`, `_last_verified_registration_timestamp_seconds`,
`_check_timestamp_seconds`, `_emergency_close_active`). **Not yet
scraped**: `values/kube-prometheus-stack.yaml` has no textfile collector
and node-exporter runs as a pod, so wiring it needs a hostPath mount +
`--collector.textfile.directory` plus a `PrometheusRule` including a
stale-check rule (`time() - pia_gateway_check_timestamp_seconds > 300`)
so a stopped timer alerts even when the last value was healthy. Until
that exists, a dead controller is only visible as silence — check
`systemctl list-timers 'pia-gateway-*'` and `status.json`'s timestamp.
### Credentials on the host
`/etc/pia-gateway/credentials.env` holds the PIA account username and
password in plaintext, root-only. This is the explicit tradeoff the
design requires: recovery must work while the laptop is off, and PIA's
token is only valid for 24 h, so an unattended host needs a renewable
credential. The program reads the file at runtime and never places
values in argv, environment, logs, notifications, metrics or state; core
dumps are disabled (`LimitCORE=0`) and unexpected exceptions are logged
by class name only. **Rotation:** change `PIA_PASSWORD` in `.env`, run
the playbook (the file is re-rendered with `no_log`), then `pia-gateway
register` (or wait for the next cycle). **Revocation:** `-e
pia_remove_host_credentials=true` removes the file (recovery then fails
closed with `auth`/credentials errors and alerts); change the PIA
password; disabling `pia_auto_recovery_enabled` alone never deletes it.
If an on-host secret is unacceptable, the alternative is an always-on
external secret broker — not a 24-hour token.
### Residual risk (read before trusting it unattended)
- Safety drift is contained by the emergency DROP + unreachable route,
not repaired: the kill switch's own rule/jump can be deleted by
something outside this role (a manual `ufw reload` flushes `FORWARD`,
removing both the jump **and** the emergency rule). The controller
re-inserts the emergency rule on its next pass (≤ 60 s) and every boot
re-seeds the kill switch; that 60 s window after an out-of-band flush
is the residual leak risk, and it predates this work.
- Verification is handshake-based. It proves the tunnel is talking to
PIA, not that a VLAN 50 packet leaves via PIA; the end-to-end check is
the SOCKS-proxy/egress-IP step in the live drill.
- The alert-bridge key on minisforum is the cluster's shared key.
- `unknown` (a read command failing) is logged/alerted but never acted
on, by design.
### Rollout (nothing below has been run live yet)
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
table pia`, `iptables -S FORWARD | head -3`, `iptables -S PIA-VLAN50`,
`journalctl -u pia-gateway-healthcheck -n 5`, `curl --socks5-hostname
10.10.40.20:1080 https://ifconfig.me` from the LAN, `wg show wg0`,
`kubectl get nodes`. Confirm console access.
2. Deploy with timers off: `ansible-playbook ... -K -J -e
pia_auto_recovery_enabled=false`. Check `ls -l /etc/pia-gateway
/var/lib/pia-gateway`, `pia-gateway status`, `systemctl list-timers
'pia-gateway-*'` (recover/renew inactive). Because the host has no
verified baseline yet, this run **registers through the shared path**
(same-key, explicit restart + handshake verification) — watch the
task output; if it fails, `-e pia_registration_backend=legacy`
restores the previous behaviour.
3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy
pass makes no changes (`journalctl -u pia-gateway-recover -n 5`).
4. Drill the silent death. First get the **live WireGuard UDP endpoint**
from the interface itself — **not** port 1337, which is only the
`addKey` registration API port and has nothing to do with the tunnel's
own data-plane port (PIA WireGuard servers commonly use a different
port per server):
```bash
sudo wg show pia-wg endpoints # <peer-pubkey>\t<ip>:<port> — this ip:port is the block target
```
Then, on minisforum, leave `pia-wg` administratively up and drop only
that path:
```bash
sudo iptables -I OUTPUT -d <endpoint ip> -p udp --dport <endpoint port> -j DROP
```
Expect: 3 stale passes → restart → (still blocked) registration to a
different server → recovery; record times from the journal. To test
backoff, block every candidate server before recovery can reach it —
since each candidate's actual WireGuard port isn't known until its own
`addKey` response arrives (only the registration port, 1337, is fixed
and shared), block by IP alone rather than guessing a port:
```bash
sudo iptables -I OUTPUT -d <candidate ip> -p udp -j DROP # repeat per candidate IP, or -e pia_region=hk to narrow the set first
```
Remove the rule(s) and confirm autonomous recovery. `iptables -D`
every injected rule afterwards — check `iptables -S OUTPUT` is back to
its pre-drill state, not just that the tunnel recovered.
5. Exercise: a failed first server (block one IP only), invalid
credentials (temporarily wrong `PIA_PASSWORD` → expect `auth_failure`,
no rotation, error alert), alert-bridge down, `pia-gateway register`
while a pass runs (expect it to wait), reboot with a pending backoff,
`pause`/`resume`.
6. During every transition, capture on `enp1s0` with an explicit
**egress-only** direction filter:
```bash
sudo tcpdump -ni enp1s0 -Q out src net 10.10.50.0/24
```
`-Q out` matters: VLAN 50 traffic **arriving** at minisforum on
`enp1s0` (from the Flint router, for policy routing into table `pia`)
is normal and expected — a plain `src net 10.10.50.0/24` filter with
no direction would match that inbound traffic too and could hide a
real leak in the noise, or be misread as one. Only a hit in the `out`
direction — a VLAN 50-sourced packet actually being transmitted out
`enp1s0` toward the LAN/Internet instead of via `pia-wg` — is a leak,
and this must stay completely silent through every failure and
recovery transition. Also capture on `pia-wg` itself to confirm
traffic **is** flowing there while open. Verify a request through
`10.10.40.20:1080` and qBittorrent/JDownloader afterwards; check IPv6
separately; confirm `wg0`, the host default route, Technitium and k3s.
7. Only then `-e pia_renewal_enabled=true`; `pia-gateway renew
--force-due`; confirm `last_verified_registration.epoch` advances only
on verification. Wire the Prometheus stale-check before calling the
feature done. Record actual times and captures in the incident file.
## Rollback
No tag-based rollback is defined — this role doesn't have an "absent"
mode. Roll back manually on minisforum (reverses this role without
**Partial rollback (keep the gateway, drop only the automation):**
`ansible-playbook ... -e pia_auto_recovery_enabled=false` stops and
disables both timers and leaves the program, credentials, health check,
kill switch and `pia-wg` in place. Add `-e pia_remove_host_credentials=true`
to also delete the on-host PIA credentials; nothing else deletes them.
No tag-based full rollback is defined — this role doesn't have an
"absent" mode. Roll back manually on minisforum (reverses this role without
touching `wg0`, k3s, or the host default route):
```bash
sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \
/etc/systemd/system/pia-gateway-healthcheck.service \
/etc/systemd/system/pia-gateway-healthcheck.timer
/etc/systemd/system/pia-gateway-healthcheck.timer \
/etc/systemd/system/pia-gateway-recover.service \
/etc/systemd/system/pia-gateway-recover.timer \
/etc/systemd/system/pia-gateway-renew.service \
/etc/systemd/system/pia-gateway-renew.timer \
/etc/systemd/system/pia-gateway-onfailure.service
sudo systemctl daemon-reload
sudo iptables -D FORWARD -s 10.10.50.0/24 -m comment --comment pia-gateway-emergency -j DROP 2>/dev/null || true
sudo rm -rf /etc/pia-gateway /var/lib/pia-gateway /usr/local/sbin/pia-gateway /run/lock/pia-gateway.lock # includes the on-host PIA credentials
sudo ip rule del from 10.10.50.0/24 table pia priority 100
sudo ip route flush table pia
sudo sed -i '/^[0-9]\+\s\+pia$/d' /etc/iproute2/rt_tables
@ -225,7 +551,7 @@ sudo iptables -F PIA-VLAN50
sudo iptables -X PIA-VLAN50
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp
sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.new /etc/wireguard/pia-ca.crt
sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.conf.bak-* /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.bak-* /etc/wireguard/pia-wg.key.new* /etc/wireguard/pia-ca.crt
```
Inspection commands to confirm rollback actually took (read-only):

View File

@ -123,3 +123,64 @@ pia_mss_clamp_enabled: true
# see tasks/healthcheck.yaml — it must never itself open the kill switch.
pia_healthcheck_interval_sec: 60
pia_healthcheck_handshake_max_age_sec: 180
# ── Automatic recovery (files/pia_gateway.py, tasks/controller.yaml) ────────
#
# Which implementation performs PIA registration when this role runs:
# shared — `pia-gateway register` on the host (the same code the unattended
# recovery/renewal timers use; activates and VERIFIES a handshake
# before reporting success).
# legacy — the original in-Ansible token/addKey flow (tasks/register.yaml).
# Kept only until the shared path has passed a live manual
# re-registration; then it is to be deleted (see README).
pia_registration_backend: shared
# Master switch for the reactive recovery timer. false stops+disables the
# timer but leaves the program, credentials, kill switch, health check and
# pia-wg in place (controlled rollback). Renewal has its own switch below.
pia_auto_recovery_enabled: true
# Proactive same-key re-registration when the last VERIFIED registration is
# older than pia_renewal_after_days. Off until the reactive path has passed
# its live drills (README "Rollout"). Needs a verified baseline in state —
# established by the shared registration path, never inferred from file
# mtimes.
pia_renewal_enabled: false
pia_renewal_after_days: 21
pia_renewal_calendar: "*-*-* 03:30:00"
pia_renewal_jitter_sec: 1800
# Recovery state machine tunables (seconds unless noted).
pia_recovery_failure_threshold: 3
pia_recovery_startup_grace_sec: 120
pia_recovery_restart_wait_sec: 90
pia_recovery_activation_wait_sec: 90
pia_recovery_max_duration_sec: 600
pia_recovery_backoff_schedule_sec: [300, 900, 1800, 3600]
pia_recovery_max_candidates_per_cycle: 6
pia_recovery_max_rotation_candidates: 3
pia_recovery_http_connect_timeout_sec: 10
pia_recovery_http_total_timeout_sec: 30
pia_recovery_backup_keep: 5
pia_recovery_lock_wait_sec: 600
# Where the on-host controller keeps things.
pia_gateway_program: /usr/local/sbin/pia-gateway
pia_gateway_etc_dir: /etc/pia-gateway
pia_gateway_state_dir: /var/lib/pia-gateway
# alert-bridge (manifests/home-services/alert-bridge*.yaml) delivery for
# state-transition events. Its API has ONE shared bearer key (no per-caller
# tokens — confirmed against the alert-bridge source), so the same key the
# cluster uses is provisioned to minisforum's root-only credentials file
# when pia_alert_bridge_api_key is set (host_vars reads it from .env). Set
# the URL to null to disable delivery; events still go to the journal.
pia_alert_bridge_url: "http://alert-bridge.home.arpa/alerts"
pia_alert_source: "pia-gateway@minisforum"
pia_alert_bridge_api_key: ""
# Deliberate credential removal. Disabling automation never deletes the
# on-host PIA credentials on its own; set this true (one run) to remove
# {{ pia_gateway_etc_dir }}/credentials.env, e.g. when decommissioning or
# after rotating the PIA password elsewhere.
pia_remove_host_credentials: false

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,98 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml (before registration — the shared
# registration backend is this program)
# Description: Installs the on-host controller (files/pia_gateway.py), its
# root-only runtime config and credentials, and the systemd units for
# the recovery / renewal timers. Timers are only ENABLED at the end of
# tasks/main.yaml, after the kill switch, interface and a verified
# registration exist — this file never starts them.
- name: Create the controller config directory (root-only)
ansible.builtin.file:
path: "{{ pia_gateway_etc_dir }}"
state: directory
mode: "0700"
owner: root
group: root
- name: Create the controller state directory (root-only)
ansible.builtin.file:
path: "{{ pia_gateway_state_dir }}"
state: directory
mode: "0700"
owner: root
group: root
- name: Install the pia-gateway program
ansible.builtin.copy:
src: pia_gateway.py
dest: "{{ pia_gateway_program }}"
mode: "0700"
owner: root
group: root
- name: Render the controller runtime config
ansible.builtin.template:
src: pia-gateway-config.json.j2
dest: "{{ pia_gateway_etc_dir }}/config.json"
mode: "0600"
owner: root
group: root
register: pia_gateway_config_result
- name: Validate the rendered config loads
ansible.builtin.command:
cmd: "{{ pia_gateway_program }} --config {{ pia_gateway_etc_dir }}/config.json status"
register: pia_gateway_config_check
changed_when: false
failed_when: pia_gateway_config_check.rc not in [0, 1, 2]
# `status`'s own source (cmd_status in files/pia_gateway.py) only ever
# returns one of three documented exits: 0 (healthy), 1 (unhealthy —
# the expected result on a fresh install, since the kill switch and
# pia-wg interface don't exist yet at this point in the run), or 2
# (state file corrupt — not a config problem either). Allowlisting
# those three and failing on EVERYTHING else (5 = bad config JSON/
# unknown key from main()'s own Config.load(), 3/4 = exits this
# command cannot legitimately return, 126/127/a crash) means a real
# config regression is never masked by matching only one "bad" code —
# the previous version of this task only failed on rc==5 and silently
# ignored any other unexpected exit.
- name: Provision PIA credentials for unattended re-registration (root-only)
ansible.builtin.template:
src: pia-gateway-credentials.env.j2
dest: "{{ pia_gateway_etc_dir }}/credentials.env"
mode: "0600"
owner: root
group: root
no_log: true
when: not (pia_remove_host_credentials | bool)
# Only PIA_USER/PIA_PASSWORD (and the alert-bridge key when set) — never
# the whole .env, the vault password, or any Kubernetes secret. This is
# the deliberate availability/security tradeoff documented in the README:
# a host that must re-register while the laptop is off needs the account
# secret locally.
- name: Remove the on-host PIA credentials (explicit request only)
ansible.builtin.file:
path: "{{ pia_gateway_etc_dir }}/credentials.env"
state: absent
when: pia_remove_host_credentials | bool
- name: Deploy controller systemd units
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
owner: root
group: root
loop:
- pia-gateway-recover.service
- pia-gateway-recover.timer
- pia-gateway-renew.service
- pia-gateway-renew.timer
- pia-gateway-onfailure.service
notify: Reload systemd daemon
# The reload runs at the first flush_handlers in main.yaml, before any
# of these units is enabled; the enable tasks also pass daemon_reload.

View File

@ -1,17 +1,17 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml
# Description: Installs the observability-only health check (script,
# oneshot service, timer). See templates/pia-gateway-healthcheck.sh.j2
# for exactly what it checks and why it never remediates.
# Description: Installs the observability-only health check (oneshot
# service + timer). Since the auto-recovery work it runs
# `pia-gateway check` — the same status classification the recovery
# controller uses — instead of the earlier standalone bash script, so
# the two can never disagree about what "healthy" means. It still never
# remediates.
- name: Deploy the health check script
ansible.builtin.template:
src: pia-gateway-healthcheck.sh.j2
dest: /usr/local/sbin/pia-gateway-healthcheck.sh
mode: "0700"
owner: root
group: root
- name: Remove the superseded bash health check script
ansible.builtin.file:
path: /usr/local/sbin/pia-gateway-healthcheck.sh
state: absent
- name: Deploy the health check systemd service
ansible.builtin.template:
@ -40,3 +40,4 @@
name: pia-gateway-healthcheck.timer
enabled: true
state: started
daemon_reload: true

View File

@ -2,10 +2,23 @@
# Part of role: pia-gateway
# Called by: ansible/playbooks/pia-gateway.yaml
# Description: Orchestrates minisforum's PIA WireGuard egress gateway for
# VPN VLAN 50 — registration, boot-ordered kill switch + policy routing,
# the WireGuard interface itself, and health checks. Split into included
# files because each stage has a distinct idempotency/ordering concern;
# see the comment at the top of each included file.
# VPN VLAN 50 — kill switch + policy routing, the on-host recovery
# controller, registration, the WireGuard interface itself, and health
# checks. Split into included files because each stage has a distinct
# idempotency/ordering concern; see the comment at the top of each.
#
# Ordering (changed with the auto-recovery work, deliberately):
# 1. tooling, baseline assertions
# 2. controller program + config + credentials (NOT its timers)
# 3. quiesce the recovery/renewal timers for the rest of the run so a
# timer-driven restart can never race an Ansible handler restart
# 4. kill switch + routing + firewall, handlers flushed so the closed
# state is live BEFORE anything can activate a tunnel
# 5. registration (shared backend activates + verifies under the lock;
# the legacy backend only writes the config and notifies a restart)
# 6. interface unit enabled, health check installed, handlers flushed
# 7. timers enabled/disabled per pia_auto_recovery_enabled /
# pia_renewal_enabled
- name: Install WireGuard tooling and iproute2
ansible.builtin.apt:
@ -13,24 +26,57 @@
- wireguard
- wireguard-tools
- iproute2
- python3-requests
- python3
state: present
update_cache: true
# python3-requests is no longer needed: the controller uses only the
# standard library.
- name: Pin reverse-path filtering for {{ pia_wg_interface }}
ansible.posix.sysctl:
name: "net.ipv4.conf.{{ pia_wg_interface }}.rp_filter"
value: "{{ pia_rp_filter_mode | string }}"
sysctl_file: /etc/sysctl.d/60-pia-gateway.conf
reload: false # interface does not exist yet on a first run; applied for real after the WG interface comes up (handlers/main.yaml)
reload: false # interface does not exist yet on a first run; applied for real after the WG interface comes up
register: pia_rp_filter_pinned
- name: Confirm host default route and ip_forward are untouched
ansible.builtin.include_tasks: assert-baseline.yaml
- name: Register minisforum with PIA and obtain WireGuard peer config
ansible.builtin.include_tasks: register.yaml
no_log: true # PIA credentials and the derived private key pass through this include
- name: Install the recovery controller, its config and credentials
ansible.builtin.include_tasks: controller.yaml
- name: Check whether the recovery timer is installed from a previous run
ansible.builtin.stat:
path: /etc/systemd/system/pia-gateway-recover.timer
register: pia_recover_timer_stat
- name: Quiesce the recovery and renewal timers for the duration of this run
ansible.builtin.systemd:
name: "{{ item }}"
state: stopped
loop:
- pia-gateway-recover.timer
- pia-gateway-renew.timer
when: pia_recover_timer_stat.stat.exists
failed_when: false
# failed_when: false — on the very first install the units were just
# deployed and never started; "stopping" them is a harmless no-op that
# some systemd versions report as an error for never-loaded timers.
- name: Wait for any in-flight recovery/renewal pass to finish
ansible.builtin.command:
cmd: systemctl is-active pia-gateway-recover.service pia-gateway-renew.service
register: pia_inflight
changed_when: false
failed_when: false
until: "'active' not in (pia_inflight.stdout_lines | default([]) | map('trim') | list)"
retries: 90
delay: 10
when: pia_recover_timer_stat.stat.exists
# A pass can take up to pia_recovery_max_duration_sec (+ restart waits);
# we never kill it mid-activation. Registration below also takes the
# program's own lock, so this is belt-and-braces for the handler restarts.
- name: Configure boot-ordered kill switch and policy routing
ansible.builtin.include_tasks: routing.yaml
@ -38,6 +84,18 @@
- name: Configure firewall-layer kill switch and Technitium access
ansible.builtin.include_tasks: firewall.yaml
- name: Apply kill-switch handlers before anything can activate a tunnel
ansible.builtin.meta: flush_handlers
- name: Register minisforum with PIA (shared on-host transaction)
ansible.builtin.include_tasks: register-shared.yaml
when: pia_registration_backend == "shared"
- name: Register minisforum with PIA (legacy in-Ansible flow)
ansible.builtin.include_tasks: register-legacy.yaml
no_log: true # PIA credentials and the derived private key pass through this include
when: pia_registration_backend == "legacy"
- name: Write and enable the pia-wg WireGuard interface
ansible.builtin.include_tasks: interface.yaml
@ -51,3 +109,54 @@
sysctl_file: /etc/sysctl.d/60-pia-gateway.conf
reload: true
when: pia_rp_filter_pinned is succeeded
- name: Apply remaining handlers while the timers are still quiesced
ansible.builtin.meta: flush_handlers
- name: Re-render pia-wg.conf when only the controller config changed (no PIA call)
ansible.builtin.command:
cmd: "{{ pia_gateway_program }} rewrite-config"
register: pia_rewrite_result
changed_when: pia_rewrite_result.rc == 0
failed_when: pia_rewrite_result.rc not in [0, 5]
when:
- pia_registration_backend == "shared"
- pia_gateway_config_result is changed
- not (pia_registered_this_run | default(false) | bool)
# Picks up hook-content changes (e.g. pia_mss_clamp_enabled) without a
# forced re-registration — the gap noted in the 2026-08-25 incident.
# Exit 5 = "no verified registration recorded yet" (legacy-registered
# host); that is not an error here, the next registration will render
# the current hooks.
- name: Enable the recovery timer
ansible.builtin.systemd:
name: pia-gateway-recover.timer
enabled: true
state: started
daemon_reload: true
when: pia_auto_recovery_enabled | bool
- name: Disable the recovery timer (pia_auto_recovery_enabled=false)
ansible.builtin.systemd:
name: pia-gateway-recover.timer
enabled: false
state: stopped
daemon_reload: true
when: not (pia_auto_recovery_enabled | bool)
- name: Enable the renewal timer
ansible.builtin.systemd:
name: pia-gateway-renew.timer
enabled: true
state: started
daemon_reload: true
when: (pia_auto_recovery_enabled | bool) and (pia_renewal_enabled | bool)
- name: Disable the renewal timer
ansible.builtin.systemd:
name: pia-gateway-renew.timer
enabled: false
state: stopped
daemon_reload: true
when: not ((pia_auto_recovery_enabled | bool) and (pia_renewal_enabled | bool))

View File

@ -1,8 +1,26 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml (with no_log: true — this file handles the
# PIA account password, the derived WireGuard private key, and the
# short-lived PIA auth token)
# Included by: tasks/main.yaml ONLY when pia_registration_backend ==
# "legacy" (with no_log: true — this file handles the PIA account
# password, the derived WireGuard private key, and the short-lived PIA
# auth token)
#
# LEGACY PATH — superseded by tasks/register-shared.yaml, which calls the
# on-host controller (files/pia_gateway.py) so Ansible and the unattended
# recovery/renewal timers share ONE registration implementation that also
# activates and verifies a handshake. This file is kept, unchanged, only as
# the documented fallback until the shared path has passed a live manual
# re-registration on minisforum (README "Rollout"). Delete it — and
# addkey-attempt.yaml, the `legacy` option, and pia-wg.conf.j2 — once that
# is done. Differences to be aware of while it still exists:
# - it does NOT verify a handshake (config write + handler restart only)
# - it does NOT record a verified-registration baseline in the
# controller state, so the renewal timer treats the host as "never
# verified" until the shared path runs once
# - its rotation stages a fixed `.key.new` path with `creates:` (an
# abandoned stage could be reused); the controller removes any
# `.key.new*` file before its own attempts.
#
# Description: Registers minisforum as a PIA WireGuard peer and writes
# {{ pia_wg_config_dir }}/{{ pia_wg_interface }}.conf. Skipped once that
# file exists, unless pia_force_reregister or pia_force_key_rotation is

View File

@ -0,0 +1,64 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml when pia_registration_backend == "shared"
# Description: Registers with PIA through the on-host controller
# (`pia-gateway register`), i.e. the same token/addKey/stage/activate/
# verify transaction the unattended recovery and renewal timers use.
# Runs when there is no config yet, no VERIFIED registration baseline in
# the controller's state, or an explicit force flag. Reports success only
# after a fresh WireGuard handshake was observed on the activated config
# — never on `addKey` HTTP status alone.
#
# Requires the kill switch to be active (tasks/routing.yaml runs first in
# main.yaml); the program refuses to activate otherwise (exit 3) and the
# play fails loudly rather than opening anything.
- name: Check whether a PIA WireGuard config already exists
ansible.builtin.stat:
path: "{{ pia_wg_config_dir }}/{{ pia_wg_interface }}.conf"
register: pia_wg_conf_stat
- name: Check whether the controller has a verified registration baseline
ansible.builtin.stat:
path: "{{ pia_gateway_state_dir }}/state.json"
register: pia_state_stat
- name: Read the controller state (no secrets are stored in it)
ansible.builtin.slurp:
src: "{{ pia_gateway_state_dir }}/state.json"
register: pia_state_raw
when: pia_state_stat.stat.exists
- name: Decide whether registration is needed this run
ansible.builtin.set_fact:
pia_has_baseline: >-
{{ pia_state_stat.stat.exists
and ((pia_state_raw.content | b64decode | from_json).last_verified_registration is not none) }}
pia_registration_mode: "{{ 'new-key' if (pia_force_key_rotation | bool) else 'same-key' }}"
pia_region_args: "{{ ['--region', pia_region] if (pia_region is not none and (pia_region | string | trim | length) > 0) else [] }}"
- name: Run the shared PIA registration transaction
ansible.builtin.command:
argv: "{{ [pia_gateway_program, 'register', '--via', 'ansible', '--mode', pia_registration_mode] + pia_region_args }}"
register: pia_register_result
changed_when: true
failed_when: pia_register_result.rc != 0
when: >-
(pia_force_reregister | bool)
or (pia_force_key_rotation | bool)
or not pia_wg_conf_stat.stat.exists
or not (pia_has_baseline | bool)
# argv form: no shell, no secrets in arguments (credentials are read from
# the root-only file by the program itself), so this task is deliberately
# NOT no_log — its stdout is the secret-free JSON summary and the
# per-candidate diagnostics, which are exactly what an operator needs
# when a run fails. The lock is taken inside the program (waits up to
# pia_recovery_lock_wait_sec for an in-flight timer run to finish).
- name: Record that registration happened this run
ansible.builtin.set_fact:
pia_registered_this_run: "{{ pia_register_result is not skipped }}"
- name: Show the registration summary
ansible.builtin.debug:
msg: "{{ pia_register_result.stdout_lines | default(['skipped — config and verified baseline already present']) }}"

View File

@ -0,0 +1,36 @@
{
"interface": {{ pia_wg_interface | to_json }},
"config_dir": {{ pia_wg_config_dir | to_json }},
"state_dir": {{ pia_gateway_state_dir | to_json }},
"credentials_file": {{ (pia_gateway_etc_dir + '/credentials.env') | to_json }},
"ca_file": {{ (pia_wg_config_dir + '/pia-ca.crt') | to_json }},
"lock_file": "/run/lock/pia-gateway.lock",
"route_table": {{ pia_route_table_name | to_json }},
"rule_priority": {{ pia_rule_priority | int }},
"vlan50_subnet": {{ pia_vlan50_subnet | to_json }},
"iptables_chain": {{ pia_iptables_chain | to_json }},
"lan_interface": {{ pia_lan_interface | to_json }},
"mss_clamp_enabled": {{ pia_mss_clamp_enabled | bool | to_json }},
"region_candidates": {{ pia_region_candidates | to_json }},
"handshake_max_age_sec": {{ pia_healthcheck_handshake_max_age_sec | int }},
"failure_threshold": {{ pia_recovery_failure_threshold | int }},
"startup_grace_sec": {{ pia_recovery_startup_grace_sec | int }},
"restart_wait_sec": {{ pia_recovery_restart_wait_sec | int }},
"activation_wait_sec": {{ pia_recovery_activation_wait_sec | int }},
"recovery_max_duration_sec": {{ pia_recovery_max_duration_sec | int }},
"backoff_schedule_sec": {{ pia_recovery_backoff_schedule_sec | to_json }},
"backoff_jitter_fraction": 0.1,
"max_candidates_per_cycle": {{ pia_recovery_max_candidates_per_cycle | int }},
"max_rotation_candidates": {{ pia_recovery_max_rotation_candidates | int }},
"renewal_after_days": {{ pia_renewal_after_days | int }},
"http_connect_timeout_sec": {{ pia_recovery_http_connect_timeout_sec | int }},
"http_total_timeout_sec": {{ pia_recovery_http_total_timeout_sec | int }},
"clock_skew_allowance_sec": 60,
"backup_keep": {{ pia_recovery_backup_keep | int }},
"lock_wait_sec": {{ pia_recovery_lock_wait_sec | int }},
"alert_bridge_url": {{ pia_alert_bridge_url | to_json }},
"alert_source": {{ pia_alert_source | to_json }},
"metrics_file": {{ (pia_gateway_state_dir + '/metrics.prom') | to_json }},
"status_file": {{ (pia_gateway_state_dir + '/status.json') | to_json }},
"killswitch_unit": "pia-killswitch.service"
}

View File

@ -0,0 +1,7 @@
# Managed by Ansible (role: pia-gateway). Root-only (0600). Read at runtime by
# /usr/local/sbin/pia-gateway; never passed via argv or environment.
PIA_USER={{ pia_user }}
PIA_PASSWORD={{ pia_password }}
{% if pia_alert_bridge_api_key | default('') | length > 0 %}
ALERT_BRIDGE_API_KEY={{ pia_alert_bridge_api_key }}
{% endif %}

View File

@ -1,7 +1,12 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
# Read-only: `pia-gateway check` classifies status with the same code the
# recovery controller uses, logs it, and refreshes the status/metrics files.
# It never restarts, registers, or touches routing/firewall state.
[Unit]
Description=PIA gateway health check (observability only, no remediation)
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pia-gateway-healthcheck.sh
ExecStart={{ pia_gateway_program }} check
SuccessExitStatus=1
LimitCORE=0

View File

@ -1,110 +0,0 @@
#!/bin/bash
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Observability only. This script NEVER modifies routing, firewall, or
# WireGuard state — a failed check must close the path (already true by
# construction, via the kill switch in tasks/routing.yaml/firewall.yaml),
# never "recover" it. Output goes to the journal
# (journalctl -u pia-gateway-healthcheck); wire alerting off of that
# separately (see plan.md Phase 9) rather than adding remediation here.
#
# The "route decision" check below confirms what the kernel FIB *would*
# do for a VLAN 50 source — it does not send a packet and is not proof of
# actual PIA egress or identity. That proof requires the packet-capture
# test matrix in plan.md Phase 5; do not treat a clean run of this script
# as substituting for it.
set -uo pipefail
WG_IF="{{ pia_wg_interface }}"
TABLE="{{ pia_route_table_name }}"
SUBNET="{{ pia_vlan50_subnet }}"
PRIORITY="{{ pia_rule_priority }}"
CHAIN="{{ pia_iptables_chain }}"
MAX_HANDSHAKE_AGE={{ pia_healthcheck_handshake_max_age_sec }}
PROBE_SRC="${SUBNET%.*}.2" # a representative VLAN 50 address for the route-decision probe only; never actually used as a source
fail=0
# Informational only (not pass/fail) — which region/server this host is
# actually registered against right now, per the comment
# tasks/register.yaml writes into the generated config. Since PIA
# WireGuard registration now falls back across multiple regions
# (pia_region_candidates in defaults/main.yaml), "which one is live"
# isn't implied by config alone — it's whichever one answered OK last
# time the playbook ran, not necessarily the first-configured one.
configured_peer=$(grep -m1 '^# PIA WireGuard server actually used' "{{ pia_wg_config_dir }}/{{ pia_wg_interface }}.conf" 2>/dev/null || true)
if [[ -n "${configured_peer}" ]]; then
echo "INFO configured-peer: ${configured_peer#\# }"
fi
if ip link show "${WG_IF}" up &>/dev/null; then
echo "PASS interface: ${WG_IF} is up"
else
echo "FAIL interface: ${WG_IF} is not up"
fail=1
fi
if command -v wg &>/dev/null && wg show "${WG_IF}" latest-handshakes &>/dev/null; then
handshake_epoch=$(wg show "${WG_IF}" latest-handshakes 2>/dev/null | awk '{print $2}')
now=$(date +%s)
if [[ -n "${handshake_epoch}" && "${handshake_epoch}" -gt 0 ]]; then
age=$((now - handshake_epoch))
if (( age <= MAX_HANDSHAKE_AGE )); then
echo "PASS handshake: ${age}s old (max ${MAX_HANDSHAKE_AGE}s)"
else
echo "FAIL handshake: ${age}s old, exceeds max ${MAX_HANDSHAKE_AGE}s"
fail=1
fi
else
echo "FAIL handshake: no handshake recorded yet"
fail=1
fi
else
echo "WARN handshake: unable to query wg show (interface absent or wg missing)"
fail=1
fi
if ip rule show | grep -qE "^${PRIORITY}:[[:space:]]*from ${SUBNET} lookup ${TABLE}\$"; then
echo "PASS rule: from ${SUBNET} lookup ${TABLE} present at priority ${PRIORITY}"
else
echo "FAIL rule: from ${SUBNET} lookup ${TABLE} missing"
fail=1
fi
default_route=$(ip route show table "${TABLE}" 2>/dev/null | grep '^default' || true)
if [[ "${default_route}" == *"dev ${WG_IF}"* ]]; then
echo "PASS route: table ${TABLE} default is via ${WG_IF} (open)"
elif [[ "${default_route}" == *unreachable* ]] || ip route show table "${TABLE}" 2>/dev/null | grep -q '^unreachable default'; then
echo "PASS route: table ${TABLE} default is unreachable (closed, kill switch engaged)"
else
echo "FAIL route: table ${TABLE} has no default route at all (neither open via ${WG_IF} nor a terminal unreachable) — investigate immediately"
fail=1
fi
# Not a check on FORWARD's own default policy — that's deliberately left
# alone (see tasks/firewall.yaml for why: minisforum runs Flannel, which
# needs its own broad FORWARD ACCEPTs). What actually matters is that the
# dedicated chain exists, is jumped into first for this source, and ends
# in a real DROP.
if iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
echo "PASS firewall: FORWARD jumps to ${CHAIN} for ${SUBNET}"
else
echo "FAIL firewall: FORWARD does not jump to ${CHAIN} for ${SUBNET}"
fail=1
fi
if iptables -S "${CHAIN}" 2>/dev/null | grep -qE '^-A '"${CHAIN}"' -j DROP$'; then
echo "PASS firewall: ${CHAIN} ends in an unconditional DROP"
else
echo "FAIL firewall: ${CHAIN} has no unconditional DROP catch-all"
fail=1
fi
route_decision=$(ip route get 1.1.1.1 from "${PROBE_SRC}" 2>&1 || true)
if [[ "${route_decision}" == *"dev ${WG_IF}"* ]]; then
echo "PASS route-decision (not proof of live egress — see header comment): ${SUBNET} sourced traffic resolves via ${WG_IF}"
else
echo "WARN route-decision (not proof of live egress — see header comment): ${route_decision}"
fi
exit "${fail}"

View File

@ -0,0 +1,12 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
# Fired by OnFailure= of the recovery/renewal units when they crash or refuse
# to run (corrupt state). Delivery goes through alert-bridge if configured;
# a delivery failure is logged and never retried here (the next successful
# recover pass flushes its own pending queue).
[Unit]
Description=PIA gateway controller failure notifier
[Service]
Type=oneshot
ExecStart={{ pia_gateway_program }} notify --level error --message "pia-gateway: a controller unit failed on minisforum (journalctl -u pia-gateway-recover -u pia-gateway-renew)"
LimitCORE=0

View File

@ -0,0 +1,19 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# One bounded recovery pass (files/pia_gateway.py `recover`). Exit codes 1
# (unhealthy, decision made) and 3 (safety drift handled with an emergency
# close + alert) are normal outcomes, not unit failures — only a crash or a
# corrupt state file (2) is, and that triggers the OnFailure notifier.
[Unit]
Description=PIA gateway automatic recovery pass
After=pia-killswitch.service network-online.target
Wants=network-online.target
OnFailure=pia-gateway-onfailure.service
[Service]
Type=oneshot
ExecStart={{ pia_gateway_program }} recover
SuccessExitStatus=1 3
TimeoutStartSec={{ (pia_recovery_max_duration_sec | int) + 300 }}
LimitCORE=0
PrivateTmp=yes

View File

@ -0,0 +1,13 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
# systemd never starts a second instance while pia-gateway-recover.service is
# still running, and the program also holds /run/lock/pia-gateway.lock.
[Unit]
Description=Run the PIA gateway recovery pass every {{ pia_healthcheck_interval_sec }}s
[Timer]
OnBootSec=90
OnUnitActiveSec={{ pia_healthcheck_interval_sec }}
AccuracySec=5s
[Install]
WantedBy=timers.target

View File

@ -0,0 +1,14 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
[Unit]
Description=PIA gateway proactive registration renewal
After=pia-killswitch.service network-online.target
Wants=network-online.target
OnFailure=pia-gateway-onfailure.service
[Service]
Type=oneshot
ExecStart={{ pia_gateway_program }} renew
SuccessExitStatus=1
TimeoutStartSec={{ (pia_recovery_max_duration_sec | int) + 300 }}
LimitCORE=0
PrivateTmp=yes

View File

@ -0,0 +1,11 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
[Unit]
Description=Daily check whether the PIA registration is older than {{ pia_renewal_after_days }} days
[Timer]
OnCalendar={{ pia_renewal_calendar }}
RandomizedDelaySec={{ pia_renewal_jitter_sec }}
Persistent=true
[Install]
WantedBy=timers.target

File diff suppressed because it is too large Load Diff