fix: address code review on pia-gateway auto-recovery controller
Some checks failed
validate / lint (push) Failing after 2s

- Serialize every mutating command (recover, renew, register,
  rewrite-config, pause, resume, state-reset) on the shared lock,
  acquired BEFORE state is read rather than after: reading first and
  locking only around the write let a slower command silently clobber
  whatever a faster one wrote in between — in particular, an operator
  pause could be lost to an in-flight recovery pass that read state
  before the pause ran and only saved its own copy back afterwards.
- Fix emergency_close() to reposition its VLAN 50 DROP to the first
  effective FORWARD position when it exists but has been pushed down
  by something else (e.g. an out-of-band ufw reload) — a bare `-C`
  existence check said nothing about position.
- Tighten wait_for_verified()'s handshake freshness check from >= to >
  since_epoch: wg show and since_epoch are both whole seconds, so a
  handshake at exactly the same second as since_epoch is
  indistinguishable from a stale, pre-activation one and must not
  count as proof of a new one.
- Fix the Ansible config-validation task to fail on any exit code
  outside status's three documented health exits (0/1/2), not just on
  5 — verified live against the real program (no wg/ip/iptables on
  this dev machine reproduces the "nothing configured yet" first-run
  case; status correctly returns 1, and broken/missing config.json
  correctly returns 5 and is now caught).
- Correct the live fault-injection drill in the role README: port
  1337 is only the addKey registration API port, not the tunnel's own
  WireGuard UDP endpoint (derive that from `wg show pia-wg endpoints`
  instead), and make the packet-capture check direction-aware
  (`tcpdump -Q out`) so it can't conflate a real egress leak with
  VLAN 50's normal inbound arrival on enp1s0.

Added 6 tests (86 total, all passing): emergency-close repositioning
past a preceding ACCEPT, pause/resume blocking on an in-flight
lock holder and surviving into a subsequent recovery pass, and the
handshake boundary at exactly/one-second-either-side of since_epoch.

yamllint, ansible-playbook --syntax-check, and ansible-lint (diffed
against main — no new finding categories) all pass. No live changes;
nothing pushed to main.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Nik Afiq 2026-09-22 15:51:02 +09:00
parent f76b1e6f13
commit 87a9303cc3
22 changed files with 4298 additions and 159 deletions

3
.gitignore vendored
View File

@ -8,4 +8,5 @@ kubeconfig*
*.kubeconfig *.kubeconfig
*.swp *.swp
*.swo *.swo
*~ *~
__pycache__/

View File

@ -2,7 +2,7 @@
**Date:** 2026-08-25 to 2026-08-27 (discovered/resolved across this window; underlying triggers occurred 2026-08-24/25) **Date:** 2026-08-25 to 2026-08-27 (discovered/resolved across this window; underlying triggers occurred 2026-08-24/25)
**Systems:** `minisforum` (PIA WireGuard gateway), `nik-debian` (VLAN 50 workloads: `browser-vpn-proxy`, `qbittorrent`, `jdownloader`) **Systems:** `minisforum` (PIA WireGuard gateway), `nik-debian` (VLAN 50 workloads: `browser-vpn-proxy`, `qbittorrent`, `jdownloader`)
**Status:** All four issues resolved live, with code fixes committed for three of them (qBittorrent, MSS clamping, `wg-quick` restart idempotency). The remaining gap (PIA tunnel handshake-death alerting/remediation) is documented with prevention options, deliberately not yet implemented — see "Open follow-up" below. **Status:** All four issues resolved live, with code fixes committed for three of them (qBittorrent, MSS clamping, `wg-quick` restart idempotency). The remaining gap (PIA tunnel handshake-death alerting/remediation) is documented with prevention options, deliberately not yet implemented — see "Open follow-up" below. **Incident 1 has since recurred (2026-09-22) — see its "Recurrences" subsection.**
## Summary ## Summary
@ -89,6 +89,15 @@ ansible-playbook -i ansible/inventory.yaml \
This hit a real, separate, now-fixed bug on the way: `ansible/roles/pia-gateway/tasks/register.yaml`'s `when:` conditions used `pia_force_reregister`/`pia_force_key_rotation` directly in boolean expressions. Passed via the CLI's plain `key=value` form (`-e pia_force_reregister=true`, exactly what the role's own README documents), Ansible sets the variable as the **string** `"true"`, not a real boolean — and recent `ansible-core` rejects using a string directly in a `when:` boolean expression (`Conditional result (True) was derived from value of type 'str'`). Fixed by wrapping every use in `| bool` (commit `e757850`), which correctly coerces both real booleans (the `defaults/main.yaml` case) and `"true"`/`"false"` strings (the CLI-override case). Verified with an isolated local test playbook exercising both cases before trusting it against the live host. This hit a real, separate, now-fixed bug on the way: `ansible/roles/pia-gateway/tasks/register.yaml`'s `when:` conditions used `pia_force_reregister`/`pia_force_key_rotation` directly in boolean expressions. Passed via the CLI's plain `key=value` form (`-e pia_force_reregister=true`, exactly what the role's own README documents), Ansible sets the variable as the **string** `"true"`, not a real boolean — and recent `ansible-core` rejects using a string directly in a `when:` boolean expression (`Conditional result (True) was derived from value of type 'str'`). Fixed by wrapping every use in `| bool` (commit `e757850`), which correctly coerces both real booleans (the `defaults/main.yaml` case) and `"true"`/`"false"` strings (the CLI-override case). Verified with an isolated local test playbook exercising both cases before trusting it against the live host.
### Recurrences
| Date | Discovered by | Pre-fix evidence | Fix | Notes |
| --- | --- | --- | --- | --- |
| 2026-08-25 | User: proxy not working after nik-debian reboot | Handshake 7h29m old; healthcheck logging `FAIL handshake` every 60s; bare `wg-quick` restart did **not** recover | `pia_force_reregister=true` | This report |
| 2026-09-22 (~09:50 JST) | User: browser can't use the SOCKS5 proxy | **Not captured** — the re-registration was run before `wg show` was checked, so handshake age and healthcheck journal were not recorded; bare restart not attempted | `pia_force_reregister=true` (~10:06 JST) | Kubernetes side verified healthy first (`browser-vpn-proxy` pod, Service, both `svclb` pods all Running). Post-fix `latest-handshakes` = `1790039182` (10:06:22 JST), ~47s before the check — fresh. Two doc gaps hit on the way: root README's command omitted `-J` (fails with "no vault secrets found") and nothing said the paths are repo-root-relative (fails from `ansible/`). Both fixed in the README. |
~28 days between the two. Not enough to claim a fixed PIA-side registration lifetime, but enough that the next occurrence should be checked against that hypothesis first — if it lands around 2026-10-20, a scheduled proactive `pia_force_reregister` run (every ~3 weeks) becomes the cheapest prevention, independent of whichever alerting option below is chosen. Next time: **run the diagnostic commands before the fix** (root README, "Tunnel silently dead") so the age and journal are on record.
### Known gap / prevention (not yet implemented — deferred by request) ### Known gap / prevention (not yet implemented — deferred by request)
`ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2` (run via `pia-gateway-healthcheck.timer`, every 60s) **already correctly detected this failure the entire time** — every single run logged `FAIL handshake: ...s old, exceeds max 180s`. The script is explicitly designed as "observability only, no remediation" (its own header comment says so), which is a deliberate prior design choice, not an oversight. The actual gap is that nothing consumes that detection — no alert, no dashboard, nothing — so a real, correctly-detected outage sat unnoticed until a user happened to test the affected service directly. `ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2` (run via `pia-gateway-healthcheck.timer`, every 60s) **already correctly detected this failure the entire time** — every single run logged `FAIL handshake: ...s old, exceeds max 180s`. The script is explicitly designed as "observability only, no remediation" (its own header comment says so), which is a deliberate prior design choice, not an oversight. The actual gap is that nothing consumes that detection — no alert, no dashboard, nothing — so a real, correctly-detected outage sat unnoticed until a user happened to test the affected service directly.
@ -238,7 +247,7 @@ None identified for this specific failure mode going forward — the fix address
## Open follow-up ## Open follow-up
- **Decide and implement PIA tunnel remediation/alerting** (Incident 1's "Known gap" section above) — explicitly deferred by the user to a later date, options documented above. - **Decide and implement PIA tunnel remediation/alerting** (Incident 1's "Known gap" section above) — explicitly deferred by the user to a later date, options documented above. **Recurred 2026-09-22** (see Incident 1's "Recurrences" table). **Implemented 2026-09-22 (option 3 + proactive renewal)** on branch `pia-gateway-auto-recovery`, per `PLAN-pia-gateway-auto-recovery.md`: an on-host controller (`ansible/roles/pia-gateway/files/pia_gateway.py`, `pia-gateway-recover.timer`) that restarts, then re-registers with the existing key, then makes one evidence-gated key rotation, verifying a fresh handshake at each step, with persisted bounded backoff, pause/resume, emergency close on kill-switch drift, and alert-bridge notifications; plus a 21-day renewal timer (off by default). 78 offline tests pass. **Not yet deployed or drilled live** — the controlled rollout is in the role README ("Rollout"); the legacy in-Ansible registration path remains selectable (`pia_registration_backend=legacy`) until the shared path passes a live re-registration. Still open after that: node-exporter textfile scrape + a Prometheus stale-check rule so a stopped timer is distinguishable from a healthy gateway.
- **Consider splitting "rewrite config from template" from "re-register with PIA" in the role** (Incident 4's root cause #1) — currently there's no way to pick up a template-only default change without forcing a full re-registration. - **Consider splitting "rewrite config from template" from "re-register with PIA" in the role** (Incident 4's root cause #1) — closed by the same change: `pia-gateway rewrite-config` re-renders `pia-wg.conf` from the recorded registration without a PIA call, and the role runs it automatically when only the controller config changed.
- Confirm whether a newer `lscr.io/linuxserver/qbittorrent` tag already includes the upstream fix for #24164, as a potential alternative/complement to the init-container workaround. - Confirm whether a newer `lscr.io/linuxserver/qbittorrent` tag already includes the upstream fix for #24164, as a potential alternative/complement to the init-container workaround.
- If it recurs, try to catch what actually triggers qBittorrent's unclean shutdown in the first place (Incident 2's root cause section) — nothing in this investigation identified it. - If it recurs, try to catch what actually triggers qBittorrent's unclean shutdown in the first place (Incident 2's root cause section) — nothing in this investigation identified it.

View File

@ -305,14 +305,79 @@ piece's own README for exact commands and rollback:
routing/firewall state and is ordered `Before=` `wg-quick@pia-wg.service`; routing/firewall state and is ordered `Before=` `wg-quick@pia-wg.service`;
both are `WantedBy=multi-user.target` so a normal boot orders them both are `WantedBy=multi-user.target` so a normal boot orders them
correctly with no manual step. `pia-gateway-healthcheck.timer` runs correctly with no manual step. `pia-gateway-healthcheck.timer` runs
independently and only observes — see independently and only observes (`pia-gateway check`).
`ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2`.
**PIA endpoint/key rotation**: re-run `ansible/playbooks/pia-gateway.yaml **Automatic recovery** (`ansible/roles/pia-gateway/README.md` →
-e pia_force_reregister=true -K` — this regenerates the local keypair and "Automatic recovery" — read that for the state machine, alerts,
re-registers with PIA, then rewrites `pia-wg.conf` (handler-triggered residual risk and the live-drill rollout): `pia-gateway-recover.timer`
restart). Rotating the PIA account password itself just needs on minisforum runs a bounded pass every 60 s — after 3 stale checks it
`ansible-vault edit ansible/group_vars/all/vault.yaml` first. restarts `pia-wg`, escalates to a same-key PIA re-registration, then to
one staged key rotation on evidence, verifies a fresh handshake before
calling anything fixed, backs off 5→15→30→60 min on failure, and keeps
VLAN 50 fail-closed throughout. Operator commands on minisforum:
```bash
sudo pia-gateway status # class, handshake age, backoff, last verified registration, pause
sudo pia-gateway pause --reason "..." # stop recovery+renewal (persists across reboot); kill switch untouched
sudo pia-gateway resume
sudo pia-gateway retry-now # one pass ignoring threshold/backoff
sudo journalctl -u pia-gateway-recover -n 50
```
Status as of 2026-09-22: implemented and tested offline (78 unit tests),
**not yet deployed or drilled live** — see the role README "Rollout".
Renewal (`pia_renewal_enabled`) stays off until the drills pass.
**PIA re-registration / key rotation** — from the repo root (the paths
are repo-root-relative — running from `ansible/` fails with "playbook
could not be found"):
```bash
# re-register with the EXISTING key (what fixed both incidents):
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml \
-e pia_force_reregister=true -K -J
# stage a NEW keypair, register it, promote it only after a verified handshake:
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml \
-e pia_force_key_rotation=true -K -J
```
`pia_force_reregister` **reuses** the on-disk key (an earlier version of
this paragraph said it regenerated the keypair — it never did; that is
`pia_force_key_rotation`). Both run the shared on-host transaction
(`pia-gateway register`), which restarts and **verifies a handshake**
even when the rendered config is byte-identical. `-J` is required
because `ansible/group_vars/all/vault.yaml` is vault-encrypted and
loaded for every host; the PIA credentials themselves are **not** in
the vault — `ansible/host_vars/minisforum.yaml` reads `PIA_USER`/
`PIA_PASSWORD` from the repo-root `.env`, and the role now also
provisions those two values to minisforum's root-only
`/etc/pia-gateway/credentials.env` so recovery works unattended.
Rotating the PIA password: edit `.env`, re-run the playbook, re-register.
**Tunnel silently dead — recurring, known, now auto-recovered**: the
symptom is "the proxy doesn't work" (browser through `10.10.40.20:1080`
fails, qBittorrent/JDownloader have no egress) while everything in
Kubernetes looks healthy and `pia-wg` on minisforum also looks up
(interface present, `wg-quick@pia-wg` `active (exited)`, routes in
place). The kill switch is doing its job — VLAN 50 fails closed rather
than leaking. With the controller deployed, expect an alert and a
recovery within ~10 minutes; check `sudo pia-gateway status` first. If
it is paused, in backoff, or the controller is not deployed, diagnose
before fixing so the age gets recorded:
```bash
ssh minisforum "sudo wg show pia-wg latest-handshakes" # epoch seconds; >180s old = dead
ssh minisforum "sudo journalctl -u pia-gateway-healthcheck -n 5"
```
then `sudo pia-gateway retry-now` on minisforum, or the re-registration
command above. A bare `systemctl restart wg-quick@pia-wg` was **not**
enough the first time (PIA had stopped recognising the registered key).
Occurrences so far — 2026-08-25 (full write-up in
`INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md`, Incident 1) and
2026-09-22 (~28 days later; pre-fix handshake age not captured, fix
confirmed by a fresh handshake immediately after). The ~28-day cadence
is the hypothesis behind the 21-day proactive renewal timer.
**Maintenance / taking the gateway down deliberately**: `sudo systemctl **Maintenance / taking the gateway down deliberately**: `sudo systemctl
stop wg-quick@pia-wg` on minisforum closes VLAN 50 Internet egress stop wg-quick@pia-wg` on minisforum closes VLAN 50 Internet egress

View File

@ -12,3 +12,12 @@ ansible_python_interpreter: /usr/bin/python3.13
# playbook triggers evaluation of these two vars. # playbook triggers evaluation of these two vars.
pia_user: "{{ lookup('ansible.builtin.file', playbook_dir + '/../../.env') | regex_search('^PIA_USER=(.*)$', '\\1', multiline=True) | first }}" pia_user: "{{ lookup('ansible.builtin.file', playbook_dir + '/../../.env') | regex_search('^PIA_USER=(.*)$', '\\1', multiline=True) | first }}"
pia_password: "{{ lookup('ansible.builtin.file', playbook_dir + '/../../.env') | regex_search('^PIA_PASSWORD=(.*)$', '\\1', multiline=True) | first }}" pia_password: "{{ lookup('ansible.builtin.file', playbook_dir + '/../../.env') | regex_search('^PIA_PASSWORD=(.*)$', '\\1', multiline=True) | first }}"
# Since the auto-recovery work these two ARE copied to minisforum, into the
# root-only /etc/pia-gateway/credentials.env (tasks/controller.yaml,
# no_log) — the host must be able to re-register with PIA unattended. See
# the role README "Credentials on the host" for the tradeoff and rotation.
#
# Optional: alert-bridge's shared bearer key so the controller can post
# state transitions to Discord. Empty (no ALERT_BRIDGE_API_KEY in .env)
# disables delivery; events still go to the journal.
pia_alert_bridge_api_key: "{{ (lookup('ansible.builtin.file', playbook_dir + '/../../.env') | regex_search('^ALERT_BRIDGE_API_KEY=(.*)$', '\\1', multiline=True) | default([''], true)) | first }}"

View File

@ -6,8 +6,11 @@ Phase 2 of `~/repo/homelab/plan.md`.
## Before the first real run ## Before the first real run
1. **The PIA API calls in `tasks/register.yaml` have gone through three 1. **The PIA API calls have gone through three rounds of correction
rounds of correction against real live testing (2026-08-24):** against real live testing (2026-08-24). They now live in
`files/pia_gateway.py` (`PiaApi`), with the original in-Ansible flow
kept as `tasks/register-legacy.yaml` until the shared path is proven
live — see "Automatic recovery" below:**
- `serverlist.piaservers.net/vpninfo/servers/v6`: works. Region values - `serverlist.piaservers.net/vpninfo/servers/v6`: works. Region values
everywhere in this role are the API's **id** field (e.g. `japan`, everywhere in this role are the API's **id** field (e.g. `japan`,
`hk`, `taiwan`), never a display name ("JP Tokyo", "Hong Kong", `hk`, `taiwan`), never a display name ("JP Tokyo", "Hong Kong",
@ -39,9 +42,14 @@ Phase 2 of `~/repo/homelab/plan.md`.
keys `manifests/media/pia-secret.sh` already reads) — `ansible/ keys `manifests/media/pia-secret.sh` already reads) — `ansible/
host_vars/minisforum.yaml` reads them directly from there via a host_vars/minisforum.yaml` reads them directly from there via a
`lookup('ansible.builtin.file', ...)` + regex, deliberately not `lookup('ansible.builtin.file', ...)` + regex, deliberately not
ansible-vault, so this needs no vault password at all. The lookup runs ansible-vault. The lookup runs on the control node; `.env` itself is
on the control node, so `.env` never touches minisforum and is never never copied and never committed (gitignored). **Since the
committed (already gitignored). auto-recovery work, only `PIA_USER`/`PIA_PASSWORD` (and, if present,
`ALERT_BRIDGE_API_KEY`) are provisioned to minisforum's root-only
`/etc/pia-gateway/credentials.env`** so the host can re-register
unattended — see "Credentials on the host" below. The playbook still
needs `-J` because `group_vars/all/vault.yaml` is vault-encrypted and
loaded for every host, not because the PIA credentials are in it.
3. Confirm console/recovery access to `minisforum` (IPMI/physical/other 3. Confirm console/recovery access to `minisforum` (IPMI/physical/other
out-of-band path) before applying — this role changes host routing and out-of-band path) before applying — this role changes host routing and
firewall policy. If SSH becomes unreachable, use that path. firewall policy. If SSH becomes unreachable, use that path.
@ -111,12 +119,13 @@ stdin, or a complete response body. If Hong Kong and Taiwan fail exactly
the same way Tokyo did, that's a strong signal the problem is shared the same way Tokyo did, that's a strong signal the problem is shared
across all three (the request shape, the account credentials, or across all three (the request shape, the account credentials, or
minisforum's own network path) rather than one region being down — see minisforum's own network path) rather than one region being down — see
`tasks/register.yaml`'s header comment. `tasks/register-legacy.yaml`'s header comment (the controller logs the
same per-candidate `kind=`/`detail=` classification).
**Worked example (2026-08-24):** exactly that happened — all 7 servers **Worked example (2026-08-24):** exactly that happened — all 7 servers
across all 3 regions came back `HTTP failure (non-2xx response)`, curl across all 3 regions came back `HTTP failure (non-2xx response)`, curl
exit 22, `The requested URL returned error: 401`, uniformly. The cause exit 22, `The requested URL returned error: 401`, uniformly. The cause
was in `addkey-attempt.yaml`'s own curl task, not PIA or any region: was in `addkey-attempt.yaml`'s own curl task (legacy path), not PIA or any region:
`ansible.builtin.command`'s `stdin` argument appends a trailing newline `ansible.builtin.command`'s `stdin` argument appends a trailing newline
by default (`stdin_add_newline` defaults to `true`), and by default (`stdin_add_newline` defaults to `true`), and
`--data-urlencode pt@-` does not strip it — every server was receiving `--data-urlencode pt@-` does not strip it — every server was receiving
@ -191,10 +200,15 @@ real.
config change that alters `PostUp`/`PreDown` content can never break config change that alters `PostUp`/`PreDown` content can never break
a restart this way again — confirmed live, 2026-08-27. a restart this way again — confirmed live, 2026-08-27.
- Installs an observability-only health check (`pia-gateway-healthcheck - Installs an observability-only health check (`pia-gateway-healthcheck
.timer`, every `pia_healthcheck_interval_sec`) that logs interface, .timer`, every `pia_healthcheck_interval_sec`) that runs `pia-gateway
handshake age, rule/route, and firewall-policy state to the journal. It check` — the same status classifier the recovery controller uses — and
never remediates — see the script's header comment for why, and for what logs interface, handshake age, rule/route ordering and firewall state to
its "route-decision" check does and does not prove. the journal plus `/var/lib/pia-gateway/status.json` / `metrics.prom`.
It never remediates. Neither it nor the controller sends a packet
through the tunnel: a "healthy" class is a fresh handshake plus intact
kill-switch invariants, not proof of egress identity.
- Installs the automatic recovery controller and its timers — see
"Automatic recovery" below.
## What it deliberately does not do ## What it deliberately does not do
@ -205,18 +219,330 @@ real.
- Does not modify `ansible/roles/wireguard` (the separate `wg0` home-VPN - Does not modify `ansible/roles/wireguard` (the separate `wg0` home-VPN
server role) or its interface. server role) or its interface.
## Known failure mode: the session dies while the interface looks fine
Seen twice so far, ~28 days apart — 2026-08-25 (root-caused in
`INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md`, Incident 1) and
2026-09-22. `pia-wg`'s handshake with PIA's server stops renewing;
the interface, `wg-quick@pia-wg` (`active (exited)` is normal for
`wg-quick`), the `pia` routing table and all firewall rules stay exactly
as `PostUp` left them. The kill switch's "interface down → unreachable
route" trigger never fires because the interface never goes down, so
VLAN 50 fails closed — correctly, but silently.
On 2026-08-25 a plain `systemctl restart wg-quick@pia-wg` did not produce
a new handshake and only a fresh `addKey` registration did — strongly
suggesting PIA had dropped the registered key server-side. The
2026-09-22 occurrence went straight to re-registration (bare restart
untested). The underlying trigger is still not confirmed; the ~4-week gap
is consistent with a fixed registration lifetime on PIA's side, which is
what the proactive renewal timer below is for. This failure mode is now
handled automatically — see the next section.
## Automatic recovery
`files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway`
(root, `0700`, standard library only — runs on the host's `python3.13`).
It is **one** implementation of PIA registration shared by the unattended
timers and by this role's own registration step, so the two can never
drift. Layout on the host:
| Path | Purpose |
| --- | --- |
| `/etc/pia-gateway/config.json` (`0600`) | rendered from role defaults (`templates/pia-gateway-config.json.j2`) |
| `/etc/pia-gateway/credentials.env` (`0600`) | `PIA_USER`, `PIA_PASSWORD`, optional `ALERT_BRIDGE_API_KEY` — see "Credentials on the host" |
| `/var/lib/pia-gateway/state.json` (`0600`) | versioned controller state (atomic tmp+fsync+rename); **no secrets** |
| `/var/lib/pia-gateway/status.json`, `metrics.prom` (`0644`) | last classification, for humans/monitoring |
| `/run/lock/pia-gateway.lock` | one `flock` shared by every mutating action (timers, `register`, `rewrite-config`); released by the kernel if a process dies |
| `pia-gateway-recover.timer` → `.service` | every 60 s, one bounded recovery pass (`recover`) |
| `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) |
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
### Status classes
Every pass classifies the gateway into exactly one of: `healthy`,
`boot_grace` (stale within 120 s of a boot or of an intentional restart —
not counted), `paused`, `tunnel_stale` (handshake 0 / older than
`pia_healthcheck_handshake_max_age_sec` / in the future beyond 60 s of
skew), `tunnel_absent` (interface down/missing, expected peer missing,
fresh handshake but table route closed), `registration_or_endpoint_failure`
/ `host_connectivity_failure` (stale **and** in backoff after a cycle
failed that way), `safety_drift`, `unknown` (a read command failed — no
action taken, alert after 3). `status.json` carries both the decision
class and the raw `tunnel_class`, so "paused" never hides a dead tunnel.
Safety invariants checked on every pass, all of which must hold before
any restart/registration is attempted: the priority-100 source rule; a
default in table `pia` that is either `dev pia-wg` (open) or
`unreachable` (closed) — *no* default at all is a failure, not "closed";
the FORWARD jump to `PIA-VLAN50` is the **first effective** rule (an
emergency rule of our own is the only thing allowed above it); the chain
ends in an unconditional `DROP` and every `ACCEPT` before it is scoped to
`pia-wg`; the host default route is still via `enp1s0`.
### Recovery state machine (`recover`, every 60 s)
| State | Action |
| --- | --- |
| healthy | reset counters; if a backoff was pending, log/notify "healthy again" |
| 1–2 consecutive stale/absent | count only |
| 3 consecutive (`pia_recovery_failure_threshold`) | save pre-fix evidence (age, endpoint, reasons) → **stage 1**: `systemctl restart wg-quick@pia-wg` once, wait up to 90 s for two consecutive healthy classifications with a handshake newer than the attempt start |
| stage 1 fails | **stage 2**: serverlist → token → `addKey` with the *existing* key against candidates in `pia_region_candidates` order (max `pia_recovery_max_candidates_per_cycle`), each accepted response staged as a validated config, activated, and verified the same way; a candidate that answers `OK` but never handshakes is skipped for the next one |
| stage 2: API accepted the key but nothing handshook | **stage 3**: exactly one staged fresh-key attempt (`pia-wg.key.new.<ts>.<pid>`, promoted only after verification, previous key backed up). Skipped when the failure was uniform (auth error, host connectivity, every API call failing the same way) |
| any failure | previous config restored (best effort) and re-activated; gateway stays closed; backoff 5 → 15 → 30 → 60 min (±10 % jitter, capped) persisted in state; classified failure recorded; alert |
| verified recovery | stage, region/server, elapsed time and pre-fix age recorded in `last_recovery`; `last_verified_registration` updated when a registration happened; one "recovered" alert |
| safety drift | **no** restart/registration. Insert `-I FORWARD 1 -s 10.10.50.0/24 -m comment --comment pia-gateway-emergency -j DROP`, then `ip route replace unreachable default table pia`, verify the DROP is first, alert `error`. If the rule already exists but isn't first any more (something else prepended a rule ahead of it, e.g. an out-of-band `ufw reload`), it's deleted and reinserted at position 1 rather than left where it drifted to — a mere existence check (`iptables -C`) says nothing about position. Left in place until a later pass sees the invariants intact, then removed under the lock. Automatic repair of the kill switch itself is deliberately **not** attempted (see "Residual risk") |
| paused (`pia-gateway pause`) | checks and notifications continue; never restart/register/renew. Safety drift is **still** contained by the emergency close while paused — pausing suspends recovery, never the fail-closed guarantee |
| corrupt/unknown-schema state file | refuse to act (exit 2, unit failure → alert). `pia-gateway state-reset --confirm` moves the file aside and starts fresh |
One pass is bounded by `pia_recovery_max_duration_sec` (600 s) inside
the program and `TimeoutStartSec` (900 s) in the unit; systemd never
starts a second instance while one runs, and the lock makes a manual
`register` wait (up to `pia_recovery_lock_wait_sec`) rather than
interrupt an activation. Reboots: the boot id is tracked, a new boot gets
`pia_recovery_startup_grace_sec` before stale counts; retry/backoff state
survives because it is on disk. A backward wall-clock step (NTP) that
leaves a persisted retry deadline further out than the backoff cap is
treated as a clock jump and the deadline is cleared rather than stalling
recovery.
`restart` returning 0, `active (exited)`, or `addKey` returning `status:
OK` are never treated as success on their own — only an observed fresh
handshake on the activated config is, and specifically one reported
**strictly after** the moment activation started: a handshake in the
same whole second as that moment doesn't count either, since `wg show`
only has one-second resolution and there'd be no way to tell it apart
from a stale, pre-activation handshake that happens to share the second.
Every mutating command — `recover`, `renew`, `register`,
`rewrite-config`, `pause`, `resume`, `state-reset` — takes the same
`/run/lock/pia-gateway.lock` **before** reading state, not after: reading
state first and only locking around the write would let a slower command
silently overwrite whatever a faster one wrote in between (in particular,
it's how a `pause` could otherwise be lost to an in-flight recovery pass
that read state before the pause ran and only saves its own copy back
once it finishes). `pause`/`resume`/`register`/`rewrite-config`/
`state-reset` wait up to `pia_recovery_lock_wait_sec` for the lock and
exit 4 if it's still busy; `recover`/`renew` (the timers) never wait —
they just skip that tick if something else currently holds it.
### Operator commands (all on minisforum, as root)
```bash
pia-gateway status # human summary incl. backoff, last verified registration, pause
pia-gateway check # what the health-check timer runs (read-only, exit 1 = unhealthy)
pia-gateway check --json # machine-readable
pia-gateway pause --reason "maintenance" # persists across reboot; suppresses recover AND renew; kill switch untouched
pia-gateway resume
pia-gateway retry-now # one recovery pass ignoring threshold and backoff
pia-gateway register [--mode same-key|new-key] [--region hk] # explicit transaction, verified, under the lock
pia-gateway rewrite-config # re-render pia-wg.conf from the recorded registration (hook changes), no API call
pia-gateway renew --force-due # exercise renewal without waiting 21 days
pia-gateway state-reset --confirm # only after reading the corrupt file it moves aside
journalctl -u pia-gateway-recover -u pia-gateway-renew -u pia-gateway-healthcheck -n 100
cat /var/lib/pia-gateway/status.json
```
Through Ansible (repo root, `-K -J`):
```bash
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml -K -J # idempotent; registers only if no config or no verified baseline
ansible-playbook ... -e pia_force_reregister=true # re-register with the EXISTING key (same-key)
ansible-playbook ... -e pia_force_key_rotation=true # stage a NEW key, register it, promote only after verification
ansible-playbook ... -e pia_region=hk # single-region override (either of the above)
ansible-playbook ... -e pia_auto_recovery_enabled=false # stop+disable both timers, keep everything else (rollback)
ansible-playbook ... -e pia_renewal_enabled=true # after the live drills — enables the daily renewal timer
ansible-playbook ... -e pia_remove_host_credentials=true # deliberate removal of /etc/pia-gateway/credentials.env
```
`pia_force_reregister` **reuses** the on-disk key; only
`pia_force_key_rotation` generates a new one (the root README used to say
otherwise — corrected). A normal run also quiesces both timers, waits for
any in-flight pass, and re-enables them after handlers, so an Ansible
restart never races an automatic one. A run whose only change is
`config.json` (e.g. flipping `pia_mss_clamp_enabled`) re-renders the
WireGuard config via `rewrite-config` without touching PIA — the
"template-only change needs a full re-registration" gap from the
2026-08-25 incident is closed.
### Proactive renewal (`renew`)
Off until enabled. When on, once a day it re-registers with the
**existing** key if `last_verified_registration` is older than
`pia_renewal_after_days` (21) — or missing, which is why the baseline
must come from a *verified* run of the shared path, never from a file
mtime. Same transaction, same lock, same verification. A failed renewal
leaves the baseline untouched and enters the same bounded backoff (it
does not silently wait another 21 days); if the tunnel is already
unhealthy it defers to the recovery path. Renewal restarts `pia-wg`, so
qBittorrent/JDownloader connections drop for a few seconds at ~03:30 —
verify they reconnect during the drill. Routine key rotation is not
done: rotation happens only in recovery stage 3, on evidence.
### Alerts and monitoring
Events (not every 60 s): recovery started, recovered, recovery failed +
backoff (`warn`; `error` for auth failure), safety drift (`error`),
emergency rule removed, renewal failed, status unknown ×3, state corrupt,
unit crashed (via `OnFailure`). Delivery is `POST` to
`pia_alert_bridge_url` with the shared alert-bridge bearer key
(`{"source": "pia-gateway@minisforum", "message": ..., "level": ...}` —
schema read from `~/repo/home-service/alert-bridge`, which has **no**
per-caller tokens, so the cluster's key is what gets provisioned;
accepted tradeoff). 5 s timeout; failures are queued in state (bounded,
20) and flushed on the next pass; they never block recovery.
`metrics.prom` is node-exporter textfile format
(`pia_gateway_healthy`, `_tunnel_ok`, `_safety_ok`,
`_handshake_age_seconds`, `_consecutive_failures`, `_paused`,
`_recovery_in_backoff`, `_last_verified_registration_timestamp_seconds`,
`_check_timestamp_seconds`, `_emergency_close_active`). **Not yet
scraped**: `values/kube-prometheus-stack.yaml` has no textfile collector
and node-exporter runs as a pod, so wiring it needs a hostPath mount +
`--collector.textfile.directory` plus a `PrometheusRule` including a
stale-check rule (`time() - pia_gateway_check_timestamp_seconds > 300`)
so a stopped timer alerts even when the last value was healthy. Until
that exists, a dead controller is only visible as silence — check
`systemctl list-timers 'pia-gateway-*'` and `status.json`'s timestamp.
### Credentials on the host
`/etc/pia-gateway/credentials.env` holds the PIA account username and
password in plaintext, root-only. This is the explicit tradeoff the
design requires: recovery must work while the laptop is off, and PIA's
token is only valid for 24 h, so an unattended host needs a renewable
credential. The program reads the file at runtime and never places
values in argv, environment, logs, notifications, metrics or state; core
dumps are disabled (`LimitCORE=0`) and unexpected exceptions are logged
by class name only. **Rotation:** change `PIA_PASSWORD` in `.env`, run
the playbook (the file is re-rendered with `no_log`), then `pia-gateway
register` (or wait for the next cycle). **Revocation:** `-e
pia_remove_host_credentials=true` removes the file (recovery then fails
closed with `auth`/credentials errors and alerts); change the PIA
password; disabling `pia_auto_recovery_enabled` alone never deletes it.
If an on-host secret is unacceptable, the alternative is an always-on
external secret broker — not a 24-hour token.
### Residual risk (read before trusting it unattended)
- Safety drift is contained by the emergency DROP + unreachable route,
not repaired: the kill switch's own rule/jump can be deleted by
something outside this role (a manual `ufw reload` flushes `FORWARD`,
removing both the jump **and** the emergency rule). The controller
re-inserts the emergency rule on its next pass (≤ 60 s) and every boot
re-seeds the kill switch; that 60 s window after an out-of-band flush
is the residual leak risk, and it predates this work.
- Verification is handshake-based. It proves the tunnel is talking to
PIA, not that a VLAN 50 packet leaves via PIA; the end-to-end check is
the SOCKS-proxy/egress-IP step in the live drill.
- The alert-bridge key on minisforum is the cluster's shared key.
- `unknown` (a read command failing) is logged/alerted but never acted
on, by design.
### Rollout (nothing below has been run live yet)
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
table pia`, `iptables -S FORWARD | head -3`, `iptables -S PIA-VLAN50`,
`journalctl -u pia-gateway-healthcheck -n 5`, `curl --socks5-hostname
10.10.40.20:1080 https://ifconfig.me` from the LAN, `wg show wg0`,
`kubectl get nodes`. Confirm console access.
2. Deploy with timers off: `ansible-playbook ... -K -J -e
pia_auto_recovery_enabled=false`. Check `ls -l /etc/pia-gateway
/var/lib/pia-gateway`, `pia-gateway status`, `systemctl list-timers
'pia-gateway-*'` (recover/renew inactive). Because the host has no
verified baseline yet, this run **registers through the shared path**
(same-key, explicit restart + handshake verification) — watch the
task output; if it fails, `-e pia_registration_backend=legacy`
restores the previous behaviour.
3. Enable: `ansible-playbook ... -K -J` (recovery on). Confirm one healthy
pass makes no changes (`journalctl -u pia-gateway-recover -n 5`).
4. Drill the silent death. First get the **live WireGuard UDP endpoint**
from the interface itself — **not** port 1337, which is only the
`addKey` registration API port and has nothing to do with the tunnel's
own data-plane port (PIA WireGuard servers commonly use a different
port per server):
```bash
sudo wg show pia-wg endpoints # <peer-pubkey>\t<ip>:<port> — this ip:port is the block target
```
Then, on minisforum, leave `pia-wg` administratively up and drop only
that path:
```bash
sudo iptables -I OUTPUT -d <endpoint ip> -p udp --dport <endpoint port> -j DROP
```
Expect: 3 stale passes → restart → (still blocked) registration to a
different server → recovery; record times from the journal. To test
backoff, block every candidate server before recovery can reach it —
since each candidate's actual WireGuard port isn't known until its own
`addKey` response arrives (only the registration port, 1337, is fixed
and shared), block by IP alone rather than guessing a port:
```bash
sudo iptables -I OUTPUT -d <candidate ip> -p udp -j DROP # repeat per candidate IP, or -e pia_region=hk to narrow the set first
```
Remove the rule(s) and confirm autonomous recovery. `iptables -D`
every injected rule afterwards — check `iptables -S OUTPUT` is back to
its pre-drill state, not just that the tunnel recovered.
5. Exercise: a failed first server (block one IP only), invalid
credentials (temporarily wrong `PIA_PASSWORD` → expect `auth_failure`,
no rotation, error alert), alert-bridge down, `pia-gateway register`
while a pass runs (expect it to wait), reboot with a pending backoff,
`pause`/`resume`.
6. During every transition, capture on `enp1s0` with an explicit
**egress-only** direction filter:
```bash
sudo tcpdump -ni enp1s0 -Q out src net 10.10.50.0/24
```
`-Q out` matters: VLAN 50 traffic **arriving** at minisforum on
`enp1s0` (from the Flint router, for policy routing into table `pia`)
is normal and expected — a plain `src net 10.10.50.0/24` filter with
no direction would match that inbound traffic too and could hide a
real leak in the noise, or be misread as one. Only a hit in the `out`
direction — a VLAN 50-sourced packet actually being transmitted out
`enp1s0` toward the LAN/Internet instead of via `pia-wg` — is a leak,
and this must stay completely silent through every failure and
recovery transition. Also capture on `pia-wg` itself to confirm
traffic **is** flowing there while open. Verify a request through
`10.10.40.20:1080` and qBittorrent/JDownloader afterwards; check IPv6
separately; confirm `wg0`, the host default route, Technitium and k3s.
7. Only then `-e pia_renewal_enabled=true`; `pia-gateway renew
--force-due`; confirm `last_verified_registration.epoch` advances only
on verification. Wire the Prometheus stale-check before calling the
feature done. Record actual times and captures in the incident file.
## Rollback ## Rollback
No tag-based rollback is defined — this role doesn't have an "absent" **Partial rollback (keep the gateway, drop only the automation):**
mode. Roll back manually on minisforum (reverses this role without `ansible-playbook ... -e pia_auto_recovery_enabled=false` stops and
disables both timers and leaves the program, credentials, health check,
kill switch and `pia-wg` in place. Add `-e pia_remove_host_credentials=true`
to also delete the on-host PIA credentials; nothing else deletes them.
No tag-based full rollback is defined — this role doesn't have an
"absent" mode. Roll back manually on minisforum (reverses this role without
touching `wg0`, k3s, or the host default route): touching `wg0`, k3s, or the host default route):
```bash ```bash
sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \ sudo rm -f /etc/systemd/system/pia-killswitch.service \
/etc/systemd/system/pia-gateway-healthcheck.service \ /etc/systemd/system/pia-gateway-healthcheck.service \
/etc/systemd/system/pia-gateway-healthcheck.timer /etc/systemd/system/pia-gateway-healthcheck.timer \
/etc/systemd/system/pia-gateway-recover.service \
/etc/systemd/system/pia-gateway-recover.timer \
/etc/systemd/system/pia-gateway-renew.service \
/etc/systemd/system/pia-gateway-renew.timer \
/etc/systemd/system/pia-gateway-onfailure.service
sudo systemctl daemon-reload sudo systemctl daemon-reload
sudo iptables -D FORWARD -s 10.10.50.0/24 -m comment --comment pia-gateway-emergency -j DROP 2>/dev/null || true
sudo rm -rf /etc/pia-gateway /var/lib/pia-gateway /usr/local/sbin/pia-gateway /run/lock/pia-gateway.lock # includes the on-host PIA credentials
sudo ip rule del from 10.10.50.0/24 table pia priority 100 sudo ip rule del from 10.10.50.0/24 table pia priority 100
sudo ip route flush table pia sudo ip route flush table pia
sudo sed -i '/^[0-9]\+\s\+pia$/d' /etc/iproute2/rt_tables sudo sed -i '/^[0-9]\+\s\+pia$/d' /etc/iproute2/rt_tables
@ -225,7 +551,7 @@ sudo iptables -F PIA-VLAN50
sudo iptables -X PIA-VLAN50 sudo iptables -X PIA-VLAN50
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp
sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.new /etc/wireguard/pia-ca.crt sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.conf.bak-* /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.bak-* /etc/wireguard/pia-wg.key.new* /etc/wireguard/pia-ca.crt
``` ```
Inspection commands to confirm rollback actually took (read-only): Inspection commands to confirm rollback actually took (read-only):

View File

@ -123,3 +123,64 @@ pia_mss_clamp_enabled: true
# see tasks/healthcheck.yaml — it must never itself open the kill switch. # see tasks/healthcheck.yaml — it must never itself open the kill switch.
pia_healthcheck_interval_sec: 60 pia_healthcheck_interval_sec: 60
pia_healthcheck_handshake_max_age_sec: 180 pia_healthcheck_handshake_max_age_sec: 180
# ── Automatic recovery (files/pia_gateway.py, tasks/controller.yaml) ────────
#
# Which implementation performs PIA registration when this role runs:
# shared — `pia-gateway register` on the host (the same code the unattended
# recovery/renewal timers use; activates and VERIFIES a handshake
# before reporting success).
# legacy — the original in-Ansible token/addKey flow (tasks/register.yaml).
# Kept only until the shared path has passed a live manual
# re-registration; then it is to be deleted (see README).
pia_registration_backend: shared
# Master switch for the reactive recovery timer. false stops+disables the
# timer but leaves the program, credentials, kill switch, health check and
# pia-wg in place (controlled rollback). Renewal has its own switch below.
pia_auto_recovery_enabled: true
# Proactive same-key re-registration when the last VERIFIED registration is
# older than pia_renewal_after_days. Off until the reactive path has passed
# its live drills (README "Rollout"). Needs a verified baseline in state —
# established by the shared registration path, never inferred from file
# mtimes.
pia_renewal_enabled: false
pia_renewal_after_days: 21
pia_renewal_calendar: "*-*-* 03:30:00"
pia_renewal_jitter_sec: 1800
# Recovery state machine tunables (seconds unless noted).
pia_recovery_failure_threshold: 3
pia_recovery_startup_grace_sec: 120
pia_recovery_restart_wait_sec: 90
pia_recovery_activation_wait_sec: 90
pia_recovery_max_duration_sec: 600
pia_recovery_backoff_schedule_sec: [300, 900, 1800, 3600]
pia_recovery_max_candidates_per_cycle: 6
pia_recovery_max_rotation_candidates: 3
pia_recovery_http_connect_timeout_sec: 10
pia_recovery_http_total_timeout_sec: 30
pia_recovery_backup_keep: 5
pia_recovery_lock_wait_sec: 600
# Where the on-host controller keeps things.
pia_gateway_program: /usr/local/sbin/pia-gateway
pia_gateway_etc_dir: /etc/pia-gateway
pia_gateway_state_dir: /var/lib/pia-gateway
# alert-bridge (manifests/home-services/alert-bridge*.yaml) delivery for
# state-transition events. Its API has ONE shared bearer key (no per-caller
# tokens — confirmed against the alert-bridge source), so the same key the
# cluster uses is provisioned to minisforum's root-only credentials file
# when pia_alert_bridge_api_key is set (host_vars reads it from .env). Set
# the URL to null to disable delivery; events still go to the journal.
pia_alert_bridge_url: "http://alert-bridge.home.arpa/alerts"
pia_alert_source: "pia-gateway@minisforum"
pia_alert_bridge_api_key: ""
# Deliberate credential removal. Disabling automation never deletes the
# on-host PIA credentials on its own; set this true (one run) to remove
# {{ pia_gateway_etc_dir }}/credentials.env, e.g. when decommissioning or
# after rotating the PIA password elsewhere.
pia_remove_host_credentials: false

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,98 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml (before registration — the shared
# registration backend is this program)
# Description: Installs the on-host controller (files/pia_gateway.py), its
# root-only runtime config and credentials, and the systemd units for
# the recovery / renewal timers. Timers are only ENABLED at the end of
# tasks/main.yaml, after the kill switch, interface and a verified
# registration exist — this file never starts them.
- name: Create the controller config directory (root-only)
ansible.builtin.file:
path: "{{ pia_gateway_etc_dir }}"
state: directory
mode: "0700"
owner: root
group: root
- name: Create the controller state directory (root-only)
ansible.builtin.file:
path: "{{ pia_gateway_state_dir }}"
state: directory
mode: "0700"
owner: root
group: root
- name: Install the pia-gateway program
ansible.builtin.copy:
src: pia_gateway.py
dest: "{{ pia_gateway_program }}"
mode: "0700"
owner: root
group: root
- name: Render the controller runtime config
ansible.builtin.template:
src: pia-gateway-config.json.j2
dest: "{{ pia_gateway_etc_dir }}/config.json"
mode: "0600"
owner: root
group: root
register: pia_gateway_config_result
- name: Validate the rendered config loads
ansible.builtin.command:
cmd: "{{ pia_gateway_program }} --config {{ pia_gateway_etc_dir }}/config.json status"
register: pia_gateway_config_check
changed_when: false
failed_when: pia_gateway_config_check.rc not in [0, 1, 2]
# `status`'s own source (cmd_status in files/pia_gateway.py) only ever
# returns one of three documented exits: 0 (healthy), 1 (unhealthy —
# the expected result on a fresh install, since the kill switch and
# pia-wg interface don't exist yet at this point in the run), or 2
# (state file corrupt — not a config problem either). Allowlisting
# those three and failing on EVERYTHING else (5 = bad config JSON/
# unknown key from main()'s own Config.load(), 3/4 = exits this
# command cannot legitimately return, 126/127/a crash) means a real
# config regression is never masked by matching only one "bad" code —
# the previous version of this task only failed on rc==5 and silently
# ignored any other unexpected exit.
- name: Provision PIA credentials for unattended re-registration (root-only)
ansible.builtin.template:
src: pia-gateway-credentials.env.j2
dest: "{{ pia_gateway_etc_dir }}/credentials.env"
mode: "0600"
owner: root
group: root
no_log: true
when: not (pia_remove_host_credentials | bool)
# Only PIA_USER/PIA_PASSWORD (and the alert-bridge key when set) — never
# the whole .env, the vault password, or any Kubernetes secret. This is
# the deliberate availability/security tradeoff documented in the README:
# a host that must re-register while the laptop is off needs the account
# secret locally.
- name: Remove the on-host PIA credentials (explicit request only)
ansible.builtin.file:
path: "{{ pia_gateway_etc_dir }}/credentials.env"
state: absent
when: pia_remove_host_credentials | bool
- name: Deploy controller systemd units
ansible.builtin.template:
src: "{{ item }}.j2"
dest: "/etc/systemd/system/{{ item }}"
mode: "0644"
owner: root
group: root
loop:
- pia-gateway-recover.service
- pia-gateway-recover.timer
- pia-gateway-renew.service
- pia-gateway-renew.timer
- pia-gateway-onfailure.service
notify: Reload systemd daemon
# The reload runs at the first flush_handlers in main.yaml, before any
# of these units is enabled; the enable tasks also pass daemon_reload.

View File

@ -1,17 +1,17 @@
--- ---
# Part of role: pia-gateway # Part of role: pia-gateway
# Included by: tasks/main.yaml # Included by: tasks/main.yaml
# Description: Installs the observability-only health check (script, # Description: Installs the observability-only health check (oneshot
# oneshot service, timer). See templates/pia-gateway-healthcheck.sh.j2 # service + timer). Since the auto-recovery work it runs
# for exactly what it checks and why it never remediates. # `pia-gateway check` — the same status classification the recovery
# controller uses — instead of the earlier standalone bash script, so
# the two can never disagree about what "healthy" means. It still never
# remediates.
- name: Deploy the health check script - name: Remove the superseded bash health check script
ansible.builtin.template: ansible.builtin.file:
src: pia-gateway-healthcheck.sh.j2 path: /usr/local/sbin/pia-gateway-healthcheck.sh
dest: /usr/local/sbin/pia-gateway-healthcheck.sh state: absent
mode: "0700"
owner: root
group: root
- name: Deploy the health check systemd service - name: Deploy the health check systemd service
ansible.builtin.template: ansible.builtin.template:
@ -40,3 +40,4 @@
name: pia-gateway-healthcheck.timer name: pia-gateway-healthcheck.timer
enabled: true enabled: true
state: started state: started
daemon_reload: true

View File

@ -2,10 +2,23 @@
# Part of role: pia-gateway # Part of role: pia-gateway
# Called by: ansible/playbooks/pia-gateway.yaml # Called by: ansible/playbooks/pia-gateway.yaml
# Description: Orchestrates minisforum's PIA WireGuard egress gateway for # Description: Orchestrates minisforum's PIA WireGuard egress gateway for
# VPN VLAN 50 — registration, boot-ordered kill switch + policy routing, # VPN VLAN 50 — kill switch + policy routing, the on-host recovery
# the WireGuard interface itself, and health checks. Split into included # controller, registration, the WireGuard interface itself, and health
# files because each stage has a distinct idempotency/ordering concern; # checks. Split into included files because each stage has a distinct
# see the comment at the top of each included file. # idempotency/ordering concern; see the comment at the top of each.
#
# Ordering (changed with the auto-recovery work, deliberately):
# 1. tooling, baseline assertions
# 2. controller program + config + credentials (NOT its timers)
# 3. quiesce the recovery/renewal timers for the rest of the run so a
# timer-driven restart can never race an Ansible handler restart
# 4. kill switch + routing + firewall, handlers flushed so the closed
# state is live BEFORE anything can activate a tunnel
# 5. registration (shared backend activates + verifies under the lock;
# the legacy backend only writes the config and notifies a restart)
# 6. interface unit enabled, health check installed, handlers flushed
# 7. timers enabled/disabled per pia_auto_recovery_enabled /
# pia_renewal_enabled
- name: Install WireGuard tooling and iproute2 - name: Install WireGuard tooling and iproute2
ansible.builtin.apt: ansible.builtin.apt:
@ -13,24 +26,57 @@
- wireguard - wireguard
- wireguard-tools - wireguard-tools
- iproute2 - iproute2
- python3-requests - python3
state: present state: present
update_cache: true update_cache: true
# python3-requests is no longer needed: the controller uses only the
# standard library.
- name: Pin reverse-path filtering for {{ pia_wg_interface }} - name: Pin reverse-path filtering for {{ pia_wg_interface }}
ansible.posix.sysctl: ansible.posix.sysctl:
name: "net.ipv4.conf.{{ pia_wg_interface }}.rp_filter" name: "net.ipv4.conf.{{ pia_wg_interface }}.rp_filter"
value: "{{ pia_rp_filter_mode | string }}" value: "{{ pia_rp_filter_mode | string }}"
sysctl_file: /etc/sysctl.d/60-pia-gateway.conf sysctl_file: /etc/sysctl.d/60-pia-gateway.conf
reload: false # interface does not exist yet on a first run; applied for real after the WG interface comes up (handlers/main.yaml) reload: false # interface does not exist yet on a first run; applied for real after the WG interface comes up
register: pia_rp_filter_pinned register: pia_rp_filter_pinned
- name: Confirm host default route and ip_forward are untouched - name: Confirm host default route and ip_forward are untouched
ansible.builtin.include_tasks: assert-baseline.yaml ansible.builtin.include_tasks: assert-baseline.yaml
- name: Register minisforum with PIA and obtain WireGuard peer config - name: Install the recovery controller, its config and credentials
ansible.builtin.include_tasks: register.yaml ansible.builtin.include_tasks: controller.yaml
no_log: true # PIA credentials and the derived private key pass through this include
- name: Check whether the recovery timer is installed from a previous run
ansible.builtin.stat:
path: /etc/systemd/system/pia-gateway-recover.timer
register: pia_recover_timer_stat
- name: Quiesce the recovery and renewal timers for the duration of this run
ansible.builtin.systemd:
name: "{{ item }}"
state: stopped
loop:
- pia-gateway-recover.timer
- pia-gateway-renew.timer
when: pia_recover_timer_stat.stat.exists
failed_when: false
# failed_when: false — on the very first install the units were just
# deployed and never started; "stopping" them is a harmless no-op that
# some systemd versions report as an error for never-loaded timers.
- name: Wait for any in-flight recovery/renewal pass to finish
ansible.builtin.command:
cmd: systemctl is-active pia-gateway-recover.service pia-gateway-renew.service
register: pia_inflight
changed_when: false
failed_when: false
until: "'active' not in (pia_inflight.stdout_lines | default([]) | map('trim') | list)"
retries: 90
delay: 10
when: pia_recover_timer_stat.stat.exists
# A pass can take up to pia_recovery_max_duration_sec (+ restart waits);
# we never kill it mid-activation. Registration below also takes the
# program's own lock, so this is belt-and-braces for the handler restarts.
- name: Configure boot-ordered kill switch and policy routing - name: Configure boot-ordered kill switch and policy routing
ansible.builtin.include_tasks: routing.yaml ansible.builtin.include_tasks: routing.yaml
@ -38,6 +84,18 @@
- name: Configure firewall-layer kill switch and Technitium access - name: Configure firewall-layer kill switch and Technitium access
ansible.builtin.include_tasks: firewall.yaml ansible.builtin.include_tasks: firewall.yaml
- name: Apply kill-switch handlers before anything can activate a tunnel
ansible.builtin.meta: flush_handlers
- name: Register minisforum with PIA (shared on-host transaction)
ansible.builtin.include_tasks: register-shared.yaml
when: pia_registration_backend == "shared"
- name: Register minisforum with PIA (legacy in-Ansible flow)
ansible.builtin.include_tasks: register-legacy.yaml
no_log: true # PIA credentials and the derived private key pass through this include
when: pia_registration_backend == "legacy"
- name: Write and enable the pia-wg WireGuard interface - name: Write and enable the pia-wg WireGuard interface
ansible.builtin.include_tasks: interface.yaml ansible.builtin.include_tasks: interface.yaml
@ -51,3 +109,54 @@
sysctl_file: /etc/sysctl.d/60-pia-gateway.conf sysctl_file: /etc/sysctl.d/60-pia-gateway.conf
reload: true reload: true
when: pia_rp_filter_pinned is succeeded when: pia_rp_filter_pinned is succeeded
- name: Apply remaining handlers while the timers are still quiesced
ansible.builtin.meta: flush_handlers
- name: Re-render pia-wg.conf when only the controller config changed (no PIA call)
ansible.builtin.command:
cmd: "{{ pia_gateway_program }} rewrite-config"
register: pia_rewrite_result
changed_when: pia_rewrite_result.rc == 0
failed_when: pia_rewrite_result.rc not in [0, 5]
when:
- pia_registration_backend == "shared"
- pia_gateway_config_result is changed
- not (pia_registered_this_run | default(false) | bool)
# Picks up hook-content changes (e.g. pia_mss_clamp_enabled) without a
# forced re-registration — the gap noted in the 2026-08-25 incident.
# Exit 5 = "no verified registration recorded yet" (legacy-registered
# host); that is not an error here, the next registration will render
# the current hooks.
- name: Enable the recovery timer
ansible.builtin.systemd:
name: pia-gateway-recover.timer
enabled: true
state: started
daemon_reload: true
when: pia_auto_recovery_enabled | bool
- name: Disable the recovery timer (pia_auto_recovery_enabled=false)
ansible.builtin.systemd:
name: pia-gateway-recover.timer
enabled: false
state: stopped
daemon_reload: true
when: not (pia_auto_recovery_enabled | bool)
- name: Enable the renewal timer
ansible.builtin.systemd:
name: pia-gateway-renew.timer
enabled: true
state: started
daemon_reload: true
when: (pia_auto_recovery_enabled | bool) and (pia_renewal_enabled | bool)
- name: Disable the renewal timer
ansible.builtin.systemd:
name: pia-gateway-renew.timer
enabled: false
state: stopped
daemon_reload: true
when: not ((pia_auto_recovery_enabled | bool) and (pia_renewal_enabled | bool))

View File

@ -1,8 +1,26 @@
--- ---
# Part of role: pia-gateway # Part of role: pia-gateway
# Included by: tasks/main.yaml (with no_log: true — this file handles the # Included by: tasks/main.yaml ONLY when pia_registration_backend ==
# PIA account password, the derived WireGuard private key, and the # "legacy" (with no_log: true — this file handles the PIA account
# short-lived PIA auth token) # password, the derived WireGuard private key, and the short-lived PIA
# auth token)
#
# LEGACY PATH — superseded by tasks/register-shared.yaml, which calls the
# on-host controller (files/pia_gateway.py) so Ansible and the unattended
# recovery/renewal timers share ONE registration implementation that also
# activates and verifies a handshake. This file is kept, unchanged, only as
# the documented fallback until the shared path has passed a live manual
# re-registration on minisforum (README "Rollout"). Delete it — and
# addkey-attempt.yaml, the `legacy` option, and pia-wg.conf.j2 — once that
# is done. Differences to be aware of while it still exists:
# - it does NOT verify a handshake (config write + handler restart only)
# - it does NOT record a verified-registration baseline in the
# controller state, so the renewal timer treats the host as "never
# verified" until the shared path runs once
# - its rotation stages a fixed `.key.new` path with `creates:` (an
# abandoned stage could be reused); the controller removes any
# `.key.new*` file before its own attempts.
#
# Description: Registers minisforum as a PIA WireGuard peer and writes # Description: Registers minisforum as a PIA WireGuard peer and writes
# {{ pia_wg_config_dir }}/{{ pia_wg_interface }}.conf. Skipped once that # {{ pia_wg_config_dir }}/{{ pia_wg_interface }}.conf. Skipped once that
# file exists, unless pia_force_reregister or pia_force_key_rotation is # file exists, unless pia_force_reregister or pia_force_key_rotation is

View File

@ -0,0 +1,64 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml when pia_registration_backend == "shared"
# Description: Registers with PIA through the on-host controller
# (`pia-gateway register`), i.e. the same token/addKey/stage/activate/
# verify transaction the unattended recovery and renewal timers use.
# Runs when there is no config yet, no VERIFIED registration baseline in
# the controller's state, or an explicit force flag. Reports success only
# after a fresh WireGuard handshake was observed on the activated config
# — never on `addKey` HTTP status alone.
#
# Requires the kill switch to be active (tasks/routing.yaml runs first in
# main.yaml); the program refuses to activate otherwise (exit 3) and the
# play fails loudly rather than opening anything.
- name: Check whether a PIA WireGuard config already exists
ansible.builtin.stat:
path: "{{ pia_wg_config_dir }}/{{ pia_wg_interface }}.conf"
register: pia_wg_conf_stat
- name: Check whether the controller has a verified registration baseline
ansible.builtin.stat:
path: "{{ pia_gateway_state_dir }}/state.json"
register: pia_state_stat
- name: Read the controller state (no secrets are stored in it)
ansible.builtin.slurp:
src: "{{ pia_gateway_state_dir }}/state.json"
register: pia_state_raw
when: pia_state_stat.stat.exists
- name: Decide whether registration is needed this run
ansible.builtin.set_fact:
pia_has_baseline: >-
{{ pia_state_stat.stat.exists
and ((pia_state_raw.content | b64decode | from_json).last_verified_registration is not none) }}
pia_registration_mode: "{{ 'new-key' if (pia_force_key_rotation | bool) else 'same-key' }}"
pia_region_args: "{{ ['--region', pia_region] if (pia_region is not none and (pia_region | string | trim | length) > 0) else [] }}"
- name: Run the shared PIA registration transaction
ansible.builtin.command:
argv: "{{ [pia_gateway_program, 'register', '--via', 'ansible', '--mode', pia_registration_mode] + pia_region_args }}"
register: pia_register_result
changed_when: true
failed_when: pia_register_result.rc != 0
when: >-
(pia_force_reregister | bool)
or (pia_force_key_rotation | bool)
or not pia_wg_conf_stat.stat.exists
or not (pia_has_baseline | bool)
# argv form: no shell, no secrets in arguments (credentials are read from
# the root-only file by the program itself), so this task is deliberately
# NOT no_log — its stdout is the secret-free JSON summary and the
# per-candidate diagnostics, which are exactly what an operator needs
# when a run fails. The lock is taken inside the program (waits up to
# pia_recovery_lock_wait_sec for an in-flight timer run to finish).
- name: Record that registration happened this run
ansible.builtin.set_fact:
pia_registered_this_run: "{{ pia_register_result is not skipped }}"
- name: Show the registration summary
ansible.builtin.debug:
msg: "{{ pia_register_result.stdout_lines | default(['skipped — config and verified baseline already present']) }}"

View File

@ -0,0 +1,36 @@
{
"interface": {{ pia_wg_interface | to_json }},
"config_dir": {{ pia_wg_config_dir | to_json }},
"state_dir": {{ pia_gateway_state_dir | to_json }},
"credentials_file": {{ (pia_gateway_etc_dir + '/credentials.env') | to_json }},
"ca_file": {{ (pia_wg_config_dir + '/pia-ca.crt') | to_json }},
"lock_file": "/run/lock/pia-gateway.lock",
"route_table": {{ pia_route_table_name | to_json }},
"rule_priority": {{ pia_rule_priority | int }},
"vlan50_subnet": {{ pia_vlan50_subnet | to_json }},
"iptables_chain": {{ pia_iptables_chain | to_json }},
"lan_interface": {{ pia_lan_interface | to_json }},
"mss_clamp_enabled": {{ pia_mss_clamp_enabled | bool | to_json }},
"region_candidates": {{ pia_region_candidates | to_json }},
"handshake_max_age_sec": {{ pia_healthcheck_handshake_max_age_sec | int }},
"failure_threshold": {{ pia_recovery_failure_threshold | int }},
"startup_grace_sec": {{ pia_recovery_startup_grace_sec | int }},
"restart_wait_sec": {{ pia_recovery_restart_wait_sec | int }},
"activation_wait_sec": {{ pia_recovery_activation_wait_sec | int }},
"recovery_max_duration_sec": {{ pia_recovery_max_duration_sec | int }},
"backoff_schedule_sec": {{ pia_recovery_backoff_schedule_sec | to_json }},
"backoff_jitter_fraction": 0.1,
"max_candidates_per_cycle": {{ pia_recovery_max_candidates_per_cycle | int }},
"max_rotation_candidates": {{ pia_recovery_max_rotation_candidates | int }},
"renewal_after_days": {{ pia_renewal_after_days | int }},
"http_connect_timeout_sec": {{ pia_recovery_http_connect_timeout_sec | int }},
"http_total_timeout_sec": {{ pia_recovery_http_total_timeout_sec | int }},
"clock_skew_allowance_sec": 60,
"backup_keep": {{ pia_recovery_backup_keep | int }},
"lock_wait_sec": {{ pia_recovery_lock_wait_sec | int }},
"alert_bridge_url": {{ pia_alert_bridge_url | to_json }},
"alert_source": {{ pia_alert_source | to_json }},
"metrics_file": {{ (pia_gateway_state_dir + '/metrics.prom') | to_json }},
"status_file": {{ (pia_gateway_state_dir + '/status.json') | to_json }},
"killswitch_unit": "pia-killswitch.service"
}

View File

@ -0,0 +1,7 @@
# Managed by Ansible (role: pia-gateway). Root-only (0600). Read at runtime by
# /usr/local/sbin/pia-gateway; never passed via argv or environment.
PIA_USER={{ pia_user }}
PIA_PASSWORD={{ pia_password }}
{% if pia_alert_bridge_api_key | default('') | length > 0 %}
ALERT_BRIDGE_API_KEY={{ pia_alert_bridge_api_key }}
{% endif %}

View File

@ -1,7 +1,12 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand. # Managed by Ansible (role: pia-gateway). Do not edit by hand.
# Read-only: `pia-gateway check` classifies status with the same code the
# recovery controller uses, logs it, and refreshes the status/metrics files.
# It never restarts, registers, or touches routing/firewall state.
[Unit] [Unit]
Description=PIA gateway health check (observability only, no remediation) Description=PIA gateway health check (observability only, no remediation)
[Service] [Service]
Type=oneshot Type=oneshot
ExecStart=/usr/local/sbin/pia-gateway-healthcheck.sh ExecStart={{ pia_gateway_program }} check
SuccessExitStatus=1
LimitCORE=0

View File

@ -1,110 +0,0 @@
#!/bin/bash
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Observability only. This script NEVER modifies routing, firewall, or
# WireGuard state — a failed check must close the path (already true by
# construction, via the kill switch in tasks/routing.yaml/firewall.yaml),
# never "recover" it. Output goes to the journal
# (journalctl -u pia-gateway-healthcheck); wire alerting off of that
# separately (see plan.md Phase 9) rather than adding remediation here.
#
# The "route decision" check below confirms what the kernel FIB *would*
# do for a VLAN 50 source — it does not send a packet and is not proof of
# actual PIA egress or identity. That proof requires the packet-capture
# test matrix in plan.md Phase 5; do not treat a clean run of this script
# as substituting for it.
set -uo pipefail
WG_IF="{{ pia_wg_interface }}"
TABLE="{{ pia_route_table_name }}"
SUBNET="{{ pia_vlan50_subnet }}"
PRIORITY="{{ pia_rule_priority }}"
CHAIN="{{ pia_iptables_chain }}"
MAX_HANDSHAKE_AGE={{ pia_healthcheck_handshake_max_age_sec }}
PROBE_SRC="${SUBNET%.*}.2" # a representative VLAN 50 address for the route-decision probe only; never actually used as a source
fail=0
# Informational only (not pass/fail) — which region/server this host is
# actually registered against right now, per the comment
# tasks/register.yaml writes into the generated config. Since PIA
# WireGuard registration now falls back across multiple regions
# (pia_region_candidates in defaults/main.yaml), "which one is live"
# isn't implied by config alone — it's whichever one answered OK last
# time the playbook ran, not necessarily the first-configured one.
configured_peer=$(grep -m1 '^# PIA WireGuard server actually used' "{{ pia_wg_config_dir }}/{{ pia_wg_interface }}.conf" 2>/dev/null || true)
if [[ -n "${configured_peer}" ]]; then
echo "INFO configured-peer: ${configured_peer#\# }"
fi
if ip link show "${WG_IF}" up &>/dev/null; then
echo "PASS interface: ${WG_IF} is up"
else
echo "FAIL interface: ${WG_IF} is not up"
fail=1
fi
if command -v wg &>/dev/null && wg show "${WG_IF}" latest-handshakes &>/dev/null; then
handshake_epoch=$(wg show "${WG_IF}" latest-handshakes 2>/dev/null | awk '{print $2}')
now=$(date +%s)
if [[ -n "${handshake_epoch}" && "${handshake_epoch}" -gt 0 ]]; then
age=$((now - handshake_epoch))
if (( age <= MAX_HANDSHAKE_AGE )); then
echo "PASS handshake: ${age}s old (max ${MAX_HANDSHAKE_AGE}s)"
else
echo "FAIL handshake: ${age}s old, exceeds max ${MAX_HANDSHAKE_AGE}s"
fail=1
fi
else
echo "FAIL handshake: no handshake recorded yet"
fail=1
fi
else
echo "WARN handshake: unable to query wg show (interface absent or wg missing)"
fail=1
fi
if ip rule show | grep -qE "^${PRIORITY}:[[:space:]]*from ${SUBNET} lookup ${TABLE}\$"; then
echo "PASS rule: from ${SUBNET} lookup ${TABLE} present at priority ${PRIORITY}"
else
echo "FAIL rule: from ${SUBNET} lookup ${TABLE} missing"
fail=1
fi
default_route=$(ip route show table "${TABLE}" 2>/dev/null | grep '^default' || true)
if [[ "${default_route}" == *"dev ${WG_IF}"* ]]; then
echo "PASS route: table ${TABLE} default is via ${WG_IF} (open)"
elif [[ "${default_route}" == *unreachable* ]] || ip route show table "${TABLE}" 2>/dev/null | grep -q '^unreachable default'; then
echo "PASS route: table ${TABLE} default is unreachable (closed, kill switch engaged)"
else
echo "FAIL route: table ${TABLE} has no default route at all (neither open via ${WG_IF} nor a terminal unreachable) — investigate immediately"
fail=1
fi
# Not a check on FORWARD's own default policy — that's deliberately left
# alone (see tasks/firewall.yaml for why: minisforum runs Flannel, which
# needs its own broad FORWARD ACCEPTs). What actually matters is that the
# dedicated chain exists, is jumped into first for this source, and ends
# in a real DROP.
if iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
echo "PASS firewall: FORWARD jumps to ${CHAIN} for ${SUBNET}"
else
echo "FAIL firewall: FORWARD does not jump to ${CHAIN} for ${SUBNET}"
fail=1
fi
if iptables -S "${CHAIN}" 2>/dev/null | grep -qE '^-A '"${CHAIN}"' -j DROP$'; then
echo "PASS firewall: ${CHAIN} ends in an unconditional DROP"
else
echo "FAIL firewall: ${CHAIN} has no unconditional DROP catch-all"
fail=1
fi
route_decision=$(ip route get 1.1.1.1 from "${PROBE_SRC}" 2>&1 || true)
if [[ "${route_decision}" == *"dev ${WG_IF}"* ]]; then
echo "PASS route-decision (not proof of live egress — see header comment): ${SUBNET} sourced traffic resolves via ${WG_IF}"
else
echo "WARN route-decision (not proof of live egress — see header comment): ${route_decision}"
fi
exit "${fail}"

View File

@ -0,0 +1,12 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
# Fired by OnFailure= of the recovery/renewal units when they crash or refuse
# to run (corrupt state). Delivery goes through alert-bridge if configured;
# a delivery failure is logged and never retried here (the next successful
# recover pass flushes its own pending queue).
[Unit]
Description=PIA gateway controller failure notifier
[Service]
Type=oneshot
ExecStart={{ pia_gateway_program }} notify --level error --message "pia-gateway: a controller unit failed on minisforum (journalctl -u pia-gateway-recover -u pia-gateway-renew)"
LimitCORE=0

View File

@ -0,0 +1,19 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# One bounded recovery pass (files/pia_gateway.py `recover`). Exit codes 1
# (unhealthy, decision made) and 3 (safety drift handled with an emergency
# close + alert) are normal outcomes, not unit failures — only a crash or a
# corrupt state file (2) is, and that triggers the OnFailure notifier.
[Unit]
Description=PIA gateway automatic recovery pass
After=pia-killswitch.service network-online.target
Wants=network-online.target
OnFailure=pia-gateway-onfailure.service
[Service]
Type=oneshot
ExecStart={{ pia_gateway_program }} recover
SuccessExitStatus=1 3
TimeoutStartSec={{ (pia_recovery_max_duration_sec | int) + 300 }}
LimitCORE=0
PrivateTmp=yes

View File

@ -0,0 +1,13 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
# systemd never starts a second instance while pia-gateway-recover.service is
# still running, and the program also holds /run/lock/pia-gateway.lock.
[Unit]
Description=Run the PIA gateway recovery pass every {{ pia_healthcheck_interval_sec }}s
[Timer]
OnBootSec=90
OnUnitActiveSec={{ pia_healthcheck_interval_sec }}
AccuracySec=5s
[Install]
WantedBy=timers.target

View File

@ -0,0 +1,14 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
[Unit]
Description=PIA gateway proactive registration renewal
After=pia-killswitch.service network-online.target
Wants=network-online.target
OnFailure=pia-gateway-onfailure.service
[Service]
Type=oneshot
ExecStart={{ pia_gateway_program }} renew
SuccessExitStatus=1
TimeoutStartSec={{ (pia_recovery_max_duration_sec | int) + 300 }}
LimitCORE=0
PrivateTmp=yes

View File

@ -0,0 +1,11 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
[Unit]
Description=Daily check whether the PIA registration is older than {{ pia_renewal_after_days }} days
[Timer]
OnCalendar={{ pia_renewal_calendar }}
RandomizedDelaySec={{ pia_renewal_jitter_sec }}
Persistent=true
[Install]
WantedBy=timers.target

File diff suppressed because it is too large Load Diff