pia_forward_bypass_chains held up against real production conditions, not just a short recheck: 1528 healthy passes, 0 SAFETY DRIFT events, 0 safety_drift classifications since the fix was deployed (2026-09-22 20:37 -> 2026-09-23 22:02). kube-router kept resyncing throughout; the bypass tolerance handled it silently every time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
828 lines
50 KiB
Markdown
828 lines
50 KiB
Markdown
# pia-gateway
|
||
|
||
Makes `minisforum` a PIA WireGuard egress gateway for VPN VLAN 50
|
||
(`10.10.50.0/24`), without changing its own default route. Implements
|
||
Phase 2 of `~/repo/homelab/plan.md`.
|
||
|
||
## Before the first real run
|
||
|
||
1. **The PIA API calls have gone through three rounds of correction
|
||
against real live testing (2026-08-24). They now live in
|
||
`files/pia_gateway.py` (`PiaApi`), with the original in-Ansible flow
|
||
kept as `tasks/register-legacy.yaml` until the shared path is proven
|
||
live — see "Automatic recovery" below:**
|
||
- `serverlist.piaservers.net/vpninfo/servers/v6`: works. Region values
|
||
everywhere in this role are the API's **id** field (e.g. `japan`,
|
||
`hk`, `taiwan`), never a display name ("JP Tokyo", "Hong Kong",
|
||
"Taiwan" — those are Gluetun's labels, not something this role or
|
||
PIA's API accepts).
|
||
- Token acquisition originally guessed a regional-meta-server
|
||
endpoint (`/authv3/generateToken`) that turned out not to be the
|
||
real flow at all — it was rebuilt from PIA's actual
|
||
`pia-foss/manual-connections` `get_token.sh` (fetched and read in
|
||
full): a single fixed `POST
|
||
https://www.privateinternetaccess.com/api/client/v2/token`,
|
||
multipart `username`/`password`, **normal system CA validation**
|
||
(no `--cacert`, no `validate_certs: false`), independent of region
|
||
entirely — only WireGuard server *selection* is regional. Not yet
|
||
exercised with real credentials end to end.
|
||
- `{wg}:1337/addKey` (WireGuard key registration) — confirmed against
|
||
`connect_to_wireguard_with_token.sh` (fetched and read in full):
|
||
this one genuinely does need PIA's own CA bundle
|
||
(`files/pia-ca.crt`, `--cacert`) and `--connect-to`. Originally
|
||
failed over only across servers *within* a single fixed region
|
||
(`pia_region: japan`) — in practice all three JP Tokyo servers
|
||
failed the same way, one after another, so failover now also
|
||
crosses regions: every server in Hong Kong, then every server in
|
||
Taiwan, then every server in JP Tokyo (`pia_region_candidates` in
|
||
`defaults/main.yaml`), stopping at the first `status: OK`. Token
|
||
passed via stdin rather than argv. See `tasks/addkey-attempt.yaml`
|
||
and "Region fallback" below.
|
||
2. Make sure the repo-root `.env` has `PIA_USER`/`PIA_PASSWORD` set (same
|
||
keys `manifests/media/pia-secret.sh` already reads) — `ansible/
|
||
host_vars/minisforum.yaml` reads them directly from there via a
|
||
`lookup('ansible.builtin.file', ...)` + regex, deliberately not
|
||
ansible-vault. The lookup runs on the control node; `.env` itself is
|
||
never copied and never committed (gitignored). **Since the
|
||
auto-recovery work, only `PIA_USER`/`PIA_PASSWORD` (and, if present,
|
||
`ALERT_BRIDGE_API_KEY`) are provisioned to minisforum's root-only
|
||
`/etc/pia-gateway/credentials.env`** so the host can re-register
|
||
unattended — see "Credentials on the host" below. The playbook still
|
||
needs `-J` because `group_vars/all/vault.yaml` is vault-encrypted and
|
||
loaded for every host, not because the PIA credentials are in it.
|
||
3. Confirm console/recovery access to `minisforum` (IPMI/physical/other
|
||
out-of-band path) before applying — this role changes host routing and
|
||
firewall policy. If SSH becomes unreachable, use that path.
|
||
4. `ufw status verbose` and `wg show` (existing `wg0`) were not inspected
|
||
live before writing this role (no passwordless sudo in discovery) —
|
||
spot-check that `wg0` (the separate home-VPN role) is unaffected after
|
||
applying.
|
||
|
||
## Region fallback
|
||
|
||
Registration tries every WireGuard server in every configured region, in
|
||
order, stopping at the first `status: OK`:
|
||
|
||
```text
|
||
hk (Hong Kong):
|
||
wg server 1
|
||
wg server 2
|
||
taiwan (Taiwan):
|
||
wg server 1
|
||
wg server 2
|
||
japan (JP Tokyo):
|
||
wg server 1
|
||
wg server 2
|
||
wg server 3
|
||
```
|
||
|
||
The actual number and order of servers within each region always comes
|
||
from the live serverlist response — never hardcoded here.
|
||
|
||
**Automatic fallback** (default — tries `pia_region_candidates` in
|
||
order):
|
||
|
||
```bash
|
||
ansible-playbook -i ansible/inventory.yaml \
|
||
ansible/playbooks/pia-gateway.yaml -K -J
|
||
```
|
||
|
||
**Forced single region**, for troubleshooting one region in isolation —
|
||
bypasses `pia_region_candidates` entirely, does not fall back to the
|
||
others:
|
||
|
||
```bash
|
||
ansible-playbook -i ansible/inventory.yaml \
|
||
ansible/playbooks/pia-gateway.yaml -K -J \
|
||
-e pia_region=hk
|
||
```
|
||
|
||
Both `pia_region` and every entry in `pia_region_candidates` are PIA API
|
||
**ids** — `hk`, `taiwan`, `japan` — not the display names Gluetun's
|
||
`SERVER_REGIONS` config uses ("Hong Kong", "Taiwan", "JP Tokyo"). Passing
|
||
a display name here matches nothing and fails the "region exists"
|
||
assertion before any network call is even made.
|
||
|
||
Which region/server actually ended up registered is recorded as
|
||
`pia_region_used`/`pia_region_name_used`/`pia_wg_server_used`, written
|
||
into `pia-wg.conf`'s own `[Peer]` comment, and surfaced by the health
|
||
check (`INFO configured-peer: ...`) — it is whichever one answered first,
|
||
not necessarily `pia_region_candidates[0]`.
|
||
|
||
If every region fails: `journalctl` (or the play's own output) has one
|
||
`debug` line per failed attempt, each showing region name+id, server
|
||
hostname+IP, curl exit code, a classified reason (connection timeout /
|
||
TLS-SSL failure / HTTP failure / empty response / malformed JSON /
|
||
parsed-but-not-OK status, never a generic catch-all), and sanitized
|
||
stderr — never the token, credentials, the full curl invocation, request
|
||
stdin, or a complete response body. If Hong Kong and Taiwan fail exactly
|
||
the same way Tokyo did, that's a strong signal the problem is shared
|
||
across all three (the request shape, the account credentials, or
|
||
minisforum's own network path) rather than one region being down — see
|
||
`tasks/register-legacy.yaml`'s header comment (the controller logs the
|
||
same per-candidate `kind=`/`detail=` classification).
|
||
|
||
**Worked example (2026-08-24):** exactly that happened — all 7 servers
|
||
across all 3 regions came back `HTTP failure (non-2xx response)`, curl
|
||
exit 22, `The requested URL returned error: 401`, uniformly. The cause
|
||
was in `addkey-attempt.yaml`'s own curl task (legacy path), not PIA or any region:
|
||
`ansible.builtin.command`'s `stdin` argument appends a trailing newline
|
||
by default (`stdin_add_newline` defaults to `true`), and
|
||
`--data-urlencode pt@-` does not strip it — every server was receiving
|
||
`"<real token>\n"` (URL-encoded, so a trailing `%0A`) as `pt`, a
|
||
different and invalid value, and correctly rejecting it. Fixed with
|
||
`stdin_add_newline: false` on that task. `connect_to_wireguard_with_token
|
||
.sh` never hits this because it passes the token as a literal shell
|
||
variable, not via stdin — the stdin delivery here is this repo's own
|
||
addition (to keep the token out of `ps`), so it needed the flag PIA's
|
||
reference script never had to think about. Left as a worked example
|
||
because "every candidate failed identically" pointing at one shared bug
|
||
in *this* code, not PIA, is exactly the diagnostic story this section
|
||
promises — and it happened to be true the first time it was tested for
|
||
real.
|
||
|
||
## What it does
|
||
|
||
- Registers minisforum as a PIA WireGuard peer — see "Region fallback"
|
||
above — and writes `/etc/wireguard/pia-wg.conf` (`Table = off` — this
|
||
role owns all routing for the interface, not wg-quick).
|
||
- Installs `pia-killswitch.service`, ordered `Before=
|
||
wg-quick@pia-wg.service`, that seeds a closed state at every boot (and
|
||
after any UFW reload — see handlers/main.yaml): the
|
||
`from 10.10.50.0/24 lookup pia` rule, a terminal `unreachable default`
|
||
route in the `pia` table (only when `pia-wg` isn't already up — see
|
||
the script's own comment for why), and a dedicated `PIA-VLAN50`
|
||
iptables chain jumped into by a single rule at the very top of
|
||
`FORWARD`, matching only source `10.10.50.0/24`, ending in an
|
||
unconditional rate-limited log+drop. This is deliberately **not** a
|
||
global `FORWARD` default-policy change — minisforum runs Flannel,
|
||
which needs its own broad `FORWARD` ACCEPTs for pod traffic (source
|
||
`10.42.0.0/16`, disjoint from VLAN 50) — so the kill switch is scoped
|
||
to a chain Flannel/k3s traffic can never enter, rather than risking a
|
||
chain-wide policy that was never actually proven safe against it.
|
||
Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW
|
||
**input** allow for 53/tcp+udp — it's locally-terminated traffic, not
|
||
routed — see `tasks/firewall.yaml`.
|
||
|
||
**The FORWARD jump's position is actively re-asserted on every run, not
|
||
just seeded once** — found live, 2026-09-22, on the very first
|
||
deployment of the auto-recovery controller. minisforum's `FORWARD`
|
||
chain default policy is `DROP` (kube-router's doing, not this role's —
|
||
correcting an assumption an earlier version of this doc made), and
|
||
kube-router and `dockerd` each independently reassert their **own**
|
||
jump at position 1 whenever they resync or restart, silently pushing
|
||
this rule down without removing it (kube-router doesn't even dedupe
|
||
its own older copy when it does — a duplicate of *its* rule was found
|
||
live too). The script used to check only whether its rule *existed*
|
||
(`iptables -C`) before deciding not to touch anything; existence isn't
|
||
proof of position, so once pushed down it stayed down until something
|
||
else fixed it. It now checks the actual first rule and repositions
|
||
(delete + reinsert at 1) if it isn't this one. Confirmed safe to do
|
||
unconditionally by reading kube-router's and Docker's actual chain
|
||
contents rather than assuming: `KUBE-ROUTER-FORWARD` only matches
|
||
specific pod `/32` addresses (disjoint from `10.10.50.0/24`),
|
||
`DOCKER-USER` was empty, and `DOCKER-FORWARD`'s one direct rule is
|
||
scoped to `-i docker0` (VLAN 50 traffic arrives via `enp1s0`) — none
|
||
of them can intercept VLAN 50 traffic regardless of order, so moving
|
||
this rule back to position 1 relative to them changes nothing for pod
|
||
or Docker-bridge traffic. The controller's own safety check
|
||
(`forward_jump_first`) was working as intended when it refused to
|
||
register against the drifted state — this is a fix to the thing it
|
||
checks, not a loosening of the check itself.
|
||
|
||
**This recurs far more often than "boot-scale," and the fix above only
|
||
helped once.** The reposition logic is correct, but it only *ran* on
|
||
the existing triggers (boot, a UFW reload, an Ansible-detected content
|
||
change to the script). Confirmed live: after that fix landed and got
|
||
one `pia-gateway register` attempt past this check, the very next
|
||
attempt — roughly an hour later, no reboot, no UFW reload, no script
|
||
change in between — hit the identical failure again. kube-router
|
||
resyncs on ordinary Kubernetes API events (this cluster's CronJobs —
|
||
`ca-sync`, `porkbun-ddns`, `gitea-backup` — each create and delete a
|
||
Pod every few minutes, each a plausible trigger), not just at
|
||
boot/restart scale as originally assumed. `pia-killswitch-reassert
|
||
.timer` (every `pia_healthcheck_interval_sec`, unconditional, not
|
||
gated by `pia_auto_recovery_enabled`) exists specifically to close this
|
||
gap by giving the already-correct, already-idempotent script regular
|
||
chances to run instead of only sporadic ones.
|
||
|
||
**The reassert timer alone wasn't the full fix either — it created a
|
||
new failure mode.** Confirmed live, 2026-09-22, minutes after enabling
|
||
`pia-gateway-recover.timer` for the first time (rollout step 3):
|
||
`pia-gateway-recover` and `pia-killswitch-reassert` are two
|
||
independently-scheduled 60s timers with no coordination between them.
|
||
kube-router turned out to resync (and re-push our jump down) roughly
|
||
every 60-90s on this cluster — far more often than "rare, boot-scale"
|
||
— so a `recover` pass would periodically sample `iptables -S FORWARD`
|
||
in the window between a kube-router push-down and the next reassert
|
||
fix, correctly detect real (if momentary) drift, and call
|
||
`emergency_close()` — which unconditionally forces table `pia`'s route
|
||
to `unreachable` and resets `consecutive_failures` to 0 in the same
|
||
step. The result was a self-reinforcing outage loop, observed directly
|
||
in the journal:
|
||
|
||
```
|
||
19:38:58 3rd consecutive unhealthy check → recovery_cycle → restart
|
||
19:39:09 recovered via restart (10s) — route open, handshake fresh
|
||
19:40:03 drift detected again ("FORWARD jump ... not first")
|
||
→ emergency close → route forced closed, counter reset to 0
|
||
19:41:08 drift again → emergency close again → route stays closed
|
||
```
|
||
|
||
VLAN 50 was open for roughly 10 seconds out of every 60-90 second
|
||
cycle. Three consecutive real failures were needed to trigger the
|
||
restart that reopens the route, but a fresh drift-and-emergency-close
|
||
kept arriving before three could accumulate — the counter reset every
|
||
time. This is not a bug in the detection (the drift was real, if
|
||
transient) — it's that treating *every* momentary reordering as a full
|
||
safety emergency was stricter than the actual invariant required,
|
||
given kube-router's own resync cadence on this specific host.
|
||
|
||
**Fix: `pia_forward_bypass_chains` (`Classifier
|
||
._forward_jump_effectively_first` in `files/pia_gateway.py`).** The
|
||
safety check now tolerates a configured, named FORWARD chain sitting
|
||
ahead of our jump — but only for as long as that chain's *own current
|
||
content* re-verifies, live, on every single check, as structurally
|
||
unable to match `10.10.50.0/24` (every `-s`/`-d` value on every rule
|
||
disjoint from our subnet; a bare/unconditional rule, a negated match,
|
||
or any construct the checker doesn't explicitly recognize fails
|
||
closed, not open). This is not a one-time trust of kube-router — if
|
||
its rules ever changed to include something that could actually
|
||
intercept VLAN 50 traffic, or an unrecognized chain/rule appeared
|
||
ahead of ours instead, it still fails closed exactly as before. Scoped
|
||
to `KUBE-ROUTER-FORWARD` only (not `DOCKER-USER`/`DOCKER-FORWARD` —
|
||
see `pia_forward_bypass_chains`'s own comment in `defaults/main.yaml`
|
||
for why only kube-router earned the exception). An emergency close and
|
||
a hard stop remain the outcome for anything not on this narrow,
|
||
content-reverified allowlist. Immediate live mitigation while this fix
|
||
was being built: `sudo systemctl stop pia-gateway-recover.timer`, then
|
||
a manual `sudo systemctl restart wg-quick@pia-wg` to reopen the route
|
||
(`pia-killswitch-reassert.timer` was left running throughout — it was
|
||
never the problem).
|
||
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
|
||
top of that closed baseline: default route via `pia-wg` in the `pia`
|
||
table, ACCEPT + established/related return inserted into the
|
||
`PIA-VLAN50` chain (not `FORWARD` directly), and source-NAT/MASQUERADE
|
||
scoped to `10.10.50.0/24` on `pia-wg` only.
|
||
- Clamps TCP MSS on the `pia-wg` forward path (`pia_mss_clamp_enabled:
|
||
true`, `iptables -t mangle ... TCPMSS --clamp-mss-to-pmtu`, scoped to
|
||
SYN packets sourced from `10.10.50.0/24` outbound on `pia-wg` only, in
|
||
`PostUp`/`PreDown` alongside the rules above). Confirmed needed live
|
||
(2026-08-25), not enabled speculatively: a real download hung with a
|
||
TLS read timeout on the larger handshake response while small
|
||
requests worked fine, and a direct `ping -M do -s 1450` test from a
|
||
VLAN 50 pod confirmed the real path MTU is `pia-wg`'s `1420` (the ICMP
|
||
"Frag needed" reply arrives correctly from minisforum itself — our own
|
||
side relays PMTU discovery fine, so the black hole is further out, on
|
||
PIA's network or the remote server's own path, where clamping the MSS
|
||
up front avoids needing that ICMP round-trip at all).
|
||
- `pia-wg.conf`'s `PreDown` lines are deliberately tolerant of the rule
|
||
they're removing not existing (`iptables -D ... 2>/dev/null || true`)
|
||
— enabling MSS clamping live (above) exposed why this matters:
|
||
`systemctl restart wg-quick@pia-wg` tears down the *currently running*
|
||
interface using whatever `PreDown` lines are on disk *right now* — if
|
||
Ansible already rewrote the config with a new/changed rule before the
|
||
restart handler fires, the live interface (brought up under the *old*
|
||
rules) won't have whatever the new `PreDown` line is trying to delete.
|
||
`wg-quick`'s own `execute_hooks()` aborts the entire down/up sequence
|
||
on the first failing hook (confirmed against its real source, not
|
||
assumed), so one non-idempotent `-D` used to turn any future
|
||
`PostUp`/`PreDown` content change into a broken restart *and* an
|
||
orphaned interface (the never-reached built-in `ip link delete` step
|
||
left `pia-wg` existing but unconfigured, which then made the
|
||
following `wg-quick up` fail too with `` `pia-wg' already exists``) —
|
||
requiring manual recovery (`ip link delete dev pia-wg` before a fresh
|
||
`systemctl start`). Every `-D` line now tolerates this by design, so a
|
||
config change that alters `PostUp`/`PreDown` content can never break
|
||
a restart this way again — confirmed live, 2026-08-27.
|
||
- Installs an observability-only health check (`pia-gateway-healthcheck
|
||
.timer`, every `pia_healthcheck_interval_sec`) that runs `pia-gateway
|
||
check` — the same status classifier the recovery controller uses — and
|
||
logs interface, handshake age, rule/route ordering and firewall state to
|
||
the journal plus `/var/lib/pia-gateway/status.json` / `metrics.prom`.
|
||
It never remediates. Neither it nor the controller sends a packet
|
||
through the tunnel: a "healthy" class is a fresh handshake plus intact
|
||
kill-switch invariants, not proof of egress identity.
|
||
- Installs the automatic recovery controller and its timers — see
|
||
"Automatic recovery" below.
|
||
|
||
## What it deliberately does not do
|
||
|
||
- Does not touch minisforum's own default route (asserted at the start of
|
||
every run — `tasks/assert-baseline.yaml` fails loudly if that's already
|
||
wrong).
|
||
- Does not enable IPv6 forwarding or any IPv6 handling for VLAN 50.
|
||
- Does not modify `ansible/roles/wireguard` (the separate `wg0` home-VPN
|
||
server role) or its interface.
|
||
|
||
## Known failure mode: the session dies while the interface looks fine
|
||
|
||
Seen twice so far, ~28 days apart — 2026-08-25 (root-caused in
|
||
`INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md`, Incident 1) and
|
||
2026-09-22. `pia-wg`'s handshake with PIA's server stops renewing;
|
||
the interface, `wg-quick@pia-wg` (`active (exited)` is normal for
|
||
`wg-quick`), the `pia` routing table and all firewall rules stay exactly
|
||
as `PostUp` left them. The kill switch's "interface down → unreachable
|
||
route" trigger never fires because the interface never goes down, so
|
||
VLAN 50 fails closed — correctly, but silently.
|
||
|
||
On 2026-08-25 a plain `systemctl restart wg-quick@pia-wg` did not produce
|
||
a new handshake and only a fresh `addKey` registration did — strongly
|
||
suggesting PIA had dropped the registered key server-side. The
|
||
2026-09-22 occurrence went straight to re-registration (bare restart
|
||
untested). The underlying trigger is still not confirmed; the ~4-week gap
|
||
is consistent with a fixed registration lifetime on PIA's side, which is
|
||
what the proactive renewal timer below is for. This failure mode is now
|
||
handled automatically — see the next section.
|
||
|
||
## Automatic recovery
|
||
|
||
`files/pia_gateway.py` is installed as `/usr/local/sbin/pia-gateway`
|
||
(root, `0700`, runs on the host's `python3.13`). Almost entirely standard
|
||
library — the one exception is `addKey` specifically, which shells out to
|
||
`curl` (now an explicit `apt` dependency of this role); see
|
||
`HttpClient.get_pinned_curl`'s own docstring and "PIA's CA certificate and
|
||
why addKey uses curl, not Python's `ssl`" below for why. It is **one**
|
||
implementation of PIA registration shared by the unattended timers and by
|
||
this role's own registration step, so the two can never drift. Layout on
|
||
the host:
|
||
|
||
| Path | Purpose |
|
||
| --- | --- |
|
||
| `/etc/pia-gateway/config.json` (`0600`) | rendered from role defaults (`templates/pia-gateway-config.json.j2`) |
|
||
| `/etc/pia-gateway/credentials.env` (`0600`) | `PIA_USER`, `PIA_PASSWORD`, optional `ALERT_BRIDGE_API_KEY` — see "Credentials on the host" |
|
||
| `/var/lib/pia-gateway/state.json` (`0600`) | versioned controller state (atomic tmp+fsync+rename); **no secrets** |
|
||
| `/var/lib/pia-gateway/status.json`, `metrics.prom` (`0644`) | last classification, for humans/monitoring |
|
||
| `/run/lock/pia-gateway.lock` | one `flock` shared by every mutating action (timers, `register`, `rewrite-config`); released by the kernel if a process dies |
|
||
| `pia-gateway-recover.timer` → `.service` | every 60 s, one bounded recovery pass (`recover`) |
|
||
| `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) |
|
||
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
|
||
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
|
||
| `pia-killswitch-reassert.timer` → `.service` | every `pia_healthcheck_interval_sec`, re-runs `pia-killswitch.sh` (idempotent). **Not** gated by `pia_auto_recovery_enabled` — see the "FORWARD jump's position" note above and the service unit's own header for why |
|
||
|
||
### PIA's CA certificate and why addKey uses curl, not Python's `ssl`
|
||
|
||
Found live, 2026-09-22, on the very first `pia-gateway register` attempt
|
||
after the FORWARD-jump fix above got past the kill-switch check:
|
||
`addKey` failed against every candidate with `kind=tls detail=certificate
|
||
verification failed`. Root-caused directly on minisforum:
|
||
|
||
```
|
||
$ sudo python3 -c '...ctx.wrap_socket(sock, server_hostname="Server-11669-10a")...'
|
||
SSLCertVerificationError verify_message= Basic Constraints of CA cert not marked critical
|
||
```
|
||
|
||
PIA's own CA certificate (`files/pia-ca.crt`) has a `CA:TRUE` Basic
|
||
Constraints extension that isn't flagged `critical` — a real RFC 5280
|
||
non-conformance, confirmed **not** a stale vendored copy: a fresh fetch
|
||
from `pia-foss/manual-connections` on GitHub is byte-identical (same
|
||
`32e9b1d1...` SHA-256 already recorded in the file's own header) and has
|
||
the identical defect. This is PIA's problem, not ours, and not fixable by
|
||
re-vendoring. Recent OpenSSL (3.2+; confirmed live against minisforum's
|
||
`3.5.7`) rejects this by default — a check that used to be opt-in only
|
||
under `X509_V_FLAG_X509_STRICT` and has since been promoted into the
|
||
default chain-verification path. `openssl s_client` on this repo's own
|
||
dev machine (LibreSSL) and this same host's `curl` both accept the exact
|
||
same certificate/chain/CA file without complaint — only Python's `ssl`
|
||
module, using this host's system OpenSSL, rejects it. That's also why
|
||
this went unnoticed until the auto-recovery work: the legacy Ansible
|
||
path's three successful registrations (2026-08-24, 08-25, and this
|
||
morning) all went through `curl --cacert` (`addkey-attempt.yaml`), never
|
||
through Python's `ssl` module — this was the *first* time that code path
|
||
ever ran for real.
|
||
|
||
Python's stdlib `ssl` module has no custom-verify-callback hook (a
|
||
long-standing, deliberate upstream gap — unlike Go's
|
||
`tls.Config.VerifyPeerCertificate` or OpenSSL's own C API), so there is
|
||
no clean way to tolerate just this one known, third-party-controlled
|
||
defect while still fully validating everything else (chain, expiry,
|
||
hostname/SAN). Rather than weakening `verify_mode` for the whole
|
||
connection or adding a new dependency (`pyOpenSSL`, for a real verify
|
||
callback — considered, rejected as a bigger lift than necessary) to work
|
||
around a gap in a library we don't control, `addKey` specifically now
|
||
shells out to `curl` (`HttpClient.get_pinned_curl`, `curl` added to this
|
||
role's `apt` install list) — the same tool PIA's own reference client
|
||
(`connect_to_wireguard_with_token.sh`) already uses for this exact
|
||
endpoint, with `--cacert` and never `-k`/`--insecure`, so this is a
|
||
different, already-proven-correct TLS backend for the one endpoint that
|
||
needs it, not a security downgrade. `serverlist` and the token endpoint
|
||
are unaffected (they present public WebPKI certs, which don't have this
|
||
problem) and still use the stdlib `ssl` path. Verified: a real subprocess
|
||
`curl` call from this repo's dev machine against the actual failing
|
||
candidate (`Server-11669-10a` / `69.33.198.97`) gets a genuine HTTP
|
||
response from PIA (401, expected — a dummy token was used), proving TLS
|
||
validation succeeds via this path where it failed via Python's `ssl`.
|
||
curl's own exit codes for a `--fail`-triggered rejection aren't fully
|
||
portable either — confirmed live: a curl built against SecureTransport
|
||
(this dev machine) reports exit 56, not the "usual" 22 GNU/OpenSSL curl
|
||
uses for the identical case — so classification checks curl's own error
|
||
text as a second signal alongside the exit code, not the exit code alone.
|
||
|
||
### Status classes
|
||
|
||
Every pass classifies the gateway into exactly one of: `healthy`,
|
||
`boot_grace` (stale within 120 s of a boot or of an intentional restart —
|
||
not counted), `paused`, `tunnel_stale` (handshake 0 / older than
|
||
`pia_healthcheck_handshake_max_age_sec` / in the future beyond 60 s of
|
||
skew), `tunnel_absent` (interface down/missing, expected peer missing,
|
||
fresh handshake but table route closed), `registration_or_endpoint_failure`
|
||
/ `host_connectivity_failure` (stale **and** in backoff after a cycle
|
||
failed that way), `safety_drift`, `unknown` (a read command failed — no
|
||
action taken, alert after 3). `status.json` carries both the decision
|
||
class and the raw `tunnel_class`, so "paused" never hides a dead tunnel.
|
||
|
||
Safety invariants checked on every pass, all of which must hold before
|
||
any restart/registration is attempted: the priority-100 source rule; a
|
||
default in table `pia` that is either `dev pia-wg` (open) or
|
||
`unreachable` (closed) — *no* default at all is a failure, not "closed";
|
||
the FORWARD jump to `PIA-VLAN50` is the **first effective** rule (an
|
||
emergency rule of our own is the only thing allowed above it); the chain
|
||
ends in an unconditional `DROP` and every `ACCEPT` before it is scoped to
|
||
`pia-wg`; the host default route is still via `enp1s0`.
|
||
|
||
### Recovery state machine (`recover`, every 60 s)
|
||
|
||
| State | Action |
|
||
| --- | --- |
|
||
| healthy | reset counters; if a backoff was pending, log/notify "healthy again" |
|
||
| 1–2 consecutive stale/absent | count only |
|
||
| 3 consecutive (`pia_recovery_failure_threshold`) | save pre-fix evidence (age, endpoint, reasons) → **stage 1**: `systemctl restart wg-quick@pia-wg` once, wait up to 90 s for two consecutive healthy classifications with a handshake newer than the attempt start |
|
||
| stage 1 fails | **stage 2**: serverlist → token → `addKey` with the *existing* key against candidates in `pia_region_candidates` order (max `pia_recovery_max_candidates_per_cycle`), each accepted response staged as a validated config, activated, and verified the same way; a candidate that answers `OK` but never handshakes is skipped for the next one |
|
||
| stage 2: API accepted the key but nothing handshook | **stage 3**: exactly one staged fresh-key attempt (`pia-wg.key.new.<ts>.<pid>`, promoted only after verification, previous key backed up). Skipped when the failure was uniform (auth error, host connectivity, every API call failing the same way) |
|
||
| any failure | previous config restored (best effort) and re-activated; gateway stays closed; backoff 5 → 15 → 30 → 60 min (±10 % jitter, capped) persisted in state; classified failure recorded; alert |
|
||
| verified recovery | stage, region/server, elapsed time and pre-fix age recorded in `last_recovery`; `last_verified_registration` updated when a registration happened; one "recovered" alert |
|
||
| safety drift | **no** restart/registration. Insert `-I FORWARD 1 -s 10.10.50.0/24 -m comment --comment pia-gateway-emergency -j DROP`, then `ip route replace unreachable default table pia`, verify the DROP is first, alert `error`. If the rule already exists but isn't first any more (something else prepended a rule ahead of it, e.g. an out-of-band `ufw reload`), it's deleted and reinserted at position 1 rather than left where it drifted to — a mere existence check (`iptables -C`) says nothing about position. Left in place until a later pass sees the invariants intact, then removed under the lock. Automatic repair of the kill switch itself is deliberately **not** attempted (see "Residual risk") |
|
||
| paused (`pia-gateway pause`) | checks and notifications continue; never restart/register/renew. Safety drift is **still** contained by the emergency close while paused — pausing suspends recovery, never the fail-closed guarantee |
|
||
| corrupt/unknown-schema state file | refuse to act (exit 2, unit failure → alert). `pia-gateway state-reset --confirm` moves the file aside and starts fresh |
|
||
|
||
One pass is bounded by `pia_recovery_max_duration_sec` (600 s) inside
|
||
the program and `TimeoutStartSec` (900 s) in the unit; systemd never
|
||
starts a second instance while one runs, and the lock makes a manual
|
||
`register` wait (up to `pia_recovery_lock_wait_sec`) rather than
|
||
interrupt an activation. Reboots: the boot id is tracked, a new boot gets
|
||
`pia_recovery_startup_grace_sec` before stale counts; retry/backoff state
|
||
survives because it is on disk. A backward wall-clock step (NTP) that
|
||
leaves a persisted retry deadline further out than the backoff cap is
|
||
treated as a clock jump and the deadline is cleared rather than stalling
|
||
recovery.
|
||
|
||
`restart` returning 0, `active (exited)`, or `addKey` returning `status:
|
||
OK` are never treated as success on their own — only an observed fresh
|
||
handshake on the activated config is, and specifically one reported
|
||
**strictly after** the moment activation started: a handshake in the
|
||
same whole second as that moment doesn't count either, since `wg show`
|
||
only has one-second resolution and there'd be no way to tell it apart
|
||
from a stale, pre-activation handshake that happens to share the second.
|
||
|
||
Every mutating command — `recover`, `renew`, `register`,
|
||
`rewrite-config`, `pause`, `resume`, `state-reset` — takes the same
|
||
`/run/lock/pia-gateway.lock` **before** reading state, not after: reading
|
||
state first and only locking around the write would let a slower command
|
||
silently overwrite whatever a faster one wrote in between (in particular,
|
||
it's how a `pause` could otherwise be lost to an in-flight recovery pass
|
||
that read state before the pause ran and only saves its own copy back
|
||
once it finishes). `pause`/`resume`/`register`/`rewrite-config`/
|
||
`state-reset` wait up to `pia_recovery_lock_wait_sec` for the lock and
|
||
exit 4 if it's still busy; `recover`/`renew` (the timers) never wait —
|
||
they just skip that tick if something else currently holds it.
|
||
|
||
### Operator commands (all on minisforum, as root)
|
||
|
||
```bash
|
||
pia-gateway status # human summary incl. backoff, last verified registration, pause
|
||
pia-gateway check # what the health-check timer runs (read-only, exit 1 = unhealthy)
|
||
pia-gateway check --json # machine-readable
|
||
pia-gateway pause --reason "maintenance" # persists across reboot; suppresses recover AND renew; kill switch untouched
|
||
pia-gateway resume
|
||
pia-gateway retry-now # one recovery pass ignoring threshold and backoff
|
||
pia-gateway register [--mode same-key|new-key] [--region hk] # explicit transaction, verified, under the lock
|
||
pia-gateway rewrite-config # re-render pia-wg.conf from the recorded registration (hook changes), no API call
|
||
pia-gateway renew --force-due # exercise renewal without waiting 21 days
|
||
pia-gateway state-reset --confirm # only after reading the corrupt file it moves aside
|
||
journalctl -u pia-gateway-recover -u pia-gateway-renew -u pia-gateway-healthcheck -n 100
|
||
cat /var/lib/pia-gateway/status.json
|
||
```
|
||
|
||
Through Ansible (repo root, `-K -J`):
|
||
|
||
```bash
|
||
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml -K -J # idempotent; registers only if no config or no verified baseline
|
||
ansible-playbook ... -e pia_force_reregister=true # re-register with the EXISTING key (same-key)
|
||
ansible-playbook ... -e pia_force_key_rotation=true # stage a NEW key, register it, promote only after verification
|
||
ansible-playbook ... -e pia_region=hk # single-region override (either of the above)
|
||
ansible-playbook ... -e pia_auto_recovery_enabled=false # stop+disable both timers, keep everything else (rollback)
|
||
ansible-playbook ... -e pia_renewal_enabled=true # after the live drills — enables the daily renewal timer
|
||
ansible-playbook ... -e pia_remove_host_credentials=true # deliberate removal of /etc/pia-gateway/credentials.env
|
||
```
|
||
|
||
`pia_force_reregister` **reuses** the on-disk key; only
|
||
`pia_force_key_rotation` generates a new one (the root README used to say
|
||
otherwise — corrected). A normal run also quiesces both timers, waits for
|
||
any in-flight pass, and re-enables them after handlers, so an Ansible
|
||
restart never races an automatic one. A run whose only change is
|
||
`config.json` (e.g. flipping `pia_mss_clamp_enabled`) re-renders the
|
||
WireGuard config via `rewrite-config` without touching PIA — the
|
||
"template-only change needs a full re-registration" gap from the
|
||
2026-08-25 incident is closed.
|
||
|
||
### Proactive renewal (`renew`)
|
||
|
||
Off until enabled. When on, once a day it re-registers with the
|
||
**existing** key if `last_verified_registration` is older than
|
||
`pia_renewal_after_days` (21) — or missing, which is why the baseline
|
||
must come from a *verified* run of the shared path, never from a file
|
||
mtime. Same transaction, same lock, same verification. A failed renewal
|
||
leaves the baseline untouched and enters the same bounded backoff (it
|
||
does not silently wait another 21 days); if the tunnel is already
|
||
unhealthy it defers to the recovery path. Renewal restarts `pia-wg`, so
|
||
qBittorrent/JDownloader connections drop for a few seconds at ~03:30 —
|
||
verify they reconnect during the drill. Routine key rotation is not
|
||
done: rotation happens only in recovery stage 3, on evidence.
|
||
|
||
### Alerts and monitoring
|
||
|
||
Events (not every 60 s): recovery started, recovered, recovery failed +
|
||
backoff (`warn`; `error` for auth failure), safety drift (`error`),
|
||
emergency rule removed, renewal failed, status unknown ×3, state corrupt,
|
||
unit crashed (via `OnFailure`). Delivery is `POST` to
|
||
`pia_alert_bridge_url` with the shared alert-bridge bearer key
|
||
(`{"source": "pia-gateway@minisforum", "message": ..., "level": ...}` —
|
||
schema read from `~/repo/home-service/alert-bridge`, which has **no**
|
||
per-caller tokens, so the cluster's key is what gets provisioned;
|
||
accepted tradeoff). 5 s timeout; failures are queued in state (bounded,
|
||
20) and flushed on the next pass; they never block recovery.
|
||
|
||
`metrics.prom` is node-exporter textfile format
|
||
(`pia_gateway_healthy`, `_tunnel_ok`, `_safety_ok`,
|
||
`_handshake_age_seconds`, `_consecutive_failures`, `_paused`,
|
||
`_recovery_in_backoff`, `_last_verified_registration_timestamp_seconds`,
|
||
`_check_timestamp_seconds`, `_emergency_close_active`). **Not yet
|
||
scraped**: `values/kube-prometheus-stack.yaml` has no textfile collector
|
||
and node-exporter runs as a pod, so wiring it needs a hostPath mount +
|
||
`--collector.textfile.directory` plus a `PrometheusRule` including a
|
||
stale-check rule (`time() - pia_gateway_check_timestamp_seconds > 300`)
|
||
so a stopped timer alerts even when the last value was healthy. Until
|
||
that exists, a dead controller is only visible as silence — check
|
||
`systemctl list-timers 'pia-gateway-*'` and `status.json`'s timestamp.
|
||
|
||
### Credentials on the host
|
||
|
||
`/etc/pia-gateway/credentials.env` holds the PIA account username and
|
||
password in plaintext, root-only. This is the explicit tradeoff the
|
||
design requires: recovery must work while the laptop is off, and PIA's
|
||
token is only valid for 24 h, so an unattended host needs a renewable
|
||
credential. The program reads the file at runtime and never places
|
||
values in argv, environment, logs, notifications, metrics or state; core
|
||
dumps are disabled (`LimitCORE=0`) and unexpected exceptions are logged
|
||
by class name only. **Rotation:** change `PIA_PASSWORD` in `.env`, run
|
||
the playbook (the file is re-rendered with `no_log`), then `pia-gateway
|
||
register` (or wait for the next cycle). **Revocation:** `-e
|
||
pia_remove_host_credentials=true` removes the file (recovery then fails
|
||
closed with `auth`/credentials errors and alerts); change the PIA
|
||
password; disabling `pia_auto_recovery_enabled` alone never deletes it.
|
||
If an on-host secret is unacceptable, the alternative is an always-on
|
||
external secret broker — not a 24-hour token.
|
||
|
||
### Residual risk (read before trusting it unattended)
|
||
|
||
- Safety drift is contained by the emergency DROP + unreachable route,
|
||
not repaired: the kill switch's own rule/jump can be deleted by
|
||
something outside this role (a manual `ufw reload` flushes `FORWARD`,
|
||
removing both the jump **and** the emergency rule). The controller
|
||
re-inserts the emergency rule on its next pass (≤ 60 s) and every boot
|
||
re-seeds the kill switch; that 60 s window after an out-of-band flush
|
||
is the residual leak risk, and it predates this work.
|
||
- Verification is handshake-based. It proves the tunnel is talking to
|
||
PIA, not that a VLAN 50 packet leaves via PIA; the end-to-end check is
|
||
the SOCKS-proxy/egress-IP step in the live drill.
|
||
- The alert-bridge key on minisforum is the cluster's shared key.
|
||
- `unknown` (a read command failing) is logged/alerted but never acted
|
||
on, by design.
|
||
|
||
### Rollout status (2026-09-23)
|
||
|
||
Steps 1-3 are done and verified live. Step 3 required two attempts —
|
||
the first caused a real (safely fail-closed) outage, root-caused and
|
||
fixed same-day; the second was verified against a full day of real
|
||
production conditions, not just a short window.
|
||
|
||
- **Step 2, attempt 1** failed: `pia-gateway register` correctly refused
|
||
with `safety invariants failing ... FORWARD jump to kill-switch chain
|
||
is not first` — a real, pre-existing drift (see the "FORWARD jump's
|
||
position is actively re-asserted" note above), not a false alarm from
|
||
this session's work, and not the "session dies" handshake issue
|
||
either. Root-caused and fixed in `pia-killswitch.sh.j2`.
|
||
- **Attempt 2**, after that fix, got past the kill-switch check and
|
||
failed differently: `addKey` rejected every candidate with
|
||
`kind=tls detail=certificate verification failed`. Root-caused and
|
||
fixed — see "PIA's CA certificate and why addKey uses curl, not
|
||
Python's `ssl`" above.
|
||
- **Attempt 3**, roughly an hour after attempt 1's fix with no reboot,
|
||
UFW reload, or script change in between, hit the *identical* FORWARD-
|
||
jump failure again — proving the drift recurs on a timescale of
|
||
minutes under this cluster's normal CronJob/pod churn, not just at
|
||
boot-scale events as originally assumed. Unblocked live with a manual
|
||
`sudo systemctl restart pia-killswitch.service`; `pia-killswitch
|
||
-reassert.timer` added as the durable fix.
|
||
- **Attempt 4** succeeded cleanly end to end: kill-switch check passed,
|
||
`addKey` accepted (`hk`/`Server-11669-10a`), and a **fresh handshake
|
||
was verified** (age 10s) — the first-ever successful shared-path
|
||
registration. `pia-gateway status` confirmed `healthy`/`safety_ok:
|
||
True`, `iptables -S FORWARD` confirmed the jump first, and
|
||
`pia-killswitch-reassert.timer` confirmed cycling correctly (7
|
||
successful runs, ~60s apart, in the journal). **Step 2 is done.**
|
||
- **Step 3** (`ansible-playbook ... -K -J`, no override — enables
|
||
`pia-gateway-recover.timer` for the first time) deployed cleanly
|
||
(`Run the shared PIA registration transaction` correctly skipped, a
|
||
verified baseline already existed) but caused a **real live outage**
|
||
within minutes: `pia-gateway-recover` and `pia-killswitch-reassert`,
|
||
two independent 60s timers, raced — kube-router's actual resync
|
||
cadence (~60-90s) turned out to be far more frequent than assumed, and
|
||
every momentary drift-then-self-correct was treated as a full safety
|
||
emergency, forcibly closing the route and resetting the 3-strikes
|
||
counter before it could accumulate. VLAN 50 was open for ~10s out of
|
||
every 60-90s cycle for several minutes, confirmed via Discord alerts
|
||
from `alert-bridge` (working end to end, first real trigger) and the
|
||
full `pia-gateway-recover`/`pia-killswitch-reassert` journals.
|
||
Mitigated live: `sudo systemctl stop pia-gateway-recover.timer` +
|
||
manual `sudo systemctl restart wg-quick@pia-wg`. Root-caused and fixed
|
||
same-day — see the "reassert timer alone wasn't the full fix" addendum
|
||
above (`pia_forward_bypass_chains`, content-verified live on every
|
||
check).
|
||
- **Step 3, re-attempt**, with the bypass fix deployed (`ansible-playbook
|
||
... -K -J`; config re-rendered with `forward_bypass_chains:
|
||
["KUBE-ROUTER-FORWARD"]`, `rewrite-config` correctly re-rendered
|
||
`pia-wg.conf` from the config-only change without a PIA call since a
|
||
baseline already existed, recovery timer re-enabled). Left running
|
||
unattended for a full **~25.5 hours** before re-checking, deliberately
|
||
— the original incident took several minutes and multiple cycles to
|
||
manifest, so a short recheck window would have proven nothing.
|
||
Confirmed via exact counts, not inference: `journalctl -u
|
||
pia-gateway-recover --since "2026-09-22 20:37:00"` showed **1528**
|
||
`class=healthy` passes (matches the expected ~1530 at 60s cadence —
|
||
the timer ran continuously the whole window), **0** `SAFETY DRIFT`
|
||
lines, **0** `class=safety_drift` classifications. `pia-gateway
|
||
status` at recheck time: `healthy`, `safety_ok: True`,
|
||
`emergency_close_active: False`, `consecutive_failures: 0`,
|
||
`last recovery: 1d ago restart` — the last restart was the one from
|
||
*before* this fix; none has been needed since. **Step 3 is done.**
|
||
kube-router did keep resyncing during this window (that's the whole
|
||
premise of the fix); the bypass tolerance handled it silently every
|
||
time, with no observable effect on the tunnel.
|
||
|
||
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
|
||
— the original `head -3` suggestion is what hid the first issue; the
|
||
drift was 4 rules in, not at the top. **Use `head -2`, not `head -1`**,
|
||
for any quick re-check — `-S FORWARD`'s first line is always the `-P`
|
||
policy line, never a real rule (this cost a round-trip during the
|
||
investigation above).
|
||
|
||
1. Baseline on minisforum: `wg show pia-wg`, `ip rule`, `ip route show
|
||
table pia`, `iptables -S FORWARD`, `iptables -S PIA-VLAN50`,
|
||
`journalctl -u pia-gateway-healthcheck -n 5`, `curl --socks5-hostname
|
||
10.10.40.20:1080 https://ifconfig.me` from the LAN, `wg show wg0`,
|
||
`kubectl get nodes`. Confirm console access.
|
||
2. Deploy with timers off: `ansible-playbook ... -K -J -e
|
||
pia_auto_recovery_enabled=false`. Check `ls -l /etc/pia-gateway
|
||
/var/lib/pia-gateway`, `pia-gateway status`, `systemctl list-timers
|
||
'pia-gateway-*'` (recover/renew inactive). Because the host has no
|
||
verified baseline yet, this run **registers through the shared path**
|
||
(same-key, explicit restart + handshake verification) — watch the
|
||
task output; if it fails, `-e pia_registration_backend=legacy`
|
||
restores the previous behaviour. If it fails specifically with
|
||
`FORWARD jump to kill-switch chain is not first`, that's the drift
|
||
above — the fixed template should correct it via the normal "Restart
|
||
pia-killswitch" handler on the same run (the template content
|
||
changing is enough to trigger it); re-running the identical command
|
||
should then get past registration. Confirm with `iptables -S FORWARD`
|
||
afterwards that the jump is genuinely first, not just that the retry
|
||
succeeded.
|
||
3. Enable: `ansible-playbook ... -K -J` (recovery on; also redeploys
|
||
`pia_forward_bypass_chains` into `config.json` if not already applied
|
||
— confirm with `sudo pia-gateway --config /etc/pia-gateway/config.json
|
||
status` or just `cat /etc/pia-gateway/config.json`). **Watch longer
|
||
than one pass** — the live incident here took 2-3 cycles (several
|
||
minutes) to manifest, not the first check. `journalctl -u
|
||
pia-gateway-recover -f` for at least 5-10 minutes; expect `class=
|
||
healthy` throughout, with **zero** `SAFETY DRIFT` lines, even while
|
||
`pia-killswitch-reassert` keeps cycling in parallel
|
||
(`journalctl -u pia-killswitch-reassert -f` alongside it). If a
|
||
`bypass chain KUBE-ROUTER-FORWARD content no longer verified safe`
|
||
reason ever appears, treat it as real — it means kube-router's rules
|
||
changed to include something that could actually match VLAN 50
|
||
traffic, not a false positive to relax further.
|
||
4. Drill the silent death. First get the **live WireGuard UDP endpoint**
|
||
from the interface itself — **not** port 1337, which is only the
|
||
`addKey` registration API port and has nothing to do with the tunnel's
|
||
own data-plane port (PIA WireGuard servers commonly use a different
|
||
port per server):
|
||
|
||
```bash
|
||
sudo wg show pia-wg endpoints # <peer-pubkey>\t<ip>:<port> — this ip:port is the block target
|
||
```
|
||
|
||
Then, on minisforum, leave `pia-wg` administratively up and drop only
|
||
that path:
|
||
|
||
```bash
|
||
sudo iptables -I OUTPUT -d <endpoint ip> -p udp --dport <endpoint port> -j DROP
|
||
```
|
||
|
||
Expect: 3 stale passes → restart → (still blocked) registration to a
|
||
different server → recovery; record times from the journal. To test
|
||
backoff, block every candidate server before recovery can reach it —
|
||
since each candidate's actual WireGuard port isn't known until its own
|
||
`addKey` response arrives (only the registration port, 1337, is fixed
|
||
and shared), block by IP alone rather than guessing a port:
|
||
|
||
```bash
|
||
sudo iptables -I OUTPUT -d <candidate ip> -p udp -j DROP # repeat per candidate IP, or -e pia_region=hk to narrow the set first
|
||
```
|
||
|
||
Remove the rule(s) and confirm autonomous recovery. `iptables -D`
|
||
every injected rule afterwards — check `iptables -S OUTPUT` is back to
|
||
its pre-drill state, not just that the tunnel recovered.
|
||
5. Exercise: a failed first server (block one IP only), invalid
|
||
credentials (temporarily wrong `PIA_PASSWORD` → expect `auth_failure`,
|
||
no rotation, error alert), alert-bridge down, `pia-gateway register`
|
||
while a pass runs (expect it to wait), reboot with a pending backoff,
|
||
`pause`/`resume`.
|
||
6. During every transition, capture on `enp1s0` with an explicit
|
||
**egress-only** direction filter:
|
||
|
||
```bash
|
||
sudo tcpdump -ni enp1s0 -Q out src net 10.10.50.0/24
|
||
```
|
||
|
||
`-Q out` matters: VLAN 50 traffic **arriving** at minisforum on
|
||
`enp1s0` (from the Flint router, for policy routing into table `pia`)
|
||
is normal and expected — a plain `src net 10.10.50.0/24` filter with
|
||
no direction would match that inbound traffic too and could hide a
|
||
real leak in the noise, or be misread as one. Only a hit in the `out`
|
||
direction — a VLAN 50-sourced packet actually being transmitted out
|
||
`enp1s0` toward the LAN/Internet instead of via `pia-wg` — is a leak,
|
||
and this must stay completely silent through every failure and
|
||
recovery transition. Also capture on `pia-wg` itself to confirm
|
||
traffic **is** flowing there while open. Verify a request through
|
||
`10.10.40.20:1080` and qBittorrent/JDownloader afterwards; check IPv6
|
||
separately; confirm `wg0`, the host default route, Technitium and k3s.
|
||
7. Only then `-e pia_renewal_enabled=true`; `pia-gateway renew
|
||
--force-due`; confirm `last_verified_registration.epoch` advances only
|
||
on verification. Wire the Prometheus stale-check before calling the
|
||
feature done. Record actual times and captures in the incident file.
|
||
|
||
## Rollback
|
||
|
||
**Partial rollback (keep the gateway, drop only the automation):**
|
||
`ansible-playbook ... -e pia_auto_recovery_enabled=false` stops and
|
||
disables both timers and leaves the program, credentials, health check,
|
||
kill switch and `pia-wg` in place. Add `-e pia_remove_host_credentials=true`
|
||
to also delete the on-host PIA credentials; nothing else deletes them.
|
||
|
||
No tag-based full rollback is defined — this role doesn't have an
|
||
"absent" mode. Roll back manually on minisforum (reverses this role without
|
||
touching `wg0`, k3s, or the host default route):
|
||
|
||
```bash
|
||
sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown
|
||
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-killswitch-reassert.timer pia-gateway-healthcheck.timer
|
||
sudo rm -f /etc/systemd/system/pia-killswitch.service \
|
||
/etc/systemd/system/pia-killswitch-reassert.service \
|
||
/etc/systemd/system/pia-killswitch-reassert.timer \
|
||
/etc/systemd/system/pia-gateway-healthcheck.service \
|
||
/etc/systemd/system/pia-gateway-healthcheck.timer \
|
||
/etc/systemd/system/pia-gateway-recover.service \
|
||
/etc/systemd/system/pia-gateway-recover.timer \
|
||
/etc/systemd/system/pia-gateway-renew.service \
|
||
/etc/systemd/system/pia-gateway-renew.timer \
|
||
/etc/systemd/system/pia-gateway-onfailure.service
|
||
sudo systemctl daemon-reload
|
||
sudo iptables -D FORWARD -s 10.10.50.0/24 -m comment --comment pia-gateway-emergency -j DROP 2>/dev/null || true
|
||
sudo rm -rf /etc/pia-gateway /var/lib/pia-gateway /usr/local/sbin/pia-gateway /run/lock/pia-gateway.lock # includes the on-host PIA credentials
|
||
sudo ip rule del from 10.10.50.0/24 table pia priority 100
|
||
sudo ip route flush table pia
|
||
sudo sed -i '/^[0-9]\+\s\+pia$/d' /etc/iproute2/rt_tables
|
||
sudo iptables -D FORWARD -s 10.10.50.0/24 -j PIA-VLAN50
|
||
sudo iptables -F PIA-VLAN50
|
||
sudo iptables -X PIA-VLAN50
|
||
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp
|
||
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp
|
||
sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.conf.bak-* /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.bak-* /etc/wireguard/pia-wg.key.new* /etc/wireguard/pia-ca.crt
|
||
```
|
||
|
||
Inspection commands to confirm rollback actually took (read-only):
|
||
|
||
```bash
|
||
sudo iptables -S FORWARD | grep PIA-VLAN50 # expect: no output
|
||
sudo iptables -L PIA-VLAN50 # expect: "iptables: No chain/target/match by that name"
|
||
sudo ufw status verbose | grep 10.10.50 # expect: no output
|
||
ip rule show | grep pia # expect: no output
|
||
ip route show table pia # expect: empty/error (table gone)
|
||
```
|
||
|
||
Does not remove `pia-credentials`/`pia-credentials-sealed.yaml` (the K8s
|
||
Secret used by the Gluetun sidecars) — that's a separate, unrelated
|
||
secret and rollback path. Does not touch `ansible/roles/common`'s own
|
||
UFW rules (Flannel pod-to-pod, pod-to-Technitium) — this role never
|
||
modified those.
|