# pia-gateway

Makes `minisforum` a PIA WireGuard egress gateway for VPN VLAN 50
(`10.10.50.0/24`), without changing its own default route. Implements
Phase 2 of `~/repo/homelab/plan.md`.

## Before the first real run

1. **The PIA API calls in `tasks/register.yaml` have gone through three
   rounds of correction against real live testing (2026-08-24):**
   - `serverlist.piaservers.net/vpninfo/servers/v6`: works. Region values
     everywhere in this role are the API's **id** field (e.g. `japan`,
     `hk`, `taiwan`), never a display name ("JP Tokyo", "Hong Kong",
     "Taiwan" — those are Gluetun's labels, not something this role or
     PIA's API accepts).
   - Token acquisition originally guessed a regional-meta-server
     endpoint (`/authv3/generateToken`) that turned out not to be the
     real flow at all — it was rebuilt from PIA's actual
     `pia-foss/manual-connections` `get_token.sh` (fetched and read in
     full): a single fixed `POST
     https://www.privateinternetaccess.com/api/client/v2/token`,
     multipart `username`/`password`, **normal system CA validation**
     (no `--cacert`, no `validate_certs: false`), independent of region
     entirely — only WireGuard server *selection* is regional. Not yet
     exercised with real credentials end to end.
   - `{wg}:1337/addKey` (WireGuard key registration) — confirmed against
     `connect_to_wireguard_with_token.sh` (fetched and read in full):
     this one genuinely does need PIA's own CA bundle
     (`files/pia-ca.crt`, `--cacert`) and `--connect-to`. Originally
     failed over only across servers *within* a single fixed region
     (`pia_region: japan`) — in practice all three JP Tokyo servers
     failed the same way, one after another, so failover now also
     crosses regions: every server in Hong Kong, then every server in
     Taiwan, then every server in JP Tokyo (`pia_region_candidates` in
     `defaults/main.yaml`), stopping at the first `status: OK`. Token
     passed via stdin rather than argv. See `tasks/addkey-attempt.yaml`
     and "Region fallback" below.
2. Make sure the repo-root `.env` has `PIA_USER`/`PIA_PASSWORD` set (same
   keys `manifests/media/pia-secret.sh` already reads) — `ansible/
   host_vars/minisforum.yaml` reads them directly from there via a
   `lookup('ansible.builtin.file', ...)` + regex, deliberately not
   ansible-vault, so this needs no vault password at all. The lookup runs
   on the control node, so `.env` never touches minisforum and is never
   committed (already gitignored).
3. Confirm console/recovery access to `minisforum` (IPMI/physical/other
   out-of-band path) before applying — this role changes host routing and
   firewall policy. If SSH becomes unreachable, use that path.
4. `ufw status verbose` and `wg show` (existing `wg0`) were not inspected
   live before writing this role (no passwordless sudo in discovery) —
   spot-check that `wg0` (the separate home-VPN role) is unaffected after
   applying.

## Region fallback

Registration tries every WireGuard server in every configured region, in
order, stopping at the first `status: OK`:

```text
hk (Hong Kong):
  wg server 1
  wg server 2
taiwan (Taiwan):
  wg server 1
  wg server 2
japan (JP Tokyo):
  wg server 1
  wg server 2
  wg server 3
```

The actual number and order of servers within each region always comes
from the live serverlist response — never hardcoded here.

**Automatic fallback** (default — tries `pia_region_candidates` in
order):

```bash
ansible-playbook -i ansible/inventory.yaml \
  ansible/playbooks/pia-gateway.yaml -K -J
```

**Forced single region**, for troubleshooting one region in isolation —
bypasses `pia_region_candidates` entirely, does not fall back to the
others:

```bash
ansible-playbook -i ansible/inventory.yaml \
  ansible/playbooks/pia-gateway.yaml -K -J \
  -e pia_region=hk
```

Both `pia_region` and every entry in `pia_region_candidates` are PIA API
**ids** — `hk`, `taiwan`, `japan` — not the display names Gluetun's
`SERVER_REGIONS` config uses ("Hong Kong", "Taiwan", "JP Tokyo"). Passing
a display name here matches nothing and fails the "region exists"
assertion before any network call is even made.

Which region/server actually ended up registered is recorded as
`pia_region_used`/`pia_region_name_used`/`pia_wg_server_used`, written
into `pia-wg.conf`'s own `[Peer]` comment, and surfaced by the health
check (`INFO configured-peer: ...`) — it is whichever one answered first,
not necessarily `pia_region_candidates[0]`.

If every region fails: `journalctl` (or the play's own output) has one
`debug` line per failed attempt, each showing region name+id, server
hostname+IP, curl exit code, a classified reason (connection timeout /
TLS-SSL failure / HTTP failure / empty response / malformed JSON /
parsed-but-not-OK status, never a generic catch-all), and sanitized
stderr — never the token, credentials, the full curl invocation, request
stdin, or a complete response body. If Hong Kong and Taiwan fail exactly
the same way Tokyo did, that's a strong signal the problem is shared
across all three (the request shape, the account credentials, or
minisforum's own network path) rather than one region being down — see
`tasks/register.yaml`'s header comment.

**Worked example (2026-08-24):** exactly that happened — all 7 servers
across all 3 regions came back `HTTP failure (non-2xx response)`, curl
exit 22, `The requested URL returned error: 401`, uniformly. The cause
was in `addkey-attempt.yaml`'s own curl task, not PIA or any region:
`ansible.builtin.command`'s `stdin` argument appends a trailing newline
by default (`stdin_add_newline` defaults to `true`), and
`--data-urlencode pt@-` does not strip it — every server was receiving
`"<real token>\n"` (URL-encoded, so a trailing `%0A`) as `pt`, a
different and invalid value, and correctly rejecting it. Fixed with
`stdin_add_newline: false` on that task. `connect_to_wireguard_with_token
.sh` never hits this because it passes the token as a literal shell
variable, not via stdin — the stdin delivery here is this repo's own
addition (to keep the token out of `ps`), so it needed the flag PIA's
reference script never had to think about. Left as a worked example
because "every candidate failed identically" pointing at one shared bug
in *this* code, not PIA, is exactly the diagnostic story this section
promises — and it happened to be true the first time it was tested for
real.

## What it does

- Registers minisforum as a PIA WireGuard peer — see "Region fallback"
  above — and writes `/etc/wireguard/pia-wg.conf` (`Table = off` — this
  role owns all routing for the interface, not wg-quick).
- Installs `pia-killswitch.service`, ordered `Before=
  wg-quick@pia-wg.service`, that seeds a closed state at every boot (and
  after any UFW reload — see handlers/main.yaml): the
  `from 10.10.50.0/24 lookup pia` rule, a terminal `unreachable default`
  route in the `pia` table (only when `pia-wg` isn't already up — see
  the script's own comment for why), and a dedicated `PIA-VLAN50`
  iptables chain jumped into by a single rule at the very top of
  `FORWARD`, matching only source `10.10.50.0/24`, ending in an
  unconditional rate-limited log+drop. This is deliberately **not** a
  global `FORWARD` default-policy change — minisforum runs Flannel,
  which needs its own broad `FORWARD` ACCEPTs for pod traffic (source
  `10.42.0.0/16`, disjoint from VLAN 50) — so the kill switch is scoped
  to a chain Flannel/k3s traffic can never enter, rather than risking a
  chain-wide policy that was never actually proven safe against it.
  Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW
  **input** allow for 53/tcp+udp — it's locally-terminated traffic, not
  routed — see `tasks/firewall.yaml`.
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
  top of that closed baseline: default route via `pia-wg` in the `pia`
  table, ACCEPT + established/related return inserted into the
  `PIA-VLAN50` chain (not `FORWARD` directly), and source-NAT/MASQUERADE
  scoped to `10.10.50.0/24` on `pia-wg` only.
- Clamps TCP MSS on the `pia-wg` forward path (`pia_mss_clamp_enabled:
  true`, `iptables -t mangle ... TCPMSS --clamp-mss-to-pmtu`, scoped to
  SYN packets sourced from `10.10.50.0/24` outbound on `pia-wg` only, in
  `PostUp`/`PreDown` alongside the rules above). Confirmed needed live
  (2026-08-25), not enabled speculatively: a real download hung with a
  TLS read timeout on the larger handshake response while small
  requests worked fine, and a direct `ping -M do -s 1450` test from a
  VLAN 50 pod confirmed the real path MTU is `pia-wg`'s `1420` (the ICMP
  "Frag needed" reply arrives correctly from minisforum itself — our own
  side relays PMTU discovery fine, so the black hole is further out, on
  PIA's network or the remote server's own path, where clamping the MSS
  up front avoids needing that ICMP round-trip at all).
- `pia-wg.conf`'s `PreDown` lines are deliberately tolerant of the rule
  they're removing not existing (`iptables -D ... 2>/dev/null || true`)
  — enabling MSS clamping live (above) exposed why this matters:
  `systemctl restart wg-quick@pia-wg` tears down the *currently running*
  interface using whatever `PreDown` lines are on disk *right now* — if
  Ansible already rewrote the config with a new/changed rule before the
  restart handler fires, the live interface (brought up under the *old*
  rules) won't have whatever the new `PreDown` line is trying to delete.
  `wg-quick`'s own `execute_hooks()` aborts the entire down/up sequence
  on the first failing hook (confirmed against its real source, not
  assumed), so one non-idempotent `-D` used to turn any future
  `PostUp`/`PreDown` content change into a broken restart *and* an
  orphaned interface (the never-reached built-in `ip link delete` step
  left `pia-wg` existing but unconfigured, which then made the
  following `wg-quick up` fail too with `` `pia-wg' already exists``) —
  requiring manual recovery (`ip link delete dev pia-wg` before a fresh
  `systemctl start`). Every `-D` line now tolerates this by design, so a
  config change that alters `PostUp`/`PreDown` content can never break
  a restart this way again — confirmed live, 2026-08-27.
- Installs an observability-only health check (`pia-gateway-healthcheck
  .timer`, every `pia_healthcheck_interval_sec`) that logs interface,
  handshake age, rule/route, and firewall-policy state to the journal. It
  never remediates — see the script's header comment for why, and for what
  its "route-decision" check does and does not prove.

## What it deliberately does not do

- Does not touch minisforum's own default route (asserted at the start of
  every run — `tasks/assert-baseline.yaml` fails loudly if that's already
  wrong).
- Does not enable IPv6 forwarding or any IPv6 handling for VLAN 50.
- Does not modify `ansible/roles/wireguard` (the separate `wg0` home-VPN
  server role) or its interface.

## Rollback

No tag-based rollback is defined — this role doesn't have an "absent"
mode. Roll back manually on minisforum (reverses this role without
touching `wg0`, k3s, or the host default route):

```bash
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \
           /etc/systemd/system/pia-gateway-healthcheck.service \
           /etc/systemd/system/pia-gateway-healthcheck.timer
sudo systemctl daemon-reload
sudo ip rule del from 10.10.50.0/24 table pia priority 100
sudo ip route flush table pia
sudo sed -i '/^[0-9]\+\s\+pia$/d' /etc/iproute2/rt_tables
sudo iptables -D FORWARD -s 10.10.50.0/24 -j PIA-VLAN50
sudo iptables -F PIA-VLAN50
sudo iptables -X PIA-VLAN50
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp
sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.new /etc/wireguard/pia-ca.crt
```

Inspection commands to confirm rollback actually took (read-only):

```bash
sudo iptables -S FORWARD | grep PIA-VLAN50   # expect: no output
sudo iptables -L PIA-VLAN50                  # expect: "iptables: No chain/target/match by that name"
sudo ufw status verbose | grep 10.10.50      # expect: no output
ip rule show | grep pia                      # expect: no output
ip route show table pia                      # expect: empty/error (table gone)
```

Does not remove `pia-credentials`/`pia-credentials-sealed.yaml` (the K8s
Secret used by the Gluetun sidecars) — that's a separate, unrelated
secret and rollback path. Does not touch `ansible/roles/common`'s own
UFW rules (Flannel pod-to-pod, pod-to-Technitium) — this role never
modified those.