homelab/ansible/roles/pia-gateway/README.md

# pia-gateway

Makes `minisforum` a PIA WireGuard egress gateway for VPN VLAN 50
(`10.10.50.0/24`), without changing its own default route. Implements
Phase 2 of `~/repo/homelab/plan.md`.

## Before the first real run

1. **The PIA API calls in `tasks/register.yaml` have gone through three
   rounds of correction against real live testing (2026-08-24):**
   - `serverlist.piaservers.net/vpninfo/servers/v6`: works. Region values
     everywhere in this role are the API's **id** field (e.g. `japan`,
     `hk`, `taiwan`), never a display name ("JP Tokyo", "Hong Kong",
     "Taiwan" — those are Gluetun's labels, not something this role or
     PIA's API accepts).
   - Token acquisition originally guessed a regional-meta-server
     endpoint (`/authv3/generateToken`) that turned out not to be the
     real flow at all — it was rebuilt from PIA's actual
     `pia-foss/manual-connections` `get_token.sh` (fetched and read in
     full): a single fixed `POST
     https://www.privateinternetaccess.com/api/client/v2/token`,
     multipart `username`/`password`, **normal system CA validation**
     (no `--cacert`, no `validate_certs: false`), independent of region
     entirely — only WireGuard server *selection* is regional. Not yet
     exercised with real credentials end to end.
   - `{wg}:1337/addKey` (WireGuard key registration) — confirmed against
     `connect_to_wireguard_with_token.sh` (fetched and read in full):
     this one genuinely does need PIA's own CA bundle
     (`files/pia-ca.crt`, `--cacert`) and `--connect-to`. Originally
     failed over only across servers *within* a single fixed region
     (`pia_region: japan`) — in practice all three JP Tokyo servers
     failed the same way, one after another, so failover now also
     crosses regions: every server in Hong Kong, then every server in
     Taiwan, then every server in JP Tokyo (`pia_region_candidates` in
     `defaults/main.yaml`), stopping at the first `status: OK`. Token
     passed via stdin rather than argv. See `tasks/addkey-attempt.yaml`
     and "Region fallback" below.
2. Make sure the repo-root `.env` has `PIA_USER`/`PIA_PASSWORD` set (same
   keys `manifests/media/pia-secret.sh` already reads) — `ansible/
   host_vars/minisforum.yaml` reads them directly from there via a
   `lookup('ansible.builtin.file', ...)` + regex, deliberately not
   ansible-vault, so this needs no vault password at all. The lookup runs
   on the control node, so `.env` never touches minisforum and is never
   committed (already gitignored).
3. Confirm console/recovery access to `minisforum` (IPMI/physical/other
   out-of-band path) before applying — this role changes host routing and
   firewall policy. If SSH becomes unreachable, use that path.
4. `ufw status verbose` and `wg show` (existing `wg0`) were not inspected
   live before writing this role (no passwordless sudo in discovery) —
   spot-check that `wg0` (the separate home-VPN role) is unaffected after
   applying.

## Region fallback

Registration tries every WireGuard server in every configured region, in
order, stopping at the first `status: OK`:

```text
hk (Hong Kong):
  wg server 1
  wg server 2
taiwan (Taiwan):
  wg server 1
  wg server 2
japan (JP Tokyo):
  wg server 1
  wg server 2
  wg server 3
```

The actual number and order of servers within each region always comes
from the live serverlist response — never hardcoded here.

**Automatic fallback** (default — tries `pia_region_candidates` in
order):

```bash
ansible-playbook -i ansible/inventory.yaml \
  ansible/playbooks/pia-gateway.yaml -K -J
```

**Forced single region**, for troubleshooting one region in isolation —
bypasses `pia_region_candidates` entirely, does not fall back to the
others:

```bash
ansible-playbook -i ansible/inventory.yaml \
  ansible/playbooks/pia-gateway.yaml -K -J \
  -e pia_region=hk
```

Both `pia_region` and every entry in `pia_region_candidates` are PIA API
**ids** — `hk`, `taiwan`, `japan` — not the display names Gluetun's
`SERVER_REGIONS` config uses ("Hong Kong", "Taiwan", "JP Tokyo"). Passing
a display name here matches nothing and fails the "region exists"
assertion before any network call is even made.

Which region/server actually ended up registered is recorded as
`pia_region_used`/`pia_region_name_used`/`pia_wg_server_used`, written
into `pia-wg.conf`'s own `[Peer]` comment, and surfaced by the health
check (`INFO configured-peer: ...`) — it is whichever one answered first,
not necessarily `pia_region_candidates[0]`.

If every region fails: `journalctl` (or the play's own output) has one
`debug` line per failed attempt, each showing region name+id, server
hostname+IP, curl exit code, a classified reason (connection timeout /
TLS-SSL failure / HTTP failure / empty response / malformed JSON /
parsed-but-not-OK status, never a generic catch-all), and sanitized
stderr — never the token, credentials, the full curl invocation, request
stdin, or a complete response body. If Hong Kong and Taiwan fail exactly
the same way Tokyo did, that's a strong signal the problem is shared
across all three (the request shape, the account credentials, or
minisforum's own network path) rather than one region being down — see
`tasks/register.yaml`'s header comment.

**Worked example (2026-08-24):** exactly that happened — all 7 servers
across all 3 regions came back `HTTP failure (non-2xx response)`, curl
exit 22, `The requested URL returned error: 401`, uniformly. The cause
was in `addkey-attempt.yaml`'s own curl task, not PIA or any region:
`ansible.builtin.command`'s `stdin` argument appends a trailing newline
by default (`stdin_add_newline` defaults to `true`), and
`--data-urlencode pt@-` does not strip it — every server was receiving
`"<real token>\n"` (URL-encoded, so a trailing `%0A`) as `pt`, a
different and invalid value, and correctly rejecting it. Fixed with
`stdin_add_newline: false` on that task. `connect_to_wireguard_with_token
.sh` never hits this because it passes the token as a literal shell
variable, not via stdin — the stdin delivery here is this repo's own
addition (to keep the token out of `ps`), so it needed the flag PIA's
reference script never had to think about. Left as a worked example
because "every candidate failed identically" pointing at one shared bug
in *this* code, not PIA, is exactly the diagnostic story this section
promises — and it happened to be true the first time it was tested for
real.

## What it does

- Registers minisforum as a PIA WireGuard peer — see "Region fallback"
  above — and writes `/etc/wireguard/pia-wg.conf` (`Table = off` — this
  role owns all routing for the interface, not wg-quick).
- Installs `pia-killswitch.service`, ordered `Before=
  wg-quick@pia-wg.service`, that seeds a closed state at every boot (and
  after any UFW reload — see handlers/main.yaml): the
  `from 10.10.50.0/24 lookup pia` rule, a terminal `unreachable default`
  route in the `pia` table (only when `pia-wg` isn't already up — see
  the script's own comment for why), and a dedicated `PIA-VLAN50`
  iptables chain jumped into by a single rule at the very top of
  `FORWARD`, matching only source `10.10.50.0/24`, ending in an
  unconditional rate-limited log+drop. This is deliberately **not** a
  global `FORWARD` default-policy change — minisforum runs Flannel,
  which needs its own broad `FORWARD` ACCEPTs for pod traffic (source
  `10.42.0.0/16`, disjoint from VLAN 50) — so the kill switch is scoped
  to a chain Flannel/k3s traffic can never enter, rather than risking a
  chain-wide policy that was never actually proven safe against it.
  Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW
  **input** allow for 53/tcp+udp — it's locally-terminated traffic, not
  routed — see `tasks/firewall.yaml`.
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
  top of that closed baseline: default route via `pia-wg` in the `pia`
  table, ACCEPT + established/related return inserted into the
  `PIA-VLAN50` chain (not `FORWARD` directly), and source-NAT/MASQUERADE
  scoped to `10.10.50.0/24` on `pia-wg` only.
- Installs an observability-only health check (`pia-gateway-healthcheck
  .timer`, every `pia_healthcheck_interval_sec`) that logs interface,
  handshake age, rule/route, and firewall-policy state to the journal. It
  never remediates — see the script's header comment for why, and for what
  its "route-decision" check does and does not prove.

## What it deliberately does not do

- Does not touch minisforum's own default route (asserted at the start of
  every run — `tasks/assert-baseline.yaml` fails loudly if that's already
  wrong).
- Does not enable IPv6 forwarding or any IPv6 handling for VLAN 50.
- Does not enable MSS clamping (`pia_mss_clamp_enabled: false` by
  default) — flip only after Phase 5 canary MTU testing shows it's
  actually needed.
- Does not modify `ansible/roles/wireguard` (the separate `wg0` home-VPN
  server role) or its interface.

## Rollback

No tag-based rollback is defined — this role doesn't have an "absent"
mode. Roll back manually on minisforum (reverses this role without
touching `wg0`, k3s, or the host default route):

```bash
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \
           /etc/systemd/system/pia-gateway-healthcheck.service \
           /etc/systemd/system/pia-gateway-healthcheck.timer
sudo systemctl daemon-reload
sudo ip rule del from 10.10.50.0/24 table pia priority 100
sudo ip route flush table pia
sudo sed -i '/^[0-9]\+\s\+pia$/d' /etc/iproute2/rt_tables
sudo iptables -D FORWARD -s 10.10.50.0/24 -j PIA-VLAN50
sudo iptables -F PIA-VLAN50
sudo iptables -X PIA-VLAN50
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp
sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.new /etc/wireguard/pia-ca.crt
```

Inspection commands to confirm rollback actually took (read-only):

```bash
sudo iptables -S FORWARD | grep PIA-VLAN50   # expect: no output
sudo iptables -L PIA-VLAN50                  # expect: "iptables: No chain/target/match by that name"
sudo ufw status verbose | grep 10.10.50      # expect: no output
ip rule show | grep pia                      # expect: no output
ip route show table pia                      # expect: empty/error (table gone)
```

Does not remove `pia-credentials`/`pia-credentials-sealed.yaml` (the K8s
Secret used by the Gluetun sidecars) — that's a separate, unrelated
secret and rollback path. Does not touch `ansible/roles/common`'s own
UFW rules (Flannel pod-to-pod, pod-to-Technitium) — this role never
modified those.