homelab/ansible/roles/pia-gateway/README.md

pia-gateway

Makes minisforum a PIA WireGuard egress gateway for VPN VLAN 50 (10.10.50.0/24), without changing its own default route. Implements Phase 2 of ~/repo/homelab/plan.md.

Before the first real run

1. The PIA API calls in tasks/register.yaml have gone through three rounds of correction against real live testing (2026-08-24):
   • serverlist.piaservers.net/vpninfo/servers/v6: works. Region values everywhere in this role are the API's id field (e.g. japan, hk, taiwan), never a display name ("JP Tokyo", "Hong Kong", "Taiwan" — those are Gluetun's labels, not something this role or PIA's API accepts).
   • Token acquisition originally guessed a regional-meta-server endpoint (/authv3/generateToken) that turned out not to be the real flow at all — it was rebuilt from PIA's actual pia-foss/manual-connections get_token.sh (fetched and read in full): a single fixed POST https://www.privateinternetaccess.com/api/client/v2/token, multipart username/password, normal system CA validation (no --cacert, no validate_certs: false), independent of region entirely — only WireGuard server selection is regional. Not yet exercised with real credentials end to end.
   • {wg}:1337/addKey (WireGuard key registration) — confirmed against connect_to_wireguard_with_token.sh (fetched and read in full): this one genuinely does need PIA's own CA bundle (files/pia-ca.crt, --cacert) and --connect-to. Originally failed over only across servers within a single fixed region (pia_region: japan) — in practice all three JP Tokyo servers failed the same way, one after another, so failover now also crosses regions: every server in Hong Kong, then every server in Taiwan, then every server in JP Tokyo (pia_region_candidates in defaults/main.yaml), stopping at the first status: OK. Token passed via stdin rather than argv. See tasks/addkey-attempt.yaml and "Region fallback" below.
2. Make sure the repo-root .env has PIA_USER/PIA_PASSWORD set (same keys manifests/media/pia-secret.sh already reads) — ansible/ host_vars/minisforum.yaml reads them directly from there via a lookup('ansible.builtin.file', ...) + regex, deliberately not ansible-vault, so this needs no vault password at all. The lookup runs on the control node, so .env never touches minisforum and is never committed (already gitignored).
3. Confirm console/recovery access to minisforum (IPMI/physical/other out-of-band path) before applying — this role changes host routing and firewall policy. If SSH becomes unreachable, use that path.
4. ufw status verbose and wg show (existing wg0) were not inspected live before writing this role (no passwordless sudo in discovery) — spot-check that wg0 (the separate home-VPN role) is unaffected after applying.

Region fallback

Registration tries every WireGuard server in every configured region, in order, stopping at the first status: OK:

hk (Hong Kong):
  wg server 1
  wg server 2
taiwan (Taiwan):
  wg server 1
  wg server 2
japan (JP Tokyo):
  wg server 1
  wg server 2
  wg server 3

The actual number and order of servers within each region always comes from the live serverlist response — never hardcoded here.

Automatic fallback (default — tries pia_region_candidates in order):

ansible-playbook -i ansible/inventory.yaml \
  ansible/playbooks/pia-gateway.yaml -K -J

Forced single region, for troubleshooting one region in isolation — bypasses pia_region_candidates entirely, does not fall back to the others:

ansible-playbook -i ansible/inventory.yaml \
  ansible/playbooks/pia-gateway.yaml -K -J \
  -e pia_region=hk

Both pia_region and every entry in pia_region_candidates are PIA API ids — hk, taiwan, japan — not the display names Gluetun's SERVER_REGIONS config uses ("Hong Kong", "Taiwan", "JP Tokyo"). Passing a display name here matches nothing and fails the "region exists" assertion before any network call is even made.

Which region/server actually ended up registered is recorded as pia_region_used/pia_region_name_used/pia_wg_server_used, written into pia-wg.conf's own [Peer] comment, and surfaced by the health check (INFO configured-peer: ...) — it is whichever one answered first, not necessarily pia_region_candidates[0].

If every region fails: journalctl (or the play's own output) has one debug line per failed attempt, each showing region name+id, server hostname+IP, curl exit code, a classified reason (connection timeout / TLS-SSL failure / HTTP failure / empty response / malformed JSON / parsed-but-not-OK status, never a generic catch-all), and sanitized stderr — never the token, credentials, the full curl invocation, request stdin, or a complete response body. If Hong Kong and Taiwan fail exactly the same way Tokyo did, that's a strong signal the problem is shared across all three (the request shape, the account credentials, or minisforum's own network path) rather than one region being down — see tasks/register.yaml's header comment.

Worked example (2026-08-24): exactly that happened — all 7 servers across all 3 regions came back HTTP failure (non-2xx response), curl exit 22, The requested URL returned error: 401, uniformly. The cause was in addkey-attempt.yaml's own curl task, not PIA or any region: ansible.builtin.command's stdin argument appends a trailing newline by default (stdin_add_newline defaults to true), and --data-urlencode pt@- does not strip it — every server was receiving "<real token>\n" (URL-encoded, so a trailing %0A) as pt, a different and invalid value, and correctly rejecting it. Fixed with stdin_add_newline: false on that task. connect_to_wireguard_with_token .sh never hits this because it passes the token as a literal shell variable, not via stdin — the stdin delivery here is this repo's own addition (to keep the token out of ps), so it needed the flag PIA's reference script never had to think about. Left as a worked example because "every candidate failed identically" pointing at one shared bug in this code, not PIA, is exactly the diagnostic story this section promises — and it happened to be true the first time it was tested for real.

What it does

• Registers minisforum as a PIA WireGuard peer — see "Region fallback" above — and writes /etc/wireguard/pia-wg.conf (Table = off — this role owns all routing for the interface, not wg-quick).
• Installs pia-killswitch.service, ordered Before= wg-quick@pia-wg.service, that seeds a closed state at every boot (and after any UFW reload — see handlers/main.yaml): the from 10.10.50.0/24 lookup pia rule, a terminal unreachable default route in the pia table (only when pia-wg isn't already up — see the script's own comment for why), and a dedicated PIA-VLAN50 iptables chain jumped into by a single rule at the very top of FORWARD, matching only source 10.10.50.0/24, ending in an unconditional rate-limited log+drop. This is deliberately not a global FORWARD default-policy change — minisforum runs Flannel, which needs its own broad FORWARD ACCEPTs for pod traffic (source 10.42.0.0/16, disjoint from VLAN 50) — so the kill switch is scoped to a chain Flannel/k3s traffic can never enter, rather than risking a chain-wide policy that was never actually proven safe against it. Technitium (10.10.40.53, minisforum's own address) gets a normal UFW input allow for 53/tcp+udp — it's locally-terminated traffic, not routed — see tasks/firewall.yaml.
• pia-wg.conf's own PostUp/PreDown open/close the narrower path on top of that closed baseline: default route via pia-wg in the pia table, ACCEPT + established/related return inserted into the PIA-VLAN50 chain (not FORWARD directly), and source-NAT/MASQUERADE scoped to 10.10.50.0/24 on pia-wg only.
• Installs an observability-only health check (pia-gateway-healthcheck .timer, every pia_healthcheck_interval_sec) that logs interface, handshake age, rule/route, and firewall-policy state to the journal. It never remediates — see the script's header comment for why, and for what its "route-decision" check does and does not prove.

What it deliberately does not do

• Does not touch minisforum's own default route (asserted at the start of every run — tasks/assert-baseline.yaml fails loudly if that's already wrong).
• Does not enable IPv6 forwarding or any IPv6 handling for VLAN 50.
• Does not enable MSS clamping (pia_mss_clamp_enabled: false by default) — flip only after Phase 5 canary MTU testing shows it's actually needed.
• Does not modify ansible/roles/wireguard (the separate wg0 home-VPN server role) or its interface.

Rollback

No tag-based rollback is defined — this role doesn't have an "absent" mode. Roll back manually on minisforum (reverses this role without touching wg0, k3s, or the host default route):

sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \
           /etc/systemd/system/pia-gateway-healthcheck.service \
           /etc/systemd/system/pia-gateway-healthcheck.timer
sudo systemctl daemon-reload
sudo ip rule del from 10.10.50.0/24 table pia priority 100
sudo ip route flush table pia
sudo sed -i '/^[0-9]\+\s\+pia$/d' /etc/iproute2/rt_tables
sudo iptables -D FORWARD -s 10.10.50.0/24 -j PIA-VLAN50
sudo iptables -F PIA-VLAN50
sudo iptables -X PIA-VLAN50
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp
sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.new /etc/wireguard/pia-ca.crt

Inspection commands to confirm rollback actually took (read-only):

sudo iptables -S FORWARD | grep PIA-VLAN50   # expect: no output
sudo iptables -L PIA-VLAN50                  # expect: "iptables: No chain/target/match by that name"
sudo ufw status verbose | grep 10.10.50      # expect: no output
ip rule show | grep pia                      # expect: no output
ip route show table pia                      # expect: empty/error (table gone)

Does not remove pia-credentials/pia-credentials-sealed.yaml (the K8s Secret used by the Gluetun sidecars) — that's a separate, unrelated secret and rollback path. Does not touch ansible/roles/common's own UFW rules (Flannel pod-to-pod, pod-to-Technitium) — this role never modified those.