227 lines
11 KiB
Markdown
227 lines
11 KiB
Markdown
# pia-gateway
|
|
|
|
Makes `minisforum` a PIA WireGuard egress gateway for VPN VLAN 50
|
|
(`10.10.50.0/24`), without changing its own default route. Implements
|
|
Phase 2 of `~/repo/homelab/plan.md`.
|
|
|
|
## Before the first real run
|
|
|
|
1. **The PIA API calls in `tasks/register.yaml` have gone through three
|
|
rounds of correction against real live testing (2026-08-24):**
|
|
- `serverlist.piaservers.net/vpninfo/servers/v6`: works. Region values
|
|
everywhere in this role are the API's **id** field (e.g. `japan`,
|
|
`hk`, `taiwan`), never a display name ("JP Tokyo", "Hong Kong",
|
|
"Taiwan" — those are Gluetun's labels, not something this role or
|
|
PIA's API accepts).
|
|
- Token acquisition originally guessed a regional-meta-server
|
|
endpoint (`/authv3/generateToken`) that turned out not to be the
|
|
real flow at all — it was rebuilt from PIA's actual
|
|
`pia-foss/manual-connections` `get_token.sh` (fetched and read in
|
|
full): a single fixed `POST
|
|
https://www.privateinternetaccess.com/api/client/v2/token`,
|
|
multipart `username`/`password`, **normal system CA validation**
|
|
(no `--cacert`, no `validate_certs: false`), independent of region
|
|
entirely — only WireGuard server *selection* is regional. Not yet
|
|
exercised with real credentials end to end.
|
|
- `{wg}:1337/addKey` (WireGuard key registration) — confirmed against
|
|
`connect_to_wireguard_with_token.sh` (fetched and read in full):
|
|
this one genuinely does need PIA's own CA bundle
|
|
(`files/pia-ca.crt`, `--cacert`) and `--connect-to`. Originally
|
|
failed over only across servers *within* a single fixed region
|
|
(`pia_region: japan`) — in practice all three JP Tokyo servers
|
|
failed the same way, one after another, so failover now also
|
|
crosses regions: every server in Hong Kong, then every server in
|
|
Taiwan, then every server in JP Tokyo (`pia_region_candidates` in
|
|
`defaults/main.yaml`), stopping at the first `status: OK`. Token
|
|
passed via stdin rather than argv. See `tasks/addkey-attempt.yaml`
|
|
and "Region fallback" below.
|
|
2. Make sure the repo-root `.env` has `PIA_USER`/`PIA_PASSWORD` set (same
|
|
keys `manifests/media/pia-secret.sh` already reads) — `ansible/
|
|
host_vars/minisforum.yaml` reads them directly from there via a
|
|
`lookup('ansible.builtin.file', ...)` + regex, deliberately not
|
|
ansible-vault, so this needs no vault password at all. The lookup runs
|
|
on the control node, so `.env` never touches minisforum and is never
|
|
committed (already gitignored).
|
|
3. Confirm console/recovery access to `minisforum` (IPMI/physical/other
|
|
out-of-band path) before applying — this role changes host routing and
|
|
firewall policy. If SSH becomes unreachable, use that path.
|
|
4. `ufw status verbose` and `wg show` (existing `wg0`) were not inspected
|
|
live before writing this role (no passwordless sudo in discovery) —
|
|
spot-check that `wg0` (the separate home-VPN role) is unaffected after
|
|
applying.
|
|
|
|
## Region fallback
|
|
|
|
Registration tries every WireGuard server in every configured region, in
|
|
order, stopping at the first `status: OK`:
|
|
|
|
```text
|
|
hk (Hong Kong):
|
|
wg server 1
|
|
wg server 2
|
|
taiwan (Taiwan):
|
|
wg server 1
|
|
wg server 2
|
|
japan (JP Tokyo):
|
|
wg server 1
|
|
wg server 2
|
|
wg server 3
|
|
```
|
|
|
|
The actual number and order of servers within each region always comes
|
|
from the live serverlist response — never hardcoded here.
|
|
|
|
**Automatic fallback** (default — tries `pia_region_candidates` in
|
|
order):
|
|
|
|
```bash
|
|
ansible-playbook -i ansible/inventory.yaml \
|
|
ansible/playbooks/pia-gateway.yaml -K -J
|
|
```
|
|
|
|
**Forced single region**, for troubleshooting one region in isolation —
|
|
bypasses `pia_region_candidates` entirely, does not fall back to the
|
|
others:
|
|
|
|
```bash
|
|
ansible-playbook -i ansible/inventory.yaml \
|
|
ansible/playbooks/pia-gateway.yaml -K -J \
|
|
-e pia_region=hk
|
|
```
|
|
|
|
Both `pia_region` and every entry in `pia_region_candidates` are PIA API
|
|
**ids** — `hk`, `taiwan`, `japan` — not the display names Gluetun's
|
|
`SERVER_REGIONS` config uses ("Hong Kong", "Taiwan", "JP Tokyo"). Passing
|
|
a display name here matches nothing and fails the "region exists"
|
|
assertion before any network call is even made.
|
|
|
|
Which region/server actually ended up registered is recorded as
|
|
`pia_region_used`/`pia_region_name_used`/`pia_wg_server_used`, written
|
|
into `pia-wg.conf`'s own `[Peer]` comment, and surfaced by the health
|
|
check (`INFO configured-peer: ...`) — it is whichever one answered first,
|
|
not necessarily `pia_region_candidates[0]`.
|
|
|
|
If every region fails: `journalctl` (or the play's own output) has one
|
|
`debug` line per failed attempt, each showing region name+id, server
|
|
hostname+IP, curl exit code, a classified reason (connection timeout /
|
|
TLS-SSL failure / HTTP failure / empty response / malformed JSON /
|
|
parsed-but-not-OK status, never a generic catch-all), and sanitized
|
|
stderr — never the token, credentials, the full curl invocation, request
|
|
stdin, or a complete response body. If Hong Kong and Taiwan fail exactly
|
|
the same way Tokyo did, that's a strong signal the problem is shared
|
|
across all three (the request shape, the account credentials, or
|
|
minisforum's own network path) rather than one region being down — see
|
|
`tasks/register.yaml`'s header comment.
|
|
|
|
**Worked example (2026-08-24):** exactly that happened — all 7 servers
|
|
across all 3 regions came back `HTTP failure (non-2xx response)`, curl
|
|
exit 22, `The requested URL returned error: 401`, uniformly. The cause
|
|
was in `addkey-attempt.yaml`'s own curl task, not PIA or any region:
|
|
`ansible.builtin.command`'s `stdin` argument appends a trailing newline
|
|
by default (`stdin_add_newline` defaults to `true`), and
|
|
`--data-urlencode pt@-` does not strip it — every server was receiving
|
|
`"<real token>\n"` (URL-encoded, so a trailing `%0A`) as `pt`, a
|
|
different and invalid value, and correctly rejecting it. Fixed with
|
|
`stdin_add_newline: false` on that task. `connect_to_wireguard_with_token
|
|
.sh` never hits this because it passes the token as a literal shell
|
|
variable, not via stdin — the stdin delivery here is this repo's own
|
|
addition (to keep the token out of `ps`), so it needed the flag PIA's
|
|
reference script never had to think about. Left as a worked example
|
|
because "every candidate failed identically" pointing at one shared bug
|
|
in *this* code, not PIA, is exactly the diagnostic story this section
|
|
promises — and it happened to be true the first time it was tested for
|
|
real.
|
|
|
|
## What it does
|
|
|
|
- Registers minisforum as a PIA WireGuard peer — see "Region fallback"
|
|
above — and writes `/etc/wireguard/pia-wg.conf` (`Table = off` — this
|
|
role owns all routing for the interface, not wg-quick).
|
|
- Installs `pia-killswitch.service`, ordered `Before=
|
|
wg-quick@pia-wg.service`, that seeds a closed state at every boot (and
|
|
after any UFW reload — see handlers/main.yaml): the
|
|
`from 10.10.50.0/24 lookup pia` rule, a terminal `unreachable default`
|
|
route in the `pia` table (only when `pia-wg` isn't already up — see
|
|
the script's own comment for why), and a dedicated `PIA-VLAN50`
|
|
iptables chain jumped into by a single rule at the very top of
|
|
`FORWARD`, matching only source `10.10.50.0/24`, ending in an
|
|
unconditional rate-limited log+drop. This is deliberately **not** a
|
|
global `FORWARD` default-policy change — minisforum runs Flannel,
|
|
which needs its own broad `FORWARD` ACCEPTs for pod traffic (source
|
|
`10.42.0.0/16`, disjoint from VLAN 50) — so the kill switch is scoped
|
|
to a chain Flannel/k3s traffic can never enter, rather than risking a
|
|
chain-wide policy that was never actually proven safe against it.
|
|
Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW
|
|
**input** allow for 53/tcp+udp — it's locally-terminated traffic, not
|
|
routed — see `tasks/firewall.yaml`.
|
|
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
|
|
top of that closed baseline: default route via `pia-wg` in the `pia`
|
|
table, ACCEPT + established/related return inserted into the
|
|
`PIA-VLAN50` chain (not `FORWARD` directly), and source-NAT/MASQUERADE
|
|
scoped to `10.10.50.0/24` on `pia-wg` only.
|
|
- Clamps TCP MSS on the `pia-wg` forward path (`pia_mss_clamp_enabled:
|
|
true`, `iptables -t mangle ... TCPMSS --clamp-mss-to-pmtu`, scoped to
|
|
SYN packets sourced from `10.10.50.0/24` outbound on `pia-wg` only, in
|
|
`PostUp`/`PreDown` alongside the rules above). Confirmed needed live
|
|
(2026-08-25), not enabled speculatively: a real download hung with a
|
|
TLS read timeout on the larger handshake response while small
|
|
requests worked fine, and a direct `ping -M do -s 1450` test from a
|
|
VLAN 50 pod confirmed the real path MTU is `pia-wg`'s `1420` (the ICMP
|
|
"Frag needed" reply arrives correctly from minisforum itself — our own
|
|
side relays PMTU discovery fine, so the black hole is further out, on
|
|
PIA's network or the remote server's own path, where clamping the MSS
|
|
up front avoids needing that ICMP round-trip at all).
|
|
- Installs an observability-only health check (`pia-gateway-healthcheck
|
|
.timer`, every `pia_healthcheck_interval_sec`) that logs interface,
|
|
handshake age, rule/route, and firewall-policy state to the journal. It
|
|
never remediates — see the script's header comment for why, and for what
|
|
its "route-decision" check does and does not prove.
|
|
|
|
## What it deliberately does not do
|
|
|
|
- Does not touch minisforum's own default route (asserted at the start of
|
|
every run — `tasks/assert-baseline.yaml` fails loudly if that's already
|
|
wrong).
|
|
- Does not enable IPv6 forwarding or any IPv6 handling for VLAN 50.
|
|
- Does not modify `ansible/roles/wireguard` (the separate `wg0` home-VPN
|
|
server role) or its interface.
|
|
|
|
## Rollback
|
|
|
|
No tag-based rollback is defined — this role doesn't have an "absent"
|
|
mode. Roll back manually on minisforum (reverses this role without
|
|
touching `wg0`, k3s, or the host default route):
|
|
|
|
```bash
|
|
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
|
|
sudo rm -f /etc/systemd/system/pia-killswitch.service \
|
|
/etc/systemd/system/pia-gateway-healthcheck.service \
|
|
/etc/systemd/system/pia-gateway-healthcheck.timer
|
|
sudo systemctl daemon-reload
|
|
sudo ip rule del from 10.10.50.0/24 table pia priority 100
|
|
sudo ip route flush table pia
|
|
sudo sed -i '/^[0-9]\+\s\+pia$/d' /etc/iproute2/rt_tables
|
|
sudo iptables -D FORWARD -s 10.10.50.0/24 -j PIA-VLAN50
|
|
sudo iptables -F PIA-VLAN50
|
|
sudo iptables -X PIA-VLAN50
|
|
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp
|
|
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp
|
|
sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.new /etc/wireguard/pia-ca.crt
|
|
```
|
|
|
|
Inspection commands to confirm rollback actually took (read-only):
|
|
|
|
```bash
|
|
sudo iptables -S FORWARD | grep PIA-VLAN50 # expect: no output
|
|
sudo iptables -L PIA-VLAN50 # expect: "iptables: No chain/target/match by that name"
|
|
sudo ufw status verbose | grep 10.10.50 # expect: no output
|
|
ip rule show | grep pia # expect: no output
|
|
ip route show table pia # expect: empty/error (table gone)
|
|
```
|
|
|
|
Does not remove `pia-credentials`/`pia-credentials-sealed.yaml` (the K8s
|
|
Secret used by the Gluetun sidecars) — that's a separate, unrelated
|
|
secret and rollback path. Does not touch `ansible/roles/common`'s own
|
|
UFW rules (Flannel pod-to-pod, pod-to-Technitium) — this role never
|
|
modified those.
|