diff --git a/README.md b/README.md index bdf1cac..e2b9ed0 100644 --- a/README.md +++ b/README.md @@ -231,8 +231,78 @@ mobileconfig profile. The `ca-sync` CronJob updates those files from the `*.nik4nao.com` certificates use the Let's Encrypt issuers in `manifests/cert-manager/cluster-issuer-letsencrypt.yaml`. +## VPN VLAN 50 (PIA gateway) — status and runbook + +`minisforum` is a PIA WireGuard egress gateway for a dedicated VPN VLAN +(`10.10.50.0/24`), giving `nik-debian`'s selected media workloads a second, +PIA-routed network path (`net1`) alongside their normal cluster networking +(`eth0`) — full design in `plan.md` (repo root). **Live as of 2026-08-24**: +every layer below is applied and independently verified, including the +Phase 5 canary's full validation/failure-test matrix and both media +workloads' own fail-closed kill-switch test (stopping `wg-quick@pia-wg` +on minisforum visibly stops their egress, no fallback to `eth0`). Current +PIA endpoint is Hong Kong (`pia_region_candidates`' first entry — +whichever region actually answers is recorded per-host, see +`ansible/roles/pia-gateway/README.md`'s "Region fallback"). See each +piece's own README for exact commands and rollback: + +| Layer | Where | Rollback | +| --- | --- | --- | +| Minisforum PIA gateway (WireGuard, policy routing, kill switch, health check) | `ansible/roles/pia-gateway/` | that role's README | +| nik-debian VLAN 50 parent (`enp1s0.50`) | `ansible/roles/vlan50-parent/` | that role's README | +| Multus, VLAN 50 NAD, canary | `manifests/multus/` (manual-sync Argo CD Application) | that directory's README | +| qBittorrent / JDownloader VLAN 50 attachment | `manifests/media/qbittorrent.yaml`, `jdownloader.yaml` | no file-based copy kept — restore the pre-migration Gluetun manifest from git history (`git show :manifests/media/qbittorrent.yaml`, likewise for jdownloader.yaml) and `kubectl apply` it; both migration commits are tagged in their own message | + +**Boot ordering** (minisforum): `pia-killswitch.service` seeds the closed +routing/firewall state and is ordered `Before=` `wg-quick@pia-wg.service`; +both are `WantedBy=multi-user.target` so a normal boot orders them +correctly with no manual step. `pia-gateway-healthcheck.timer` runs +independently and only observes — see +`ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2`. + +**PIA endpoint/key rotation**: re-run `ansible/playbooks/pia-gateway.yaml +-e pia_force_reregister=true -K` — this regenerates the local keypair and +re-registers with PIA, then rewrites `pia-wg.conf` (handler-triggered +restart). Rotating the PIA account password itself just needs +`ansible-vault edit ansible/group_vars/all/vault.yaml` first. + +**Maintenance / taking the gateway down deliberately**: `sudo systemctl +stop wg-quick@pia-wg` on minisforum closes VLAN 50 Internet egress +immediately (kill switch stays engaged) without touching minisforum's own +networking, k3s, or any other workload — this is the same fail-closed path +exercised by the Phase 5 failure tests in `plan.md`. + +**Multus troubleshooting**: check `kubectl -n kube-system get pods -l +app=multus -o wide` (should be Running, only on `nik-debian`), then +`/var/lib/rancher/k3s/agent/etc/cni/net.d/` on nik-debian for a generated +`00-multus.conf` — if it's missing, Multus isn't actually in the CNI +chain regardless of pod status. See `manifests/multus/README.md` for the +k3s-specific path gotchas that make this different from a stock Multus +install. + +**Monitoring**: currently journald only +(`journalctl -u pia-gateway-healthcheck` on minisforum) — nothing is wired +into the existing kube-prometheus-stack (`values/kube-prometheus-stack +.yaml` has no custom node-exporter textfile-collector or +`additionalPrometheusRulesMap` configured, and none was added by this +work). Wiring real alerting would mean enabling node-exporter's textfile +collector for that host and adding a `PrometheusRule` — not done, flagged +here as the natural next step rather than guessed at. + +**Rollback (full)**: work through the table above bottom-to-top — restore +qBittorrent/JDownloader from git history first (fastest to undo, keeps +service running — see the table's Rollback column), then remove the +Multus/NAD resources, then `ansible/roles/vlan50-parent`'s rollback, then +`ansible/roles/pia-gateway`'s rollback. None of these steps touch +`pia-credentials`, PVCs, or the normal k3s/Argo CD control plane. + ## Gotchas +- **`manifests/media/qbittorrent.yaml` and `jdownloader.yaml` deploy the + moment they're pushed to `main`** — the `media` Argo CD Application + already has `selfHeal`/`automated` sync, unlike the new `multus` + Application (deliberately manual-sync). Double-check which gate applies + before pushing changes to either file. - Argo CD Applications mostly set `prune: false`; removing resources from Git may require manual cleanup. - Gitea uses a manual public `IngressRoute`; `values/gitea.yaml` has no diff --git a/manifests/README.md b/manifests/README.md index abcb055..f27e7c9 100644 --- a/manifests/README.md +++ b/manifests/README.md @@ -16,6 +16,7 @@ subdirectories are consumed by Argo CD Applications in `argocd/apps`. | `homeassistant/` | Home Assistant external service and ingress | | `media/` | Jellyfin, qBittorrent, JDownloader ([sending links](media/jdownloader.md)) | | `monitoring/` | Monitoring PVs, Grafana datasource, Grafana/Auth OIDC secrets | +| `multus/` | Multus CNI, VLAN 50 NetworkAttachmentDefinition, canary ([details](multus/README.md)) — manual-sync Application, see `argocd/apps/multus.yaml` | | `network/` | Pi-hole secrets, DDNS, Traefik dashboard, external host ingresses | | `portfolio/` | Portfolio deployment, ingress, registry pull secret |