nik/homelab
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: nik:main
Branches Tags
nik:main
nik:restore-public-ingress
..
compare: nik:restore-public-ingress
Branches Tags
nik:main
nik:restore-public-ingress

1 Commits
main ... restore-pu

Author SHA1 Message Date
Nik Afiq
435f62d96f Revert "fix: remove Ingress and TLS configurations for Watch Party and Portfolio services" 
This reverts commit 3657519b17c16888fbde0946ba5c65c855330d2b.
 2026-08-07 16:28:23 +09:00
75 changed files with 630 additions and 4219 deletions
Show Stats Expand all files Collapse all files

13
CLAUDE.md
View File

@ -45,10 +45,10 @@ Hosts (see `ansible/inventory.yaml`):
| Host | IP | Role |
| --- | --- | --- |
| `minisforum` | `10.10.40.53` | K3s server, Traefik entrypoint, primary app node |
| `debian` | `10.10.40.20` | K3s agent, NFS storage (`/mnt/storage`), secondary Pi-hole |
| `mac-mini` | `10.10.40.30` | Standalone Docker host — Watch Party, Ollama (not in the K3s cluster) |
| `gpu-node` | `10.10.40.12` | K3s agent with NVIDIA GPU, tainted `spot=true:NoSchedule`, labeled `node-role: gpu`; runs Ollama directly on the host, not as a pod |
| `minisforum` | `192.168.7.77` | K3s server, Traefik entrypoint, primary app node |
| `debian` | `192.168.7.183` | K3s agent, NFS storage (`/mnt/storage`), secondary Pi-hole |
| `mac-mini` | `192.168.7.96` | Standalone Docker host — Watch Party, Ollama (not in the K3s cluster) |
| `gpu-node` | `192.168.7.98` | K3s agent with NVIDIA GPU, tainted `spot=true:NoSchedule`, labeled `node-role: gpu`; runs Ollama directly on the host, not as a pod |
K3s's bundled Traefik and ingress controller are disabled — Traefik is
installed and managed through Argo CD instead. Internal services live under
@ -140,8 +140,3 @@ otherwise it reports spurious `unknown-module` errors for `community.general`/
`manifests/network/pihole-debian-patch.sh`.
- Dashy's manifest ships an empty ConfigMap shell; real config only lands
after running `manifests/core/apply-dashy-config.sh`.
- A newly pushed Argo CD `Application` doesn't create its namespace
instantly — `app-of-apps` only picks it up on its next Git poll. A
namespace-scoped secret script run immediately after pushing can fail
with `namespaces "<ns>" not found` until then (or until a forced
refresh).

244
INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md
View File

@ -1,244 +0,0 @@
# Incident report: VLAN 50 egress outage, qBittorrent crash loop, and a PMTU black hole
**Date:** 2026-08-25 to 2026-08-27 (discovered/resolved across this window; underlying triggers occurred 2026-08-24/25)
**Systems:** `minisforum` (PIA WireGuard gateway), `nik-debian` (VLAN 50 workloads: `browser-vpn-proxy`, `qbittorrent`, `jdownloader`)
**Status:** All four issues resolved live, with code fixes committed for three of them (qBittorrent, MSS clamping, `wg-quick` restart idempotency). The remaining gap (PIA tunnel handshake-death alerting/remediation) is documented with prevention options, deliberately not yet implemented — see "Open follow-up" below.
## Summary
Four **independent** problems, found across the same troubleshooting session:
1. **minisforum's PIA WireGuard tunnel (`pia-wg`) silently died** — the interface, systemd unit, and routes all stayed "up" and looked healthy, but the encrypted session with PIA's server had stopped working hours earlier. This black-holed all VLAN 50 egress (`browser-vpn-proxy`, `qbittorrent`, `jdownloader`) — safely (no leak to `eth0`), but silently (nothing paged anyone).
2. **qBittorrent (5.2.0) got stuck in an internal crash loop** on `nik-debian`, unrelated to the tunnel issue and pre-dating it. A known upstream bug in qBittorrent's single-instance lock file handling caused it to restart every ~1-2 seconds, forever, without ever opening its WebUI.
3. **A pre-existing PMTU black hole** between `pia-wg` (MTU `1420`) and the rest of the path caused real downloads to hang indefinitely once the tunnel itself was working again — small requests succeeded, but any connection needing a larger response (like a real download's TLS handshake) silently never got one. This was a known, anticipated gap (`pia_mss_clamp_enabled` existed in the role from the start, deliberately left off pending real evidence) — now confirmed and fixed.
4. **Deploying the fix for #3 broke `wg-quick@pia-wg`'s restart**, because tearing down the live interface with newly-changed `PreDown` rules it never actually had installed made the whole teardown abort partway, orphaning the interface. Fixed at the root by making `PreDown` idempotent — a general hazard for any future config change, not specific to MSS clamping.
All four were found during the same troubleshooting session because a user report ("the proxy isn't working after I rebooted nik-debian") triggered an investigation broad enough to surface all of them in turn — #3 only became visible *after* #1 was fixed (a dead tunnel has no traffic to black-hole), and #4 only surfaced while actually deploying #3's fix. **None of the four share a root cause** — see "Were these related?" below.
## Timeline
All timestamps as observed from their own source; see the timezone caveat below before doing precise cross-host arithmetic.
| When (as logged) | Source | Event |
| --- | --- | --- |
| 2026-08-24 23:07:21Z | `nik-debian` node `Ready` condition | `nik-debian` rejoins the cluster after a reboot |
| 2026-08-24 23:31:28 (pod-local clock) | `browser-vpn-proxy` container log | Fresh pod starts, SOCKS5 proxy begins accepting connections |
| 2026-08-24 23:32:08 (pod-local clock) | `qbittorrent`/`jdownloader` pod logs | Both pods' init containers complete, main containers start |
| ~2026-08-25 08:35 (pod-local clock, per file mtimes) | qBittorrent's own log-rotation file timestamps | qBittorrent's internal crash loop begins (first evidence of rapid log rotation) — **this predates the tunnel's death below** |
| ~2026-08-25 13:15 JST (back-calculated from the healthcheck's own handshake-age readings) | minisforum, `pia-gateway-healthcheck` | `pia-wg`'s WireGuard session actually goes dead (last real handshake) |
| 2026-08-25 11:04-11:05 (pod-local clock) | `browser-vpn-proxy` container log | First visible symptoms: `broken pipe`, then `connection timed out` reaching an external site |
| 2026-08-25 20:42-20:46 JST | minisforum, `journalctl -u pia-gateway-healthcheck` | Healthcheck has been correctly detecting `FAIL handshake: ...s old, exceeds max 180s` every 60s the entire time — just never surfaced to a human |
| 2026-08-25 (user report) | User | Reports proxy not working after a reboot of `nik-debian` |
| 2026-08-25, live troubleshooting | This session | Both root causes diagnosed and fixed (see below) |
**Timezone caveat:** minisforum's `systemctl`/`journalctl` output explicitly shows `JST`. The container/pod log timestamps pasted during this investigation did not carry an explicit timezone marker, so their exact offset from minisforum's JST clock isn't confirmed here — treat cross-host time deltas in this report as approximate, not to-the-second precise. The *relative* ordering of events on the same host (e.g. "qBittorrent's loop started before the tunnel died") is solid, since the reasoning for that only compares timestamps that share a clock.
## Were these related?
No — evidence points at three independent causes, not a shared trigger:
- qBittorrent's crash loop started (~08:35, pod-local clock) well before the PIA tunnel actually died (~13:15 JST, minisforum's clock) and well before egress failures were even visible (~11:04, pod-local clock, on a *different* pod).
- The qBittorrent crash loop's own root cause (a stale single-instance lock file) has nothing to do with WireGuard, PIA, or networking at all.
- The PMTU black hole (Incident 3) is a pre-existing structural gap in the tunnel's own MTU handling — not something the reboot, the tunnel death, or qBittorrent's bug caused. It was simply **not visible** while the tunnel was dead (Incident 1) — no working tunnel meant no traffic to black-hole — and only surfaced once Incident 1 was fixed and something tried to move a large enough response across the path again.
- All three problems happened to be sitting there, undetected, until a user report about a fourth, unrelated-seeming thing (`browser-vpn-proxy` after a reboot) triggered an investigation broad enough to notice all of them in sequence.
The reboot of `nik-debian` itself is **not** the confirmed trigger for any of the three:
- The PIA tunnel runs on a completely different host (minisforum), which was not rebooted.
- qBittorrent's crash loop started roughly 9 hours *after* the reboot, not immediately following it — something else (unidentified — see below) caused qBittorrent's underlying unclean shutdown that left the stale lock file in the first place. The reboot is a plausible but unconfirmed contributor; it was not caught in the act.
- The PMTU black hole is a property of `pia-wg`'s own fixed `mtu 1420` versus the rest of the path — a static configuration characteristic, not something a reboot would change.
## Incident 1: PIA WireGuard tunnel silently died ("zombie tunnel")
### Symptom
All three VLAN 50 workloads lost egress simultaneously. `qbittorrent`'s and `jdownloader`'s sidecar containers failed with `OSError: [Errno 101] Network unreachable` — an immediate "no route" response, not a timeout. `browser-vpn-proxy` showed the same underlying cause but surfaced as a plain timeout from outside the cluster.
### Root cause
`wg-quick@pia-wg.service` was `active (exited)` (this is *normal* for `wg-quick` — it's a one-shot config script, not a long-running daemon) and the `pia-wg` interface existed with its usual `PostUp`-installed routes and iptables rules. But `sudo wg show pia-wg` showed:
```
latest handshake: 7 hours, 29 minutes, 29 seconds ago
persistent keepalive: every 25 seconds
```
WireGuard's own protocol re-handshakes roughly every 2 minutes under normal operation, and a 25-second keepalive is configured specifically to keep that happening even with no application traffic. A handshake this stale means the actual encrypted session with PIA's server (`69.33.198.63:1337`) had been dead for hours — the local interface, routes, and firewall rules `wg-quick` set up were all still present and "correct," but nothing was actually getting through. `ip route show table pia` confirmed the route was the normal *working* route (`default dev pia-wg scope link`), **not** the kill switch's `unreachable` route — meaning the kill switch's own "interface down → block" mechanism never triggered, because the interface never actually went down. It just stopped working while looking fine.
**What actually broke the session was not conclusively identified.** `ping -c3 69.33.198.63` succeeded cleanly (0% loss, ~70ms) from minisforum's normal path, ruling out "minisforum's WAN is broken" or "that IP is completely unreachable." The most likely explanations, in rough order of plausibility, none confirmed:
1. **PIA's registered-key expiry.** A plain `sudo systemctl restart wg-quick@pia-wg` (which only recreates the local interface with the *same* key) did **not** produce a new handshake even after 10+ seconds — strongly suggesting PIA's server no longer recognized this specific registered public key. Re-registering from scratch (`-e pia_force_reregister=true`, generating a fresh keypair and calling PIA's `addKey` API again) *did* restore a working tunnel immediately. This is the strongest evidence: a fresh registration worked where a bare restart did not.
2. A server-side outage specific to that one PIA edge server (`69.33.198.63`) — plausible, but the successful fresh re-registration landed on a *different* exit IP in the same Hong Kong region, so this wasn't independently distinguished from (1).
3. A transient NAT/UDP path issue on minisforum's own WAN — considered less likely given `ping` (a different protocol, admittedly) worked fine, and a restart-without-reregistration should have been enough to recover from a purely local NAT hiccup, but wasn't.
### How to reproduce
Not independently reproduced on demand — this was diagnosed from a live occurrence, not triggered experimentally. The **diagnostic signature**, if it happens again, is:
```bash
sudo wg show pia-wg # interface "looks" fine, but check "latest handshake"
ip route show table pia # shows the *working* route, not "unreachable" — the kill switch did NOT engage
sudo systemctl restart wg-quick@pia-wg
sudo wg show pia-wg latest-handshakes # if this stays at 0 (never) for more than ~15-20s after a restart, a bare restart isn't enough — re-registration is likely needed
```
### How it was fixed (this occurrence)
```bash
ansible-playbook -i ansible/inventory.yaml \
ansible/playbooks/pia-gateway.yaml -K -J \
-e pia_force_reregister=true
```
This hit a real, separate, now-fixed bug on the way: `ansible/roles/pia-gateway/tasks/register.yaml`'s `when:` conditions used `pia_force_reregister`/`pia_force_key_rotation` directly in boolean expressions. Passed via the CLI's plain `key=value` form (`-e pia_force_reregister=true`, exactly what the role's own README documents), Ansible sets the variable as the **string** `"true"`, not a real boolean — and recent `ansible-core` rejects using a string directly in a `when:` boolean expression (`Conditional result (True) was derived from value of type 'str'`). Fixed by wrapping every use in `| bool` (commit `e757850`), which correctly coerces both real booleans (the `defaults/main.yaml` case) and `"true"`/`"false"` strings (the CLI-override case). Verified with an isolated local test playbook exercising both cases before trusting it against the live host.
### Known gap / prevention (not yet implemented — deferred by request)
`ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2` (run via `pia-gateway-healthcheck.timer`, every 60s) **already correctly detected this failure the entire time** — every single run logged `FAIL handshake: ...s old, exceeds max 180s`. The script is explicitly designed as "observability only, no remediation" (its own header comment says so), which is a deliberate prior design choice, not an oversight. The actual gap is that nothing consumes that detection — no alert, no dashboard, nothing — so a real, correctly-detected outage sat unnoticed until a user happened to test the affected service directly.
Options, not yet decided:
1. **Alert only (recommended by this session, not yet chosen by the user):** keep the "no remediation" design, but have the healthcheck (or a small wrapper) notify a human after a few consecutive failures. Two ways to actually send it:
- Via `alert-bridge.home.arpa`'s `POST /alerts` (already exists, already designed for exactly this — a per-caller bearer token instead of distributing the raw Discord webhook secret to every host). Its exact request schema lives in the private `gitea.nik4nao.com/nik/alert-bridge` source, not visible from outside — confirm it before wiring anything.
- A direct `curl` to the Discord webhook URL from minisforum. Simpler, no schema to reverse-engineer, but means minisforum needs its own copy of the raw webhook secret rather than a scoped token — a step away from the existing pattern.
2. **Auto-restart on sustained failure:** after N consecutive failed checks, automatically `systemctl restart wg-quick@pia-wg`. Fixes it without a human noticing, but is a real architecture change to a script deliberately built to never remediate — and masks a recurring problem instead of surfacing it, if the underlying cause (e.g. PIA-side key expiry) keeps happening.
3. **Both:** attempt auto-restart, but still alert either way, so there's always a record even when unattended recovery works.
4. Given evidence pointed at *registration* expiry rather than a simple interface restart being sufficient, any remediation path should probably escalate to `pia_force_reregister=true`-equivalent logic, not just a bare `wg-quick` restart, if a restart-only attempt doesn't restore a fresh handshake within a short grace period.
## Incident 2: qBittorrent 5.2.0 stale-lock crash loop
### Symptom
`qbittorrent` pod showed `1/2 Ready` — the `qbittorrent` container was `Running` (no restarts, no crash reported at the Kubernetes level) but its readiness probe failed continuously: `dial tcp <pod-ip>:8080: connect: connection refused`. This reproduced identically on a **freshly recreated pod** (2.5 minutes old), ruling out anything specific to the original (12-hour-old, post-reboot) pod instance.
### Root cause
`kubectl exec ... ps aux` showed `qbittorrent-nox` genuinely running, consuming ~80% CPU. Its own log file (`/config/qBittorrent/logs/qbittorrent.log` — the container's stdout only ever showed the linuxserver.io init banner, qBittorrent logs to a file, not stdout) showed:
```
(N) 2026-08-25T21:08:00 - qBittorrent termination initiated
(N) 2026-08-25T21:08:00 - qBittorrent is now ready to exit
(N) 2026-08-25T21:08:01 - qBittorrent v5.2.0 started. Process ID: 1376
(N) 2026-08-25T21:08:01 - Using config directory: /config/qBittorrent
(N) 2026-08-25T21:08:01 - qBittorrent termination initiated
(N) 2026-08-25T21:08:01 - qBittorrent is now ready to exit
(N) 2026-08-25T21:08:03 - qBittorrent v5.2.0 started. Process ID: 1384
...
```
A self-perpetuating crash loop, cycling every 1-2 seconds (PID incrementing by 8 each time — matching the container's process-spawning pattern), with the process **voluntarily** exiting immediately after every start (no crash, no signal — it logs its own "termination initiated"). The `/config/qBittorrent/` directory contained a directory's worth of hundreds of rotated log files (`qbittorrent.log.bak1` through `.bak414`+), each ~66-67KB, many piling up within minutes of each other — direct evidence of sustained rapid restart cycling, not a one-off.
This exactly matches a known, confirmed upstream bug: **[qbittorrent/qBittorrent#24164](https://github.com/qbittorrent/qBittorrent/issues/24164)** (and its duplicate, #24185). qBittorrent 5.2.0 switched its single-instance lock from the old fcntl-based `QtLockedFile` (a 0-byte lock file) to Qt's `QLockFile`, which expects the lock file to contain a PID + hostname. If the *previous* shutdown wasn't clean and left a lock file the new version can't parse/verify, 5.2.0 assumes another instance holds it, tries to hand off over the (also stale) `ipc-socket`, that fails too, and it exits — then immediately restarts (supervised by the container's own `s6` process manager) and repeats forever.
**What triggered the original unclean shutdown is not confirmed.** The lock/IPC files were both dated `2026-08-25 08:05` (pod-local clock), about 30 minutes before the crash-loop pattern is first visible in the rotated-log timestamps (`08:35`) — consistent with *something* restarting the qBittorrent process around `08:05` without a clean shutdown, but what that something was (a Kubernetes-level event, an OOM condition, a manual action, something else) wasn't caught in the act and isn't identified here.
One structural factor worth naming even though it's not fully confirmed: qBittorrent's lock file embeds a **hostname**, and every Kubernetes pod restart gets a **new** hostname (the pod name). That's exactly the kind of environment where a lock's embedded hostname can't reliably be trusted to "match" across restarts — plausibly why this surfaces more readily here than it would on a stable bare-metal or VM host with a fixed hostname, though this wasn't independently isolated as *the* determining factor versus the unclean-shutdown precondition alone.
### How to reproduce
Not reproduced from a clean slate in this session — confirmed via matching log signatures against the upstream issue, not an independent from-scratch trigger. Based on the upstream report, the reproduction conditions are: qBittorrent 5.2.0 starts with a `lockfile`/`ipc-socket` already present in its config directory from a *previous*, unclean shutdown (e.g. the container was SIGKILLed rather than given a chance to shut down gracefully). The diagnostic signature if it happens again:
```bash
kubectl -n downloads logs <qbittorrent-pod> -c qbittorrent -c ip-reporter --tail=5 # stdout goes quiet right after the LSIO init banner
kubectl -n downloads exec <qbittorrent-pod> -c qbittorrent -- ps aux # qbittorrent-nox IS running, high CPU, PID climbing on repeat checks
kubectl -n downloads exec <qbittorrent-pod> -c qbittorrent -- tail -n 20 /config/qBittorrent/logs/qbittorrent.log # the "started / termination initiated / ready to exit" cycle
```
### How it was fixed
**This occurrence (live remediation):**
```bash
kubectl -n downloads exec <pod> -c qbittorrent -- rm -f /config/qBittorrent/lockfile /config/qBittorrent/ipc-socket
```
The next `s6`-supervised restart (within 1-2 seconds, given the loop's own cycle time) found no lock file, acquired a fresh one, and started normally.
**Prevention (implemented, committed):** added a new `qbittorrent-clear-stale-lock` init container to `manifests/media/qbittorrent.yaml` that unconditionally removes `lockfile`/`ipc-socket` before qBittorrent starts, on every pod start. This is safe specifically because this Deployment uses `strategy: Recreate` with a single replica and a `ReadWriteOnce` PVC — Kubernetes itself guarantees the previous instance is fully terminated before a new pod's init containers ever run, so there is no scenario in this specific topology where the lock could be legitimately held by a still-running peer. (This reasoning would **not** automatically transfer to a `RollingUpdate` strategy or multiple replicas — don't copy this pattern there without re-checking the concurrency assumption.)
### Residual risk
The upstream issue references a fix PR (#24218) intended to make qBittorrent itself clean up a stale lock automatically. Whether the pinned image (`lscr.io/linuxserver/qbittorrent:5.2.0`) includes that fix wasn't checked — if it doesn't, our init-container workaround remains necessary until either the image is updated past the point where the upstream fix landed, or that's independently confirmed. What caused the *original* unclean shutdown (the actual precondition for hitting this bug at all) remains unidentified — the fix here prevents the *symptom* (permanent crash loop) from persisting, but doesn't address whatever causes qBittorrent's shutdowns to be unclean in the first place, if that keeps happening.
## Incident 3: PMTU black hole on the `pia-wg` forward path
### Symptom
After Incident 1 was fixed (fresh PIA registration, working tunnel), JDownloader still couldn't actually download anything: a link showed status "Running" but no progress and no speed, indefinitely. The app itself was healthy, and basic connectivity worked (its own self-update fetched a small file from an external CDN without any problem).
### Root cause
JDownloader's own per-host plugin log (`subyshare.com_jd.plugins.hoster.SubyShareCom.log.0`) showed the real error: connecting to the actual download server succeeded, JDownloader sent its TLS `ClientHello`, and then nothing came back —
```
javax.net.ssl.SSLException: Read timed out
Caused by: java.net.SocketTimeoutException: Read timed out
ConnectIP: sbs237.sbsf.tech/195.201.192.190:182
```
— a 60-second read timeout waiting for the server's handshake response. Small round-trips (a version-check HTTP fetch, plain ICMP pings) worked fine throughout; only a connection needing to move a larger response hung.
This is the textbook signature of a path-MTU (PMTU) black hole. `ansible/roles/pia-gateway/templates/pia-wg.conf.j2` sets `pia-wg`'s interface MTU to `1420` (WireGuard's own encapsulation overhead requires a lower MTU than the outer network's `1500`) — confirmed directly, not assumed, via a real test: `ping -M do -s 1450 -c 3 1.1.1.1` from a VLAN 50 pod returned
```
From 10.10.40.53 icmp_seq=1 Frag needed and DF set (mtu = 1420)
```
i.e. minisforum correctly tells the sender to fragment/shrink, and the pod's own kernel correctly honors that — **our own side of the path relays PMTU discovery correctly.** That means the actual black hole is further out: somewhere between minisforum's tunnel and the real download server (PIA's own network, or the server's own path), where an equivalent "fragmentation needed" notification would need to travel much further back to the *server* (not us) and evidently isn't making it, so the server's larger response packets just silently vanish instead of ever being resent smaller.
This gap was **anticipated from the start**, not newly introduced: `ansible/roles/pia-gateway/defaults/main.yaml` already had a `pia_mss_clamp_enabled` flag and the corresponding `iptables -t mangle ... TCPMSS --clamp-mss-to-pmtu` rule already written in the WireGuard config template — deliberately left `false` by `plan.md`'s own explicit instruction ("do not guess this... flip only after ... MTU testing shows it's actually needed"). It most likely wasn't caught by earlier canary/qBittorrent/browser-vpn-proxy testing because those tests mostly exercised small request/response pairs (`curl ifconfig.me`, `dig`, small API calls) — this only surfaces on a connection that needs to move a larger response, like a real file download's TLS handshake.
### How to reproduce
```bash
kubectl -n downloads exec vlan50-canary -c netshoot -- ping -M do -s 1392 -c 3 1.1.1.1 # works — under the real MTU
kubectl -n downloads exec vlan50-canary -c netshoot -- ping -M do -s 1450 -c 3 1.1.1.1 # fails locally with "Frag needed... mtu = 1420" — this is the confirmation test, not a guess
```
Application-level symptom: any download/request whose response exceeds ~1420 bytes (accounting for WireGuard/IP/TCP overhead) will hang indefinitely rather than erroring cleanly, while small requests keep working — making this easy to miss if testing only checks basic connectivity.
### How it was fixed
Enabled the already-built toggle rather than writing new mechanism: `ansible/roles/pia-gateway/defaults/main.yaml`'s `pia_mss_clamp_enabled` flipped from `false` to `true`. This clamps the TCP MSS on SYN packets outbound on `pia-wg` for `10.10.50.0/24` traffic, so both ends of any new TCP connection negotiate a small-enough segment size from the start — sidestepping the need for a working PMTU-discovery ICMP round-trip entirely. Applying it needed a second, unplanned fix — see Incident 4 immediately below, discovered in the process of actually deploying this one. Confirmed working: after Incident 4's recovery, all three VLAN 50 workloads' egress checks passed and the specific JDownloader download that had been stuck actually completed.
### Residual risk
MSS clamping fixes new TCP connections going forward; it doesn't retroactively fix anything already stuck — confirmed in practice, the originally-stuck download needed a manual retry after the fix landed. The scope is intentionally narrow (`10.10.50.0/24` outbound on `pia-wg` only, SYN packets only) to match the existing kill-switch/firewall pattern of "own chain, only touch what's ours" — this should not affect any other traffic on minisforum.
## Incident 4: enabling MSS clamping broke `wg-quick` restart (discovered while fixing Incident 3)
### Symptom
Running the Ansible playbook to actually apply the MSS-clamping default change (no `-e` override, since the change was now a committed default) completed with `changed=0` — the config file was never rewritten at all, because `pia-wg.conf` already existed and the registration block that writes it is gated behind `pia_force_reregister or pia_force_key_rotation or not pia_wg_conf_stat.stat.exists`; a plain re-run with the file already present skips that entire block. Re-running with `-e pia_force_reregister=true` this time got further, but then failed outright:
```
RUNNING HANDLER [pia-gateway : Restart pia-wg]
[ERROR]: Task failed: Module failed: Unable to restart service wg-quick@pia-wg: Job for wg-quick@pia-wg.service failed because the control process exited with error code.
```
### Root cause
Two separate, now both understood:
1. **The config-write gate.** The role currently bundles "rewrite `pia-wg.conf` from the template" together with "actually re-register with PIA" under one `when:` condition — there's no path to pick up a template-only change (like flipping `pia_mss_clamp_enabled`) without forcing a full PIA re-registration too. Not fixed here; noted as a design gap (see Open follow-up).
2. **`systemctl restart` tore down the live interface using rules it never had.** `journalctl` showed exactly why: `PreDown`'s new `iptables -t mangle -D FORWARD ... TCPMSS` line failed with `iptables: Bad rule (does a matching rule exist in that chain?)` — because the *currently running* interface had been brought up under the *old* config (before MSS clamping existed), so its `PostUp` never added that mangle rule in the first place. Confirmed against `wg-quick`'s real source (`execute_hooks()` in `wireguard-tools`'s `linux.bash`): each hook line runs via `(eval "$hook")`, and `wg-quick`'s own `set -e` aborts the entire down/up sequence on the first failing hook. Since that failing line was the *last* `PreDown` line, everything before it (the kill-switch's `unreachable` route seed, the `MASQUERADE`/`ACCEPT` rule deletes) had already run successfully — confirmed live: `ip route show table pia` showed `unreachable default` (fail-closed, not leaking) — but `wg-quick`'s own built-in `ip link delete dev pia-wg` step, which normally runs *after* all custom `PreDown` lines, never got reached. The interface was left orphaned: still present at the kernel level, but stripped of its firewall rules. The subsequent `wg-quick up` then failed too, immediately, with `` `pia-wg' already exists``, since `ip link add` can't create an interface that's still there.
This is a **general** hazard, not specific to MSS clamping: Ansible's `template` task overwrites `pia-wg.conf` on disk before the "Restart pia-wg" handler fires, so *any* future change to what `PostUp`/`PreDown` actually do will hit the identical failure the next time the config changes while the tunnel is already up.
### How to reproduce
Change any `PostUp`/`PreDown` rule content in `pia-wg.conf.j2` (add, remove, or reorder a rule) while `pia-wg` is currently running under the *old* version of that content, then `systemctl restart wg-quick@pia-wg` (or let Ansible's handler do it). If any new/changed `PreDown` line tries to `-D` something the live interface never actually added, the restart fails and the interface is left orphaned.
### How it was fixed
**This occurrence (live recovery):**
```bash
sudo ip link delete dev pia-wg
sudo systemctl start wg-quick@pia-wg
```
Directly removing the orphaned interface (the exact step the aborted teardown was one line away from doing itself) let a fresh `wg-quick up` succeed cleanly, this time actually applying the new `PostUp` rules including the MSS clamp. Confirmed via `iptables -t mangle -S FORWARD` showing the rule actually installed, and a handshake timestamp matching the moment of the restart (not stale).
**Prevention (implemented, committed):** made every `PreDown` `iptables -D` line in `pia-wg.conf.j2` tolerate the rule it's removing not existing (`2>/dev/null || true`), verified safe against `wg-quick`'s real source (a hook line's compound exit status becomes 0, so `set -e` no longer treats "nothing to delete" as fatal) and verified by rendering the template directly with Jinja2 before trusting it. The first `PreDown` line (`ip route replace unreachable default table pia`) didn't need this — `route replace` doesn't fail due to prior state the way `-D` does. This makes `PreDown` fully idempotent, so this exact failure mode can't recur for *any* future `PostUp`/`PreDown` content change — no separate "self-heal after the fact" layer needed, since the restart now just succeeds correctly every time.
### Residual risk
None identified for this specific failure mode going forward — the fix addresses the root cause directly rather than adding a recovery layer on top of a still-fragile restart. The separate config-write gating issue (item 1 above) remains open.
## Open follow-up
- **Decide and implement PIA tunnel remediation/alerting** (Incident 1's "Known gap" section above) — explicitly deferred by the user to a later date, options documented above.
- **Consider splitting "rewrite config from template" from "re-register with PIA" in the role** (Incident 4's root cause #1) — currently there's no way to pick up a template-only default change without forcing a full re-registration.
- Confirm whether a newer `lscr.io/linuxserver/qbittorrent` tag already includes the upstream fix for #24164, as a potential alternative/complement to the init-container workaround.
- If it recurs, try to catch what actually triggers qBittorrent's unclean shutdown in the first place (Incident 2's root cause section) — nothing in this investigation identified it.

145
README.md
View File

@ -8,10 +8,10 @@ in Kubernetes manifests plus Helm values.
| Host | IP | Role |
| --- | --- | --- |
| `minisforum` | `10.10.40.53` | K3s server, Traefik entrypoint, primary app node |
| `debian` / `nik-debian` | `10.10.40.20` | K3s agent, NFS storage, secondary Pi-hole |
| `mac-mini` | `10.10.40.30` | Standalone services such as Watch Party and Ollama |
| `gpu-node` / `nik-gpu` | `10.10.40.12` | K3s agent with NVIDIA GPU passthrough, spot-tainted; runs Ollama directly on the host |
| `minisforum` | `192.168.7.77` | K3s server, Traefik entrypoint, primary app node |
| `debian` / `nik-debian` | `192.168.7.183` | K3s agent, NFS storage, secondary Pi-hole |
| `mac-mini` | `192.168.7.96` | Standalone services such as Watch Party and Ollama |
| `gpu-node` / `nik-gpu` | `192.168.7.98` | K3s agent with NVIDIA GPU passthrough, spot-tainted; runs Ollama directly on the host |
The cluster uses Traefik instead of the bundled K3s ingress controller. Internal
services are published under `home.arpa` with certificates from an internal CA.
@ -40,7 +40,7 @@ Public services under `nik4nao.com` use Let's Encrypt.
| Git and CI | Gitea, Gitea Actions runner, registry pull secrets, Gitea backup CronJob |
| Media | Jellyfin, qBittorrent, JDownloader |
| Home services | Home Assistant ingress, HA gateway, AI gateway, Discord bot, Alexa bridge, Alert bridge |
| Public apps | Portfolio, me.nik4nao.com, Watch Party ingress to the Mac Mini |
| Public apps | Portfolio, Watch Party ingress to the Mac Mini |
| Dashboard | Dashy |
## Important URLs
@ -64,7 +64,6 @@ Public services under `nik4nao.com` use Let's Encrypt.
| `http://alert-bridge.home.arpa` | Alert bridge (Discord webhook relay) | Plain HTTP |
| `https://watch-party.nik4nao.com` | Watch Party on Mac Mini | Let's Encrypt |
| `https://nik4nao.com` | Portfolio | Let's Encrypt |
| `https://me.nik4nao.com` | me.nik4nao.com (terminal-style personal site) | Let's Encrypt |
| `http://ca.home.arpa` | Internal CA installer | Plain HTTP |
`home.arpa` names are defined explicitly in `values/pihole.yaml` and
@ -191,7 +190,6 @@ bash manifests/home-services/registry-secret.sh
bash manifests/monitoring/grafana-secret.sh
bash manifests/network/ddns-secret.sh
bash manifests/portfolio/registry-secret.sh
bash manifests/me-nik4nao-com/registry-secret.sh
```
Sealed Secret regeneration:
@ -221,12 +219,11 @@ state that must live on known disks:
| `/home/nik/backups` on `debian` | NFS export for Gitea's backup CronJob (separate from `/mnt/storage`) |
The Debian NFS server exports both `/mnt/storage` and `/home/nik/backups` to
`10.10.40.53`.
`192.168.7.77`.
## TLS and Trust
`*.home.arpa` certificates use `internal-ca-issuer` from `manifests/cert-manager`,
which signs off the self-signed root `internal-ca-cert` (`cert-manager` namespace).
`*.home.arpa` certificates use `internal-ca-issuer` from `manifests/cert-manager`.
The CA installer at `http://ca.home.arpa` serves `ca.crt` and an iOS/macOS
mobileconfig profile. The `ca-sync` CronJob updates those files from the
`cert-manager/internal-ca-cert` secret when the CA changes.
@ -234,124 +231,8 @@ mobileconfig profile. The `ca-sync` CronJob updates those files from the
`*.nik4nao.com` certificates use the Let's Encrypt issuers in
`manifests/cert-manager/cluster-issuer-letsencrypt.yaml`.
### Renewing the internal CA
`internal-ca-cert`'s default `privateKey.rotationPolicy` is `Never`, so a
normal renewal reuses the existing private key — every already-issued leaf
cert and every device that already trusts the CA keep working unchanged.
**Deleting the `internal-ca-cert` secret instead of renewing it in place
generates a brand-new key pair**, which invalidates every leaf cert and every
device's trust simultaneously (this happened 2026-09-02: leaf certs issued
before the deletion failed `openssl verify` against the recreated CA
entirely, and Jellyfin/Kavita/etc. all showed as insecure on client devices).
To renew correctly:
```bash
cmctl renew certificate internal-ca-cert -n cert-manager
```
This reissues the CA certificate (new serial/validity) while keeping the same
key, so it should not require any downstream action. If the key ever *does*
change (intentional rotation, or the secret was lost and had to be
recreated), do both of the following — the CA rotating does **not**
automatically cascade to either:
1. **Force-reissue every leaf cert**, since each `Certificate` renews on its
own independent schedule and won't notice the issuing CA changed:
```bash
for c in $(kubectl get certificate -A -o json | \
jq -r '.items[] | select(.spec.issuerRef.name=="internal-ca-issuer") | "\(.metadata.namespace)/\(.metadata.name)"'); do
ns="${c%%/*}"; name="${c##*/}"
secret=$(kubectl get certificate "$name" -n "$ns" -o jsonpath='{.spec.secretName}')
kubectl delete secret "$secret" -n "$ns"
done
```
Verify afterward that every leaf chains to the current CA:
```bash
kubectl get secret internal-ca-cert -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/current-ca.crt
openssl verify -CAfile /tmp/current-ca.crt <(kubectl get secret <leaf-secret> -n <ns> -o jsonpath='{.data.tls\.crt}' | base64 -d)
```
2. **Re-trust the CA on every client device** — visit `http://ca.home.arpa`,
reinstall the mobileconfig, and re-enable full trust under Settings →
General → About → Certificate Trust Settings (iOS/macOS). The old trust
profile does not update itself; nothing pushes the new CA to devices.
## VPN VLAN 50 (PIA gateway) — status and runbook
`minisforum` is a PIA WireGuard egress gateway for a dedicated VPN VLAN
(`10.10.50.0/24`), giving `nik-debian`'s selected media workloads a second,
PIA-routed network path (`net1`) alongside their normal cluster networking
(`eth0`) — full design in `plan.md` (repo root). **Live as of 2026-08-24**:
every layer below is applied and independently verified, including the
Phase 5 canary's full validation/failure-test matrix and both media
workloads' own fail-closed kill-switch test (stopping `wg-quick@pia-wg`
on minisforum visibly stops their egress, no fallback to `eth0`). Current
PIA endpoint is Hong Kong (`pia_region_candidates`' first entry —
whichever region actually answers is recorded per-host, see
`ansible/roles/pia-gateway/README.md`'s "Region fallback"). See each
piece's own README for exact commands and rollback:
| Layer | Where | Rollback |
| --- | --- | --- |
| Minisforum PIA gateway (WireGuard, policy routing, kill switch, health check) | `ansible/roles/pia-gateway/` | that role's README |
| nik-debian VLAN 50 parent (`enp1s0.50`) | `ansible/roles/vlan50-parent/` | that role's README |
| Multus, VLAN 50 NAD, canary | `manifests/multus/` (manual-sync Argo CD Application) | that directory's README |
| qBittorrent VLAN 50 attachment | `manifests/media/qbittorrent.yaml` | no file-based copy kept — `git show 28d0628:manifests/media/qbittorrent.yaml \| kubectl apply -f -` restores the last pre-migration (Gluetun-based) version |
| JDownloader VLAN 50 attachment | `manifests/media/jdownloader.yaml` | same approach — `git show 2f7ec5f:manifests/media/jdownloader.yaml \| kubectl apply -f -` |
| browser-vpn-proxy (SOCKS5, no auth — see its own header for why) | `manifests/multus/30-browser-vpn-proxy.yaml` (manual-sync Argo CD Application) | that directory's README — replaces the standalone `manifests/media/gluetun-proxy.yaml` (removed 2026-08-24; `git show 462a9d9:manifests/media/gluetun-proxy.yaml \| kubectl apply -f -` restores it, NodePort `30888`, if ever needed) |
**Boot ordering** (minisforum): `pia-killswitch.service` seeds the closed
routing/firewall state and is ordered `Before=` `wg-quick@pia-wg.service`;
both are `WantedBy=multi-user.target` so a normal boot orders them
correctly with no manual step. `pia-gateway-healthcheck.timer` runs
independently and only observes — see
`ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2`.
**PIA endpoint/key rotation**: re-run `ansible/playbooks/pia-gateway.yaml
-e pia_force_reregister=true -K` — this regenerates the local keypair and
re-registers with PIA, then rewrites `pia-wg.conf` (handler-triggered
restart). Rotating the PIA account password itself just needs
`ansible-vault edit ansible/group_vars/all/vault.yaml` first.
**Maintenance / taking the gateway down deliberately**: `sudo systemctl
stop wg-quick@pia-wg` on minisforum closes VLAN 50 Internet egress
immediately (kill switch stays engaged) without touching minisforum's own
networking, k3s, or any other workload — this is the same fail-closed path
exercised by the Phase 5 failure tests in `plan.md`.
**Multus troubleshooting**: check `kubectl -n kube-system get pods -l
app=multus -o wide` (should be Running, only on `nik-debian`), then
`/var/lib/rancher/k3s/agent/etc/cni/net.d/` on nik-debian for a generated
`00-multus.conf` — if it's missing, Multus isn't actually in the CNI
chain regardless of pod status. See `manifests/multus/README.md` for the
k3s-specific path gotchas that make this different from a stock Multus
install.
**Monitoring**: currently journald only
(`journalctl -u pia-gateway-healthcheck` on minisforum) — nothing is wired
into the existing kube-prometheus-stack (`values/kube-prometheus-stack
.yaml` has no custom node-exporter textfile-collector or
`additionalPrometheusRulesMap` configured, and none was added by this
work). Wiring real alerting would mean enabling node-exporter's textfile
collector for that host and adding a `PrometheusRule` — not done, flagged
here as the natural next step rather than guessed at.
**Rollback (full)**: work through the table above bottom-to-top — restore
browser-vpn-proxy/qBittorrent/JDownloader from git history first (fastest
to undo, keeps service running — see the table's Rollback column), then
remove the Multus/NAD resources, then `ansible/roles/vlan50-parent`'s
rollback, then `ansible/roles/pia-gateway`'s rollback. None of these
steps touch `pia-credentials`, PVCs, or the normal k3s/Argo CD control
plane.
## Gotchas
- **`manifests/media/qbittorrent.yaml` and `jdownloader.yaml` deploy the
moment they're pushed to `main`** — the `media` Argo CD Application
already has `selfHeal`/`automated` sync, unlike the new `multus`
Application (deliberately manual-sync). Double-check which gate applies
before pushing changes to either file.
- Argo CD Applications mostly set `prune: false`; removing resources from Git may
require manual cleanup.
- Gitea uses a manual public `IngressRoute`; `values/gitea.yaml` has no
@ -375,20 +256,8 @@ plane.
- Authentik forward-auth depends on the `Cookie` header in
`authRequestHeaders`; removing it causes redirect loops.
- CoreDNS custom zone snippets must use the `.server` key suffix.
- Renewing `internal-ca-cert` by deleting its secret (instead of `cmctl
renew`) generates a new key pair, silently breaking every leaf cert and
every device's trust at once — see "Renewing the internal CA" under [TLS
and Trust](#tls-and-trust) for the recovery/prevention procedure.
- The Dashy manifest contains an empty ConfigMap shell; use
`manifests/core/apply-dashy-config.sh` to load the real config.
- A newly pushed Argo CD `Application` (a new file under `argocd/apps/`)
doesn't create its namespace/resources the instant you push — `app-of-apps`
only picks it up on its next Git poll. Running a namespace-scoped
`registry-secret.sh` (or anything else assuming the namespace already
exists) right after pushing can fail with `namespaces "<ns>" not found`;
either wait for the next poll or force one: `kubectl patch application
app-of-apps -n argocd --type merge -p
'{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'`.
See the scoped READMEs in `ansible/`, `argocd/`, and `manifests/` for workflows
specific to those directories.

318
REFACTOR_PLAN.md Normal file
View File

@ -0,0 +1,318 @@
# Homelab Repo Audit & Refactor Plan
Living document. Produced by a full read-only audit of the repo (Ansible, Argo
CD, manifests, values, config, router). Nothing was applied, synced,
committed, or pushed as part of producing this. We work through the
"Migration Plan" stages one at a time, referencing this file; update the
Status column as stages land.
**Audit constraints honored**: `.env` was never opened; `router/backup-base.tar.gz`
was never extracted; no `kubectl apply` / `helm install` / Argo CD sync /
Ansible-against-real-hosts was run. All findings are grounded in file
citations, and the highest-severity ones were independently re-verified
(not just taken from sub-agent research output) before being recorded here.
**Immich removed entirely, 2026-07-23**: not used enough to justify chasing
its stale-NFS-handle issue. `manifests/media/immich.yaml` (+ its secret
script/sealed secret), `immich.home.arpa` DNS entries, its Dashy tile, and all
other repo references are gone. Findings #16 and #20 that mentioned Immich
are now moot for it specifically (kept as historical record, not rewritten).
Live namespace/PVCs need a manual `kubectl delete namespace immich` to
actually free the data -- Argo CD's `media` Application won't do this itself
(`prune: false`).
## Stage Tracker
| Stage | Scope | Status |
|---|---|---|
| 1 | Safety fixes & secret hygiene | Files done, committed locally, **not pushed**. Actual secret rotation (new password/API key values + live DB password change) still needs the user — see note below. |
| 2 | Validation tooling & clean baseline | Done. `yamllint`/`ansible-lint`/`kubeconform` installed by user; `.yamllint.yml`/`.ansible-lint` added; baseline clean (0 yamllint issues, `kubeconform`: 117+20 resources valid, all playbooks pass `--syntax-check`). Remaining `ansible-lint` findings are either pre-existing role/var naming conventions (out of scope — would require repo-wide renames) or collections-not-installed noise that resolves once `ansible-galaxy collection install -r ansible/requirements.yml` is run. |
| 3 | Remove confirmed junk | Done, committed locally. `.DS_Store` confirmed never tracked (no action needed). |
| 4 | Ansible cleanup | Done, committed locally. |
| 5 | Argo CD bootstrap normalization | Files done, committed locally, **not pushed — highest-risk stage**. `authentik.yaml`'s chart version is a placeholder needing your input (`helm list -n authentik`); every new/changed Application here needs `kubectl diff` one at a time before/after enabling, not a blind batch push. |
| 6 | Values/DNS consolidation | Done, committed locally, **not pushed**. |
| 7 | Kubernetes correctness & security | Resource requests/limits + probes done (finding #16), committed locally, **not pushed**. `gitea-backup` RBAC narrowing (#11) done as part of Stage 5's commit. Grafana `runAsNonRoot`/`fsGroup` test (#25) and `:latest` image pinning not done — both need a live test window / registry inspection I didn't do unprompted. |
| 8 | Documentation & DR runbook | Done, committed locally. |
| 9 | CI & Claude Code guidance | Done, committed locally, **not pushed** (new CI automation surface). |
**Correction, 2026-07-23**: the original `existingSecret`-based fix for Finding #3 was wrong and caused a real Gitea outage. Confirmed directly from the chart source (`gitea/helm-gitea` `templates/gitea/config.yaml`): `{{ set .Values.gitea.config.database "PASSWD" .Values.postgresql.global.postgresql.auth.password }}` — Gitea's own `[database]` config is *always* built from this plaintext values field, regardless of `existingSecret` (which only affects the bundled postgresql subchart's own credential provisioning, a separate concern). Removing the plaintext field made it silently fall back to the chart's built-in default password, breaking every fresh Gitea pod's DB connection. `values/gitea.yaml` is back to a plaintext (rotated) password with a comment explaining why, and `manifests/gitea/gitea-postgres-secret.sh` was removed. Getting this fully off plaintext would need a Config Management Plugin or similar — not attempted here; flagging as a real open item if it matters enough to invest in.
Dashy's API key fix (env-injection via `manifests/core/apply-dashy-config.sh`) is unaffected by this and still needs you to rotate the actual key value.
### New findings from this session's live-cluster checks (not in the original audit)
- **Grafana and Loki's static PVs are orphaned.** `manifests/monitoring/monitoring-pvs.yaml` defines `grafana-pv`/`loki-pv` hostPath PVs, but live `kubectl get pv` shows both sitting `Available` (unbound) — their Helm-managed PVCs got dynamically provisioned via `local-path` instead, unlike `prometheus-pv` which binds correctly. Data isn't lost, just not where the README/manifest imply. Needs a decision: bind properly (data migration required) or drop the orphaned static PVs and document reality. Not yet actioned.
- **Finding #24 (GPU passthrough) is resolved as working, not broken.** Live check on node `nik-gpu`: `nvidia.com/gpu: "1"` allocatable, `nvidia` runtime handler registered in containerd, `nvidia-device-plugin` pod `Running`. The orphaned-looking `k3s-containerd-config.toml.j2` template is a non-issue in practice — no action needed.
- **Unknown #1 (Grafana PVC) resolved**: dynamically provisioned via `local-path`, not the static PV — see the orphaned-PV finding above, same root cause.
- Node hostnames in the live cluster are `nik-debian`/`nik-gpu`, not `debian`/`gpu-node` as in `ansible/inventory.yaml` — cosmetic (K3s registers by actual hostname), doesn't affect anything, just noted for anyone cross-referencing `kubectl get nodes` against the docs.
Open decisions needed from the user before/during the relevant stage:
- Finding #5 (`router/backup-base.tar.gz`): needs manual review outside this
workflow; may imply git-history scrub / router credential rotation.
- Finding #9 (K3s version skew): which version is canonical (`v1.32.2+k3s1`
vs `v1.32.4+k3s1`)?
- Finding #18: is `prune:true` on `home-services`/`otel-collector`/`tempo`
and `selfHeal:false` on both pihole Applications intentional policy?
- Finding #25: Grafana `runAsUser: 0` — worth testing `fsGroup`-only instead?
---
## Phase 1 — Current-State Architecture
```
Ansible (hosts) Argo CD (cluster)
───────────────── ──────────────────────────────────────────
minisforum (K3s server, .77) ──┐
debian (K3s agent, .183) ────┼─▶ K3s cluster ──▶ helm install argocd (manual, README.md:92-103)
gpu-node (K3s agent, .98) ──┘ │ │
mac-mini (standalone Docker) │ ▼
│ kubectl apply -f manifests/argocd/app-of-apps.yaml (manual, one-time)
│ │
│ ▼
│ Application: app-of-apps → watches argocd/apps/*.yaml
│ │
│ ┌───────────┴────────────────────────────┐
│ ▼ ▼
│ Helm-chart Applications Raw-manifest Applications
│ (chart + values/<name>.yaml) (path: manifests/<area>)
│ traefik, cert-manager*, gitea, core, media, home-services,
│ pihole, pihole-debian, monitoring, portfolio, homeassistant,
│ loki, tempo, otel-collector, *-secrets (sealed only)
│ sealed-secrets, argocd(self)
│
└── NOT reconciled by Argo CD at all:
manifests/cert-manager/ (ClusterIssuers)
manifests/authentik/ + values/authentik.yaml
4 files in manifests/network/ (non-sealed)
→ applied by hand, outside GitOps
```
Host-level services that never touch the K3s cluster: Home Assistant (Docker
Compose + systemd on `minisforum`, `ansible/roles/homeassistant/`), Watch
Party and Ollama (`mac-mini`), Ollama (`gpu-node`) — all Ansible-managed.
Bootstrap sequence as documented (`README.md:71-105`, `argocd/README.md:7-24`):
Ansible playbooks → manual `helm install argocd` → manual `kubectl apply` of
app-of-apps → everything else via Git. **This documented sequence is
incomplete** — it never mentions installing Authentik or the cert-manager
`ClusterIssuer`s, both required for the rest of the stack to actually work
(see Critical findings below).
## Executive Summary
The repo is coherently designed for a single-operator homelab: Ansible for
host bootstrap, Argo CD app-of-apps for cluster reconciliation, a clean
two-pattern secrets model (sealed-secrets for committed ciphertext, runtime
scripts for `.env`-sourced live secrets), and a genuinely useful pair of
existing Claude Code subagents (`infra-change-reviewer`, `secrets-leak-scanner`)
that already encode a lot of hard-won tribal knowledge. There's no
fashionable-but-wrong abstraction problem here — the biggest risks are **gaps
between what the documentation/GitOps model claims and what's actually wired
up**: two foundational subsystems (cert-manager's ClusterIssuers, and the
entire Authentik SSO stack) are invisible to Argo CD despite everything else
depending on them, a live database password and a live third-party API key
are committed in plaintext, and two high-value internal certificates are
silently issued by the wrong CA chain. None of this requires a restructure to
fix — it requires closing specific, identifiable gaps. Ansible is in good
shape except for a dual-`ansible.cfg` trap that breaks documented commands if
run from the wrong directory, and one likely-broken kubeconfig-fetch task in
the K3s bootstrap role. No CI or local validation tooling exists yet, which
is the main reason several of these gaps went unnoticed.
## Prioritized Findings
### Critical
| # | Finding | Evidence | Impact | Recommendation |
|---|---|---|---|---|
| 1 | `manifests/cert-manager/` (both `ClusterIssuer`s + the internal CA `Certificate`) is never referenced by any Argo CD `Application` — confirmed by grep across all of `argocd/apps/*.yaml`, zero `path:` hits | `argocd/apps/cert-manager.yaml` only installs the Helm chart/controller; no sibling Application points at `manifests/cert-manager/` | Every TLS certificate in the cluster depends on issuers that exist only because someone once ran `kubectl apply` by hand. A cold rebuild following the documented bootstrap steps will **not** restore them, and nothing detects/corrects drift on the live ones. | Add `argocd/apps/cert-manager-config.yaml` (path: `manifests/cert-manager`, sync-wave before consumers). See Stage 5. |
| 2 | `manifests/authentik/` (7 files: ingress, proxy outpost, middleware, 4 secret scripts) and `values/authentik.yaml` are never referenced by any Application, and Authentik's manual `helm upgrade --install` isn't even in the documented bootstrap steps | Confirmed by the same grep sweep; `values/authentik.yaml:1`'s header comment is the *only* place the install command lives | The SSO IdP gating Argo CD, Grafana, and Gitea OAuth is entirely outside GitOps and undocumented as a bootstrap step. Disaster recovery would silently fail to restore login for everything behind it. | Add `argocd/apps/authentik.yaml` (chart) + `argocd/apps/authentik-config.yaml` (path: `manifests/authentik`), mirroring the cert-manager two-Application pattern. |
| 3 | Plaintext PostgreSQL password committed | `values/gitea.yaml:48` — `password: gitea-db-password` under `postgresql.global.postgresql.auth.password`, no `existingSecret` used (contrast `values/gitea.yaml:11`, which correctly uses `existingSecret: gitea-admin-secret`) | A real credential sits in git history now, readable by anyone with repo access. Contradicts the repo's own stated no-plaintext-secrets rule and `.claude/agents/secrets-leak-scanner.md`'s invariant. | Rotate the DB password, convert to a `*-secret.sh` → sealed-secret pattern (mirror `manifests/media/immich-postgres-secret.sh`). Treat the current value as compromised since it's in history. |
| 4 | Live third-party API key committed in plaintext | `config/dashy/conf.yaml:63` and `:70` — a weather-widget API key used twice | Anyone with repo read access (or Dashy's own JS, which ships client-side) can extract and reuse the key. | Rotate the key at the provider; inject at apply-time from `.env` via `manifests/core/apply-dashy-config.sh` instead of hardcoding in `config/dashy/conf.yaml`. |
| 5 | `router/backup-base.tar.gz` is tracked in git (added in commit `e7718ce`, ~8.8KB compressed) | Confirmed tracked via `git ls-files`; **not opened**, per safety constraints | `router/uci-base.conf` itself (which was read) is a sanitized base config — WiFi disabled with default open SSID, no VPN configured, sentinel password field — which makes it likely the fuller `.tar.gz` backup is where any real WiFi PSK/VPN key/admin credential would actually live. Cannot confirm what's inside without extracting it. | **Action needed from user**: inspect this archive outside this workflow. If it contains live credentials, both git-history scrubbing and router credential rotation should be considered. |
| 6 | Two high-value internal certificates are issued by the wrong CA | `manifests/authentik/authentik-ingress.yaml:11-12` (`auth.home.arpa`) and `manifests/network/traefik-dashboard-ingress.yaml:65-66` (`traefik.home.arpa`) both set `issuerRef.name: internal-ca` — the root self-signed issuer — instead of `internal-ca-issuer`, the chained issuer every other internal `Certificate` uses (verified pattern in `manifests/homeassistant/homeassistant.yaml:31`, `manifests/home-services/certs.yaml`, `manifests/argocd/argocd.yaml:9`, media manifests) | Devices that installed the CA via the `ca-installer`/`ca-sync` flow (`manifests/README.md:36-42`) will **not** trust these two certs — browser TLS warnings on exactly the SSO login and the Traefik dashboard. Compounded by #1/#2: neither file is under GitOps, so nothing corrects this automatically. | Change `issuerRef.name` to `internal-ca-issuer` in both files. |
### High
| # | Finding | Evidence | Impact | Recommendation |
|---|---|---|---|---|
| 7 | Dual `ansible.cfg` breaks every documented playbook command if run from `ansible/` instead of repo root — empirically reproduced, not just theorized | `ansible.cfg` (root) vs `ansible/ansible.cfg`; confirmed via `ansible-config dump` that only one loads at a time, and reproduced two independent failures (path-doubling, then role-not-found) plus a silent loss of `host_key_checking=False` | A user who `cd`s into `ansible/` first (a very natural mistake) gets confusing, unrelated-looking errors, or worse, an interactive host-key prompt in what's meant to be scriptable. | Delete `ansible/ansible.cfg`; fold its two settings (`inject_facts_as_vars: False`, `deprecation_warnings: False`) into the root `ansible.cfg`. Single source of truth, run only from repo root (already the documented convention). |
| 8 | Likely-broken kubeconfig path in K3s bootstrap | `ansible/roles/k3s-server/tasks/main.yaml:50-62` — fetches kubeconfig to `~/.kube/config`, but the next task's `ansible.builtin.replace` targets `/tmp/k3s-minisforum.yaml`, a path nothing else in the role writes to | `replace` errors if the target doesn't exist — this task likely fails on a genuinely fresh `bootstrap-minisforum.yaml`/`setup-k3s.yaml` run, i.e. exactly the disaster-recovery path this needs to work for. **Static-analysis finding, not yet confirmed against a live re-run** — flagged as an unknown too. | Fix the path mismatch (verify what the fetch task actually names the local file and point `replace` at that same path); test on an actual rebuild before trusting this for DR. |
| 9 | K3s version defined in 3 places, already out of sync | `ansible/roles/k3s-server/defaults/main.yaml:6` and `k3s-agent/defaults/main.yaml:6` both `v1.32.2+k3s1`; `ansible/host_vars/gpu-node.yaml:35` is `v1.32.4+k3s1`. `ansible/README.md:58-59`'s upgrade doc only mentions the first two locations. | A fresh `debian`/`minisforum` (re)provision installs a different K3s version than what's apparently already running on `gpu-node` — real version-skew risk, and the documented upgrade procedure would miss the gpu-node override entirely. | Pick one canonical version, align all three, document all three locations. |
| 10 | 4 of 6 files in `manifests/network/` are excluded from Argo CD sync | `argocd/apps/network-secrets.yaml:14-15` sets `directory.include: '*-sealed.yaml'`, so `ddns-cronjob.yaml`, `glances-debian-ingress.yaml`, `traefik-dashboard-ingress.yaml`, `watch-party-ingress.yaml` are never synced | These back real, documented services (`README.md:52,63,64`) with zero drift protection or self-heal — same class of gap as #1/#2, smaller blast radius. | Add a proper `network.yaml` Application (or widen the include filter) covering these; keep `network-secrets.yaml` scoped to sealed secrets only. |
| 11 | `gitea-backup` CronJob has cluster-wide exec privileges it doesn't need | `manifests/gitea/gitea-backup.yaml:11` (`ClusterRole`) / `:23` (`ClusterRoleBinding`) grant `pods/exec` cluster-wide; the CronJob script only execs into the `gitea` namespace (lines 61-72) | The backup ServiceAccount can exec into any pod in any namespace — real privilege-escalation surface for a component whose only job is backing up one namespace. | Replace with a namespaced `Role`/`RoleBinding` scoped to `gitea`. |
| 12 | `homeassistant` Application/manifest namespace mismatch | `argocd/apps/homeassistant.yaml:14` targets/creates namespace `homeassistant`; every resource in `manifests/homeassistant/homeassistant.yaml` is hardcoded to `namespace: default` (lines 6,17,27,40) — verified directly | Argo CD creates and "manages" an empty, unused `homeassistant` namespace while the real resources live in `default`. Confusing, latent bug, no current functional harm. | Change `destination.namespace` to `default` and drop `CreateNamespace=true` (no runtime-affecting change — resources already live in `default`). |
| 13 | `gitea-backup` namespace has no `Namespace` object anywhere and isn't auto-created | `manifests/gitea/gitea-backup.yaml:8,29,39,100` all reference namespace `gitea-backup`; the owning Application (`gitea-secrets.yaml`) targets namespace `gitea` with `CreateNamespace=true` — a different namespace | Would fail to apply on any cluster where `gitea-backup` namespace wasn't created by hand out-of-band. | Add an explicit `Namespace: gitea-backup` object to the manifest, or fold the backup CronJob into the `gitea` namespace. |
| 14 | No CI or local validation tooling anywhere in the repo | Confirmed via recursive search: no `.github/`, `.gitea/workflows/`, pre-commit config, or yamllint/ansible-lint config at any level. `yamllint`, `ansible-lint`, `kubeconform` not installed locally either (only `helm`, `kubectl`, `kubeseal`, `jq`, `ansible-playbook` are present) | Given "no staging environment," every change is validated by pushing and watching Argo CD/production react. This is the root cause that let findings #3, #4, #20 go unnoticed. | See Stage 2. |
| 15 | Missing `requirements.yml` + undocumented collection dependency | No `requirements.yml` anywhere; `README.md:76-78` only tells a new operator to install `community.general`/`ansible.posix`, but `ansible/roles/glances/tasks/main.yaml:24` and `ansible/roles/watch-party/tasks/main.yaml:27` require `community.docker`, never mentioned | A fresh workstation following the documented bootstrap verbatim gets a "module not found" failure on `setup-glances-debian.yaml`/`deploy-watch-party.yaml`. | Add `ansible/requirements.yml` pinning all three collections; update `README.md`. |
| 16 | Missing resource requests/limits and probes on a long list of workloads | `manifests/core/{dashy,glances}.yaml`, `manifests/core/ca-installer/ca-installer.yaml`, `manifests/authentik/authentik-proxy-outpost.yaml`, `manifests/media/jellyfin.yaml`, main containers in `qbittorrent.yaml`/`jdownloader.yaml` (only their `gluetun` sidecars have probes), all 4 Deployments in `manifests/media/immich.yaml` | No protection against one workload starving others on a small, fixed-capacity cluster; no automatic restart on hang for several user-facing services. Contrast: `manifests/home-services/*.yaml` and `manifests/portfolio/portfolio.yaml` do this correctly already — good templates to copy from. | See Stage 7. |
### Medium
| # | Finding | Evidence | Recommendation |
|---|---|---|---|
| 17 | Sync-wave ordering doesn't cover cert-manager/sealed-secrets vs. their consumers | Only `gitea-secrets`/`gitea`/`loki`/`monitoring-secrets`/`monitoring`/`network-secrets` carry `sync-wave` annotations; `cert-manager` and `sealed-secrets` (whose CRDs everything else's `Certificate`/`SealedSecret` objects need) are unannotated, same wave-0 as their consumers | Add `sync-wave: "-2"` to `cert-manager`/`sealed-secrets`, `-1` to their config Applications, consistent with the existing pattern. Self-heals today via retries, so this is about shortening a fresh-bootstrap flaky window, not fixing active breakage. |
| 18 | Undocumented sync-policy exceptions | `home-services`, `otel-collector`, `tempo` use `prune: true` against the repo's stated `prune: false` norm (`argocd/README.md:37`); `pihole`/`pihole-debian` are the only two with `selfHeal: false` (plausibly intentional, given the documented external-IP-loss gotcha, but never stated as policy) | Add a one-line YAML comment on each explaining the exception, or normalize them if unintentional — needs user input on intent. |
| 19 | 3 Applications use `targetRevision: HEAD` instead of the documented `main` | `home-services.yaml:12`, `otel-collector.yaml:17`, `tempo.yaml:17` | Normalize to `main`. Zero behavioral difference today; purely consistency. |
| 20 | Pi-hole DNS entries have already drifted | `values/pihole.yaml:81,85` (`immich.home.arpa`, `gluetun.home.arpa`) missing from `values/pihole-debian.yaml` | Add the two missing entries. (Going forward this is now caught by the existing `PostToolUse` hook in `.claude/settings.json` — it just doesn't catch pre-existing drift.) |
| 21 | Orphaned DNS record | `nik4nao.home.arpa` exists in both pihole values files but no Certificate/Ingress/IngressRoute anywhere references it | Confirm it's unused, then remove. |
| 22 | `ansible/README.md` mislabels the `homeassistant` role "Legacy" | `ansible/README.md:54`; but `manifests/homeassistant/homeassistant.yaml`'s `Endpoints` (hardcoded to `192.168.7.77:8123`) has nothing else in the repo standing up a listener at that address — the "legacy" Ansible role/Docker Compose deployment is the **only** thing actually serving it | Re-word the doc entry; this is load-bearing, not dead — don't let a future cleanup pass delete it. |
| 23 | `docker` role logic duplicated wholesale inside `homeassistant` role | `ansible/roles/homeassistant/tasks/main.yaml:2-49` reimplements Docker CE install (Debian apt path) instead of reusing `ansible/roles/docker/tasks/main.yaml:6-43` (Ubuntu apt path) | Parameterize `docker` role for both distros, have `homeassistant` depend on it instead of duplicating. |
| 24 | GPU passthrough likely non-functional as wired | `ansible/roles/nvidia/templates/k3s-containerd-config.toml.j2` exists but is never referenced by any task in that role; `manifests/home-services/nvidia-device-plugin.yaml:5-9` defines a `RuntimeClass handler: nvidia` that depends on exactly that containerd config existing on `gpu-node` | **Unknown pending live verification** — needs a check on `gpu-node` before deciding whether to wire up the template or remove the RuntimeClass. |
| 25 | Grafana forced to run as root | `values/kube-prometheus-stack.yaml:49-52` — `runAsNonRoot: false`, `runAsUser: 0`, likely to support `initChownData: true` fixing hostPath ownership | Test whether `fsGroup` alone (without full root) suffices; needs a live test window with rollback ready. |
| 26 | Mixed `Ingress` vs Traefik `IngressRoute` usage with no stated policy | e.g. `manifests/media/qbittorrent.yaml` uses plain `Ingress` for the app (line 176) but `IngressRoute` for `gluetun-api` in the same file (line 216) | Not broken, but only the `IngressRoute` half gets native `Middleware` support without annotation workarounds — worth standardizing on one mechanism over time, not urgent. |
### Low (grouped)
- **Ansible hygiene**: zero `tags:` usage anywhere (no selective runs possible); FQCN usage inconsistent (`wireguard` and `homeassistant` roles are 100% bare module names, `ollama` is mixed — `ansible/roles/{wireguard,homeassistant,ollama}/tasks/main.yaml`); `nvidia/tasks/main.yaml:58-62` uses `changed_when: true` unconditionally, restarting Docker every run; `gitea-runner/tasks/main.yaml:89-93` unconditionally deletes `/run/docker.sock` every run despite the task name implying a conditional check.
- **Operational secret leaks (not committed, but printed)**: `ansible/roles/k3s-server/tasks/main.yaml:46-48` prints the real K3s join token via `debug: msg:` on every run; `ansible/roles/gitea-runner/tasks/main.yaml:55-79` writes the runner registration token into a world-readable (`0644`) systemd unit file; `ansible/roles/wireguard/tasks/main.yaml:169-184` prints generated WireGuard private keys to console. None are committed to git, but all are worth tightening.
- **Doc drift**: `README.md:203-204` claims Gitea ingress is "disabled in `values/gitea.yaml`" — that file has no `ingress:` key at all (it's disabled by chart default, not an explicit setting); `README.md:174-186` groups Gitea's backups under `/mnt/storage` when they actually live on a separate NFS export (`/home/nik/backups`, per `manifests/gitea/gitea-backup.yaml:92` and `ansible/roles/nfs-server/templates/exports.j2:7`); `ansible/README.md`'s role table and playbook list omit the `docker`/`nvidia` roles, `setup-gpu-node.yaml`, and `homeassistant.yaml`; root `README.md`'s host table omits `gpu-node` entirely as a 4th host; `config/dashy/conf.yaml:2` points at `manifests/apply-dashy-config.sh`, missing the real `core/` path segment.
- **`.gitignore` gaps** (nothing currently leaking, but no coverage): `*.retry` (a live risk — Ansible drops these on playbook failure and this repo's Ansible tree is actively used), `.vault_pass*`, kubeconfig-shaped filenames, editor swap files. `.DS_Store` files exist in the working tree at repo root and `config/` despite being gitignored — confirm they're actually untracked, not just ignored-going-forward.
## Duplicated / Ambiguous / Possibly Obsolete Resources
Per the "trace references before declaring obsolete" rule — **none of the
"unsynced" items above are actually dead**; they're all in active use, just
invisible to Argo CD. The only things that look genuinely obsolete or
redundant are:
- ~~`nik4nao.home.arpa` DNS entry~~ — **WRONG, corrected 2026-07-23**: this
points at `192.168.7.183` (nik-debian's raw IP) for a host-level SMB/Samba
share, not a Kubernetes Service — there was never going to be a
Certificate/Ingress/IngressRoute for it, so "no K8s resource references it"
was the wrong test entirely. It was removed from both pihole values files
and broke real SMB access; restored. Lesson: that specific check only
proves a hostname isn't routed through Traefik, not that it's unused —
doesn't apply to raw-IP entries for non-HTTP/non-K8s services.
- `ansible/ansible.cfg` — not obsolete exactly, but its only real-world effect today is negative (breaks documented commands); candidate for deletion/merge into the root cfg.
- `ansible/roles/nvidia/templates/k3s-containerd-config.toml.j2` — orphaned template, either finish wiring it up or remove it once GPU passthrough status is confirmed live.
- `.DS_Store` at repo root and in `config/` — should never have been committed; harmless but should be `git rm --cached`.
## Security Findings Summary (redacted)
- Two real credentials committed in plaintext: a database password (`values/gitea.yaml:48`) and a third-party API key (`config/dashy/conf.yaml:63,70`). Both should be treated as compromised and rotated — no value is printed anywhere in this document.
- One tracked binary (`router/backup-base.tar.gz`) whose contents were not inspected but is flagged as the most likely place real WiFi/VPN/router-admin credentials would live, given the sibling `uci-base.conf` is sanitized. Needs review outside this workflow; rotation and/or history-scrubbing may be warranted depending on what's inside.
- No plaintext secrets found in Ansible beyond the two operational-leak items (console-printed token, world-readable systemd unit) — the vault (`ansible/group_vars/all/vault.yaml`) is genuinely encrypted and used correctly.
- RBAC: one real over-scope (`gitea-backup`'s cluster-wide `pods/exec`), no wildcard verbs/resources found anywhere else.
- Existing guardrails already partially cover this class of problem going forward: `.claude/agents/secrets-leak-scanner.md` and the `PostToolUse` hook in `.claude/settings.json` — but the scanner's stated scope reads as Ansible/manifests/Helm-values-centric and doesn't explicitly call out `config/**`, which is exactly where the Dashy API key slipped through. Worth widening its scope statement rather than adding new tooling.
## What Should Be Preserved
- The Ansible → Argo CD boundary itself is clean — no Ansible task embeds Kubernetes YAML or runs `kubectl apply` against application manifests (verified by repo-wide grep). The one-time hand-off (`helm install argocd` → `kubectl apply -f app-of-apps.yaml`) is exactly right for this scale.
- The app-of-apps + one-Application-per-file pattern (20 flat files in `argocd/apps/`) is simple, greppable, and easy to reason about at this service count — an ApplicationSet or Kustomize layer would add indirection with no real benefit here.
- The two-pattern secrets model (sealed-secrets for committed ciphertext, runtime `.env`-sourced scripts for cluster-only secrets) is coherent and consistently documented across all three READMEs.
- Exact Helm chart version pinning on every chart-backed Application — zero floating versions found.
- No deprecated Kubernetes API versions anywhere — clean.
- Idempotent `curl | sh`-style installs are consistently guarded with `creates:` across `docker`, `nvidia`, `k3s-server`, `k3s-agent`, `ollama` roles.
- `.claude/agents/infra-change-reviewer.md` and `secrets-leak-scanner.md` plus the `settings.json` hooks already encode a lot of this audit's tribal knowledge (DNS-sync rule, sealed-secret hand-edit ban, cert-issuer split, GPU taint/toleration, Ansible dual-cfg risk) — build on these, don't replace them.
- Good in-repo examples worth using as the template when fixing the bad ones: `manifests/home-services/nvidia-device-plugin.yaml` (privilege drop), `manifests/core/ca-installer/ca-sync.yaml` (tightly-scoped namespaced RBAC), `manifests/home-services/{ai-gateway,ha-gateway}.yaml` + `manifests/portfolio/portfolio.yaml` (proper resources + probes).
## Unknowns (need live cluster info or user input — not determinable from the repo alone)
1. Whether Grafana's PVC actually binds to the static `grafana-pv` hostPath or silently gets dynamically provisioned via `local-path` instead (`values/kube-prometheus-stack.yaml` sets no `storageClassName`/`volumeName` for Grafana, unlike Prometheus/Loki which do). Needs `kubectl get pvc -n monitoring` / `kubectl get pv grafana-pv -o yaml`.
2. Whether GPU passthrough is actually functional today given the orphaned containerd template (finding #24). Needs a check on `gpu-node`.
3. What's actually inside `router/backup-base.tar.gz` (finding #5) — deliberately not inspected.
4. Whether the K3s bootstrap kubeconfig-path bug (finding #8) actually breaks a fresh run, or whether there's missing context from an untraced earlier step. Static reading only — recommend confirming before relying on it for DR.
5. Whether `selfHeal: false` on the two pihole Applications is deliberate policy (tied to the external-IP-loss gotcha) or an oversight.
6. What Authentik Helm chart version is actually running — it's outside Argo CD, so there's no `targetRevision` to read; needs `helm list -n authentik` (or wherever it's installed) on the live cluster.
7. Whether `values/gitea.yaml`'s committed password and Dashy's committed API key have ever been exposed beyond the operator (repo visibility/access history) — affects how urgent rotation is.
---
## Phase 4 — Target Architecture
Given this is a single cluster, single operator, ~20-service homelab with no
near-term multi-cluster plan, two credible options were considered.
### Option A (recommended): Keep the current three-way split, close the gaps
Keep `ansible/` / `argocd/apps/` / `manifests/` / `values/` exactly as they
are structurally — this layout already matches how the system is organized
conceptually, how the existing subagents and skills reference paths, and how
the current documentation is written. Fix the specific holes (findings
above) rather than moving files around.
```
homelab/
├── ansible.cfg # single config, root-only (ansible/ansible.cfg removed)
├── ansible/
│ ├── requirements.yml # NEW — pins community.general, ansible.posix, community.docker
│ ├── inventory.yaml
│ ├── group_vars/ · host_vars/
│ ├── playbooks/
│ └── roles/
├── argocd/
│ ├── apps/ # + cert-manager-config.yaml, authentik.yaml, authentik-config.yaml
│ └── values/
├── manifests/ # unchanged tree; every subdir now has a matching Application
├── values/ # unchanged
├── config/dashy/
├── router/
├── .yamllint.yml # NEW
├── .ansible-lint # NEW (config only, tool installed separately)
└── CLAUDE.md # + validation commands section
```
**Why not restructure further**: there's no duplication cost today that a
`clusters/`/`infrastructure/`/`apps/` split or Kustomize bases would actually
reduce — each service already has exactly one Application file and at most
one values file. Introducing Kustomize overlays or ApplicationSets would add
a templating/indirection layer this repo doesn't need at 1-cluster,
~20-service scale, and would itself become something to maintain and
explain.
### Option B (not recommended now): `clusters/` + `infrastructure/` + `apps/`, colocated values
Restructure to one directory per service (e.g.
`apps/gitea/{application.yaml,values.yaml}`), split `infrastructure/`
(cert-manager, sealed-secrets, traefik, pihole) from `apps/` (user-facing
services), under a `clusters/homelab/` root to leave room for a future
second cluster. This gives slightly tighter per-service ownership (one
directory instead of two: `argocd/apps/x.yaml` + `values/x.yaml`) but
requires moving **every** Application's `source.path`, which Argo CD treats
as a new resource identity — each move is a prune-risk event that must be
done one Application at a time with `prune: false` verified beforehand,
diffed, and rolled out carefully. That's a lot of migration risk for a
benefit (one fewer directory hop per service) that doesn't solve any problem
that exists today, and it presumes a multi-cluster future that isn't
planned.
**Recommendation: Option A.** Revisit Option B only if a second cluster is
actually added or the service count grows to where cross-referencing
`argocd/apps/` and `values/` by hand becomes genuinely painful — neither is
true today.
---
## Phase 5 — Incremental Migration Plan
Every stage is designed to leave the repo in a working, deployable state at
every commit boundary. Stages that touch what Argo CD currently manages are
explicitly marked **RUNTIME-AFFECTING** and require explicit go-ahead before
pushing, per the repo's own rule that `git push` is a production deploy.
| Stage | Scope | Key changes | Runtime impact | Rollback |
|---|---|---|---|---|
| **1. Safety fixes & secret hygiene** | Findings #3, #4, #6, plus `.gitignore` gaps | Rotate + re-secret the Gitea DB password and Dashy API key; fix the two `issuerRef` values; extend `.gitignore` (`*.retry`, `.vault_pass*`, kubeconfig patterns, swap files); `git rm --cached` the `.DS_Store` files | **RUNTIME-AFFECTING** for the password rotation (Gitea/Postgres restart) and cert reissuance (brief TLS transition on 2 hostnames); everything else is zero-impact | Revert commit; for password rotation, keep old secret retrievable until new one is confirmed working |
| **2. Validation tooling & clean baseline** | New root-level config | Add `.yamllint.yml`, `.ansible-lint`; install `yamllint`/`ansible-lint`/`kubeconform` locally (asking first, since that's installing software); run baseline, fix anything trivial it finds | None — local/offline only | Uninstall tools / delete config files |
| **3. Remove confirmed junk** | `.DS_Store`, orphaned `nik4nao.home.arpa` DNS entry, doc-path typo in `config/dashy/conf.yaml:2` | Delete/clean | Negligible (DNS entry removal is additive-safe to revert) | `git revert` |
| **4. Ansible cleanup** | Findings #7, #9, #15, plus low-severity items | Merge `ansible/ansible.cfg` into root; add `requirements.yml`; align `k3s_version` across 3 locations; fix `gitea-runner` docker.sock task, `nvidia` `changed_when`, K3s token debug print, systemd unit permissions; update both READMEs (gpu-node host, docker/nvidia roles, homeassistant status) | **RUNTIME-AFFECTING** only for the `k3s_version` alignment (changes what a future node provision installs) — needs sign-off on which version is canonical; rest is inert until playbooks rerun | Git revert; k3s_version change only takes effect on next actual provision, not immediately |
| **5. Argo CD bootstrap normalization** | Findings #1, #2, #10, #12, #13, #17, #19 | New `cert-manager-config.yaml` + `authentik.yaml`/`authentik-config.yaml` Applications (first-time adoption of already-live resources — `kubectl diff` before enabling automated sync on each); widen `network-secrets` coverage or add `network.yaml`; fix homeassistant namespace; add `gitea-backup` Namespace + narrow its RBAC to a Role; add missing sync-waves; normalize `targetRevision: HEAD`→`main` | **RUNTIME-AFFECTING, highest-risk stage** — bringing previously-unmanaged live resources under Argo CD for the first time. Roll out one Application at a time, diff first, watch sync status before moving to the next | Set the newly-added Application's sync policy back to manual, or delete the Application (resources stay, since `prune: false`) |
| **6. Values/DNS consolidation** | Finding #18, #20 | Reconcile `pihole.yaml`/`pihole-debian.yaml` drift (add missing entries); document or normalize the `prune:true`/`selfHeal:false` outliers (pending intent) | **RUNTIME-AFFECTING but low-risk** — adds DNS records only, additive | Revert values file |
| **7. Kubernetes correctness & security** | Findings #11, #16, #25, #26 | Add resource requests/limits + probes to flagged workloads (one service at a time); narrow `gitea-backup` RBAC; test Grafana `runAsNonRoot`/`fsGroup`-only; pin remaining `:latest` images to currently-running tags | **RUNTIME-AFFECTING** — limits set too low can OOMKill; needs `kubectl top` baselines first (gather live before proposing numbers); roll out one service at a time with rollback ready | Revert manifest, resync |
| **8. Documentation & DR runbook** | All doc-drift items | Fix host tables, role tables, playbook lists, ingress/storage claims; add an explicit cold-rebuild runbook reflecting the now-closed GitOps gaps | None | Revert |
| **9. CI & Claude Code guidance** | New `.gitea/workflows/validate.yaml` (yamllint/ansible-lint/kubeconform on push); `CLAUDE.md` validation-commands section; widen `secrets-leak-scanner.md`'s stated scope to include `config/**`; consider extending the `PreToolUse` hook to block Claude Code from reading `.env`/`router/backup-base.tar.gz` directly | New automation surface | None to the cluster; new CI pipeline needs approval since it's a new automation surface | Delete workflow file |
Every stage gets validated with whatever's actually available
(`ansible-playbook --syntax-check`, `yamllint`, `ansible-lint`,
`kubeconform`/`kubectl apply --dry-run=client` where a live context isn't
required) before being reported done — with the concrete diff and validation
output shown, not just a claim of success.
---
## Claude Code Project Support
- **`CLAUDE.md`**: keep it, add a short "Validation commands" section once Stage 2 lands (exact `yamllint`/`ansible-lint`/`ansible-playbook --syntax-check` invocations).
- **Guard against accidental `.env`/key/backup access**: partially covered already (`.gitignore`, sealed-secret `PreToolUse` hook). Extend that hook to also block Claude Code `Read` of `.env` and `router/backup-base.tar.gz` — small addition, Stage 9.
- **One validation skill/command**: rather than a new skill, widen `secrets-leak-scanner.md`'s stated file scope to explicitly include `config/**` (the actual gap that let the Dashy API key through) — cheaper and more targeted than adding new tooling.

14
ansible/README.md
View File

@ -10,10 +10,10 @@ cluster.
| Group | Host | Purpose |
| --- | --- | --- |
| `k3s_server` | `minisforum` | K3s server at `10.10.40.53` |
| `k3s_agents` | `debian` | K3s agent and NFS storage at `10.10.40.20` |
| `mac_mini` | `mac-mini` | Docker/Ollama host at `10.10.40.30` |
| `gpu_workstation` | `gpu-node` | K3s agent with NVIDIA GPU passthrough at `10.10.40.12` (spot-tainted) |
| `k3s_server` | `minisforum` | K3s server at `192.168.7.77` |
| `k3s_agents` | `debian` | K3s agent and NFS storage at `192.168.7.183` |
| `mac_mini` | `mac-mini` | Docker/Ollama host at `192.168.7.96` |
| `gpu_workstation` | `gpu-node` | K3s agent with NVIDIA GPU passthrough at `192.168.7.98` (spot-tainted) |
All hosts use the `nik` user and the SSH key configured in `inventory.yaml`.
@ -49,8 +49,6 @@ ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/homeassistant.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/vlan50-parent.yaml -K
```
## Roles
@ -68,9 +66,7 @@ ansible-playbook -i ansible/inventory.yaml ansible/playbooks/vlan50-parent.yaml
| `glances` | Host-level Glances service |
| `ollama` | Ollama service on the Mac Mini and GPU node (branches on OS) |
| `watch-party` | Watch Party Docker Compose deployment on the Mac Mini |
| `wireguard` | WireGuard server configuration (inbound home-VPN access — phone/Mac clients) |
| `pia-gateway` | Minisforum's PIA WireGuard *egress* gateway for VPN VLAN 50 (policy routing, kill switch, health checks) — see its own README |
| `vlan50-parent` | nik-debian's tagged VLAN 50 parent interface (`enp1s0.50`) for Multus — see its own README |
| `wireguard` | WireGuard server configuration |
| `homeassistant` | Standalone Home Assistant deployment (Docker Compose + systemd on `minisforum`) — this is the **only** thing serving `ha.home.arpa`, not legacy/dead |
## Notes

13
ansible/group_vars/all/k3s.yaml
View File

@ -1,13 +0,0 @@
---
# Vars for: K3s cluster-wide DNS resolver and pod network
# Applied by: ansible/roles/k3s-server, ansible/roles/k3s-agent, ansible/roles/common
# Description: Single source of truth for the resolver IP written into
# /etc/rancher/k3s/resolv.conf on every K3s node, so CoreDNS's
# "forward . /etc/resolv.conf" always reaches Technitium regardless of
# which node CoreDNS is scheduled on, or that node's own (possibly stale)
# host resolvers. k3s_pod_cidr backs the routed UFW rules in the common
# role that allow Flannel pod-to-pod and pod-to-Technitium traffic through
# each node's routed/FORWARD chain.
k3s_dns_resolver: 10.10.40.53
k3s_pod_cidr: 10.42.0.0/16

6
ansible/host_vars/gpu-node.yaml
View File

@ -27,8 +27,8 @@ ufw_allowed_ports:
# agent's outbound connection to the k3s server on 6443, not a direct
# inbound connection).
- { port: "6443", proto: tcp, comment: "K3s API server" }
- { port: "10250", proto: tcp, comment: "Kubelet", src: "10.10.40.0/24" }
- { port: "8472", proto: udp, comment: "Flannel VXLAN", src: "10.10.40.0/24" }
- { port: "10250", proto: tcp, comment: "Kubelet" }
- { port: "8472", proto: udp, comment: "Flannel VXLAN" }
data_dirs:
- /data/tts-gateway
@ -38,7 +38,7 @@ nvidia_driver_version: "570"
cuda_version: "12-8"
# ── k3s-agent ──────────────────────────────────────────────────────────────────
k3s_server_url: "https://10.10.40.53:6443"
k3s_server_url: "https://192.168.7.77:6443"
k3s_node_token: "{{ vault_k3s_node_token }}"
# Check current cluster version with: k3s --version on minisforum

12
ansible/host_vars/minisforum.yaml
View File

@ -1,14 +1,2 @@
# Host vars for: minisforum (k3s server)
ansible_python_interpreter: /usr/bin/python3.13
# ── pia-gateway ────────────────────────────────────────────────────────────────
# Read directly from the repo-root .env (same PIA_USER/PIA_PASSWORD keys
# manifests/media/pia-secret.sh already uses) rather than ansible-vault —
# .env is already the established, gitignored, plaintext-at-rest secret
# source for this account elsewhere in the repo, and lookups run on the
# control node, so this never touches minisforum or gets committed.
# playbook_dir is always ansible/playbooks for every playbook in this
# repo, so ../../.env resolves to the repo root regardless of which
# playbook triggers evaluation of these two vars.
pia_user: "{{ lookup('ansible.builtin.file', playbook_dir + '/../../.env') | regex_search('^PIA_USER=(.*)$', '\\1', multiline=True) | first }}"
pia_password: "{{ lookup('ansible.builtin.file', playbook_dir + '/../../.env') | regex_search('^PIA_PASSWORD=(.*)$', '\\1', multiline=True) | first }}"

8
ansible/inventory.yaml
View File

@ -9,24 +9,24 @@ all:
k3s_server:
hosts:
minisforum:
ansible_host: 10.10.40.53
ansible_host: 192.168.7.77
ansible_port: 430
k3s_agents:
hosts:
debian:
ansible_host: 10.10.40.20
ansible_host: 192.168.7.183
ansible_port: 430
mac_mini:
hosts:
mac-mini:
ansible_host: 10.10.40.30
ansible_host: 192.168.7.96
ansible_python_interpreter: /usr/bin/python3
gpu_workstation:
hosts:
gpu-node:
ansible_host: 10.10.40.12
ansible_host: 192.168.7.98
ansible_port: 430
ansible_python_interpreter: /usr/bin/python3.12

6
ansible/playbooks/pia-gateway.yaml
View File

@ -1,6 +0,0 @@
---
- name: PIA WireGuard egress gateway for VPN VLAN 50
hosts: minisforum
become: true
roles:
- pia-gateway

6
ansible/playbooks/vlan50-parent.yaml
View File

@ -1,6 +0,0 @@
---
- name: Tagged VLAN 50 parent interface on nik-debian
hosts: debian
become: true
roles:
- vlan50-parent

6
ansible/roles/common/defaults/main.yaml
View File

@ -23,10 +23,8 @@ ufw_allowed_ports:
- { port: 80, proto: tcp, comment: HTTP }
- { port: 443, proto: tcp, comment: HTTPS }
- { port: 6443, proto: tcp, comment: K3s API server }
- { port: 10250, proto: tcp, comment: Kubelet, src: 10.10.40.0/24 }
- { port: 8472, proto: udp, comment: Flannel VXLAN, src: 10.10.40.0/24 }
- { port: 53, proto: udp, comment: WireGuard road-warrior DNS, src: 10.10.0.0/24 }
- { port: 53, proto: tcp, comment: WireGuard road-warrior DNS, src: 10.10.0.0/24 }
- { port: 10250, proto: tcp, comment: Kubelet }
- { port: 8472, proto: udp, comment: Flannel VXLAN }
data_dirs:
- /data/gitea

24
ansible/roles/common/tasks/main.yaml
View File

@ -1,7 +1,7 @@
---
# Part of role: common
# Called by: ansible/playbooks/bootstrap-minisforum.yaml
# Description: Sets timezone, installs base packages, creates user, hardens SSH, configures UFW (including routed pod-to-pod/pod-to-Technitium rules), and creates data directories.
# Description: Sets timezone, installs base packages, creates user, hardens SSH, configures UFW, and creates data directories.
- name: Set timezone
community.general.timezone:
@ -60,31 +60,9 @@
rule: allow
port: "{{ item.port }}"
proto: "{{ item.proto }}"
src: "{{ item.src | default('any') }}"
comment: "{{ item.comment }}"
loop: "{{ ufw_allowed_ports }}"
- name: Allow routed pod-to-pod traffic (Flannel)
community.general.ufw:
rule: allow
route: true
src: "{{ k3s_pod_cidr }}"
dest: "{{ k3s_pod_cidr }}"
comment: K3s pod-to-pod (Flannel)
- name: Allow routed pod traffic to Technitium DNS
community.general.ufw:
rule: allow
route: true
src: "{{ k3s_pod_cidr }}"
dest: "{{ k3s_dns_resolver }}"
port: "53"
proto: "{{ item }}"
comment: K3s pod DNS to Technitium
loop:
- tcp
- udp
- name: Enable UFW
community.general.ufw:
state: enabled

5
ansible/roles/k3s-agent/defaults/main.yaml
View File

@ -4,8 +4,5 @@
# Description: Default variables for the k3s-agent role including version, server URL, and join token.
k3s_version: v1.32.4+k3s1
k3s_server_url: https://10.10.40.53:6443
k3s_server_url: https://192.168.7.77:6443
k3s_node_token: ""
k3s_agent_config:
resolv-conf: /etc/rancher/k3s/resolv.conf

9
ansible/roles/k3s-agent/handlers/main.yaml
View File

@ -1,9 +0,0 @@
---
# Part of role: k3s-agent
# Called by: ansible/roles/k3s-agent/tasks/main.yaml
# Description: Restarts the K3s agent when its config.yaml or resolv.conf changes.
- name: Restart k3s-agent
ansible.builtin.service:
name: k3s-agent
state: restarted

24
ansible/roles/k3s-agent/tasks/main.yaml
View File

@ -2,29 +2,7 @@
# Part of role: k3s-agent
# Called by: ansible/playbooks/join-debian-agent.yaml
# ansible/playbooks/setup-gpu-node.yaml
# Description: Configures the K3s agent's DNS resolver, installs K3s in agent mode, joins the cluster, labels and taints the node.
- name: Create K3s config directory
ansible.builtin.file:
path: /etc/rancher/k3s
state: directory
mode: "0755"
- name: Write K3s agent config
ansible.builtin.copy:
dest: /etc/rancher/k3s/config.yaml
content: "{{ k3s_agent_config | to_nice_yaml }}"
mode: "0644"
notify: Restart k3s-agent
- name: Write K3s DNS resolver file
ansible.builtin.copy:
dest: /etc/rancher/k3s/resolv.conf
content: "nameserver {{ k3s_dns_resolver }}\n"
owner: root
group: root
mode: "0644"
notify: Restart k3s-agent
# Description: Installs K3s in agent mode, joins the cluster, labels and taints the node.
- name: Download and install K3s agent
ansible.builtin.shell:

3
ansible/roles/k3s-server/defaults/main.yaml
View File

@ -4,14 +4,13 @@
# Description: Default variables for the k3s-server role including version, IP, and server configuration.
k3s_version: v1.32.4+k3s1
k3s_server_ip: 10.10.40.53
k3s_server_ip: 192.168.7.77
k3s_server_config:
disable:
- traefik
flannel-backend: vxlan
node-ip: "{{ k3s_server_ip }}"
resolv-conf: /etc/rancher/k3s/resolv.conf
tls-san:
- "{{ k3s_server_ip }}"
- minisforum

9
ansible/roles/k3s-server/handlers/main.yaml
View File

@ -1,9 +0,0 @@
---
# Part of role: k3s-server
# Called by: ansible/roles/k3s-server/tasks/main.yaml
# Description: Restarts K3s when its config.yaml or resolv.conf changes.
- name: Restart k3s
ansible.builtin.service:
name: k3s
state: restarted

12
ansible/roles/k3s-server/tasks/main.yaml
View File

@ -1,7 +1,7 @@
---
# Part of role: k3s-server
# Called by: ansible/playbooks/setup-k3s.yaml
# Description: Installs K3s server, configures its DNS resolver, fetches kubeconfig, installs Helm, and labels the node as primary.
# Description: Installs K3s server, fetches kubeconfig, installs Helm, and labels the node as primary.
- name: Create K3s config directory
ansible.builtin.file:
@ -14,16 +14,6 @@
dest: /etc/rancher/k3s/config.yaml
content: "{{ k3s_server_config | to_nice_yaml }}"
mode: "0644"
notify: Restart k3s
- name: Write K3s DNS resolver file
ansible.builtin.copy:
dest: /etc/rancher/k3s/resolv.conf
content: "nameserver {{ k3s_dns_resolver }}\n"
owner: root
group: root
mode: "0644"
notify: Restart k3s
- name: Download and install K3s
ansible.builtin.shell:

8
ansible/roles/nfs-server/defaults/main.yaml
View File

@ -1,11 +1,7 @@
---
# Part of role: nfs-server
# Called by: ansible/playbooks/setup-nfs-debian.yaml
# Description: Default variables for the nfs-server role including export path, allowed client IP, and UFW-allowed SMB source networks.
# Description: Default variables for the nfs-server role including export path and allowed client IP.
nfs_export_path: /mnt/storage
nfs_allowed_ip: 10.10.40.53
smb_allowed_subnets:
- 10.10.10.0/24
- 10.10.40.0/24
nfs_allowed_ip: 192.168.7.77

21
ansible/roles/nfs-server/tasks/main.yaml
View File

@ -1,7 +1,7 @@
---
# Part of role: nfs-server
# Called by: ansible/playbooks/setup-nfs-debian.yaml
# Description: Installs NFS server, configures exports, allows NFS/SMB through UFW, and ensures the backup directory exists.
# Description: Installs NFS server, configures exports, and ensures the backup directory exists.
- name: Install NFS server
ansible.builtin.apt:
@ -29,25 +29,6 @@
enabled: true
become: true
- name: Allow NFS access through UFW from the k3s server
community.general.ufw:
rule: allow
port: "2049"
proto: tcp
src: "{{ nfs_allowed_ip }}"
comment: NFS from k3s server
become: true
- name: Allow SMB access through UFW from trusted networks
community.general.ufw:
rule: allow
port: "445"
proto: tcp
src: "{{ item }}"
comment: SMB (Samba)
loop: "{{ smb_allowed_subnets }}"
become: true
- name: Ensure backup directory exists with correct ownership
ansible.builtin.file:
path: /home/nik/backups/gitea

4
ansible/roles/nfs-server/templates/exports.j2
View File

@ -3,5 +3,5 @@
# /etc/exports - managed by Ansible
# NFS exports for K3s cluster
/mnt/storage 10.10.40.53(ro,sync,no_subtree_check,no_root_squash,fsid=1)
/home/nik/backups 10.10.40.53(rw,sync,no_subtree_check,no_root_squash,fsid=2)
/mnt/storage 192.168.7.77(ro,sync,no_subtree_check,no_root_squash,fsid=1)
/home/nik/backups 192.168.7.77(rw,sync,no_subtree_check,no_root_squash,fsid=2)

245
ansible/roles/pia-gateway/README.md
View File

@ -1,245 +0,0 @@
# pia-gateway
Makes `minisforum` a PIA WireGuard egress gateway for VPN VLAN 50
(`10.10.50.0/24`), without changing its own default route. Implements
Phase 2 of `~/repo/homelab/plan.md`.
## Before the first real run
1. **The PIA API calls in `tasks/register.yaml` have gone through three
rounds of correction against real live testing (2026-08-24):**
- `serverlist.piaservers.net/vpninfo/servers/v6`: works. Region values
everywhere in this role are the API's **id** field (e.g. `japan`,
`hk`, `taiwan`), never a display name ("JP Tokyo", "Hong Kong",
"Taiwan" — those are Gluetun's labels, not something this role or
PIA's API accepts).
- Token acquisition originally guessed a regional-meta-server
endpoint (`/authv3/generateToken`) that turned out not to be the
real flow at all — it was rebuilt from PIA's actual
`pia-foss/manual-connections` `get_token.sh` (fetched and read in
full): a single fixed `POST
https://www.privateinternetaccess.com/api/client/v2/token`,
multipart `username`/`password`, **normal system CA validation**
(no `--cacert`, no `validate_certs: false`), independent of region
entirely — only WireGuard server *selection* is regional. Not yet
exercised with real credentials end to end.
- `{wg}:1337/addKey` (WireGuard key registration) — confirmed against
`connect_to_wireguard_with_token.sh` (fetched and read in full):
this one genuinely does need PIA's own CA bundle
(`files/pia-ca.crt`, `--cacert`) and `--connect-to`. Originally
failed over only across servers *within* a single fixed region
(`pia_region: japan`) — in practice all three JP Tokyo servers
failed the same way, one after another, so failover now also
crosses regions: every server in Hong Kong, then every server in
Taiwan, then every server in JP Tokyo (`pia_region_candidates` in
`defaults/main.yaml`), stopping at the first `status: OK`. Token
passed via stdin rather than argv. See `tasks/addkey-attempt.yaml`
and "Region fallback" below.
2. Make sure the repo-root `.env` has `PIA_USER`/`PIA_PASSWORD` set (same
keys `manifests/media/pia-secret.sh` already reads) — `ansible/
host_vars/minisforum.yaml` reads them directly from there via a
`lookup('ansible.builtin.file', ...)` + regex, deliberately not
ansible-vault, so this needs no vault password at all. The lookup runs
on the control node, so `.env` never touches minisforum and is never
committed (already gitignored).
3. Confirm console/recovery access to `minisforum` (IPMI/physical/other
out-of-band path) before applying — this role changes host routing and
firewall policy. If SSH becomes unreachable, use that path.
4. `ufw status verbose` and `wg show` (existing `wg0`) were not inspected
live before writing this role (no passwordless sudo in discovery) —
spot-check that `wg0` (the separate home-VPN role) is unaffected after
applying.
## Region fallback
Registration tries every WireGuard server in every configured region, in
order, stopping at the first `status: OK`:
```text
hk (Hong Kong):
wg server 1
wg server 2
taiwan (Taiwan):
wg server 1
wg server 2
japan (JP Tokyo):
wg server 1
wg server 2
wg server 3
```
The actual number and order of servers within each region always comes
from the live serverlist response — never hardcoded here.
**Automatic fallback** (default — tries `pia_region_candidates` in
order):
```bash
ansible-playbook -i ansible/inventory.yaml \
ansible/playbooks/pia-gateway.yaml -K -J
```
**Forced single region**, for troubleshooting one region in isolation —
bypasses `pia_region_candidates` entirely, does not fall back to the
others:
```bash
ansible-playbook -i ansible/inventory.yaml \
ansible/playbooks/pia-gateway.yaml -K -J \
-e pia_region=hk
```
Both `pia_region` and every entry in `pia_region_candidates` are PIA API
**ids** — `hk`, `taiwan`, `japan` — not the display names Gluetun's
`SERVER_REGIONS` config uses ("Hong Kong", "Taiwan", "JP Tokyo"). Passing
a display name here matches nothing and fails the "region exists"
assertion before any network call is even made.
Which region/server actually ended up registered is recorded as
`pia_region_used`/`pia_region_name_used`/`pia_wg_server_used`, written
into `pia-wg.conf`'s own `[Peer]` comment, and surfaced by the health
check (`INFO configured-peer: ...`) — it is whichever one answered first,
not necessarily `pia_region_candidates[0]`.
If every region fails: `journalctl` (or the play's own output) has one
`debug` line per failed attempt, each showing region name+id, server
hostname+IP, curl exit code, a classified reason (connection timeout /
TLS-SSL failure / HTTP failure / empty response / malformed JSON /
parsed-but-not-OK status, never a generic catch-all), and sanitized
stderr — never the token, credentials, the full curl invocation, request
stdin, or a complete response body. If Hong Kong and Taiwan fail exactly
the same way Tokyo did, that's a strong signal the problem is shared
across all three (the request shape, the account credentials, or
minisforum's own network path) rather than one region being down — see
`tasks/register.yaml`'s header comment.
**Worked example (2026-08-24):** exactly that happened — all 7 servers
across all 3 regions came back `HTTP failure (non-2xx response)`, curl
exit 22, `The requested URL returned error: 401`, uniformly. The cause
was in `addkey-attempt.yaml`'s own curl task, not PIA or any region:
`ansible.builtin.command`'s `stdin` argument appends a trailing newline
by default (`stdin_add_newline` defaults to `true`), and
`--data-urlencode pt@-` does not strip it — every server was receiving
`"<real token>\n"` (URL-encoded, so a trailing `%0A`) as `pt`, a
different and invalid value, and correctly rejecting it. Fixed with
`stdin_add_newline: false` on that task. `connect_to_wireguard_with_token
.sh` never hits this because it passes the token as a literal shell
variable, not via stdin — the stdin delivery here is this repo's own
addition (to keep the token out of `ps`), so it needed the flag PIA's
reference script never had to think about. Left as a worked example
because "every candidate failed identically" pointing at one shared bug
in *this* code, not PIA, is exactly the diagnostic story this section
promises — and it happened to be true the first time it was tested for
real.
## What it does
- Registers minisforum as a PIA WireGuard peer — see "Region fallback"
above — and writes `/etc/wireguard/pia-wg.conf` (`Table = off` — this
role owns all routing for the interface, not wg-quick).
- Installs `pia-killswitch.service`, ordered `Before=
wg-quick@pia-wg.service`, that seeds a closed state at every boot (and
after any UFW reload — see handlers/main.yaml): the
`from 10.10.50.0/24 lookup pia` rule, a terminal `unreachable default`
route in the `pia` table (only when `pia-wg` isn't already up — see
the script's own comment for why), and a dedicated `PIA-VLAN50`
iptables chain jumped into by a single rule at the very top of
`FORWARD`, matching only source `10.10.50.0/24`, ending in an
unconditional rate-limited log+drop. This is deliberately **not** a
global `FORWARD` default-policy change — minisforum runs Flannel,
which needs its own broad `FORWARD` ACCEPTs for pod traffic (source
`10.42.0.0/16`, disjoint from VLAN 50) — so the kill switch is scoped
to a chain Flannel/k3s traffic can never enter, rather than risking a
chain-wide policy that was never actually proven safe against it.
Technitium (`10.10.40.53`, minisforum's own address) gets a normal UFW
**input** allow for 53/tcp+udp — it's locally-terminated traffic, not
routed — see `tasks/firewall.yaml`.
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
top of that closed baseline: default route via `pia-wg` in the `pia`
table, ACCEPT + established/related return inserted into the
`PIA-VLAN50` chain (not `FORWARD` directly), and source-NAT/MASQUERADE
scoped to `10.10.50.0/24` on `pia-wg` only.
- Clamps TCP MSS on the `pia-wg` forward path (`pia_mss_clamp_enabled:
true`, `iptables -t mangle ... TCPMSS --clamp-mss-to-pmtu`, scoped to
SYN packets sourced from `10.10.50.0/24` outbound on `pia-wg` only, in
`PostUp`/`PreDown` alongside the rules above). Confirmed needed live
(2026-08-25), not enabled speculatively: a real download hung with a
TLS read timeout on the larger handshake response while small
requests worked fine, and a direct `ping -M do -s 1450` test from a
VLAN 50 pod confirmed the real path MTU is `pia-wg`'s `1420` (the ICMP
"Frag needed" reply arrives correctly from minisforum itself — our own
side relays PMTU discovery fine, so the black hole is further out, on
PIA's network or the remote server's own path, where clamping the MSS
up front avoids needing that ICMP round-trip at all).
- `pia-wg.conf`'s `PreDown` lines are deliberately tolerant of the rule
they're removing not existing (`iptables -D ... 2>/dev/null || true`)
— enabling MSS clamping live (above) exposed why this matters:
`systemctl restart wg-quick@pia-wg` tears down the *currently running*
interface using whatever `PreDown` lines are on disk *right now* — if
Ansible already rewrote the config with a new/changed rule before the
restart handler fires, the live interface (brought up under the *old*
rules) won't have whatever the new `PreDown` line is trying to delete.
`wg-quick`'s own `execute_hooks()` aborts the entire down/up sequence
on the first failing hook (confirmed against its real source, not
assumed), so one non-idempotent `-D` used to turn any future
`PostUp`/`PreDown` content change into a broken restart *and* an
orphaned interface (the never-reached built-in `ip link delete` step
left `pia-wg` existing but unconfigured, which then made the
following `wg-quick up` fail too with `` `pia-wg' already exists``) —
requiring manual recovery (`ip link delete dev pia-wg` before a fresh
`systemctl start`). Every `-D` line now tolerates this by design, so a
config change that alters `PostUp`/`PreDown` content can never break
a restart this way again — confirmed live, 2026-08-27.
- Installs an observability-only health check (`pia-gateway-healthcheck
.timer`, every `pia_healthcheck_interval_sec`) that logs interface,
handshake age, rule/route, and firewall-policy state to the journal. It
never remediates — see the script's header comment for why, and for what
its "route-decision" check does and does not prove.
## What it deliberately does not do
- Does not touch minisforum's own default route (asserted at the start of
every run — `tasks/assert-baseline.yaml` fails loudly if that's already
wrong).
- Does not enable IPv6 forwarding or any IPv6 handling for VLAN 50.
- Does not modify `ansible/roles/wireguard` (the separate `wg0` home-VPN
server role) or its interface.
## Rollback
No tag-based rollback is defined — this role doesn't have an "absent"
mode. Roll back manually on minisforum (reverses this role without
touching `wg0`, k3s, or the host default route):
```bash
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \
/etc/systemd/system/pia-gateway-healthcheck.service \
/etc/systemd/system/pia-gateway-healthcheck.timer
sudo systemctl daemon-reload
sudo ip rule del from 10.10.50.0/24 table pia priority 100
sudo ip route flush table pia
sudo sed -i '/^[0-9]\+\s\+pia$/d' /etc/iproute2/rt_tables
sudo iptables -D FORWARD -s 10.10.50.0/24 -j PIA-VLAN50
sudo iptables -F PIA-VLAN50
sudo iptables -X PIA-VLAN50
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp
sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp
sudo rm -f /etc/wireguard/pia-wg.conf /etc/wireguard/pia-wg.key /etc/wireguard/pia-wg.key.new /etc/wireguard/pia-ca.crt
```
Inspection commands to confirm rollback actually took (read-only):
```bash
sudo iptables -S FORWARD | grep PIA-VLAN50 # expect: no output
sudo iptables -L PIA-VLAN50 # expect: "iptables: No chain/target/match by that name"
sudo ufw status verbose | grep 10.10.50 # expect: no output
ip rule show | grep pia # expect: no output
ip route show table pia # expect: empty/error (table gone)
```
Does not remove `pia-credentials`/`pia-credentials-sealed.yaml` (the K8s
Secret used by the Gluetun sidecars) — that's a separate, unrelated
secret and rollback path. Does not touch `ansible/roles/common`'s own
UFW rules (Flannel pod-to-pod, pod-to-Technitium) — this role never
modified those.

125
ansible/roles/pia-gateway/defaults/main.yaml
View File

@ -1,125 +0,0 @@
---
# Part of role: pia-gateway
# Called by: ansible/playbooks/pia-gateway.yaml
# Description: Default variables for minisforum's PIA WireGuard egress
# gateway for VPN VLAN 50 — policy routing, kill switch, and health
# checks. See ~/repo/homelab/plan.md for the cross-repo contract this
# role implements (fixed values below match that contract).
# PIA regions to try, in order, for WireGuard addKey registration —
# first server (in the first region that has any) to return
# `status: OK` wins. These are API region **ids** from PIA's serverlist
# (https://serverlist.piaservers.net/vpninfo/servers/v6), e.g.
# {"id": "japan", "name": "JP Tokyo", ...} — NOT Gluetun's display names
# ("JP Tokyo", "Hong Kong", "Taiwan") and not the slug you might guess
# from one (confirmed live 2026-08-24: Tokyo's id is "japan", not
# "jp_tokyo" — that matched zero regions on a first pass). Order matches
# the region set already used by the Gluetun sidecars, Hong Kong/Taiwan
# moved ahead of Tokyo after all three Tokyo servers were observed
# failing in practice — see tasks/register.yaml's header for what that
# means and doesn't mean.
pia_region_candidates:
- hk
- taiwan
- japan
# Force a SINGLE region for troubleshooting, bypassing
# pia_region_candidates entirely — e.g. `-e pia_region=hk`. This is also
# an API id, not a display name; same rule as above. Leave null (the
# default) to use the ordered candidate list. tasks/register.yaml
# computes the actual list to try (pia_effective_regions) from whichever
# of these two is set — see that file's own comment for the exact logic,
# not duplicated here to avoid the two drifting out of sync.
pia_region: null
# WireGuard interface identity. Deliberately not "wg0" — that name is
# already used by ansible/roles/wireguard, minisforum's separate *inbound*
# home-VPN server role. Colliding with it would tear down remote access.
pia_wg_interface: pia-wg
pia_wg_config_dir: /etc/wireguard
# Re-registration behaviour — two independent, unambiguous knobs (an
# earlier single pia_force_reregister conflated "re-run the API calls"
# with "get a new keypair", which isn't the same thing and isn't safe to
# conflate — the key-generation task uses `creates:`, so a naive
# "force" only re-ran the PIA API calls with the SAME existing key,
# silently, unless you also knew to delete the key file yourself):
#
# - pia_force_reregister: re-run PIA registration (serverlist lookup,
# token, addKey) even if {{ pia_wg_interface }}.conf already exists.
# Reuses whatever keypair is already on disk (generates one only if
# none exists yet). Use this to pick up a new PIA-assigned peer/server
# without changing minisforum's own WireGuard identity.
# - pia_force_key_rotation: additionally generate a NEW keypair and
# register THAT with PIA (implies pia_force_reregister). Rotation is
# safe-by-construction: the new key is generated to a *.key.new path
# and only promoted (with the old key backed up alongside a timestamp)
# after PIA has actually accepted the new key and the new config has
# been written — see tasks/register.yaml. If registration fails
# partway, the previous key/config are left completely untouched
# rather than ending up in a broken half-state.
pia_force_reregister: false
pia_force_key_rotation: false
# Dedicated policy-routing table for VLAN 50 egress. 0/253/254/255 are
# kernel-reserved (unspec/default/main/local) — this must stay clear of
# those and of any other custom table already registered on the host.
pia_route_table_id: 52
pia_route_table_name: pia
# Rule priority for "from 10.10.50.0/24 lookup pia". Must sort after the
# kernel's built-in local table (priority 0, always evaluated first, keeps
# traffic addressed to the host's own local/Technitium address delivered
# locally per plan.md) and before main (priority 32766).
pia_rule_priority: 100
# Fixed cross-repo contract (see plan.md "Fixed cross-repository
# contract") — VLAN 50 subnet and the Flint gateway that owns the return
# route back to VLAN 50. Not derived from live discovery; change only if
# the contract itself changes.
pia_vlan50_subnet: 10.10.50.0/24
pia_flint_gateway: 10.10.50.1
# Dedicated iptables chain the kill switch governs, jumped into by a
# single rule at FORWARD position 1 matching only
# {{ pia_vlan50_subnet }} — deliberately not a global FORWARD default-
# policy change, so k3s/Flannel's own FORWARD-chain traffic (source
# 10.42.0.0/16, disjoint from VLAN 50) is never affected by this role at
# all. See templates/pia-killswitch.sh.j2 for the full reasoning.
pia_iptables_chain: PIA-VLAN50
# Real parent LAN interface on minisforum — verified live 2026-08-23 via
# `ip -brief addr`; it is enp1s0, not "eth0". Used for MASQUERADE-exclusion
# bookkeeping and documentation only, since NAT/forward rules below key off
# {{ pia_wg_interface }} rather than enumerating the normal interface.
pia_lan_interface: enp1s0
# Reverse-path filtering, applied to {{ pia_wg_interface }} only. The
# kernel default on minisforum (conf.default.rp_filter) is already loose
# (2) at time of writing, so new interfaces inherit loose mode — this task
# pins it explicitly rather than relying on that inherited default holding
# across future changes. Loose mode is required because PIA's return
# traffic can legitimately arrive with routing asymmetry relative to how
# minisforum would route toward that source itself.
pia_rp_filter_mode: 2
# MSS clamping on the pia-wg forward path — ON. Confirmed needed live
# (2026-08-25), not guessed: a real JDownloader download to
# sbs237.sbsf.tech hung indefinitely (TLS read timeout waiting for the
# server's handshake response — small requests like a CDN version check
# worked fine, only the larger TLS response never arrived). Verified
# with the exact test plan.md called for before flipping this: `ping -M
# do -s 1450 -c 3 1.1.1.1` from a VLAN 50 pod returned `From 10.10.40.53
# Frag needed and DF set (mtu = 1420)` — our own PMTU discovery between
# the pod and minisforum works correctly, which means the black hole is
# further out (PIA's network or the remote server's own path), not on
# our side — exactly the class of problem MSS clamping fixes by
# avoiding the need for that ICMP round-trip at all, clamping the MSS
# at the TCP handshake instead of relying on mid-connection PMTU
# discovery that something further along the path may not relay back.
pia_mss_clamp_enabled: true
# Health check cadence and thresholds. The check only observes and logs —
# see tasks/healthcheck.yaml — it must never itself open the kill switch.
pia_healthcheck_interval_sec: 60
pia_healthcheck_handshake_max_age_sec: 180

43
ansible/roles/pia-gateway/files/pia-ca.crt
View File

@ -1,43 +0,0 @@
-----BEGIN CERTIFICATE-----
MIIHqzCCBZOgAwIBAgIJAJ0u+vODZJntMA0GCSqGSIb3DQEBDQUAMIHoMQswCQYD
VQQGEwJVUzELMAkGA1UECBMCQ0ExEzARBgNVBAcTCkxvc0FuZ2VsZXMxIDAeBgNV
BAoTF1ByaXZhdGUgSW50ZXJuZXQgQWNjZXNzMSAwHgYDVQQLExdQcml2YXRlIElu
dGVybmV0IEFjY2VzczEgMB4GA1UEAxMXUHJpdmF0ZSBJbnRlcm5ldCBBY2Nlc3Mx
IDAeBgNVBCkTF1ByaXZhdGUgSW50ZXJuZXQgQWNjZXNzMS8wLQYJKoZIhvcNAQkB
FiBzZWN1cmVAcHJpdmF0ZWludGVybmV0YWNjZXNzLmNvbTAeFw0xNDA0MTcxNzQw
MzNaFw0zNDA0MTIxNzQwMzNaMIHoMQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0Ex
EzARBgNVBAcTCkxvc0FuZ2VsZXMxIDAeBgNVBAoTF1ByaXZhdGUgSW50ZXJuZXQg
QWNjZXNzMSAwHgYDVQQLExdQcml2YXRlIEludGVybmV0IEFjY2VzczEgMB4GA1UE
AxMXUHJpdmF0ZSBJbnRlcm5ldCBBY2Nlc3MxIDAeBgNVBCkTF1ByaXZhdGUgSW50
ZXJuZXQgQWNjZXNzMS8wLQYJKoZIhvcNAQkBFiBzZWN1cmVAcHJpdmF0ZWludGVy
bmV0YWNjZXNzLmNvbTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBALVk
hjumaqBbL8aSgj6xbX1QPTfTd1qHsAZd2B97m8Vw31c/2yQgZNf5qZY0+jOIHULN
De4R9TIvyBEbvnAg/OkPw8n/+ScgYOeH876VUXzjLDBnDb8DLr/+w9oVsuDeFJ9K
V2UFM1OYX0SnkHnrYAN2QLF98ESK4NCSU01h5zkcgmQ+qKSfA9Ny0/UpsKPBFqsQ
25NvjDWFhCpeqCHKUJ4Be27CDbSl7lAkBuHMPHJs8f8xPgAbHRXZOxVCpayZ2SND
fCwsnGWpWFoMGvdMbygngCn6jA/W1VSFOlRlfLuuGe7QFfDwA0jaLCxuWt/BgZyl
p7tAzYKR8lnWmtUCPm4+BtjyVDYtDCiGBD9Z4P13RFWvJHw5aapx/5W/CuvVyI7p
Kwvc2IT+KPxCUhH1XI8ca5RN3C9NoPJJf6qpg4g0rJH3aaWkoMRrYvQ+5PXXYUzj
tRHImghRGd/ydERYoAZXuGSbPkm9Y/p2X8unLcW+F0xpJD98+ZI+tzSsI99Zs5wi
jSUGYr9/j18KHFTMQ8n+1jauc5bCCegN27dPeKXNSZ5riXFL2XX6BkY68y58UaNz
meGMiUL9BOV1iV+PMb7B7PYs7oFLjAhh0EdyvfHkrh/ZV9BEhtFa7yXp8XR0J6vz
1YV9R6DYJmLjOEbhU8N0gc3tZm4Qz39lIIG6w3FDAgMBAAGjggFUMIIBUDAdBgNV
HQ4EFgQUrsRtyWJftjpdRM0+925Y6Cl08SUwggEfBgNVHSMEggEWMIIBEoAUrsRt
yWJftjpdRM0+925Y6Cl08SWhge6kgeswgegxCzAJBgNVBAYTAlVTMQswCQYDVQQI
EwJDQTETMBEGA1UEBxMKTG9zQW5nZWxlczEgMB4GA1UEChMXUHJpdmF0ZSBJbnRl
cm5ldCBBY2Nlc3MxIDAeBgNVBAsTF1ByaXZhdGUgSW50ZXJuZXQgQWNjZXNzMSAw
HgYDVQQDExdQcml2YXRlIEludGVybmV0IEFjY2VzczEgMB4GA1UEKRMXUHJpdmF0
ZSBJbnRlcm5ldCBBY2Nlc3MxLzAtBgkqhkiG9w0BCQEWIHNlY3VyZUBwcml2YXRl
aW50ZXJuZXRhY2Nlc3MuY29tggkAnS7684Nkme0wDAYDVR0TBAUwAwEB/zANBgkq
hkiG9w0BAQ0FAAOCAgEAJsfhsPk3r8kLXLxY+v+vHzbr4ufNtqnL9/1Uuf8NrsCt
pXAoyZ0YqfbkWx3NHTZ7OE9ZRhdMP/RqHQE1p4N4Sa1nZKhTKasV6KhHDqSCt/dv
Em89xWm2MVA7nyzQxVlHa9AkcBaemcXEiyT19XdpiXOP4Vhs+J1R5m8zQOxZlV1G
tF9vsXmJqWZpOVPmZ8f35BCsYPvv4yMewnrtAC8PFEK/bOPeYcKN50bol22QYaZu
LfpkHfNiFTnfMh8sl/ablPyNY7DUNiP5DRcMdIwmfGQxR5WEQoHL3yPJ42LkB5zs
6jIm26DGNXfwura/mi105+ENH1CaROtRYwkiHb08U6qLXXJz80mWJkT90nr8Asj3
5xN2cUppg74nG3YVav/38P48T56hG1NHbYF5uOCske19F6wi9maUoto/3vEr0rnX
JUp2KODmKdvBI7co245lHBABWikk8VfejQSlCtDBXn644ZMtAdoxKNfR2WTFVEwJ
iyd1Fzx0yujuiXDROLhISLQDRjVVAvawrAtLZWYK31bY7KlezPlQnl/D9Asxe85l
8jO5+0LdJ6VyOs/Hd4w52alDW/MFySDZSfQHMTIc30hLBJ8OnCEIvluVQQ2UQvoW
+no177N9L2Y+M9TcTA62ZyMXShHQGeh20rb4kK8f+iFX8NxtdHVSkxMEFSfDDyQ=
-----END CERTIFICATE-----

46
ansible/roles/pia-gateway/handlers/main.yaml
View File

@ -1,46 +0,0 @@
---
# Part of role: pia-gateway
# Handlers fire in the order listed here (not the order notified):
# 1. systemd daemon reload must stay first so unit-file content changes
# are picked up before any restart handler below acts on them.
# 2. "Reload ufw" must come before "Restart pia-killswitch" — a UFW
# reload rebuilds ufw's own chains and, depending on how it's
# triggered, can in principle disturb rules inserted directly into
# the base FORWARD chain (our position-1 jump to
# {{ pia_iptables_chain }}) outside ufw's own bookkeeping. Restarting
# pia-killswitch immediately after re-establishes that jump and the
# chain's contents idempotently, closing the window rather than
# leaving it open until the next reboot. This only protects against
# ufw reloads that happen *through this Ansible run* (i.e., a task
# here notifies both) — a `ufw reload` run manually and separately,
# outside of Ansible entirely, is not something a handler can react
# to; that residual gap is closed only by every real boot re-running
# pia-killswitch.service fresh (WantedBy=multi-user.target).
# 3. "Restart pia-killswitch" must come before "Restart pia-wg" — if
# both fire together (e.g. pia-wg's config changed AND the
# killswitch script changed in the same run), pia-wg's own PostUp
# needs to run last so its "open" route is the final word, not
# killswitch's route-table seeding.
- name: Reload systemd daemon
ansible.builtin.systemd:
daemon_reload: true
- name: Reload ufw
community.general.ufw:
state: reloaded
- name: Restart pia-killswitch
ansible.builtin.systemd:
name: pia-killswitch.service
state: restarted
- name: Restart pia-wg
ansible.builtin.systemd:
name: "wg-quick@{{ pia_wg_interface }}"
state: restarted
- name: Restart pia-gateway-healthcheck timer
ansible.builtin.systemd:
name: pia-gateway-healthcheck.timer
state: restarted

179
ansible/roles/pia-gateway/tasks/addkey-attempt.yaml
View File

@ -1,179 +0,0 @@
---
# Part of role: pia-gateway
# Included by: tasks/register.yaml, once per candidate in pia_candidates
# (loop_var: pia_candidate — a flattened {region_id, region_name, cn, ip}
# dict, one server, built by register.yaml via subelements over every
# requested region in order), in order, until one succeeds.
#
# Every meaningful task below is individually guarded with
# `when: pia_peer is not defined` — confirmed empirically (not assumed)
# that putting this guard only on the include statement in register.yaml
# does NOT stop later loop iterations from running; ansible re-evaluates
# a `when` on tasks inside an included file per-iteration, but does not
# re-evaluate a `when` on the include statement itself the same way. The
# guard has to live here, on every task, or a candidate that comes up
# after an earlier one already worked would still get hit.
- name: "Reset per-attempt response state for {{ pia_candidate.cn }}"
ansible.builtin.set_fact:
pia_addkey_response: {}
pia_addkey_rc: -1
pia_addkey_stderr: ""
pia_addkey_parsed: null
pia_addkey_diagnosis: ""
when: pia_peer is not defined
# Explicit reset, every iteration, before this candidate's own curl
# call runs — a prior version of this file only ever *conditionally*
# set pia_addkey_parsed (rc == 0 and non-empty stdout), so a candidate
# whose curl call failed outright (rc != 0) would skip that task and
# silently keep the PREVIOUS candidate's parsed response sitting in
# pia_addkey_parsed. That's not just cosmetic: it means a stale
# {"status": "..."} from an earlier failure could read as if it came
# from the current candidate in diagnostics, or — if the guard logic
# were ever refactored slightly differently — actually be accepted for
# the wrong server. Resetting to a known-empty/null state here, every
# time, makes each iteration self-contained regardless of what
# happened before it.
- name: "Register WireGuard key with server {{ pia_candidate.cn }}"
ansible.builtin.command:
cmd: >-
curl --silent --show-error --fail --location -G
--connect-timeout 10 --max-time 30
--cacert {{ pia_wg_config_dir }}/pia-ca.crt
--connect-to {{ pia_candidate.cn }}::{{ pia_candidate.ip }}:
--data-urlencode pt@-
--data-urlencode pubkey={{ pia_public_key.stdout }}
https://{{ pia_candidate.cn }}:1337/addKey
args:
stdin: "{{ pia_auth_token }}"
stdin_add_newline: false
register: pia_addkey_response
changed_when: false
failed_when: false
no_log: true
when: pia_peer is not defined
# Same request shape as PIA's own connect_to_wireguard_with_token.sh
# for every region — WireGuard hostname for TLS/SNI, its server IP via
# --connect-to, PIA's private CA (never -k/--insecure/validate_certs:
# false), port 1337, /addKey, generated pubkey. Token passed via stdin
# (--data-urlencode pt@-) rather than as a plain argv value — verified
# empirically (this session) that curl's argv, including a literal
# "pt@-", never the token itself, is what shows up in `ps`, so the
# token is never visible to another local user on the host for the
# process's lifetime. pubkey is not sensitive (public by design) and
# stays a normal argument. No curl --retry here — retries happen by
# moving to the next candidate in the loop instead, so one slow/dead
# server can't eat the whole attempt budget retrying itself before
# failover to the next server, or the next region, gets a chance.
#
# stdin_add_newline: false is load-bearing, not cosmetic. Ansible's
# command module appends a trailing newline to `stdin` by default
# (stdin_add_newline defaults to true) — confirmed live (this session)
# against a real run: every one of 7 servers across all 3 regions
# rejected the token with a uniform HTTP 401, and reproducing the exact
# same stdin mechanism against a plain echo endpoint showed why —
# `--data-urlencode pt@-` does not strip that trailing newline, so PIA
# received "<real token>\n" (URL-encoded, with a trailing %0A) as the
# pt value: a different, invalid string. This is what actually broke
# every region identically, not a credentials or connectivity problem
# in any of them. get_token.sh/connect_to_wireguard_with_token.sh never
# hit this because they pass the token as a literal shell variable
# (`"pt=${PIA_TOKEN}"`), not via stdin — the stdin delivery mechanism
# itself was introduced in this repo specifically to keep the token out
# of `ps` output, and needs this flag to behave the same as PIA's own
# reference flow.
- name: "Capture safe diagnostics for {{ pia_candidate.cn }}"
ansible.builtin.set_fact:
pia_addkey_rc: "{{ pia_addkey_response.rc | default(-1) }}"
pia_addkey_stderr: "{{ pia_addkey_response.stderr | default('') }}"
when: pia_peer is not defined
# rc/stderr never contain the token or pubkey — same reasoning as the
# token request's diagnostics in register.yaml. Never derived from
# pia_addkey_response.cmd/.invocation (the literal argv, harmless here
# since the token isn't in it either, but still not something to print
# wholesale) — only the two specific safe fields are ever extracted.
# --- Classify the outcome into one human-readable diagnosis. Each of
# these is mutually exclusive by its `when:`, and pia_addkey_diagnosis
# was reset to "" above, so exactly one (or zero, if this candidate was
# skipped because an earlier one already succeeded) ends up set. Curl
# exit codes confirmed against curl's own documented libcurl error list,
# not guessed — 28 is CURLE_OPERATION_TIMEDOUT, 22 is
# CURLE_HTTP_RETURNED_ERROR (only possible here because of --fail), and
# the TLS set is every CURLE_SSL_*/CURLE_PEER_FAILED_VERIFICATION code
# curl currently defines (35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83).
- name: "Diagnose a connection timeout for {{ pia_candidate.cn }}"
ansible.builtin.set_fact:
pia_addkey_diagnosis: connection timeout
when: (pia_peer is not defined) and (pia_addkey_rc | int == 28)
- name: "Diagnose a TLS validation failure for {{ pia_candidate.cn }}"
ansible.builtin.set_fact:
pia_addkey_diagnosis: TLS/SSL failure
when: (pia_peer is not defined) and (pia_addkey_rc | int in [35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83])
- name: "Diagnose an HTTP failure for {{ pia_candidate.cn }}"
ansible.builtin.set_fact:
pia_addkey_diagnosis: HTTP failure (non-2xx response)
when: (pia_peer is not defined) and (pia_addkey_rc | int == 22)
- name: "Diagnose an unclassified curl error for {{ pia_candidate.cn }}"
ansible.builtin.set_fact:
pia_addkey_diagnosis: "curl error (exit {{ pia_addkey_rc }})"
when: (pia_peer is not defined) and (pia_addkey_rc | int not in [0, 22, 28, 35, 53, 54, 58, 59, 60, 66, 77, 80, 82, 83])
- name: "Parse the addKey response from {{ pia_candidate.cn }}"
ansible.builtin.set_fact:
pia_addkey_parsed: "{{ pia_addkey_response.stdout | from_json }}"
when: (pia_peer is not defined) and (pia_addkey_rc | int == 0) and (pia_addkey_response.stdout | trim | length > 0)
ignore_errors: true
# ignore_errors here on purpose: a genuinely malformed body from one
# misbehaving server must not abort the whole play — it should just
# leave pia_addkey_parsed at the null it was reset to above, so this
# candidate is diagnosed as malformed (below) and the loop moves on.
- name: "Diagnose an empty response for {{ pia_candidate.cn }}"
ansible.builtin.set_fact:
pia_addkey_diagnosis: empty response body
when: (pia_peer is not defined) and (pia_addkey_rc | int == 0) and (pia_addkey_response.stdout | trim | length == 0)
- name: "Diagnose a malformed JSON response for {{ pia_candidate.cn }}"
ansible.builtin.set_fact:
pia_addkey_diagnosis: malformed JSON response
when: (pia_peer is not defined) and (pia_addkey_rc | int == 0) and (pia_addkey_response.stdout | trim | length > 0) and (pia_addkey_parsed is none)
- name: "Diagnose a non-OK status for {{ pia_candidate.cn }}"
ansible.builtin.set_fact:
pia_addkey_diagnosis: "parsed OK but status is '{{ pia_addkey_parsed.status | default('missing') }}', not OK"
when: (pia_peer is not defined) and (pia_addkey_parsed is not none) and ((pia_addkey_parsed.status | default('')) != 'OK')
- name: "Log the attempt outcome for {{ pia_candidate.cn }}"
ansible.builtin.debug:
msg: >-
region={{ pia_candidate.region_name }} ({{ pia_candidate.region_id }})
server={{ pia_candidate.cn }} ip={{ pia_candidate.ip }}
rc={{ pia_addkey_rc }} reason=[{{ pia_addkey_diagnosis }}]
stderr=[{{ pia_addkey_stderr }}]
status={{ pia_addkey_parsed.status | default('n/a') }}
when: (pia_peer is not defined) and (pia_addkey_diagnosis | length > 0)
# Deliberately never includes: the token, username/password, the full
# curl invocation/cmd array, request stdin, the complete response body
# (only its parsed .status field, when parsing succeeded), or any key
# material beyond what pubkey already is (public, safe by design).
- name: "Accept the response from {{ pia_candidate.cn }}"
ansible.builtin.set_fact:
pia_peer: "{{ pia_addkey_parsed }}"
pia_region_used: "{{ pia_candidate.region_id }}"
pia_region_name_used: "{{ pia_candidate.region_name }}"
pia_wg_server_used: "{{ pia_candidate }}"
when: (pia_peer is not defined) and (pia_addkey_parsed is not none) and ((pia_addkey_parsed.status | default('')) == 'OK')
# Deliberately checks status == OK here, not just "did it parse" — a
# response that parses but isn't actually OK must not stop the loop
# from trying the remaining candidates. Once this fires, every
# subsequent iteration's tasks are no-ops (all guarded on
# `pia_peer is not defined`), so pia_region_used/pia_wg_server_used are
# never overwritten by a later candidate.

34
ansible/roles/pia-gateway/tasks/assert-baseline.yaml
View File

@ -1,34 +0,0 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml
# Description: Enables IP forwarding persistently (required for minisforum
# to route VLAN 50 traffic at all) and asserts the host's own default
# route is still the normal LAN gateway before any PIA-specific routing
# is touched. This is the guard behind "never change minisforum's normal
# default route globally" — it fails the play loudly instead of layering
# PIA routing on top of an already-wrong baseline.
- name: Enable IP forwarding persistently
ansible.posix.sysctl:
name: net.ipv4.ip_forward
value: "1"
sysctl_set: true
state: present
reload: true
- name: Read current default route
ansible.builtin.command: ip -4 route show default
register: pia_current_default_route
changed_when: false
- name: Assert the host default route is still the normal LAN interface
ansible.builtin.assert:
that:
- pia_current_default_route.stdout is search('dev ' + pia_lan_interface)
- not (pia_current_default_route.stdout is search('dev ' + pia_wg_interface))
fail_msg: >-
minisforum's default route is not via {{ pia_lan_interface }}
({{ pia_current_default_route.stdout }}). Refusing to continue —
this role must never run against a host whose own default route
has already been changed.
success_msg: "Host default route confirmed via {{ pia_lan_interface }}"

66
ansible/roles/pia-gateway/tasks/firewall.yaml
View File

@ -1,66 +0,0 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml
# Description: The one thing this file does now: permit VLAN 50 clients
# to reach the local Technitium listener on minisforum. Everything
# else the kill switch needs (the dedicated FORWARD-chain scoping, the
# routing-table unreachable fallback) lives in tasks/routing.yaml /
# templates/pia-killswitch.sh.j2 — see those for the actual fail-closed
# layers.
#
# Two corrections from an earlier version of this file, both from code
# review against the live topology rather than assumption:
#
# 1. Technitium (10.10.40.53) IS minisforum's own address — traffic from
# a VLAN 50 client to it terminates LOCALLY on minisforum. That's
# INPUT traffic, not FORWARD/routed traffic, so it needs a normal UFW
# input allow, not `route: true` (which adds a FORWARD-chain rule for
# traffic passing *through* the host to some other destination — the
# wrong chain for a locally-terminated flow, and it would never
# actually have matched real Technitium queries).
#
# 2. This file used to also force UFW's DEFAULT_FORWARD_POLICY to DROP,
# intended as a second kill-switch layer. That was never actually
# proven safe: minisforum is the k3s server and runs Flannel, which
# depends on the FORWARD chain for pod-to-pod and pod-to-internet
# traffic (source 10.42.0.0/16) — changing the chain-wide default
# policy risked breaking that, for a benefit (fail-closed for VLAN 50
# specifically) that's already fully covered by the dedicated
# {{ pia_iptables_chain }} chain in pia-killswitch.sh.j2, which is
# scoped to source {{ pia_vlan50_subnet }} only and can structurally
# never see Flannel's traffic (disjoint source CIDR). Removed rather
# than shipped unproven.
#
# Inspection commands (read-only):
# sudo iptables -S FORWARD | head -3 # confirm the single
# # jump to {{ pia_iptables_chain }}
# # at position 1, and that
# # the chain-wide policy
# # is still whatever it
# # was before this role
# # (ACCEPT, typically)
# sudo iptables -S {{ pia_iptables_chain }} # the kill-switch chain
# # itself: ACCEPT pair
# # when pia-wg is up,
# # LOG+DROP catch-all
# # always present
# sudo ufw status verbose | grep 53 # the Technitium INPUT
# # allow, both protocols
#
# Rollback (this file's part only — see routing.yaml/README.md for the
# rest of the role's rollback):
# sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto tcp
# sudo ufw delete allow from 10.10.50.0/24 to 10.10.40.53 port 53 proto udp
- name: Allow VLAN 50 clients to reach Technitium DNS (local INPUT traffic)
community.general.ufw:
rule: allow
direction: in
src: "{{ pia_vlan50_subnet }}"
dest: "{{ k3s_dns_resolver }}"
port: "53"
proto: "{{ item }}"
comment: VLAN 50 clients to Technitium (local, not routed)
loop:
- tcp
- udp

42
ansible/roles/pia-gateway/tasks/healthcheck.yaml
View File

@ -1,42 +0,0 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml
# Description: Installs the observability-only health check (script,
# oneshot service, timer). See templates/pia-gateway-healthcheck.sh.j2
# for exactly what it checks and why it never remediates.
- name: Deploy the health check script
ansible.builtin.template:
src: pia-gateway-healthcheck.sh.j2
dest: /usr/local/sbin/pia-gateway-healthcheck.sh
mode: "0700"
owner: root
group: root
- name: Deploy the health check systemd service
ansible.builtin.template:
src: pia-gateway-healthcheck.service.j2
dest: /etc/systemd/system/pia-gateway-healthcheck.service
mode: "0644"
owner: root
group: root
notify:
- Reload systemd daemon
- Restart pia-gateway-healthcheck timer
- name: Deploy the health check systemd timer
ansible.builtin.template:
src: pia-gateway-healthcheck.timer.j2
dest: /etc/systemd/system/pia-gateway-healthcheck.timer
mode: "0644"
owner: root
group: root
notify:
- Reload systemd daemon
- Restart pia-gateway-healthcheck timer
- name: Enable and start the health check timer
ansible.builtin.systemd:
name: pia-gateway-healthcheck.timer
enabled: true
state: started

32
ansible/roles/pia-gateway/tasks/interface.yaml
View File

@ -1,32 +0,0 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml
# Description: Enables and starts the pia-wg WireGuard interface itself.
# The config file was already written by tasks/register.yaml (only on
# first run / forced re-registration); this task just makes sure the
# systemd unit is enabled and running, and that pia-killswitch.service
# (tasks/routing.yaml) has already run first — both are WantedBy
# multi-user.target with an explicit Before= from the kill switch, so a
# normal boot orders them correctly without help here, but a first
# Ansible run on an already-booted host needs the same ordering applied
# explicitly since neither unit was part of a boot transaction yet.
- name: Confirm the kill switch is active before bringing up pia-wg
ansible.builtin.systemd:
name: pia-killswitch.service
register: pia_killswitch_status
failed_when: false
- name: Assert the kill switch is active
ansible.builtin.assert:
that:
- pia_killswitch_status.status.ActiveState == "active"
fail_msg: >-
pia-killswitch.service is not active — refusing to start
{{ pia_wg_interface }} without the kill switch already seeded.
- name: Enable and start wg-quick@{{ pia_wg_interface }}
ansible.builtin.systemd:
name: "wg-quick@{{ pia_wg_interface }}"
enabled: true
state: started

53
ansible/roles/pia-gateway/tasks/main.yaml
View File

@ -1,53 +0,0 @@
---
# Part of role: pia-gateway
# Called by: ansible/playbooks/pia-gateway.yaml
# Description: Orchestrates minisforum's PIA WireGuard egress gateway for
# VPN VLAN 50 — registration, boot-ordered kill switch + policy routing,
# the WireGuard interface itself, and health checks. Split into included
# files because each stage has a distinct idempotency/ordering concern;
# see the comment at the top of each included file.
- name: Install WireGuard tooling and iproute2
ansible.builtin.apt:
name:
- wireguard
- wireguard-tools
- iproute2
- python3-requests
state: present
update_cache: true
- name: Pin reverse-path filtering for {{ pia_wg_interface }}
ansible.posix.sysctl:
name: "net.ipv4.conf.{{ pia_wg_interface }}.rp_filter"
value: "{{ pia_rp_filter_mode | string }}"
sysctl_file: /etc/sysctl.d/60-pia-gateway.conf
reload: false # interface does not exist yet on a first run; applied for real after the WG interface comes up (handlers/main.yaml)
register: pia_rp_filter_pinned
- name: Confirm host default route and ip_forward are untouched
ansible.builtin.include_tasks: assert-baseline.yaml
- name: Register minisforum with PIA and obtain WireGuard peer config
ansible.builtin.include_tasks: register.yaml
no_log: true # PIA credentials and the derived private key pass through this include
- name: Configure boot-ordered kill switch and policy routing
ansible.builtin.include_tasks: routing.yaml
- name: Configure firewall-layer kill switch and Technitium access
ansible.builtin.include_tasks: firewall.yaml
- name: Write and enable the pia-wg WireGuard interface
ansible.builtin.include_tasks: interface.yaml
- name: Install gateway health checks
ansible.builtin.include_tasks: healthcheck.yaml
- name: Re-apply rp_filter now that {{ pia_wg_interface }} exists
ansible.posix.sysctl:
name: "net.ipv4.conf.{{ pia_wg_interface }}.rp_filter"
value: "{{ pia_rp_filter_mode | string }}"
sysctl_file: /etc/sysctl.d/60-pia-gateway.conf
reload: true
when: pia_rp_filter_pinned is succeeded

392
ansible/roles/pia-gateway/tasks/register.yaml
View File

@ -1,392 +0,0 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml (with no_log: true — this file handles the
# PIA account password, the derived WireGuard private key, and the
# short-lived PIA auth token)
# Description: Registers minisforum as a PIA WireGuard peer and writes
# {{ pia_wg_config_dir }}/{{ pia_wg_interface }}.conf. Skipped once that
# file exists, unless pia_force_reregister or pia_force_key_rotation is
# set — see defaults/main.yaml for exactly what each does.
#
# Status as of 2026-08-24 (third pass — multi-region fallback added after
# all three JP Tokyo servers were observed failing live, one after
# another, against the single-region version of this file):
# - Token acquisition (`Request a PIA auth token` below) was completely
# rebuilt after live testing against the *actual* PIA API showed the
# first version's endpoint (a regional meta server's
# /authv3/generateToken, with PIA's private CA) doesn't exist as
# documented — it's unreachable/wrong. The correct, current flow
# (confirmed against PIA's own pia-foss/manual-connections
# get_token.sh, fetched and read in full, not from memory) is a
# single fixed POST to www.privateinternetaccess.com — a normal
# public domain with a normal publicly-trusted cert — with
# username/password as multipart form fields. It does not depend on
# pia_region/pia_region_candidates at all; only WireGuard server
# SELECTION does.
# - addKey (WireGuard key registration) was already close to correct on
# the first pass and is confirmed against
# connect_to_wireguard_with_token.sh, fetched and read in full: a
# regional server IP with SNI pinned to its hostname via
# --connect-to, validated against PIA's own CA bundle (still needed
# here — only the token endpoint uses the system CA store), token via
# stdin not argv. Failing over across only the servers *within* one
# region turned out not to be enough in practice — all three JP Tokyo
# servers failed the same way, one after another. This version tries
# every server in every configured region, in order (pia_region_
# candidates, or a single pia_region override — see defaults/
# main.yaml), not just every server within a single fixed region. The
# underlying request shape sent to each server is unchanged from the
# previous pass (still connect_to_wireguard_with_token.sh's flow) —
# if Hong Kong and Taiwan fail identically to how Tokyo did, that
# points at something shared across every region (the request itself,
# credentials, or minisforum's own network path), not at Tokyo
# specifically, and the per-attempt diagnostics below are what
# distinguishes those cases from three-independent-regions-all-down.
# - serverlist fetch + parse: unchanged, exercised live and working.
- name: Determine the effective list of PIA regions to try
ansible.builtin.set_fact:
pia_effective_regions: "{{ [pia_region] if (pia_region is not none and (pia_region | trim | length) > 0) else pia_region_candidates }}"
# A single -e pia_region=<id> override takes over completely (for
# troubleshooting one region in isolation) rather than being prepended
# to the candidate list — mixing the two would make "did it use my
# override or fall through to the list" ambiguous.
- name: Assert the effective region list is usable
ansible.builtin.assert:
that:
- pia_effective_regions is defined
- pia_effective_regions | length > 0
- pia_effective_regions | select('none') | list | length == 0
- pia_effective_regions | map('string') | map('trim') | select('equalto', '') | list | length == 0
fail_msg: >-
pia_region_candidates (or a single -e pia_region override) resolved
to an empty or invalid region list:
{{ pia_effective_regions | default('undefined') }}. Set
pia_region_candidates in defaults/main.yaml to a nonempty list of
PIA region ids (not display names), or pass -e pia_region=<id> for
a single region.
- name: Check whether a PIA WireGuard config already exists
ansible.builtin.stat:
path: "{{ pia_wg_config_dir }}/{{ pia_wg_interface }}.conf"
register: pia_wg_conf_stat
- name: PIA registration
# | bool on both flags: -e pia_force_reregister=true (the plain
# key=value CLI form this role's own README documents) sets the var as
# a STRING "true", not a real boolean — recent ansible-core versions
# reject using a string directly in a `when:` boolean expression
# ("Conditional result (True) was derived from value of type 'str'"),
# confirmed live (2026-08-25) against the exact documented invocation.
# | bool coerces either a real bool (the defaults/main.yaml default,
# unaffected either way) or a "true"/"false" string (the CLI-override
# case) into an actual boolean, so both invocation styles work.
when: (pia_force_reregister | bool) or (pia_force_key_rotation | bool) or not pia_wg_conf_stat.stat.exists
block:
- name: Ensure WireGuard config directory exists
ansible.builtin.file:
path: "{{ pia_wg_config_dir }}"
state: directory
mode: "0700"
owner: root
group: root
- name: Determine which private key path this run will use
ansible.builtin.set_fact:
pia_active_key_path: "{{ (pia_wg_config_dir + '/' + pia_wg_interface + '.key.new') if (pia_force_key_rotation | bool) else (pia_wg_config_dir + '/' + pia_wg_interface + '.key') }}"
# Rotation stages the new key at a separate *.key.new path and only
# promotes it (see the "Promote the rotated key" block at the end
# of this file) after PIA has accepted it AND the new config has
# been written successfully — the previous key/config are never
# touched until both of those have actually succeeded, so a failed
# rotation leaves the working gateway exactly as it was rather
# than half-migrated.
- name: Generate minisforum's PIA WireGuard private key for this run
ansible.builtin.shell: |
set -euo pipefail
umask 077
wg genkey > {{ pia_active_key_path }}
args:
creates: "{{ pia_active_key_path }}"
- name: Set permissions on the active private key
ansible.builtin.file:
path: "{{ pia_active_key_path }}"
mode: "0600"
owner: root
group: root
- name: Read the active private key
ansible.builtin.slurp:
src: "{{ pia_active_key_path }}"
register: pia_private_key_raw
- name: Derive the public key
ansible.builtin.command: wg pubkey
args:
stdin: "{{ pia_private_key_raw.content | b64decode | trim }}"
register: pia_public_key
changed_when: false
- name: Fetch PIA's region/server list
ansible.builtin.uri:
url: https://serverlist.piaservers.net/vpninfo/servers/v6
return_content: true
register: pia_serverlist_raw
# The response body is one line of minified JSON followed by a
# detached signature block. Take just the first line (matching the
# standard `head -1` extraction used elsewhere for this endpoint) —
# do NOT split on a blank-line separator: that assumption was wrong
# in practice (observed failing against the live response) and
# silently handed the whole blob, signature included, to from_json.
- name: Parse the full region list
# Deliberately one physical line, not a folded (>-) block: an
# earlier version split a similar expression across two lines with
# the second more indented than the first, which YAML's folding
# rule ("more-indented lines are not folded") turns into a literal
# embedded newline instead of a space — that corrupted from_json's
# view of the string in practice (confirmed against the live
# endpoint) even though the newline sits outside any bracket. Keep
# this on one line if it's ever touched again.
ansible.builtin.set_fact:
pia_all_regions: "{{ (pia_serverlist_raw.content.split('\n')[0] | trim | from_json)['regions'] }}"
- name: Look up each requested region's metadata, in order
ansible.builtin.set_fact:
pia_region_lookup: "{{ (pia_region_lookup | default([])) + [pia_match] }}"
loop: "{{ pia_effective_regions }}"
loop_control:
loop_var: pia_region_id
vars:
pia_match_list: "{{ pia_all_regions | selectattr('id', 'equalto', pia_region_id) | list }}"
pia_match: "{{ pia_match_list[0] if (pia_match_list | length > 0) else {'id': pia_region_id, 'name': None, 'servers': {'wg': []}} }}"
# Builds pia_region_lookup as one entry per requested region, in
# the SAME order as pia_effective_regions (not the API's own region
# order) — that order is what makes "try hk, then taiwan, then
# japan" actually mean that. A region that doesn't exist in the
# live serverlist gets a sentinel entry (name: null, no wg
# servers) rather than being silently skipped, so it can be
# reported precisely below instead of just quietly trying one
# fewer region than requested.
- name: Identify any requested regions that don't exist in the current serverlist
ansible.builtin.set_fact:
pia_missing_regions: "{{ pia_region_lookup | selectattr('name', 'none') | map(attribute='id') | list }}"
- name: Identify any requested regions with no WireGuard servers
ansible.builtin.set_fact:
pia_empty_wg_regions: "{{ pia_region_lookup | rejectattr('name', 'none') | rejectattr('servers.wg') | map(attribute='id') | list }}"
- name: Assert every requested region exists and offers WireGuard
ansible.builtin.assert:
that:
- pia_missing_regions | length == 0
- pia_empty_wg_regions | length == 0
fail_msg: >-
One or more requested PIA regions are unusable — check each id
against the 'id' field of
https://serverlist.piaservers.net/vpninfo/servers/v6 (these are
API ids, not Gluetun's display names like "JP Tokyo"). Not
found in the current serverlist at all:
{{ pia_missing_regions | default([]) }}. Found but offer no
WireGuard servers right now:
{{ pia_empty_wg_regions | default([]) }}. Requested (in order):
{{ pia_effective_regions }}. Not silently substituting a
different region — fix pia_region_candidates/pia_region and
rerun.
- name: Build the ordered, flattened list of region+server candidates
ansible.builtin.set_fact:
pia_candidates: "{{ (pia_candidates | default([])) + [{'region_id': pia_pair.0.id, 'region_name': pia_pair.0.name, 'cn': pia_pair.1.cn, 'ip': pia_pair.1.ip}] }}"
loop: "{{ pia_region_lookup | subelements('servers.wg') }}"
loop_control:
loop_var: pia_pair
# subelements preserves both list orders: outer (region, matching
# pia_effective_regions) and inner (server, matching whatever order
# the live API returned for that region — never hardcoded here).
# Each flattened candidate keeps region id, region display name,
# WireGuard hostname, and WireGuard IP together, so
# addkey-attempt.yaml and diagnostics never need to re-look-up the
# parent region for a given server.
- name: Assert at least one candidate server was found
ansible.builtin.assert:
that:
- pia_candidates | length > 0
fail_msg: >-
No WireGuard servers found across any requested region
({{ pia_effective_regions }}) — this should be unreachable given
the region-level assert above; if you see this, the serverlist
response shape has likely changed.
- name: Deploy PIA's CA bundle (WireGuard addKey endpoint only)
ansible.builtin.copy:
src: pia-ca.crt
dest: "{{ pia_wg_config_dir }}/pia-ca.crt"
mode: "0644"
owner: root
group: root
# Only the regional addKey endpoint (1337/addKey, below) needs
# this — it presents a cert chain that does not validate against a
# normal system CA store (confirmed live: curl exit 60, "unable to
# get local issuer certificate"), and PIA's own tooling pins this
# exact bundle rather than trusting the system store. The token
# endpoint (www.privateinternetaccess.com, below) is a normal
# public domain with a normal publicly-trusted cert and uses
# validate_certs: true / the system store — do not add --cacert or
# validate_certs: false there.
#
# Source: https://raw.githubusercontent.com/pia-foss/manual-connections/master/ca.rsa.4096.crt
# Re-fetched fresh and diffed byte-for-byte identical against the
# vendored copy on 2026-08-24.
# File SHA-256 (sha256sum of the .crt file's bytes — this is what
# to compare against a fresh `curl ... | sha256sum` to verify the
# vendored copy, NOT the same thing as the X.509 certificate
# fingerprint below):
# 32e9b1d1433ea97614f2a14c6e358e3f57c0570cc9f6b2ee812699ba696c66ab
# X.509 certificate fingerprint (openssl x509 -noout -fingerprint
# -sha256 — a hash of the DER-encoded certificate structure, a
# different value from the file SHA-256 above; useful for
# comparing against a cert viewed some other way, e.g. in a
# browser or `openssl s_client`):
# SHA256 Fingerprint=1F:D2:56:58:45:6E:AB:30:41:FB:A7:7C:CD:39:8A:B8:12:4E:DC:C1:B8:B2:FC:1D:55:FD:F6:B1:BB:FC:9D:70
- name: Request a PIA auth token
ansible.builtin.uri:
url: https://www.privateinternetaccess.com/api/client/v2/token
method: POST
body_format: form-multipart
body:
username: "{{ pia_user }}"
password: "{{ pia_password }}"
validate_certs: true
return_content: true
timeout: 15
register: pia_token_response
until: pia_token_response.status | default(0) == 200
retries: 3
delay: 5
failed_when: false
no_log: true
# Fixed public endpoint, system CA validation — see get_token.sh.
# Independent of pia_region entirely.
- name: Capture safe (non-credential) diagnostics from the token request
ansible.builtin.set_fact:
pia_token_status: "{{ pia_token_response.status | default(-1) }}"
pia_token_msg: "{{ pia_token_response.msg | default('') }}"
# Deliberately NOT no_log. .status (an HTTP code) and .msg (Ansible's
# own generated "Status code was N and not [200]: ..." text) never
# contain the credential, the token, or the request body — verified
# empirically before use here, not assumed. What DOES contain
# secrets and stays behind no_log: pia_token_response itself (its
# .json/.content on a 200 IS the token; its .invocation.module_args
# is the literal username/password), and pia_user/pia_password.
- name: Fail clearly if the auth token request failed
ansible.builtin.fail:
msg: >-
PIA auth token request failed: HTTP {{ pia_token_status }}
({{ pia_token_msg }}) from
www.privateinternetaccess.com/api/client/v2/token
when: pia_token_status | int != 200
- name: Extract the auth token
ansible.builtin.set_fact:
pia_auth_token: "{{ pia_token_response.json.token | default('') }}"
no_log: true
- name: Assert the token is non-empty
ansible.builtin.assert:
that:
- pia_auth_token | length > 0
fail_msg: "PIA token response (HTTP 200) did not contain a non-empty token"
- name: Attempt WireGuard key registration against each region/server candidate, in order
ansible.builtin.include_tasks: addkey-attempt.yaml
loop: "{{ pia_candidates }}"
loop_control:
loop_var: pia_candidate
# pia_candidates is already flattened and ordered: every server in
# the first requested region, then every server in the second, and
# so on (built above via subelements over pia_region_lookup — never
# re-sorted or grouped differently here). Every meaningful task in
# addkey-attempt.yaml is individually guarded with
# `when: pia_peer is not defined` (re-evaluated fresh for each loop
# iteration) — confirmed empirically that a guard on this include
# statement itself does NOT re-evaluate per iteration and would NOT
# actually stop later candidates from being attempted after an
# earlier one already succeeded; the guard has to live on the inner
# tasks, which is also where addkey-attempt.yaml resets its
# per-attempt state so a stale parsed response from one candidate
# can never be mistaken for another's.
- name: Assert PIA accepted the registration against at least one candidate
ansible.builtin.assert:
that:
- pia_peer is defined
- pia_peer.status == "OK"
- pia_region_used is defined
- pia_wg_server_used is defined
fail_msg: >-
PIA addKey failed against every candidate: {{ pia_candidates |
length }} server(s) across {{ pia_effective_regions | length }}
region(s) ({{ pia_effective_regions }}) all failed. See the
per-attempt diagnostics logged above for exactly why each one
failed (timeout, TLS, HTTP error, empty/malformed response, or
a parsed non-OK status). If every region failed the same way,
that points at something shared across all of them — the
request itself, credentials, or minisforum's own network path —
not at any one region being down.
- name: Write the pia-wg WireGuard interface config
ansible.builtin.template:
src: pia-wg.conf.j2
dest: "{{ pia_wg_config_dir }}/{{ pia_wg_interface }}.conf"
mode: "0600"
owner: root
group: root
vars:
pia_private_key: "{{ pia_private_key_raw.content | b64decode | trim }}"
no_log: true
notify: Restart pia-wg
# ansible.builtin.template writes atomically (temp file + rename),
# so a failure partway through this specific task cannot leave a
# partially-written config — the previous file, if any, is left
# untouched. Everything that can fail on PIA's side (registration,
# both API calls) happens strictly before this point, so the
# previous working config is never at risk from those failures
# either.
- name: Promote the rotated key now that registration and config write both succeeded
when: pia_force_key_rotation | bool
block:
- name: Check whether a previous key exists to back up
ansible.builtin.stat:
path: "{{ pia_wg_config_dir }}/{{ pia_wg_interface }}.key"
register: pia_previous_key_stat
- name: Back up the previous key
ansible.builtin.command:
cmd: >-
cp -p {{ pia_wg_config_dir }}/{{ pia_wg_interface }}.key
{{ pia_wg_config_dir }}/{{ pia_wg_interface }}.key.bak-{{ ansible_date_time.iso8601_basic_short }}
when: pia_previous_key_stat.stat.exists
changed_when: true
- name: Promote the new key into place
ansible.builtin.command:
cmd: >-
mv {{ pia_wg_config_dir }}/{{ pia_wg_interface }}.key.new
{{ pia_wg_config_dir }}/{{ pia_wg_interface }}.key
changed_when: true
# Only reached after the config template task above has already
# succeeded (Ansible blocks/tasks run sequentially and stop on
# first failure) — by this point PIA has accepted the new key and
# the new config referencing it is already safely on disk, so
# promoting the key file itself (for next run's `creates:` check
# to find it under the normal, non-.new name) is the only
# remaining step, not a point where failure could strand anything.

42
ansible/roles/pia-gateway/tasks/routing.yaml
View File

@ -1,42 +0,0 @@
---
# Part of role: pia-gateway
# Included by: tasks/main.yaml
# Description: Registers the dedicated PIA policy-routing table and
# installs+enables the boot-time kill-switch service that seeds it
# (rule, terminal unreachable route, static drop rule) before
# wg-quick@{{ pia_wg_interface }} is allowed to start. See
# templates/pia-killswitch.sh.j2 for what the seeded state actually is.
- name: Register the PIA routing table name
ansible.builtin.lineinfile:
path: /etc/iproute2/rt_tables
line: "{{ pia_route_table_id }}\t{{ pia_route_table_name }}"
regexp: "^[0-9]+\\s+{{ pia_route_table_name }}$"
create: true
mode: "0644"
- name: Deploy the kill-switch seed script
ansible.builtin.template:
src: pia-killswitch.sh.j2
dest: /usr/local/sbin/pia-killswitch.sh
mode: "0700"
owner: root
group: root
notify: Restart pia-killswitch
- name: Deploy the pia-killswitch systemd unit
ansible.builtin.template:
src: pia-killswitch.service.j2
dest: /etc/systemd/system/pia-killswitch.service
mode: "0644"
owner: root
group: root
notify:
- Reload systemd daemon
- Restart pia-killswitch
- name: Enable and start the kill-switch service now
ansible.builtin.systemd:
name: pia-killswitch.service
enabled: true
state: started

7
ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.service.j2
View File

@ -1,7 +0,0 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
[Unit]
Description=PIA gateway health check (observability only, no remediation)
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pia-gateway-healthcheck.sh

110
ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2
View File

@ -1,110 +0,0 @@
#!/bin/bash
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Observability only. This script NEVER modifies routing, firewall, or
# WireGuard state — a failed check must close the path (already true by
# construction, via the kill switch in tasks/routing.yaml/firewall.yaml),
# never "recover" it. Output goes to the journal
# (journalctl -u pia-gateway-healthcheck); wire alerting off of that
# separately (see plan.md Phase 9) rather than adding remediation here.
#
# The "route decision" check below confirms what the kernel FIB *would*
# do for a VLAN 50 source — it does not send a packet and is not proof of
# actual PIA egress or identity. That proof requires the packet-capture
# test matrix in plan.md Phase 5; do not treat a clean run of this script
# as substituting for it.
set -uo pipefail
WG_IF="{{ pia_wg_interface }}"
TABLE="{{ pia_route_table_name }}"
SUBNET="{{ pia_vlan50_subnet }}"
PRIORITY="{{ pia_rule_priority }}"
CHAIN="{{ pia_iptables_chain }}"
MAX_HANDSHAKE_AGE={{ pia_healthcheck_handshake_max_age_sec }}
PROBE_SRC="${SUBNET%.*}.2" # a representative VLAN 50 address for the route-decision probe only; never actually used as a source
fail=0
# Informational only (not pass/fail) — which region/server this host is
# actually registered against right now, per the comment
# tasks/register.yaml writes into the generated config. Since PIA
# WireGuard registration now falls back across multiple regions
# (pia_region_candidates in defaults/main.yaml), "which one is live"
# isn't implied by config alone — it's whichever one answered OK last
# time the playbook ran, not necessarily the first-configured one.
configured_peer=$(grep -m1 '^# PIA WireGuard server actually used' "{{ pia_wg_config_dir }}/{{ pia_wg_interface }}.conf" 2>/dev/null || true)
if [[ -n "${configured_peer}" ]]; then
echo "INFO configured-peer: ${configured_peer#\# }"
fi
if ip link show "${WG_IF}" up &>/dev/null; then
echo "PASS interface: ${WG_IF} is up"
else
echo "FAIL interface: ${WG_IF} is not up"
fail=1
fi
if command -v wg &>/dev/null && wg show "${WG_IF}" latest-handshakes &>/dev/null; then
handshake_epoch=$(wg show "${WG_IF}" latest-handshakes 2>/dev/null | awk '{print $2}')
now=$(date +%s)
if [[ -n "${handshake_epoch}" && "${handshake_epoch}" -gt 0 ]]; then
age=$((now - handshake_epoch))
if (( age <= MAX_HANDSHAKE_AGE )); then
echo "PASS handshake: ${age}s old (max ${MAX_HANDSHAKE_AGE}s)"
else
echo "FAIL handshake: ${age}s old, exceeds max ${MAX_HANDSHAKE_AGE}s"
fail=1
fi
else
echo "FAIL handshake: no handshake recorded yet"
fail=1
fi
else
echo "WARN handshake: unable to query wg show (interface absent or wg missing)"
fail=1
fi
if ip rule show | grep -qE "^${PRIORITY}:[[:space:]]*from ${SUBNET} lookup ${TABLE}\$"; then
echo "PASS rule: from ${SUBNET} lookup ${TABLE} present at priority ${PRIORITY}"
else
echo "FAIL rule: from ${SUBNET} lookup ${TABLE} missing"
fail=1
fi
default_route=$(ip route show table "${TABLE}" 2>/dev/null | grep '^default' || true)
if [[ "${default_route}" == *"dev ${WG_IF}"* ]]; then
echo "PASS route: table ${TABLE} default is via ${WG_IF} (open)"
elif [[ "${default_route}" == *unreachable* ]] || ip route show table "${TABLE}" 2>/dev/null | grep -q '^unreachable default'; then
echo "PASS route: table ${TABLE} default is unreachable (closed, kill switch engaged)"
else
echo "FAIL route: table ${TABLE} has no default route at all (neither open via ${WG_IF} nor a terminal unreachable) — investigate immediately"
fail=1
fi
# Not a check on FORWARD's own default policy — that's deliberately left
# alone (see tasks/firewall.yaml for why: minisforum runs Flannel, which
# needs its own broad FORWARD ACCEPTs). What actually matters is that the
# dedicated chain exists, is jumped into first for this source, and ends
# in a real DROP.
if iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
echo "PASS firewall: FORWARD jumps to ${CHAIN} for ${SUBNET}"
else
echo "FAIL firewall: FORWARD does not jump to ${CHAIN} for ${SUBNET}"
fail=1
fi
if iptables -S "${CHAIN}" 2>/dev/null | grep -qE '^-A '"${CHAIN}"' -j DROP$'; then
echo "PASS firewall: ${CHAIN} ends in an unconditional DROP"
else
echo "FAIL firewall: ${CHAIN} has no unconditional DROP catch-all"
fail=1
fi
route_decision=$(ip route get 1.1.1.1 from "${PROBE_SRC}" 2>&1 || true)
if [[ "${route_decision}" == *"dev ${WG_IF}"* ]]; then
echo "PASS route-decision (not proof of live egress — see header comment): ${SUBNET} sourced traffic resolves via ${WG_IF}"
else
echo "WARN route-decision (not proof of live egress — see header comment): ${route_decision}"
fi
exit "${fail}"

11
ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.timer.j2
View File

@ -1,11 +0,0 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
[Unit]
Description=Run the PIA gateway health check every {{ pia_healthcheck_interval_sec }}s
[Timer]
OnBootSec={{ pia_healthcheck_interval_sec }}
OnUnitActiveSec={{ pia_healthcheck_interval_sec }}
AccuracySec=5s
[Install]
WantedBy=timers.target

20
ansible/roles/pia-gateway/templates/pia-killswitch.service.j2
View File

@ -1,20 +0,0 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Seeds the VLAN 50 kill switch (policy-routing rule, terminal unreachable
# route, static drop rule) before wg-quick@{{ pia_wg_interface }}.service
# is allowed to start — see the drop-in this role installs on that unit
# (wg-quick-order.conf.j2) for the other half of the ordering.
[Unit]
Description=PIA gateway kill switch (VLAN 50 policy routing seed)
Before=wg-quick@{{ pia_wg_interface }}.service
DefaultDependencies=no
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/pia-killswitch.sh
[Install]
WantedBy=multi-user.target

92
ansible/roles/pia-gateway/templates/pia-killswitch.sh.j2
View File

@ -1,92 +0,0 @@
#!/bin/bash
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Runs once at boot, before wg-quick@{{ pia_wg_interface }}.service
# (enforced by the systemd drop-in in pia-killswitch.service's
# Before=/wg-quick's Requires=After=), and again any time UFW is
# reloaded (see handlers/main.yaml — "Reload ufw" fires before "Restart
# pia-killswitch" so a reload can never leave a permanent gap). Seeds the
# closed state that PIA gateway's kill switch relies on:
#
# - a source rule sending {{ pia_vlan50_subnet }} to the dedicated
# "{{ pia_route_table_name }}" table, evaluated after the kernel's
# built-in "local" rule (priority 0) and before "main" (32766) — this
# is what keeps Technitium (10.10.40.53, a local address) delivered
# via the local table rather than pulled into policy routing.
# - a terminal "unreachable" default route in that table — but only
# when {{ pia_wg_interface }} isn't already up — so VLAN 50 traffic
# fails closed instead of falling through to main if the interface is
# down, absent, or its PostUp hasn't run yet, without this script
# stomping an already-open, healthy route on a restart unrelated to
# the tunnel's own state (see the check below).
# - a DEDICATED "{{ pia_iptables_chain }}" chain, jumped into by exactly
# one rule at the very top of FORWARD (position 1), matching only
# source {{ pia_vlan50_subnet }}. This is deliberately NOT a global
# FORWARD default-policy change: minisforum is the k3s server and
# runs Flannel, which depends on its own broad FORWARD-chain
# ACCEPTs for pod-to-pod and pod-to-internet traffic (source
# 10.42.0.0/16, entirely disjoint from 10.10.50.0/24) — changing the
# chain-wide default policy was never actually proven safe against
# that, so this scopes the kill switch to a chain that Flannet/k3s
# traffic can never enter in the first place, rather than risking it.
# Being the FIRST rule in FORWARD also means no other, later rule
# (ufw's own, Docker's, anything) can accidentally pre-empt this
# source's fate with a broader ACCEPT — the jump is unconditional for
# this source and the chain itself ends in an unconditional DROP, not
# a RETURN, so nothing after it in FORWARD is ever consulted for
# this source either.
#
# Idempotent: safe to run more than once (e.g. `systemctl restart
# pia-killswitch`, or after this script re-runs following a UFW reload)
# without creating duplicate rules or duplicate chains.
set -euo pipefail
TABLE="{{ pia_route_table_name }}"
SUBNET="{{ pia_vlan50_subnet }}"
WG_IF="{{ pia_wg_interface }}"
PRIORITY="{{ pia_rule_priority }}"
CHAIN="{{ pia_iptables_chain }}"
if ! ip rule show | grep -qE "^${PRIORITY}:[[:space:]]*from ${SUBNET} lookup ${TABLE}\$"; then
ip rule add from "${SUBNET}" table "${TABLE}" priority "${PRIORITY}"
fi
# Only seed the terminal-unreachable fallback if {{ pia_wg_interface }}
# isn't already up. This script re-runs any time it's restarted for
# reasons unrelated to the tunnel's own state (a UFW reload, a content
# fix to this script itself) — if the interface is already up and
# healthy, its own PostUp already set the real "default dev %i" route in
# this table, and blindly overwriting that back to "unreachable" here
# would break a working tunnel's routing for no reason until something
# re-runs PostUp. Only actual absence of the interface should close it.
if ! ip link show "${WG_IF}" up &>/dev/null; then
ip route replace unreachable default table "${TABLE}"
fi
# Dedicated chain: create if missing (iptables -N fails harmlessly if it
# already exists, hence || true rather than a -C-style existence check —
# there is no direct "does this chain exist" check short of grepping -L).
iptables -N "${CHAIN}" 2>/dev/null || true
# Exactly one jump from FORWARD into it, at absolute position 1.
if ! iptables -C FORWARD -s "${SUBNET}" -j "${CHAIN}" 2>/dev/null; then
iptables -I FORWARD 1 -s "${SUBNET}" -j "${CHAIN}"
fi
# Static catch-all inside the chain: anything that reaches this point
# (i.e., wasn't already ACCEPTed by a more specific rule that
# pia-wg.conf.j2's PostUp inserts ahead of these when the interface is
# up) is logged and dropped. --log-limit is not a real iptables option
# (there is no such flag — rate-limiting a LOG target is done via a
# separate -m limit match ahead of it, as below); the -C check must
# match the rule as actually inserted, term for term, or it will never
# find it and silently re-append a duplicate on every run.
if ! iptables -C "${CHAIN}" -m limit --limit 5/minute -j LOG \
--log-prefix "PIA-KILLSWITCH-DROP: " 2>/dev/null; then
iptables -A "${CHAIN}" -m limit --limit 5/minute -j LOG \
--log-prefix "PIA-KILLSWITCH-DROP: "
fi
if ! iptables -C "${CHAIN}" -j DROP 2>/dev/null; then
iptables -A "${CHAIN}" -j DROP
fi

73
ansible/roles/pia-gateway/templates/pia-wg.conf.j2
View File

@ -1,73 +0,0 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand — changes
# are overwritten on the next run of ansible/playbooks/pia-gateway.yaml.
#
# Table = off: wg-quick must NOT touch the main routing table. All routing
# for this interface lives in the dedicated "{{ pia_route_table_name }}"
# table (see tasks/routing.yaml), reached only by traffic sourced from
# {{ pia_vlan50_subnet }}. minisforum's own default route is never
# affected by this interface coming up or down.
[Interface]
PrivateKey = {{ pia_private_key }}
Address = {{ pia_peer.peer_ip }}/32
Table = off
# Kill-switch ordering: the boot-time pia-killswitch.service (see
# tasks/routing.yaml) has already seeded an "unreachable" default route in
# table {{ pia_route_table_name }} and created the dedicated
# {{ pia_iptables_chain }} chain (jumped into from FORWARD position 1 for
# source {{ pia_vlan50_subnet }} only — never touches Flannel/k3s traffic)
# ending in an unconditional LOG+DROP, before this unit ever starts
# (systemd After=/Requires=). PostUp only ever *opens* the path narrower
# than that pre-seeded closed state, by inserting ACCEPT rules ahead of
# the chain's own catch-all; PreDown removes exactly those same rules,
# re-closing it before removing anything else.
PostUp = ip route replace default dev %i table {{ pia_route_table_name }}
PostUp = iptables -I {{ pia_iptables_chain }} 1 -s {{ pia_vlan50_subnet }} -o %i -j ACCEPT
PostUp = iptables -I {{ pia_iptables_chain }} 1 -d {{ pia_vlan50_subnet }} -i %i -m state --state ESTABLISHED,RELATED -j ACCEPT
PostUp = iptables -t nat -A POSTROUTING -s {{ pia_vlan50_subnet }} -o %i -j MASQUERADE
{% if pia_mss_clamp_enabled %}
PostUp = iptables -t mangle -A FORWARD -s {{ pia_vlan50_subnet }} -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
{% endif %}
# PreDown's iptables -D lines are deliberately tolerant (2>/dev/null ||
# true) of the rule they target not existing — confirmed against
# wg-quick's own source (execute_hooks() runs each line via `(eval
# "$hook")`, and wg-quick's top-level `set -e` treats that subshell's
# exit code as fatal for the WHOLE up/down sequence): without this, a
# single -D failing (e.g. the currently-live interface was brought up
# under an OLDER config that never added the rule this line is now
# trying to remove — root-caused live, 2026-08-27, when enabling MSS
# clamping for the first time made this exact thing happen) aborts
# every PreDown line after it, including wg-quick's own built-in `ip
# link delete`, leaving an orphaned interface that then makes the
# following `wg-quick up` fail too ("pia-wg already exists") — turning
# any future PostUp/PreDown content change into a two-step manual
# recovery (`ip link delete dev pia-wg` before a fresh start) instead
# of a clean, unattended `systemctl restart`. `ip route replace` (the
# first PreDown line) doesn't need this — replace never fails due to
# prior state the way -D does.
PreDown = ip route replace unreachable default table {{ pia_route_table_name }}
PreDown = iptables -t nat -D POSTROUTING -s {{ pia_vlan50_subnet }} -o %i -j MASQUERADE 2>/dev/null || true
PreDown = iptables -D {{ pia_iptables_chain }} -d {{ pia_vlan50_subnet }} -i %i -m state --state ESTABLISHED,RELATED -j ACCEPT 2>/dev/null || true
PreDown = iptables -D {{ pia_iptables_chain }} -s {{ pia_vlan50_subnet }} -o %i -j ACCEPT 2>/dev/null || true
{% if pia_mss_clamp_enabled %}
PreDown = iptables -t mangle -D FORWARD -s {{ pia_vlan50_subnet }} -o %i -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu 2>/dev/null || true
{% endif %}
[Peer]
# PIA WireGuard server actually used — {{ pia_region_name_used }}
# ({{ pia_region_used }}), {{ pia_wg_server_used.cn }}. This is whichever
# region+server in the pia_region_candidates/pia_region fallback order
# (see defaults/main.yaml) responded first with status: OK — not
# necessarily the first-configured region. Re-run the playbook with
# pia_force_reregister (or check journalctl for the last apply's
# per-attempt diagnostics) to see what else was tried before this one.
PublicKey = {{ pia_peer.server_key }}
# The endpoint IP is the server we actually connected to
# (pia_wg_server_used, set by addkey-attempt.yaml on whichever candidate
# in servers.wg succeeded), not a field from the addKey JSON response —
# matching connect_to_wireguard_with_token.sh, which builds the Endpoint
# from its own $WG_SERVER_IP rather than trusting a response field for it.
Endpoint = {{ pia_wg_server_used.ip }}:{{ pia_peer.server_port }}
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25

41
ansible/roles/vlan50-parent/README.md
View File

@ -1,41 +0,0 @@
# vlan50-parent
Creates `enp1s0.50`, a tagged VLAN 50 parent interface on `nik-debian`, for
Multus macvlan/ipvlan attachments (Phase 3/4 of `~/repo/homelab/plan.md`).
Leaves the existing untagged VLAN 40 config (`enp1s0`'s own address,
default route, k3s node identity, host DNS) untouched.
## Before running
- Requires the managed-switch port for `nik-debian` already carrying VLAN
50 tagged in addition to its existing VLAN 40 untagged/PVID 40 — this is
the `home-network`-repo-owned switch/Flint handoff from plan.md Phase 1.
This role has no way to verify that from the host side; if the switch
isn't actually passing tagged VLAN 50 frames yet, the subinterface will
come up with link state but no VLAN 50 traffic will ever arrive.
- Confirm console/recovery access to `nik-debian` (physical/IPMI/other
out-of-band) before applying, same as any host networking change.
## What it does
- Installs the `vlan` package and loads/persists the `8021q` kernel
module.
- Writes `/etc/network/interfaces.d/enp1s0.50` — `iface ... inet manual`
with `vlan-raw-device enp1s0` and `vlan-id 50`. No IP address is ever
assigned to it.
- Brings the interface up now (`ifup`) if not already present, and
asserts afterward that it's UP, carries no IPv4/IPv6 address, and that
the host's own default route is still via `enp1s0` — never via
`enp1s0.50`.
## Rollback
```bash
sudo ifdown enp1s0.50 || true
sudo ip link delete enp1s0.50 2>/dev/null || true
sudo rm -f /etc/network/interfaces.d/enp1s0.50
sudo rm -f /etc/modules-load.d/8021q.conf
```
Leaves the `vlan` package and `8021q` module load itself in place (harmless
if unused); remove manually only if desired.

20
ansible/roles/vlan50-parent/defaults/main.yaml
View File

@ -1,20 +0,0 @@
---
# Part of role: vlan50-parent
# Called by: ansible/playbooks/vlan50-parent.yaml
# Description: Default variables for nik-debian's tagged VLAN 50 parent
# interface. Implements Phase 3 of ~/repo/homelab/plan.md. Only apply
# this role after the home-network handoff confirms the managed-switch
# port for nik-debian is VLAN 40 untagged/PVID 40 *and* VLAN 50 tagged —
# this role does not and cannot verify switch-side config itself.
# Real parent interface on nik-debian — verified live 2026-08-23 via
# `ip -brief addr`; it is enp1s0, not "eth0".
vlan50_parent_interface: enp1s0
vlan50_id: 50
vlan50_subinterface: "{{ vlan50_parent_interface }}.{{ vlan50_id }}"
# No L3 address is assigned to the subinterface by this role (plan.md:
# "prefer an UP L2 parent with no L3 address" — Multus's macvlan/ipvlan
# CNI attaches directly to it at L2; the host itself never needs an
# address on VLAN 50).
vlan50_mtu: 1500

74
ansible/roles/vlan50-parent/tasks/main.yaml
View File

@ -1,74 +0,0 @@
---
# Part of role: vlan50-parent
# Called by: ansible/playbooks/vlan50-parent.yaml
# Description: Creates nik-debian's persistent tagged VLAN 50 subinterface
# ({{ vlan50_subinterface }}) without touching its existing untagged
# VLAN 40 config (node IP, default route, k3s identity, host DNS). See
# this role's defaults/main.yaml for the switch-readiness precondition.
- name: Install the vlan package (ifupdown VLAN support)
ansible.builtin.apt:
name: vlan
state: present
update_cache: true
- name: Load the 8021q kernel module now
community.general.modprobe:
name: 8021q
state: present
- name: Persist the 8021q kernel module across reboots
ansible.builtin.copy:
dest: /etc/modules-load.d/8021q.conf
content: |
# Managed by Ansible (role: vlan50-parent).
8021q
mode: "0644"
owner: root
group: root
- name: Write the VLAN 50 subinterface definition
ansible.builtin.template:
src: vlan50.interface.j2
dest: "/etc/network/interfaces.d/{{ vlan50_subinterface }}"
mode: "0644"
owner: root
group: root
register: vlan50_iface_file
- name: Check whether the subinterface is already up
ansible.builtin.command: "ip link show {{ vlan50_subinterface }}"
register: vlan50_iface_check
changed_when: false
failed_when: false
- name: Bring up the VLAN 50 subinterface
ansible.builtin.command: "ifup {{ vlan50_subinterface }}"
when: vlan50_iface_check.rc != 0 or vlan50_iface_file is changed
- name: Confirm the subinterface is up with no L3 address
ansible.builtin.command: "ip -brief addr show {{ vlan50_subinterface }}"
register: vlan50_iface_state
changed_when: false
- name: Assert VLAN 50 parent is UP and carries no IPv4/IPv6 address
ansible.builtin.assert:
that:
- "'UP' in vlan50_iface_state.stdout or 'UNKNOWN' in vlan50_iface_state.stdout"
- vlan50_iface_state.stdout.split() | select('match', '^[0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+/') | list | length == 0
fail_msg: "{{ vlan50_subinterface }} is either down or unexpectedly carries an IPv4 address: {{ vlan50_iface_state.stdout }}"
- name: Read current default route
ansible.builtin.command: ip -4 route show default
register: vlan50_current_default_route
changed_when: false
- name: Assert the host default route is still the untagged VLAN 40 interface
ansible.builtin.assert:
that:
- vlan50_current_default_route.stdout is search('dev ' + vlan50_parent_interface + '(\s|$)')
- not (vlan50_current_default_route.stdout is search('dev ' + vlan50_subinterface + '(\s|$)'))
fail_msg: >-
nik-debian's default route is not via {{ vlan50_parent_interface }}
({{ vlan50_current_default_route.stdout }}) — VLAN 50 must never
become the host default route.

13
ansible/roles/vlan50-parent/templates/vlan50.interface.j2
View File

@ -1,13 +0,0 @@
# Managed by Ansible (role: vlan50-parent). Do not edit by hand.
#
# Tagged VLAN 50 parent for Multus macvlan/ipvlan attachments — see
# ~/repo/homelab/plan.md Phase 3/4. Deliberately "manual" (no inet
# config): this interface must never carry a host IP or become a route
# candidate. Untagged VLAN 40 on {{ vlan50_parent_interface }} itself
# (node IP, default route, k3s identity) is defined elsewhere in
# /etc/network/interfaces and is not touched by this file.
auto {{ vlan50_subinterface }}
iface {{ vlan50_subinterface }} inet manual
vlan-raw-device {{ vlan50_parent_interface }}
vlan-id {{ vlan50_id }}
mtu {{ vlan50_mtu }}

16
ansible/roles/wireguard/tasks/main.yaml
View File

@ -139,12 +139,12 @@
[Interface]
PrivateKey = {{ phone_private_key.content | b64decode | trim }}
Address = 10.10.0.2/32
DNS = 10.10.40.53
DNS = 192.168.7.77
[Peer]
PublicKey = {{ server_public_key.stdout }}
Endpoint = {{ wireguard_endpoint }}:51820
AllowedIPs = 192.168.7.0/24, 10.10.40.0/24, 10.10.0.0/24
AllowedIPs = 192.168.7.0/24, 10.10.0.0/24
PersistentKeepalive = 25
# --- Mac client config ---
@ -158,32 +158,32 @@
[Interface]
PrivateKey = {{ mac_private_key.content | b64decode | trim }}
Address = 10.10.0.3/32
DNS = 10.10.40.53
DNS = 192.168.7.77
[Peer]
PublicKey = {{ server_public_key.stdout }}
Endpoint = {{ wireguard_endpoint }}:51820
AllowedIPs = 192.168.7.0/24, 10.10.40.0/24, 10.10.0.0/24
AllowedIPs = 192.168.7.0/24, 10.10.0.0/24
PersistentKeepalive = 25
- name: Display mac client config
ansible.builtin.shell: cat /etc/wireguard/mac-client.conf
register: mac_conf
changed_when: false
when: wireguard_show_client_configs | default(false) | bool
when: wireguard_show_client_configs | default(false)
- name: Show mac client config
ansible.builtin.debug:
msg: "{{ mac_conf.stdout_lines }}"
when: wireguard_show_client_configs | default(false) | bool
when: wireguard_show_client_configs | default(false)
- name: Generate QR code for phone
ansible.builtin.shell: qrencode -t ansiutf8 < /etc/wireguard/phone-client.conf
register: phone_qr
changed_when: false
when: wireguard_show_client_configs | default(false) | bool
when: wireguard_show_client_configs | default(false)
- name: Display phone QR code
ansible.builtin.debug:
msg: "{{ phone_qr.stdout_lines }}"
when: wireguard_show_client_configs | default(false) | bool
when: wireguard_show_client_configs | default(false)

23
argocd/apps/me-nik4nao-com.yaml
View File

@ -1,23 +0,0 @@
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: me-nik4nao-com
namespace: argocd
spec:
project: default
source:
repoURL: https://gitea.nik4nao.com/nik/homelab.git
targetRevision: main
path: manifests/me-nik4nao-com
directory:
recurse: true
include: '*.yaml'
destination:
server: https://kubernetes.default.svc
namespace: me-nik4nao-com
syncPolicy:
automated:
prune: false
selfHeal: true
syncOptions:
- CreateNamespace=true

27
argocd/apps/multus.yaml
View File

@ -1,27 +0,0 @@
# Multus + VLAN 50 NAD + canary for VPN VLAN 50 (plan.md Phase 4/5).
#
# Deliberately manual sync (no syncPolicy.automated) — unlike this repo's
# other Applications, every resource here (Multus DaemonSet, NAD, canary
# pod) needs an explicit human go-ahead before it ever touches the
# cluster, and staying manual means merging this file to main does NOT
# by itself deploy anything. Use selective sync (argocd app sync multus
# --resource <group:kind:name>, or the UI's per-resource sync) to apply
# 00/01/02 (Multus itself), verify, then 10 (the NAD), verify, then 20
# (the canary) only once the user has separately approved running it.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: multus
namespace: argocd
spec:
project: default
source:
repoURL: https://gitea.nik4nao.com/nik/homelab.git
targetRevision: main
path: manifests/multus
destination:
server: https://kubernetes.default.svc
namespace: kube-system
syncPolicy:
syncOptions:
- CreateNamespace=true

2
config/dashy/conf.yaml
View File

@ -228,7 +228,7 @@ sections:
id: 1_services_dashy
- title: Ollama
icon: mdi-robot-outline
url: http://10.10.40.30:11434
url: http://192.168.7.96:11434
target: newtab
id: 2_services_ollama
- title: Glances

2
manifests/README.md
View File

@ -14,10 +14,8 @@ subdirectories are consumed by Argo CD Applications in `argocd/apps`.
| `gitea/` | Gitea storage, backup, public ingress, runner and OIDC/admin secrets |
| `home-services/` | HA gateway, AI gateway, Discord bot, service TLS, registry secret |
| `homeassistant/` | Home Assistant external service and ingress |
| `me-nik4nao-com/` | me.nik4nao.com deployment, ingress, registry pull secret |
| `media/` | Jellyfin, qBittorrent, JDownloader ([sending links](media/jdownloader.md)) |
| `monitoring/` | Monitoring PVs, Grafana datasource, Grafana/Auth OIDC secrets |
| `multus/` | Multus CNI, VLAN 50 NetworkAttachmentDefinition, canary, browser-vpn-proxy SOCKS5 proxy ([details](multus/README.md)) — manual-sync Application, see `argocd/apps/multus.yaml` |
| `network/` | Pi-hole secrets, DDNS, Traefik dashboard, external host ingresses |
| `portfolio/` | Portfolio deployment, ingress, registry pull secret |

4
manifests/core/coredns-custom.yaml
View File

@ -1,6 +1,6 @@
# Apply: kubectl apply -f manifests/core/coredns-custom.yaml
# Delete: kubectl delete -f manifests/core/coredns-custom.yaml
# Description: CoreDNS custom ConfigMap to forward .home.arpa DNS queries to Pihole at 10.10.40.53.
# Description: CoreDNS custom ConfigMap to forward .home.arpa DNS queries to Pihole at 192.168.7.77.
apiVersion: v1
kind: ConfigMap
metadata:
@ -11,5 +11,5 @@ data:
home.arpa:53 {
errors
cache 30
forward . 10.10.40.53
forward . 192.168.7.77
}

2
manifests/gitea/gitea-backup.yaml
View File

@ -96,7 +96,7 @@ spec:
accessModes:
- ReadWriteMany
nfs:
server: 10.10.40.20
server: 192.168.7.183
path: /home/nik/backups/gitea
persistentVolumeReclaimPolicy: Retain
---

2
manifests/home-services/ai-gateway.yaml
View File

@ -27,7 +27,7 @@ spec:
- name: GRPC_PORT
value: "50052"
- name: OLLAMA_URL
value: "http://10.10.40.30:11434"
value: "http://192.168.7.96:11434"
- name: OLLAMA_MODEL
value: "llama3"
- name: OLLAMA_TIMEOUT

2
manifests/homeassistant/homeassistant.yaml
View File

@ -6,7 +6,7 @@ metadata:
namespace: default
subsets:
- addresses:
- ip: 10.10.40.53
- ip: 192.168.7.77
ports:
- port: 8123
---

97
manifests/me-nik4nao-com/me-nik4nao-com.yaml
View File

@ -1,97 +0,0 @@
# Apply: kubectl apply -f manifests/me-nik4nao-com/me-nik4nao-com.yaml
# Delete: kubectl delete -f manifests/me-nik4nao-com/me-nik4nao-com.yaml
# Description: me.nik4nao.com (terminal-style personal site) deployment, service, TLS certificate, and public IngressRoute. Same pattern as manifests/portfolio.
---
apiVersion: v1
kind: Namespace
metadata:
name: me-nik4nao-com
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: me-nik4nao-com
namespace: me-nik4nao-com
spec:
replicas: 1
selector:
matchLabels:
app: me-nik4nao-com
template:
metadata:
labels:
app: me-nik4nao-com
spec:
imagePullSecrets:
- name: gitea-registry
containers:
- name: me-nik4nao-com
image: gitea.nik4nao.com/nik/me.nik4nao.com:latest
imagePullPolicy: Always
ports:
- containerPort: 80
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
name: me-nik4nao-com
namespace: me-nik4nao-com
spec:
selector:
app: me-nik4nao-com
ports:
- port: 80
targetPort: 80
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: me-nik4nao-com-tls
namespace: me-nik4nao-com
spec:
secretName: me-nik4nao-com-tls
issuerRef:
name: letsencrypt-prod
kind: ClusterIssuer
dnsNames:
- me.nik4nao.com
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: me-nik4nao-com
namespace: me-nik4nao-com
spec:
entryPoints:
- websecure
routes:
- match: Host(`me.nik4nao.com`)
kind: Rule
middlewares:
- name: me-nik4nao-com-ratelimit
services:
- name: me-nik4nao-com
port: 80
tls:
secretName: me-nik4nao-com-tls
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: me-nik4nao-com-ratelimit
namespace: me-nik4nao-com
spec:
rateLimit:
average: 50
burst: 100
period: 1m
sourceCriterion:
ipStrategy:
depth: 1

14
manifests/me-nik4nao-com/registry-secret.sh
View File

@ -1,14 +0,0 @@
#!/usr/bin/env bash
# Usage: bash manifests/me-nik4nao-com/registry-secret.sh
# Description: Creates the Gitea container registry pull secret for the me-nik4nao-com namespace
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../../.env"
kubectl create secret docker-registry gitea-registry \
--namespace me-nik4nao-com \
--docker-server="$REGISTRY_SERVER" \
--docker-username="$REGISTRY_USER" \
--docker-password="$REGISTRY_PASSWORD" \
--dry-run=client -o yaml | kubectl apply -f -

99
manifests/media/gluetun-proxy.yaml Normal file
View File

@ -0,0 +1,99 @@
# Apply: kubectl apply -f manifests/network/gluetun-proxy.yaml
# Delete: kubectl delete -f manifests/network/gluetun-proxy.yaml
# Description: Standalone Gluetun HTTP proxy for browser traffic via PIA.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: gluetun-proxy
namespace: downloads
spec:
replicas: 1
selector:
matchLabels:
app: gluetun-proxy
template:
metadata:
labels:
app: gluetun-proxy
spec:
containers:
- name: gluetun
image: qmcgaw/gluetun:v3.41.1
securityContext:
capabilities:
add:
- NET_ADMIN
env:
- name: VPN_SERVICE_PROVIDER
value: private internet access
- name: VPN_TYPE
value: openvpn
- name: SERVER_REGIONS
value: South Korea,Taiwan,Hong Kong
- name: OPENVPN_USER
valueFrom:
secretKeyRef:
name: pia-credentials
key: OPENVPN_USER
- name: OPENVPN_PASSWORD
valueFrom:
secretKeyRef:
name: pia-credentials
key: OPENVPN_PASSWORD
- name: HTTPPROXY
value: "on"
- name: HTTPPROXY_LISTENING_ADDRESS
value: ":8888"
- name: HTTPPROXY_LOG
value: "off"
- name: FIREWALL_OUTBOUND_SUBNETS
value: "10.42.0.0/16,10.43.0.0/16,192.168.7.0/24"
- name: BLOCK_IPV6
value: "on"
ports:
- containerPort: 8888
startupProbe:
exec:
command:
- /gluetun-entrypoint
- healthcheck
periodSeconds: 10
failureThreshold: 60
livenessProbe:
exec:
command:
- /gluetun-entrypoint
- healthcheck
periodSeconds: 30
failureThreshold: 3
readinessProbe:
exec:
command:
- /gluetun-entrypoint
- healthcheck
periodSeconds: 10
failureThreshold: 3
volumeMounts:
- name: tun
mountPath: /dev/net/tun
volumes:
- name: tun
hostPath:
path: /dev/net/tun
type: CharDevice
---
apiVersion: v1
kind: Service
metadata:
name: gluetun-proxy
namespace: downloads
spec:
selector:
app: gluetun-proxy
type: NodePort
ports:
- name: httpproxy
port: 8888
targetPort: 8888
nodePort: 30888

136
manifests/media/jdownloader.yaml
View File

@ -1,23 +1,6 @@
# Apply: kubectl apply -f manifests/media/jdownloader.yaml
# Delete: kubectl delete -f manifests/media/jdownloader.yaml
# Description: JDownloader deployment with Ingress at jdownloader.home.arpa.
#
# VLAN 50 MIGRATION — live since 2026-08-24, migrated after qBittorrent
# was live, validated, and soaked (plan.md's explicit ordering — the two
# were deliberately not batched). Same VLAN 50/Multus design as
# qbittorrent.yaml — see the root README's "VPN VLAN 50" section for the
# full status/runbook. One difference: JDownloader has no
# reliably-persistent, file-editable interface-bind setting the way
# qBittorrent's qBittorrent.conf does (its own preferences aren't a
# simple INI this repo can safely patch), so this migration relies on the
# network-namespace egress guard alone for enforcement, exactly as
# plan.md anticipated for JDownloader specifically ("application-level
# interface binding may be weaker or unavailable... the namespace egress
# guard is mandatory") — confirmed live via this workload's own
# fail-closed kill-switch test. `media` has selfHeal+automated sync — any
# future push to this file deploys immediately. No pre-migration Gluetun
# copy is kept on disk; roll back via git history instead — see the root
# README's Rollback table for the exact commit.
apiVersion: apps/v1
kind: Deployment
metadata:
@ -34,65 +17,66 @@ spec:
metadata:
labels:
app: jdownloader
annotations:
k8s.v1.cni.cncf.io/networks: |
[{"name": "vlan50", "namespace": "downloads", "interface": "net1", "ips": ["10.10.50.11/24"]}]
spec:
# Hard-pinned to nik-debian, not just node-role: storage — see
# qbittorrent.yaml for why.
nodeSelector:
node-role: storage
kubernetes.io/hostname: nik-debian
# No cluster DNS/CoreDNS resolution needed or provided here — see
# qbittorrent.yaml for the full reasoning (identical here: this pod
# never looks up an in-cluster service by name).
dnsPolicy: None
dnsConfig:
nameservers:
- "10.10.40.53"
initContainers:
# NET_ADMIN lives here ONLY. No qBittorrent-style config-bind
# sibling init container — see this file's header for why
# JDownloader relies on the egress guard alone.
- name: vlan50-egress-guard
image: nicolaka/netshoot:v0.11
command: ["/bin/sh", "/scripts/guard.sh"]
env:
- name: VLAN50_GATEWAY
value: "10.10.50.1"
- name: TECHNITIUM_IP
value: "10.10.40.53"
- name: POD_CIDR
value: "10.42.0.0/16"
- name: SERVICE_CIDR
value: "10.43.0.0/16"
- name: NODE_IP
value: "10.10.40.20"
- name: EXPECTED_VLAN50_IP
value: "10.10.50.11"
containers:
- name: gluetun
image: qmcgaw/gluetun:v3.41.1
securityContext:
capabilities:
drop: ["ALL"]
# NET_RAW alongside NET_ADMIN: the guard script's
# arping-based duplicate-address and gateway-reachability
# checks need it — see
# vlan50-egress-guard-script.yaml's header comment.
add: ["NET_ADMIN", "NET_RAW"]
add:
- NET_ADMIN
env:
- name: VPN_SERVICE_PROVIDER
value: private internet access
- name: VPN_TYPE
value: openvpn
- name: SERVER_REGIONS
value: Singapore,South Korea,Taiwan,Hong Kong
# value: US Seattle,US Oregon,US Silicon Valley
- name: OPENVPN_USER
valueFrom:
secretKeyRef:
name: pia-credentials
key: OPENVPN_USER
- name: OPENVPN_PASSWORD
valueFrom:
secretKeyRef:
name: pia-credentials
key: OPENVPN_PASSWORD
- name: FIREWALL_OUTBOUND_SUBNETS
value: "10.42.0.0/16,10.43.0.0/16,192.168.7.0/24"
- name: BLOCK_IPV6
value: "on"
startupProbe:
exec:
command:
- /gluetun-entrypoint
- healthcheck
periodSeconds: 10
failureThreshold: 60
livenessProbe:
exec:
command:
- /gluetun-entrypoint
- healthcheck
periodSeconds: 30
failureThreshold: 3
readinessProbe:
exec:
command:
- /gluetun-entrypoint
- healthcheck
periodSeconds: 10
failureThreshold: 3
volumeMounts:
- name: guard-script
mountPath: /scripts
containers:
- name: tun
mountPath: /dev/net/tun
- name: jdownloader
image: jlesage/jdownloader-2:latest
ports:
- containerPort: 5800
# No securityContext capability restriction here, deliberately
# — jlesage/jdownloader-2 uses the same docker-baseimage-gui
# PUID/GID pattern as linuxserver's images (USER_ID/GROUP_ID
# below): starts as root, its own init chowns /config to the
# requested uid/gid and drops privileges from there. See
# qbittorrent.yaml's app container for the fuller version of
# this reasoning — same conclusion, same kind of image.
env:
- name: USER_ID
value: "1000"
@ -128,12 +112,6 @@ spec:
image: python:3.12-alpine
ports:
- containerPort: 9667
# Unlike the jdownloader container above, this is a plain
# Python base image with no PUID/GID privilege-drop machinery
# to preserve — dropping all capabilities here is safe.
securityContext:
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 5m
@ -171,10 +149,8 @@ spec:
# .torrent files aren't handled by JDownloader's crawler (no BT
# plugin), so fetch the raw file ourselves instead of writing a
# crawljob. This container shares the pod's network namespace
# (and its vlan50-egress-guard init container's routes/rules),
# so the fetch is still VPN-routed same as JDownloader's own
# downloads.
# crawljob. This container shares gluetun's netns, so the fetch
# is still VPN-routed same as JDownloader's own downloads.
if basename.lower().endswith('.torrent'):
safe_name = basename.replace('/', '_').replace('\\', '_')
try:
@ -207,6 +183,10 @@ spec:
http.server.HTTPServer(('0.0.0.0', 9667), Handler).serve_forever()
volumes:
- name: tun
hostPath:
path: /dev/net/tun
type: CharDevice
- name: config
hostPath:
path: /data/jdownloader
@ -215,10 +195,6 @@ spec:
hostPath:
path: /mnt/storage/dl
type: Directory
- name: guard-script
configMap:
name: vlan50-egress-guard-script
defaultMode: 365 # octal 0555, r-xr-xr-x
---
apiVersion: v1
kind: Service

117
manifests/media/jellyfin-header-debug.yaml
View File

@ -1,117 +0,0 @@
# Apply: kubectl apply -f manifests/media/jellyfin-header-debug.yaml
# Delete: kubectl delete -f manifests/media/jellyfin-header-debug.yaml
# Description: TEMPORARY compat shim + diagnostic proxy in front of the real
# jellyfin Service. The Filebar iOS app (<=1.4.13) sends its auth header only
# on the legacy X-Emby-Authorization header, unquoted (e.g. Client=Filebar
# instead of Client="Filebar"), and never sends the modern Authorization
# header Jellyfin 12.0 requires to identify the client -- so every login
# gets a 400 (ArgumentNullException on request.App). This proxy rewrites
# just Filebar's requests into the header format Jellyfin expects (matching
# what SenPlayer already sends correctly) and forwards everything else
# unchanged. All requests are also logged to stdout for visibility.
#
# TODO: DELETE this file and revert the jellyfin Ingress backend to
# `jellyfin` once Filebar ships a fixed release that sends a correct
# Authorization header on its own -- this is a workaround for their bug,
# not something to keep around permanently.
apiVersion: v1
kind: ConfigMap
metadata:
name: jellyfin-debug-proxy-conf
namespace: jellyfin
data:
default.conf: |
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
log_format jf_debug '$time_iso8601 client=$remote_addr method=$request_method uri="$request_uri" '
'status=$status '
'user_agent="$http_user_agent" '
'authorization="$http_authorization" '
'x_emby_authorization="$http_x_emby_authorization" '
'x_emby_token="$http_x_emby_token" '
'x_mediabrowser_token="$http_x_mediabrowser_token"';
server {
listen 8080;
access_log /dev/stdout jf_debug;
# TEMPORARY Filebar compat shim -- see TODO in this file's header.
# Only touches requests from Filebar's broken auth header; everything
# else (SenPlayer, web UI, etc.) passes through untouched.
access_by_lua_block {
local ua = ngx.var.http_user_agent or ""
local authorization = ngx.var.http_authorization
local legacy = ngx.var.http_x_emby_authorization
if ua:find("^Filebar") and (not authorization or authorization == "") and legacy then
local client = legacy:match("Client=([^,]+)")
local device = legacy:match("Device=([^,]+)")
local deviceid = legacy:match("DeviceId=([^,]+)")
local version = legacy:match("Version=([^,]+)")
local token = legacy:match("Token=([^,]+)")
if client then
local fixed = string.format('MediaBrowser Client="%s", Device="%s", DeviceId="%s", Version="%s"',
client, device or "", deviceid or "", version or "")
if token and token ~= "" then
fixed = fixed .. string.format(', Token="%s"', token)
end
ngx.req.set_header("Authorization", fixed)
ngx.req.set_header("X-Emby-Authorization", fixed)
end
end
}
location / {
proxy_pass http://jellyfin.jellyfin.svc.cluster.local:80;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_read_timeout 3600s;
}
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: jellyfin-debug-proxy
namespace: jellyfin
spec:
replicas: 1
selector:
matchLabels:
app: jellyfin-debug-proxy
template:
metadata:
labels:
app: jellyfin-debug-proxy
spec:
containers:
- name: nginx
image: openresty/openresty:1.31.1.1-alpine
ports:
- containerPort: 8080
volumeMounts:
- name: conf
mountPath: /etc/nginx/conf.d
volumes:
- name: conf
configMap:
name: jellyfin-debug-proxy-conf
---
apiVersion: v1
kind: Service
metadata:
name: jellyfin-debug-proxy
namespace: jellyfin
spec:
selector:
app: jellyfin-debug-proxy
ports:
- port: 80
targetPort: 8080

9
manifests/media/jellyfin.yaml
View File

@ -17,7 +17,7 @@ spec:
- ReadOnlyMany
persistentVolumeReclaimPolicy: Retain
nfs:
server: 10.10.40.20
server: 192.168.7.183
path: /mnt/storage
nodeAffinity:
required:
@ -85,7 +85,7 @@ spec:
- primary
containers:
- name: jellyfin
image: jellyfin/jellyfin:12.0
image: jellyfin/jellyfin:latest
ports:
- containerPort: 8096
env:
@ -169,9 +169,6 @@ spec:
pathType: Prefix
backend:
service:
# TEMPORARY: routed through jellyfin-debug-proxy (see
# jellyfin-header-debug.yaml) to capture auth headers from
# the Filebar app. Revert to `jellyfin` once done.
name: jellyfin-debug-proxy
name: jellyfin
port:
number: 80

2
manifests/media/kavita.yaml
View File

@ -17,7 +17,7 @@ spec:
- ReadOnlyMany
persistentVolumeReclaimPolicy: Retain
nfs:
server: 10.10.40.20
server: 192.168.7.183
path: /mnt/storage/books
nodeAffinity:
required:

2
manifests/media/kavita2.yaml
View File

@ -17,7 +17,7 @@ spec:
- ReadOnlyMany
persistentVolumeReclaimPolicy: Retain
nfs:
server: 10.10.40.20
server: 192.168.7.183
path: /mnt/storage/jellyfin/other
nodeAffinity:
required:

248
manifests/media/qbittorrent.yaml
View File

@ -1,22 +1,6 @@
# Apply: kubectl apply -f manifests/media/qbittorrent.yaml
# Delete: kubectl delete -f manifests/media/qbittorrent.yaml
# Description: qBittorrent deployment with Ingress at qbittorrent.home.arpa.
#
# VLAN 50 MIGRATION — live since 2026-08-24, validated end to end
# (Phase 5 canary matrix plus this workload's own fail-closed kill-switch
# test: stopping wg-quick@pia-wg on minisforum stops its egress, no
# fallback to eth0). This Deployment attaches VLAN 50 (net1, 10.10.50.10)
# via Multus instead of running a per-pod Gluetun/OpenVPN sidecar — PIA
# egress is minisforum's job (ansible/roles/pia-gateway), reached through
# nik-debian's enp1s0.50 (ansible/roles/vlan50-parent) and the Multus NAD
# (manifests/multus/10-nad-vlan50.yaml); see the root README's "VPN VLAN
# 50" section for the full status/runbook. `media` (this file's
# Application) has selfHeal+automated sync — any future push to this file
# deploys immediately, no separate "sync" gate. No pre-migration Gluetun
# copy is kept on disk; roll back via git history instead (`git show
# <commit-before-migration>:manifests/media/qbittorrent.yaml`, then
# `kubectl apply -f -`) — see the root README's Rollback table for the
# exact commit.
apiVersion: v1
kind: Namespace
metadata:
@ -53,163 +37,61 @@ spec:
metadata:
labels:
app: qbittorrent
annotations:
k8s.v1.cni.cncf.io/networks: |
[{"name": "vlan50", "namespace": "downloads", "interface": "net1", "ips": ["10.10.50.10/24"]}]
spec:
# Hard-pinned to nik-debian, not just node-role: storage — plan.md
# requires an explicit pin, not an incidental one, since node-role:
# storage could someday apply to a second node.
nodeSelector:
node-role: storage
kubernetes.io/hostname: nik-debian
# No cluster DNS/CoreDNS resolution needed or provided here — this
# pod never looks up an in-cluster service by name (torrent
# trackers/peers and the web UI are all external or address-based).
# dnsPolicy: None + a single nameserver means Technitium sees this
# pod's queries with its real VLAN 50 source, routed via net1 per
# the egress-guard init container below — a normal ClusterFirst
# policy would instead send queries to CoreDNS over eth0, and
# Technitium would never see this workload's queries or source at
# all. Deliberately one resolver, not several — a second, un-audited
# resolver silently taking over on NXDOMAIN is exactly the kind of
# policy bypass this design exists to prevent, not something to
# leave available "just in case".
dnsPolicy: None
dnsConfig:
nameservers:
- "10.10.40.53"
initContainers:
# NET_ADMIN lives here ONLY — dropped from every app container
# below. See manifests/multus/vlan50-egress-guard-script.yaml for
# exactly what this does and does not prove.
- name: vlan50-egress-guard
image: nicolaka/netshoot:v0.11
command: ["/bin/sh", "/scripts/guard.sh"]
env:
- name: VLAN50_GATEWAY
value: "10.10.50.1"
- name: TECHNITIUM_IP
value: "10.10.40.53"
- name: POD_CIDR
value: "10.42.0.0/16"
- name: SERVICE_CIDR
value: "10.43.0.0/16"
- name: NODE_IP
value: "10.10.40.20"
- name: EXPECTED_VLAN50_IP
value: "10.10.50.10"
securityContext:
capabilities:
drop: ["ALL"]
# NET_RAW alongside NET_ADMIN: the guard script's
# arping-based duplicate-address and gateway-reachability
# checks need it — see
# vlan50-egress-guard-script.yaml's header comment.
add: ["NET_ADMIN", "NET_RAW"]
volumeMounts:
- name: guard-script
mountPath: /scripts
# Idempotently ensures qBittorrent's own config binds it to net1
# /10.10.50.10 ("bind qBittorrent to net1... using a supported
# persistent setting" — plan.md). Sets the CURRENT interface-bind
# keys for qBittorrent 5.2.0 — [BitTorrent] Session\Interface,
# Session\InterfaceName, Session\InterfaceAddress. An earlier
# version of this file set the legacy [Preferences]
# Connection\Interface* keys instead; confirmed against
# qBittorrent's own src/app/upgrade.cpp (fetched and read in
# full) that those are the pre-migration names — the migration
# table maps {new: "BitTorrent/Session/InterfaceAddress", old:
# "Preferences/Connection/InterfaceAddress"} (and likewise for
# Interface/InterfaceName), and src/base/bittorrent/
# sessionimpl.cpp confirms BITTORRENT_SESSION_KEY("Interface")/
# ("InterfaceAddress") are what the actual listening/binding
# logic reads (getListeningIPs()) — InterfaceName isn't read
# there but is set alongside the other two for consistency with
# what the GUI's own interface picker would persist. Only these
# three keys are touched; every other persisted setting
# (categories, ports, credentials, RSS) is left alone.
#
# Runs as 1000:1000 (matching PUID/PGID below), not root — this
# container only edits one file on the config PVC, so there's no
# reason for it to run as root and risk leaving root-owned files
# behind for the app container (which linuxserver's own
# PUID/PGID-based privilege drop wouldn't necessarily fix up for
# files it didn't create itself).
- name: qbittorrent-vpn-bind-config
image: nicolaka/netshoot:v0.11
securityContext:
runAsUser: 1000
runAsGroup: 1000
capabilities:
drop: ["ALL"]
command:
- /bin/sh
- -c
- |
set -eu
CONF=/config/qBittorrent/qBittorrent.conf
mkdir -p "$(dirname "$CONF")"
touch "$CONF"
grep -q '^\[BitTorrent\]' "$CONF" || printf '\n[BitTorrent]\n' >> "$CONF"
set_key() {
key="$1"; val="$2"
if grep -q "^${key}=" "$CONF"; then
sed -i "s|^${key}=.*|${key}=${val}|" "$CONF"
else
sed -i "/^\[BitTorrent\]/a ${key}=${val}" "$CONF"
fi
}
set_key 'Session\\Interface' 'net1'
set_key 'Session\\InterfaceName' 'net1'
set_key 'Session\\InterfaceAddress' '10.10.50.10'
echo "qBittorrent.conf VPN bind settings ensured"
volumeMounts:
- name: config
mountPath: /config
# Works around a real upstream qBittorrent 5.2.0 bug
# (qbittorrent/qBittorrent#24164, confirmed against the actual
# issue thread, not assumed): 5.2.0 switched its single-instance
# lock from the old fcntl-based QtLockedFile to Qt's QLockFile,
# which embeds a PID + hostname. If the previous shutdown wasn't
# clean, 5.2.0 can't parse/verify the stale lock, assumes another
# instance holds it, tries to hand off over the (also stale)
# ipc-socket, that fails too, and it exits immediately — then
# restarts, hits the same stale lock, and repeats forever (a
# crash loop, not a slow startup — confirmed live 2026-08-25 from
# this exact pod: PID incrementing every ~1-2s, 80% CPU, WebUI
# never listening). Every Kubernetes pod restart gets a new
# hostname (the pod name), which is exactly the kind of
# environment where the lock's embedded hostname can't be
# trusted to match — plausibly why this surfaces more here than
# on a stable bare-metal/VM host.
#
# Unconditionally clearing the lock on every start is safe
# specifically in this Deployment: strategy: Recreate plus a
# single replica plus a ReadWriteOnce PVC together guarantee
# Kubernetes itself never runs two instances against this config
# concurrently — by the time this init container runs, any
# previous instance is already fully terminated. So there is no
# scenario here where the lock could be legitimately held by a
# still-running peer.
- name: qbittorrent-clear-stale-lock
image: nicolaka/netshoot:v0.11
securityContext:
runAsUser: 1000
runAsGroup: 1000
capabilities:
drop: ["ALL"]
command:
- /bin/sh
- -c
- |
set -eu
rm -fv /config/qBittorrent/lockfile /config/qBittorrent/ipc-socket
echo "stale lock/IPC state cleared (if any was present)"
volumeMounts:
- name: config
mountPath: /config
containers:
- name: gluetun
image: qmcgaw/gluetun:v3.41.1
securityContext:
capabilities:
add:
- NET_ADMIN
env:
- name: VPN_SERVICE_PROVIDER
value: private internet access
- name: VPN_TYPE
value: openvpn
- name: SERVER_REGIONS
value: South Korea,Taiwan,Hong Kong
- name: OPENVPN_USER
valueFrom:
secretKeyRef:
name: pia-credentials
key: OPENVPN_USER
- name: OPENVPN_PASSWORD
valueFrom:
secretKeyRef:
name: pia-credentials
key: OPENVPN_PASSWORD
- name: FIREWALL_OUTBOUND_SUBNETS
value: "10.42.0.0/16,10.43.0.0/16,192.168.7.0/24"
- name: BLOCK_IPV6
value: "on"
startupProbe:
exec:
command:
- /gluetun-entrypoint
- healthcheck
periodSeconds: 10
failureThreshold: 60
livenessProbe:
exec:
command:
- /gluetun-entrypoint
- healthcheck
periodSeconds: 30
failureThreshold: 3
readinessProbe:
exec:
command:
- /gluetun-entrypoint
- healthcheck
periodSeconds: 10
failureThreshold: 3
volumeMounts:
- name: tun
mountPath: /dev/net/tun
- name: qbittorrent
image: lscr.io/linuxserver/qbittorrent:5.2.0
ports:
@ -223,19 +105,6 @@ spec:
value: "Asia/Tokyo"
- name: WEBUI_PORT
value: "8080"
# No securityContext capability restriction here, deliberately.
# linuxserver.io images (this one included) are built around a
# PUID/PGID pattern: the container starts as root, s6-overlay's
# own init chowns /config to the requested PUID/PGID and only
# then drops privileges to run qBittorrent itself as that user
# — that chown step needs real root capabilities (at least
# CHOWN/DAC_OVERRIDE/SETUID/SETGID) that `drop: ["ALL"]` would
# remove, breaking the image's own startup before it ever gets
# to serving anything. This container is not where NET_ADMIN or
# any other genuinely elevated capability is needed for VLAN 50
# specifically — that stays exclusively in the init container
# above — this is just not fighting an image that already
# implements its own privilege drop.
readinessProbe:
httpGet:
path: /
@ -259,13 +128,6 @@ spec:
image: python:3-alpine
ports:
- containerPort: 8888
# Unlike the qbittorrent container above, this is a plain
# Python base image with no PUID/PGID privilege-drop
# machinery to preserve — dropping all capabilities here is
# safe and doesn't fight anything the image needs to start.
securityContext:
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 5m
@ -305,6 +167,10 @@ spec:
http.server.HTTPServer(('0.0.0.0', 8888), Handler).serve_forever()
volumes:
- name: tun
hostPath:
path: /dev/net/tun
type: CharDevice
- name: config
persistentVolumeClaim:
claimName: qbittorrent-config
@ -312,10 +178,6 @@ spec:
hostPath:
path: /mnt/storage/torrents
type: Directory
- name: guard-script
configMap:
name: vlan50-egress-guard-script
defaultMode: 365 # octal 0555, r-xr-xr-x
---
apiVersion: v1
kind: Service

57
manifests/multus/00-crd.yaml
View File

@ -1,57 +0,0 @@
# Apply: kubectl apply -f manifests/multus/00-crd.yaml
# Description: NetworkAttachmentDefinition CRD. Cluster-scoped.
# argocd.argoproj.io/sync-wave -1 so it lands before Multus's own RBAC
# and DaemonSet, which reference this kind.
#
# Verbatim from k8snetworkplumbingwg/multus-cni's official
# deployments/multus-daemonset-thick.yml (commit 95a0932350f0060cd23209
# 6876e24672a1de83fa, fetched 2026-08-24 — see 02-daemonset.yaml's header
# for the same pin) — not adapted at all; this resource has nothing
# k3s-specific about it.
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: network-attachment-definitions.k8s.cni.cncf.io
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
group: k8s.cni.cncf.io
scope: Namespaced
names:
plural: network-attachment-definitions
singular: network-attachment-definition
kind: NetworkAttachmentDefinition
shortNames:
- nad
- net-attach-def
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing
Working Group to express the intent for attaching pods to one or more logical or physical
networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec'
type: object
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this represen
tation of an object. Servers should convert recognized schemas to the
latest internal value, and may reject unrecognized values. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment'
type: object
properties:
config:
description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration'
type: string

77
manifests/multus/01-rbac.yaml
View File

@ -1,77 +0,0 @@
# Apply: kubectl apply -f manifests/multus/01-rbac.yaml
# Description: ServiceAccount and cluster-scoped RBAC for the Multus
# DaemonSet (02-daemonset.yaml). sync-wave 0 — after the CRD, before
# the DaemonSet that uses this ServiceAccount.
#
# Verbatim from k8snetworkplumbingwg/multus-cni's official
# deployments/multus-daemonset-thick.yml (commit 95a0932350f0060cd23209
# 6876e24672a1de83fa, fetched 2026-08-24 — see 02-daemonset.yaml's header
# for the same pin), including the resource.k8s.io (Dynamic Resource
# Allocation) and events.k8s.io rules upstream grants — left in even
# though k3s v1.32 may not register that API group, since RBAC rules for
# a nonexistent group/resource are inert, not an error, and removing
# them would be inventing a divergence from upstream rather than fixing
# anything.
apiVersion: v1
kind: ServiceAccount
metadata:
name: multus
namespace: kube-system
annotations:
argocd.argoproj.io/sync-wave: "0"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: multus
annotations:
argocd.argoproj.io/sync-wave: "0"
rules:
- apiGroups: ["k8s.cni.cncf.io"]
resources:
- "*"
verbs:
- "*"
- apiGroups:
- ""
resources:
- pods
- pods/status
verbs:
- get
- list
- update
- watch
- apiGroups:
- "resource.k8s.io"
resources:
- resourceclaims
- resourceclaims/status
- resourceslices
verbs:
- get
- list
- apiGroups:
- ""
- events.k8s.io
resources:
- events
verbs:
- create
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: multus
annotations:
argocd.argoproj.io/sync-wave: "0"
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: multus
subjects:
- kind: ServiceAccount
name: multus
namespace: kube-system

360
manifests/multus/02-daemonset.yaml
View File

@ -1,360 +0,0 @@
# Apply: kubectl apply -f manifests/multus/02-daemonset.yaml
# Description: Multus thick-plugin DaemonSet, restricted by nodeSelector
# to nik-debian only (the sole node with a VLAN 50 trunk — deliberately
# not cluster-wide, so minisforum and nik-gpu's CNI chain is never
# touched by this at all). sync-wave 0 — after RBAC.
#
# Rebuilt from k8snetworkplumbingwg/multus-cni's official
# deployments/multus-daemonset-thick.yml, commit
# 95a0932350f0060cd232096876e24672a1de83fa (master, fetched and read in
# full 2026-08-24 — https://raw.githubusercontent.com/
# k8snetworkplumbingwg/multus-cni/master/deployments/
# multus-daemonset-thick.yml). An earlier version of this file was NOT
# a faithful copy of that manifest — it used an invented entrypoint
# (/entrypoint/cni-installer.sh, which does not exist in the real image)
# and an incomplete set of mounts/config keys, reconstructed from memory
# rather than the real source. This version changes only what k3s and
# this topology actually require, everything else (entrypoints, daemon
# config keys, volume set, RBAC-adjacent mounts) is verbatim:
#
# 1. cni/cnibin volumes' hostPath: k3s's real CNI conf/bin dirs
# (/var/lib/rancher/k3s/agent/etc/cni/net.d,
# /var/lib/rancher/k3s/data/cni — confirmed LIVE on nik-debian
# 2026-08-23; there is no /opt/cni/bin or /etc/cni/net.d on this
# host, upstream's own defaults would silently no-op on k3s
# without this). The container-side mountPaths upstream defines
# (/host/etc/cni/net.d, /host/opt/cni/bin) are unchanged — only
# which host directory backs them moves.
# 2. All OTHER hostPath volumes (host-run, host-var-lib-cni-multus,
# host-var-lib-kubelet, host-run-k8s-cni-cncf-io, host-run-netns,
# hostroot, multus-conf-dir) are left at upstream's standard paths
# — k3s does not relocate /run, /var/lib/kubelet, /run/netns, or /
# itself, only the CNI-specific directories. This assumption (that
# kubelet's own root-dir is standard on k3s) is NOT independently
# verified against a live k3s node the way the CNI paths are —
# confirm before applying if this cluster ever changes
# --root-dir on the kubelet.
# 3. nodeSelector restricting it to nik-debian (upstream is
# cluster-wide by default).
# 4. Image pinned to the v4.3.0-thick digest (upstream's own quickstart
# file uses the "snapshot-thick" moving tag, which is explicitly a
# dev/nightly build, not something to run unpinned in production).
# Digest confirmed live via the GHCR OCI Distribution API
# 2026-08-24: v4.3.0-thick and stable-thick both resolve to
# sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896
# d56ae51c9, i.e. v4.3.0 is genuinely the current stable release,
# not just a same-named coincidence.
# 5. daemon-config.json has one key added beyond upstream's own
# example: "binDir". Root-caused live (2026-08-24) after Multus
# itself came up correctly but every pod on nik-debian using the
# NAD failed FailedCreatePodSandBox with `plugin type="flannel"
# failed (add): failed to find plugin "flannel" in path
# [/opt/cni/bin]`. Read multus-cni's actual source (pinned to the
# v4.3.0 tag, not master) to confirm the mechanism rather than
# guess: pkg/server/exec_chroot.go's ChrootExec resolves a delegate
# plugin's absolute path using the DAEMON CONTAINER's own
# filesystem view (FindInPath), then execs that exact path string
# *after* chrooting the child process into "hostroot" (the real
# host root, bind-mounted). binDir defaults to "/opt/cni/bin"
# (pkg/types/conf.go's defaultBinDir) when unset — a path that is
# valid inside this container (bind-mounted to k3s's real bin dir
# below) but does not exist on the actual host filesystem at all,
# so the post-chroot exec fails even though the daemon "found" the
# plugin from its own point of view first. Confirmed the real host
# path live via SSH (flannel/bridge/host-local genuinely present at
# /var/lib/rancher/k3s/data/cni), confirmed binDir's exact JSON tag
# against pkg/server/config/generator.go's MultusConf struct at the
# v4.3.0 tag specifically (not assumed from master, in case the
# schema had drifted between versions — it hadn't, but that was
# checked, not assumed). Setting binDir alone isn't sufficient by
# itself: it also has to resolve correctly in the *pre-chroot*
# container view (FindInPath's side), which is exactly why the
# cnibin mountPath below changed too — see that comment.
# 6. The main kube-multus container mounts a second hostPath,
# k3s-cni-data, at /var/lib/rancher/k3s/data (cnibin's parent), in
# addition to cnibin itself. Root-caused live (2026-08-24), same
# incident as point 5 continued: after binDir was corrected to
# /var/lib/rancher/k3s/data/cni, pod sandbox creation still failed
# with the identical error, `failed to find plugin "flannel" in
# path [/var/lib/rancher/k3s/data/cni]` — this time from inside the
# corrected path itself. Read containernetworking/cni's actual
# source (pkg/invoke/find.go's FindInPath) to confirm the mechanism:
# it calls os.Stat on each candidate path, which follows symlinks,
# and requires the resolved target to be a regular file. A live SSH
# check on nik-debian (already done for point 5) had shown
# /var/lib/rancher/k3s/data/cni is itself a directory of SYMLINKS —
# e.g. .../data/cni/flannel -> .../data/<content-hash>/bin/cni — not
# the real binaries. The narrow cnibin mount only bind-mounts the
# cni subdirectory, so inside the container those symlinks point at
# a hash-named sibling directory that was never mounted at all, and
# os.Stat fails on the daemon container's own (pre-chroot) side —
# exactly the FindInPath call point 5 identified, just one symlink
# hop further than binDir alone could fix. Mounting the parent
# (k3s-cni-data, at the same path in-container as on the host) lets
# the symlink target resolve inside the container too, without
# changing binDir or the resolved host path at all. cnibin itself is
# kept, narrowed to the two init containers' own plain file writes
# (which never follow the symlinks and so never needed the wider
# mount) — see point 7 for the second of those two.
# 7. A cni-plugins init container, absent from upstream's own
# daemonset-thick.yml entirely, installs the standard CNI reference
# plugins (bridge, macvlan, ipvlan, vlan, host-local, loopback,
# portmap, bandwidth, firewall, dhcp, static, tuning, sbr, vrf,
# host-device, ptp, dummy) that k3s does not bundle. Root-caused live
# (2026-08-24), same incident as points 5/6 continued: with binDir
# and the mount both fixed, sandbox creation failed a third time,
# now with `failed to find plugin "macvlan" in path
# [/var/lib/rancher/k3s/data/cni]` — a genuinely absent binary this
# time, not another path/mount problem. Confirmed against k3s's own
# real build script (k3s-io/k3s's scripts/package-cli at this exact
# tag, read in full): its cni_binaries array is hardcoded to exactly
# bandwidth, bridge, firewall, flannel, host-local, loopback,
# portmap — macvlan is deliberately excluded by k3s's own project
# scope decision (k3s-io/k3s issue #9224 confirms this is
# intentional, not an oversight). k3s's own documentation
# (https://docs.k3s.io/networking/multus-ipams) names the fix for
# exactly this gap: install the missing plugins via the same
# mechanism Rancher's own rke2-multus Helm chart uses — see this
# container's own inline comment for exactly which real chart/image
# that is and how it was verified (image digest confirmed live
# against the real registry, not assumed from the tag).
#
# Do not hand-edit the daemon-config.json keys, the container
# command/args, or the mount set below without re-diffing against the
# URL above — the parts still matching upstream verbatim were not
# invented, they're copied; the binDir addition, the cnibin/k3s-cni-data
# mount split, and the added cni-plugins init container are the three
# deliberate, root-caused departures.
apiVersion: v1
kind: ConfigMap
metadata:
name: multus-daemon-config
namespace: kube-system
labels:
tier: node
app: multus
annotations:
argocd.argoproj.io/sync-wave: "0"
data:
daemon-config.json: |
{
"chrootDir": "/hostroot",
"cniVersion": "0.3.1",
"logLevel": "verbose",
"logToStderr": true,
"cniConfigDir": "/host/etc/cni/net.d",
"multusAutoconfigDir": "/host/etc/cni/net.d",
"multusConfigFile": "auto",
"socketDir": "/host/run/multus/",
"binDir": "/var/lib/rancher/k3s/data/cni"
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-multus-ds
namespace: kube-system
labels:
tier: node
app: multus
name: multus
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
selector:
matchLabels:
name: multus
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
tier: node
app: multus
name: multus
spec:
# Not upstream — restricts this DaemonSet to the one node with a
# VLAN 50 trunk. minisforum and nik-gpu never run Multus at all.
nodeSelector:
kubernetes.io/hostname: nik-debian
hostNetwork: true
hostPID: true
tolerations:
- operator: Exists
effect: NoSchedule
- operator: Exists
effect: NoExecute
serviceAccountName: multus
containers:
- name: kube-multus
# VERIFY the digest still resolves before applying — see
# header comment for how it was obtained and when.
image: "ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896d56ae51c9" # v4.3.0-thick
command: ["/usr/src/multus-cni/bin/multus-daemon"]
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: true
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- name: cni
mountPath: /host/etc/cni/net.d
# multus-daemon expects that cnibin path must be identical between pod and container host.
# e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon,
# not to any other directory, like '/opt/bin' or '/usr/bin'.
#
# On THIS host that mount is k3s-cni-data (below), not cnibin
# — see the header comment's point 6. k3s's actual plugin
# binaries live in a content-hash-named sibling directory
# (/var/lib/rancher/k3s/data/<hash>/bin/cni) and
# /var/lib/rancher/k3s/data/cni/flannel etc. are only
# *symlinks* to that; mounting just the cni/ subdirectory
# (cnibin, still used by the init container below for its own
# narrower purpose) left those symlinks dangling from this
# container's own point of view, which is exactly where
# FindInPath (invoke.FindInPath, os.Stat-based, follows
# symlinks) checks each candidate before the chroot-exec ever
# happens. Mounting the whole parent directory at an identical
# path is what makes both the symlink and its target resolve
# the same way here as they do on the real host.
- name: k3s-cni-data
mountPath: /var/lib/rancher/k3s/data
- name: host-run
mountPath: /host/run
- name: host-var-lib-cni-multus
mountPath: /var/lib/cni/multus
- name: host-var-lib-kubelet
mountPath: /var/lib/kubelet
mountPropagation: HostToContainer
- name: host-run-k8s-cni-cncf-io
mountPath: /run/k8s.cni.cncf.io
- name: host-run-netns
mountPath: /run/netns
mountPropagation: HostToContainer
- name: multus-daemon-config
mountPath: /etc/cni/net.d/multus.d
readOnly: true
- name: hostroot
mountPath: /hostroot
mountPropagation: HostToContainer
- mountPath: /etc/cni/multus/net.d
name: multus-conf-dir
env:
- name: MULTUS_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
initContainers:
- name: cni-plugins
# NOT part of upstream k8snetworkplumbingwg/multus-cni's own
# daemonset-thick.yml (that manifest assumes the standard CNI
# reference plugins already exist on the host, which is true on
# most distributions but NOT k3s — see the header comment's
# point 7). This container and image are copied from Rancher's
# own official rke2-multus Helm chart (the one k3s's own docs at
# https://docs.k3s.io/networking/multus-ipams recommend for
# exactly this situation), fetched and read in full 2026-08-24
# from the real packaged chart at https://rke2-charts.rancher.io
# /assets/rke2-multus/rke2-multus-v4.3.017.tgz — not fabricated.
image: "docker.io/rancher/hardened-cni-plugins@sha256:935b2e954695ecc153d4a671f51717dc93b613e08b5f4230ba0eb8dbf273495a" # v1.9.1-build20260819
securityContext:
privileged: true
env:
# Skip flannel specifically: k3s's own binDir already has a
# working flannel plugin (from k3s's own build, version-locked
# to this k3s release) — see header comment's points 5 and 6.
# Letting this image overwrite it with its own separately
# versioned flannel would be a needless, unverified swap of a
# plugin that already works, for zero benefit (this image is
# only needed here for the plugins k3s doesn't ship at all,
# e.g. macvlan). Matches Rancher's own chart's default
# cniplugins.skipcnis value for this exact reason.
- name: SKIP_CNI_BINARIES
value: flannel
volumeMounts:
# install-cnis.sh (this image's entrypoint, read in full) has
# a hardcoded destination of /host/opt/cni/bin — not
# configurable via any env var or arg. Mounting cnibin (this
# role's real k3s CNI bin dir, not the standard /opt/cni/bin)
# at that container path redirects the writes to the right
# place without needing a different image or a patched
# script. It copies plain regular files directly into that
# directory (not through the flannel-style symlink-to-hash-dir
# indirection those specific k3s-provided binaries use), so
# this narrower cnibin mount is sufficient here — no need for
# the wider k3s-cni-data mount the main container uses.
- name: cnibin
mountPath: /host/opt/cni/bin
- name: install-multus-binary
# Same image/digest as the main container — see header comment.
image: "ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896d56ae51c9" # v4.3.0-thick
command:
- "/usr/src/multus-cni/bin/install_multus"
- "-d"
- "/host/var/lib/rancher/k3s/data/cni"
- "-t"
- "thick"
resources:
requests:
cpu: "10m"
memory: "15Mi"
securityContext:
privileged: true
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- name: cnibin
mountPath: /host/var/lib/rancher/k3s/data/cni
mountPropagation: Bidirectional
terminationGracePeriodSeconds: 30
volumes:
- name: cni
hostPath:
# k3s CNI conf dir — NOT the standard /etc/cni/net.d.
path: /var/lib/rancher/k3s/agent/etc/cni/net.d
- name: cnibin
hostPath:
# k3s CNI bin dir — NOT the standard /opt/cni/bin. Used only
# by the install-multus-binary init container (a plain file
# write, doesn't need the wider parent — see k3s-cni-data
# below for why the main container needs more than this).
path: /var/lib/rancher/k3s/data/cni
- name: k3s-cni-data
hostPath:
# The parent of cnibin above, mounted identically in the main
# container so that k3s's plugin symlinks (e.g.
# .../data/cni/flannel -> .../data/<hash>/bin/cni) resolve
# correctly from inside the container too, not just on the
# real host — see the volumeMounts comment above for why.
path: /var/lib/rancher/k3s/data
- name: hostroot
hostPath:
path: /
- name: multus-daemon-config
configMap:
name: multus-daemon-config
items:
- key: daemon-config.json
path: daemon-config.json
- name: host-run
hostPath:
path: /run
- name: host-var-lib-cni-multus
hostPath:
path: /var/lib/cni/multus
- name: host-var-lib-kubelet
hostPath:
path: /var/lib/kubelet
- name: host-run-k8s-cni-cncf-io
hostPath:
path: /run/k8s.cni.cncf.io
- name: host-run-netns
hostPath:
path: /run/netns/
- name: multus-conf-dir
hostPath:
path: /etc/cni/multus/net.d

48
manifests/multus/10-nad-vlan50.yaml
View File

@ -1,48 +0,0 @@
# Apply: kubectl apply -f manifests/multus/10-nad-vlan50.yaml
# Description: Namespace-scoped NetworkAttachmentDefinition for VLAN 50,
# macvlan bridge mode on nik-debian's enp1s0.50 (ansible/roles/
# vlan50-parent). macvlan bridge chosen over ipvlan L2 per the
# home-network handoff confirming the managed switch accepts multiple
# source MACs on nik-debian's port — see plan.md Phase 4 / the
# coordination handoff in the session that authored this. sync-wave 1 —
# after Multus itself (00/01/02), before the canary (20) or any
# workload attachment.
#
# ipam.type is "static" deliberately, not a shared pool/allocator
# (Whereabouts etc.) — this cluster only ever attaches a handful of fixed
# workloads to VLAN 50 (qBittorrent .10, JDownloader .11, one reserved
# .12, one canary from 10.10.50.100-149), so each pod's own
# k8s.v1.cni.cncf.io/networks annotation supplies its ips/gateway
# directly; there is no dynamic allocation to arbitrate. Excluded from
# any future allocator by simply never being handed to one: .1 (Flint),
# .10/.11/.12 (static workloads), and .100-149 (canary/dynamic) are never
# referenced anywhere else in this repo.
#
# capabilities.ips: true is required by the Network Plumbing Working
# Group CNI spec for a plugin/IPAM combination to accept a runtime-
# supplied "ips" field at all (https://www.cni.dev/plugins/current/ipam/
# static/) — without it at the top level of the NAD config, the per-pod
# k8s.v1.cni.cncf.io/networks annotation's "ips" array is not a
# recognized capability and the static IPAM plugin has no defined way to
# receive it.
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: vlan50
namespace: downloads
spec:
config: |
{
"cniVersion": "0.3.1",
"name": "vlan50",
"type": "macvlan",
"master": "enp1s0.50",
"mode": "bridge",
"mtu": 1500,
"capabilities": {
"ips": true
},
"ipam": {
"type": "static"
}
}

89
manifests/multus/20-canary.yaml
View File

@ -1,89 +0,0 @@
# Apply: kubectl apply -f manifests/multus/20-canary.yaml
# Delete when done: kubectl delete -f manifests/multus/20-canary.yaml
# Description: TEMPORARY canary pod for Phase 5 validation (plan.md) —
# proves eth0/net1 routing, kill-switch failure modes, and the netns
# egress guard before qBittorrent/JDownloader are touched. Not a
# long-lived resource; delete it once the Phase 5 validation matrix has
# run. sync-wave 2 — after Multus (0) and the NAD (1).
#
# *** DO NOT SYNC/APPLY until: ***
# - Phases 0-4 have actually been applied and verified live (this file
# is drafted alongside them, not proof they work).
# - The user has explicitly approved running the canary specifically —
# per this task's own gates, Argo CD sync of this file needs the same
# explicit go-ahead as any other live change, even though the
# multus.yaml Application it lives under is manual-sync-only anyway.
#
# Image: nicolaka/netshoot:v0.11 — a widely-used network-debugging image
# (iproute2 + iptables + curl/dig/tcpdump). Tag not independently
# verified against a live pull; confirm it still resolves before
# applying, same caveat as the Multus image tag in 02-daemonset.yaml.
apiVersion: v1
kind: Pod
metadata:
name: vlan50-canary
namespace: downloads
labels:
app: vlan50-canary
annotations:
argocd.argoproj.io/sync-wave: "2"
k8s.v1.cni.cncf.io/networks: |
[{"name": "vlan50", "namespace": "downloads", "interface": "net1", "ips": ["10.10.50.100/24"]}]
spec:
nodeSelector:
kubernetes.io/hostname: nik-debian
restartPolicy: Never
# Same reasoning as qbittorrent.yaml/jdownloader.yaml: no cluster
# DNS/CoreDNS resolution needed, single Technitium resolver, routed via
# net1 so Technitium sees the real VLAN 50 source.
dnsPolicy: None
dnsConfig:
nameservers:
- "10.10.40.53"
initContainers:
- name: vlan50-egress-guard
image: nicolaka/netshoot:v0.11
command: ["/bin/sh", "/scripts/guard.sh"]
env:
- name: VLAN50_GATEWAY
value: "10.10.50.1"
- name: TECHNITIUM_IP
value: "10.10.40.53"
- name: POD_CIDR
value: "10.42.0.0/16"
- name: SERVICE_CIDR
value: "10.43.0.0/16"
- name: NODE_IP
value: "10.10.40.20"
- name: EXPECTED_VLAN50_IP
value: "10.10.50.100"
securityContext:
capabilities:
drop: ["ALL"]
# NET_RAW alongside NET_ADMIN: the guard script's arping-based
# duplicate-address and gateway-reachability checks build raw
# ARP frames, which the kernel gates on CAP_NET_RAW specifically
# — see vlan50-egress-guard-script.yaml's header comment.
add: ["NET_ADMIN", "NET_RAW"]
volumeMounts:
- name: guard-script
mountPath: /scripts
containers:
- name: netshoot
image: nicolaka/netshoot:v0.11
command: ["sleep", "infinity"]
securityContext:
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 20m
memory: 32Mi
limits:
cpu: 200m
memory: 128Mi
volumes:
- name: guard-script
configMap:
name: vlan50-egress-guard-script
defaultMode: 365 # octal 0555, r-xr-xr-x — yamllint here forbids octal literals

215
manifests/multus/30-browser-vpn-proxy.yaml
View File

@ -1,215 +0,0 @@
# sync-wave 1 — same wave as the NAD (10-nad-vlan50.yaml), the only
# thing this actually needs to exist first. NOT wave 3/after-the-canary
# as originally drafted: that was a documentation convention, not a real
# dependency, and it turned out to actively break syncing. Root-caused
# live (2026-08-24): this cluster's argocd-cm sets
# resource.customizations.ignoreResourceUpdates.all to ignore /status
# changes cluster-wide (a deliberate, existing setting, not something to
# change just for this file) — Argo's wave orchestrator waits for each
# wave's resources to report Healthy before starting the next wave, but
# a Pod that's deleted and recreated (exactly what the canary is, every
# test round) needs Argo to observe a fresh Pending -> Running /status
# transition *during that sync*, which the ignore-updates config
# prevents it from ever seeing — so the operation hangs forever on
# "waiting for healthy state of /Pod/vlan50-canary" even once the Pod is
# actually Running. Already-stable resources (the DaemonSet, RBAC, NAD)
# never hit this because their cached health from a prior sync is still
# valid when nothing about them changed. Moved here from
# manifests/multus/reserved/ (2026-08-24) now that qBittorrent and
# JDownloader have both independently passed their canary, validation,
# and soak gates — the reservation this file's earlier header described
# is satisfied. Still governed only by the `multus` Argo CD Application's
# own manual-sync gate (no automated sync there at all) — the same gate
# already used for the DaemonSet, NAD, and canary, not a new mechanism.
#
# Before applying:
# 1. Verify port 1080 below doesn't collide with anything live —
# confirmed free 2026-08-24 (`kubectl get svc -A`, checked against
# every live LoadBalancer Service's ports: gitea-ssh uses 2222,
# traefik uses 80/443).
#
# Design:
# - serjs/go-socks5-proxy (github.com/serjs/socks5-server) — a small,
# single-purpose Go SOCKS5 server, not a VPN client itself. It
# doesn't need to be, here: unlike the legacy manifests/media/
# gluetun-proxy.yaml it's replacing (which runs its own OpenVPN
# client), PIA egress for this workload comes from the same
# netns-level routing as qBittorrent/JDownloader — the init
# container below sends this pod's default route out net1/PIA, the
# SOCKS5 server itself just needs to be a plain, correctly-behaving
# proxy. Pinned to v0.0.4's digest, confirmed live against Docker
# Hub's registry API 2026-08-24:
# sha256:6828ddb2a6a93dec85209b69cd3842bd80094f7dd52f35ee1dd0f9e9165e2188
# - No auth (REQUIRE_AUTH=false), no ALLOWED_IPS — a deliberate choice,
# not an oversight. plan.md originally called for auth AND
# LAN-restriction together as a floor; discussed and explicitly
# relaxed (2026-08-24) after confirming the home network's own
# firewall already restricts this Service's port to Trusted
# (VLAN 10) and Lab (VLAN 40) only — Trusted/Lab → Lab is ALLOW,
# IoT/Guest/Management → Lab is DENY except the Technitium DNS
# exception (see ~/repo/home-network/Home Network Implementation
# Plan.md section 10). Residual risk accepted knowingly: Lab has its
# own Wi-Fi SSID with open DHCP, so "reachable" means "anyone with
# that Wi-Fi password or a device already on Trusted/Lab," not just
# deliberately provisioned devices — there is no proxy-level control
# backing this up.
# - ALLOWED_IPS is not a fallback for the above — it's removed because
# it cannot do what it looks like it does. Read this image's actual
# source (server.go, and the go-socks5 fork it replaces via go.mod's
# replace directive) to confirm, not assumed: ALLOWED_IPS values are
# parsed with net.ParseIP (not net.ParseCIDR) and matched with exact
# net.IP.Equal(), never CIDR containment. The original draft's value
# here, "10.10.40.0/24", would have failed to parse as a plain IP,
# producing a whitelist containing one nil entry — since Equal()
# against nil never matches any real client address, every single
# connection would have been rejected, including legitimate ones.
# This library has no way to express a subnet allowlist at all, only
# a list of exact individual IPs, which doesn't fit a dynamic-DHCP
# LAN. Access restriction is the OpenWrt firewall alone; there is no
# compensating netns-level (nft) inbound filter here either — one
# could be added later, but would first need confirming whether this
# Service actually preserves real client source IPs end-to-end
# (default `externalTrafficPolicy: Cluster` typically does not —
# this Service doesn't set `externalTrafficPolicy: Local` either,
# unlike values/traefik.yaml's LoadBalancer, which does specifically
# for this reason) rather than assuming it does.
# - Service is LoadBalancer (k3s's built-in ServiceLB/Klipper), port
# 1080 directly — not NodePort, at the user's request for a more
# natural port than a 30000+ NodePort value. `loadBalancerIP:
# 10.10.40.20` (nik-debian) matches values/gitea.yaml's gitea-ssh
# Service pattern exactly (`loadBalancerIP: 10.10.40.53`, minisforum)
# — confirmed by reading how Klipper actually works before adding
# this, not assumed: `loadBalancerIP` only sets the *reported*
# external IP (`.status.loadBalancer.ingress[].ip`); the underlying
# svclb DaemonSet still binds port 1080 on every node by default,
# same as gitea-ssh's `loadBalancerIP` pin does today. Actually
# scoping it to nik-debian only needs k3s's separate
# `svccontroller.k3s.cattle.io/enablelb`/`lbpool` node-label
# mechanism, which nothing in this repo uses yet — not added here,
# so this is reachable at <any node IP>:1080, functionally identical
# multi-node reachability to the NodePort version it replaced, just
# a different (and more memorable) port number. Never exposed
# publicly either way (no port-forward on Flint/the router for it,
# same as the now-decommissioned gluetun-proxy was).
#
# Browser configuration — this is the part that actually matters for
# "DNS must be resolved through the proxy": configuring a SOCKS5 proxy
# address alone is NOT enough. By default most browsers resolve
# hostnames locally (via the LAN's normal DNS) and only send the already-
# resolved IP through the proxy — that leaks every site you visit to
# your normal LAN resolver and bypasses Technitium/PIA for DNS entirely,
# defeating the point. "socks5h" (vs plain "socks5") in a proxy URL is
# the conventional way tools signal "resolve hostnames on the proxy side,
# not locally" (e.g. curl --socks5-hostname, or a proxy URL scheme of
# socks5h://). In Firefox specifically: Settings -> Network Settings ->
# Manual proxy configuration -> SOCKS Host set to this Service -> SOCKS
# v5 selected -> the checkbox "Proxy DNS when using SOCKS v5" MUST be
# checked, or Firefox resolves locally despite the proxy being SOCKS5.
# Verify this actually holds (a DNS leak test site, or packet capture
# showing no DNS queries leaving the browser's own host) before trusting
# it — do not assume the checkbox alone is sufficient without checking.
apiVersion: apps/v1
kind: Deployment
metadata:
name: browser-vpn-proxy
namespace: downloads
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: browser-vpn-proxy
template:
metadata:
labels:
app: browser-vpn-proxy
annotations:
k8s.v1.cni.cncf.io/networks: |
[{"name": "vlan50", "namespace": "downloads", "interface": "net1", "ips": ["10.10.50.12/24"]}]
spec:
nodeSelector:
node-role: storage
kubernetes.io/hostname: nik-debian
dnsPolicy: None
dnsConfig:
nameservers:
- "10.10.40.53"
initContainers:
- name: vlan50-egress-guard
image: nicolaka/netshoot:v0.11
command: ["/bin/sh", "/scripts/guard.sh"]
env:
- name: VLAN50_GATEWAY
value: "10.10.50.1"
- name: TECHNITIUM_IP
value: "10.10.40.53"
- name: POD_CIDR
value: "10.42.0.0/16"
- name: SERVICE_CIDR
value: "10.43.0.0/16"
- name: NODE_IP
value: "10.10.40.20"
- name: EXPECTED_VLAN50_IP
value: "10.10.50.12"
securityContext:
capabilities:
drop: ["ALL"]
# NET_RAW alongside NET_ADMIN: the guard script's
# arping-based duplicate-address and gateway-reachability
# checks need it — see
# vlan50-egress-guard-script.yaml's header comment.
add: ["NET_ADMIN", "NET_RAW"]
volumeMounts:
- name: guard-script
mountPath: /scripts
containers:
- name: socks5
image: "serjs/go-socks5-proxy@sha256:6828ddb2a6a93dec85209b69cd3842bd80094f7dd52f35ee1dd0f9e9165e2188" # v0.0.4
ports:
- containerPort: 1080
securityContext:
capabilities:
drop: ["ALL"]
env:
# REQUIRE_AUTH defaults to true in this image (confirmed
# against its real source, params struct in server.go) — it
# must be explicitly set to "false" to actually go
# passwordless, omitting it entirely would NOT have that
# effect. Deliberately no auth, no ALLOWED_IPS — see the
# header comment for the reasoning and what actually
# restricts access instead.
- name: REQUIRE_AUTH
value: "false"
resources:
requests:
cpu: 20m
memory: 32Mi
limits:
cpu: 200m
memory: 128Mi
volumes:
- name: guard-script
configMap:
name: vlan50-egress-guard-script
defaultMode: 365 # octal 0555, r-xr-xr-x
---
apiVersion: v1
kind: Service
metadata:
name: browser-vpn-proxy
namespace: downloads
spec:
selector:
app: browser-vpn-proxy
type: LoadBalancer
# Pinned to nik-debian's own address, matching values/gitea.yaml's
# gitea-ssh Service (loadBalancerIP: 10.10.40.53, minisforum) — see the
# header comment for what this does and does not actually restrict.
loadBalancerIP: "10.10.40.20"
ports:
- name: socks5
port: 1080
targetPort: 1080

198
manifests/multus/README.md
View File

@ -1,198 +0,0 @@
# Multus + VLAN 50 NAD + canary
Implements plan.md Phase 4/5. Reconciled by `argocd/apps/multus.yaml`,
which is deliberately **manual-sync only** — see that file's header
comment. Nothing here applies to the cluster just because it's merged to
`main`.
## Files (sync-wave order)
| File | Wave | What |
| --- | --- | --- |
| `00-crd.yaml` | -1 | `NetworkAttachmentDefinition` CRD |
| `01-rbac.yaml` | 0 | Multus ServiceAccount/ClusterRole/ClusterRoleBinding |
| `02-daemonset.yaml` | 0 | Multus thick-plugin DaemonSet — **nodeSelector restricted to `nik-debian` only**, k3s-specific CNI paths |
| `10-nad-vlan50.yaml` | 1 | VLAN 50 `NetworkAttachmentDefinition`, macvlan bridge on `enp1s0.50`, static IPAM |
| `30-browser-vpn-proxy.yaml` | 1 | `.12` — SOCKS5 proxy for browser traffic via PIA, replaces `manifests/media/gluetun-proxy.yaml`. No auth, no app-level IP allowlist (see its own header for why and what actually restricts access instead). Deliberately **not** waved after the canary — see its own header's "sync-wave 1" note for why that would hang the sync entirely — **do not sync without separate explicit approval** |
| `vlan50-egress-guard-script.yaml` | (no wave annotation — apply alongside 10) | Shared init-container script consumed by the canary, qBittorrent, JDownloader, and `browser-vpn-proxy` |
| `20-canary.yaml` | 2 | Temporary canary pod — **do not sync without separate explicit approval**, see its own header |
## Before applying anything here
1. Ansible Phases 2/3 (`ansible/roles/pia-gateway`, `ansible/roles/
vlan50-parent`) must already be live and verified — the NAD's `master:
enp1s0.50` and the whole VLAN 50 return path depend on both.
2. **Verify the Multus DaemonSet manifest** (`02-daemonset.yaml`) against
the current `k8snetworkplumbingwg/multus-cni` `deployments/multus-
daemonset-thick.yml` — see that file's own header comment for exactly
what's unverified (image tag, daemon flags) versus live-confirmed (the
k3s CNI paths).
3. Sync order matters even within this one manual-sync Application: 00 →
01 → 02, confirm the Multus pod is actually Running on `nik-debian`
and `/var/lib/rancher/k3s/agent/etc/cni/net.d` now has a generated
`00-multus.conf`, *then* 10 (and, independently, 30 — same wave, no
ordering between them), *then* — only with separate approval — 20.
`30-browser-vpn-proxy.yaml` follows the same "separate approval" gate
as 20 — no Secret precondition, it needs no credentials at all (see
its own header for why).
4. **This cluster's Argo CD ignores `/status` updates for reconciliation
purposes on every resource** (`argocd-cm`'s
`resource.customizations.ignoreResourceUpdates.all`, cluster-wide, not
specific to this Application — do not change it just for this
directory). Practical effect: a sync that wave-gates on a resource
whose health depends on a `/status` transition happening *during that
sync* — e.g. a Pod that was just deleted and recreated, going
Pending → Running — can hang forever on "waiting for healthy state of
/X", even once the resource is actually healthy live. This is why
`20-canary.yaml` no longer has anything synced after it in this
Application (see `30-browser-vpn-proxy.yaml`'s header) — don't
sync-wave anything new after the canary for the same reason. If a
sync ever hangs like this: `kubectl -n argocd patch application
multus --type=merge -p '{"operation": null}'` clears the stuck
operation (touches nothing in the actual cluster, just Argo's own
tracking) so a fresh sync can start.
## Canary lifecycle — re-running it after a Multus/CNI change
**A canary that already reached `Failed` will never rerun on its own, and
an Argo "Synced" status does not mean it passed.** `20-canary.yaml` sets
`restartPolicy: Never` and has a fixed name (`vlan50-canary`, no owning
Deployment/Job/controller) — deliberately, so nothing ever recreates it
automatically (see "Should this become a Deployment/Job instead?" below
for why that's load-bearing, not incidental). Two consequences that have
each caused real confusion live (2026-08-24), diagnosed from cluster
evidence rather than assumed:
- Once the Pod object reaches `phase: Failed`, it stays exactly as it is
forever — Kubernetes does not retry a `restartPolicy: Never` Pod, full
stop. Fixing the underlying Multus/CNI DaemonSet does **not** make this
Pod try again; it is inert. The only way to get a fresh attempt is to
delete and recreate the Pod object itself (new UID, new sandbox, new
CNI ADD from scratch).
- Argo reporting `multus` as `Synced` only means the *live Pod manifest
matches what's in Git* — for a bare Pod (not a Deployment), that
comparison is satisfied by the existing (possibly long-Failed) Pod
object as-is. Argo has no reason to touch it, so "Synced" here proves
nothing about whether the canary actually ran successfully, or even
about when it last ran. Don't read Synced as "passing."
A real example of what this produces if missed: after a DaemonSet fix
lands and rolls out, the *node* briefly passes through an inconsistent
state (old Multus daemon torn down, new one's init containers still
installing plugins, new daemon not started yet — a few seconds to low
tens of seconds depending on image pull/init time). A `FailedCreatePodSandBox`
retry loop that happens to land its next attempt inside that exact
window can produce a Pod that *looks* alive (sandbox created, init
container starts, primary `eth0` interface up) but whose secondary
`net1` attach never actually completed — no `AddedInterface` event for
it, no `k8s.v1.cni.cncf.io/network-status` annotation, the guard script's
own `net1` wait times out. That Pod is now permanently `Failed` and
permanently stale evidence about the *current* DaemonSet state — its logs
and events describe a multus-daemon instance that may no longer exist.
Treating that as proof of a live code defect (rather than recreating the
Pod first) risks chasing a bug that was already fixed.
**After any change to `02-daemonset.yaml`, `10-nad-vlan50.yaml`, or the
node's CNI binaries**, the only trustworthy re-test is:
1. Wait for the DaemonSet rollout to actually finish and be Ready —
`kubectl -n kube-system rollout status daemonset/kube-multus-ds`.
2. Delete the old (Failed) canary Pod — `kubectl -n downloads delete pod
vlan50-canary` (or via the Argo UI).
3. Recreate it fresh — `kubectl apply -f manifests/multus/20-canary.yaml`
(or a selective Argo sync of just that resource).
4. Record the **new** Pod UID (`kubectl -n downloads get pod vlan50-canary
-o jsonpath='{.metadata.uid}'`) and evaluate only logs/events tied to
that UID going forward — old aggregated events for the previous UID
are not evidence about this attempt.
### Should this become a Deployment/Job instead, so it retries itself?
Deliberately not, at least not while `ipam.type: static` has no
allocator behind it (see above) and this canary is hardcoded to
`10.10.50.100`. A controller that recreates the Pod automatically (a
`Job`, a `Deployment` with `replicas: 1`) would retry through exactly the
kind of transient window described above *without* a human confirming
the previous attempt's netns was actually torn down cleanly first — two
overlapping attempts (a slow-terminating old Pod and a freshly-started
new one) both claiming `.100` on the same L2 segment is precisely the
duplicate-address scenario the guard script's `arping -D` check now
exists to catch, not something to engineer back in via unattended
retries. Keeping this a manually recreated, controller-less Pod means
recreation only ever happens when someone is actually watching the
outcome — that property is worth more here than convenience. If this
changes (e.g. a real per-IP allocator is introduced later), revisit.
The managed switch accepts multiple source MAC addresses on nik-debian's
port (confirmed in the session that authored this plan — see the
coordination handoff in the final report of that session). macvlan bridge
is plan.md's stated preference whenever that's true, and is simpler than
ipvlan L2 (each attached pod gets its own real MAC, no L2 address-sharing
edge cases).
## Why `ipam.type: static` and no shared allocator (Whereabouts etc.)
Only ever a handful of fixed IPs on this network — qBittorrent (`.10`),
JDownloader (`.11`), `browser-vpn-proxy` (`.12`), and a single canary
(`.100`). A dynamic IPAM controller (CRD, webhook, its own failure
modes) is unjustified complexity for that; each pod's own
`k8s.v1.cni.cncf.io/networks` annotation just states its IP directly.
## Rollback
**Deleting the Kubernetes-side resources alone is not safe and can break
all new pod scheduling on nik-debian** — not just Multus-attached pods.
An earlier version of this doc claimed leaving the generated
`00-multus.conf` in place was harmless once Multus was gone; that's
wrong. Multus's "auto" config mode does **not** replace or rename the
original Flannel CNI conf file — it adds `00-multus.conf` alongside it,
which sorts first and wins by kubelet's own file-ordering convention.
`kubectl delete`ing the DaemonSet removes the running daemon and its pod,
but does **not** remove that generated conf file or the multus shim
binary the init container copied onto the host — both are host
filesystem side effects `kubectl delete` has no reach into. If
`00-multus.conf` is left behind after the daemon it depends on is gone,
containerd/kubelet will try to invoke a CNI plugin that's no longer
there for **every** new pod sandbox on that node, Multus-attached or not
— the node effectively stops being able to start any new pod until that
file is dealt with.
Correct order — host-side cleanup on nik-debian *before* the Kubernetes
side:
```bash
# 1. On nik-debian (needs root — not reachable via kubectl):
ssh -p 430 nik@10.10.40.20
sudo ls /var/lib/rancher/k3s/agent/etc/cni/net.d/
# Confirm both files are present: 00-multus.conf (generated) and the
# original Flannel conf (something like 10-flannel.conflist — the
# exact name wasn't captured during discovery; read whatever's there).
sudo cp /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conf \
/root/00-multus.conf.removed-$(date +%Y%m%d%H%M%S) # capture for diagnosis first
sudo rm /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conf
# Optional cleanup, not required for correctness once the conf file
# above is gone — the multus shim binary itself, if you want it off
# the host too:
sudo rm -f /var/lib/rancher/k3s/data/cni/multus-shim # confirm the actual filename first; do not guess-delete
# 2. Prove an ordinary pod can still be newly scheduled on nik-debian
# before touching anything else (from your workstation):
kubectl run vlan50-rollback-probe --image=busybox:1.36 --restart=Never \
--overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"nik-debian"}}}' \
-- sleep 60
kubectl wait --for=condition=Ready pod/vlan50-rollback-probe --timeout=60s
kubectl delete pod vlan50-rollback-probe
# 3. Only now, the Kubernetes-side resources, in this order:
kubectl delete -f manifests/multus/30-browser-vpn-proxy.yaml # if it was ever applied
kubectl delete -f manifests/multus/20-canary.yaml # if it was ever applied
kubectl delete -f manifests/multus/10-nad-vlan50.yaml # only after no pod still references it
kubectl delete -f manifests/multus/vlan50-egress-guard-script.yaml
kubectl delete -f manifests/multus/02-daemonset.yaml
kubectl delete -f manifests/multus/01-rbac.yaml
kubectl delete -f manifests/multus/00-crd.yaml # only after no NAD remains
```
None of this has been exercised against the live cluster — step 2's
probe pod is exactly how to prove it actually worked before considering
the rollback complete, not an optional extra.

237
manifests/multus/vlan50-egress-guard-script.yaml
View File

@ -1,237 +0,0 @@
# Apply: kubectl apply -f manifests/multus/vlan50-egress-guard-script.yaml
# Description: Shared init-container script used by every VLAN 50
# workload (canary, qBittorrent, JDownloader) to remove the automatic
# eth0 default route, add the explicit eth0/net1 routes plan.md
# requires, and install a netns-local iptables egress guard as an
# independently-enforced backstop against Internet fallback through
# eth0 — see plan.md "Pod layer" requirements and "Kubernetes routing
# and egress requirements". Not relied on alone: this is defense in
# depth alongside the explicit routing it also sets up, not a
# substitute for it.
#
# Consumed by an initContainer with NET_ADMIN (and only that container —
# app containers must not carry NET_ADMIN). Runs once per pod netns
# creation; the resulting routes/iptables state persists for the pod's
# lifetime (container restarts within the same pod do not recreate the
# netns — a full pod reschedule does, and reruns this init container
# fresh, which is the intended behavior). This has NOT been exercised
# against a live pod; treat every line as needing the Phase 5 canary
# proof (packet capture + restart test) before trusting it in production.
#
# Required env on the init container: VLAN50_GATEWAY, TECHNITIUM_IP,
# POD_CIDR, SERVICE_CIDR, NODE_IP (nik-debian's own node IP —
# kubelet-originated probe traffic to the pod arrives via the primary
# eth0 gateway and its replies must go back the same way, not out net1),
# EXPECTED_VLAN50_IP (this workload's static VLAN 50 address, without a
# prefix — .100 for the canary, .10/.11/.12 for qBittorrent/JDownloader/
# the future browser-vpn-proxy — must match the /24 the NAD's static
# IPAM assigned via the pod's own k8s.v1.cni.cncf.io/networks annotation,
# checked below rather than trusted blindly).
# Optional: EXTRA_ETH0_CIDR (space-separated, for any additional
# narrowly-required cluster path — leave unset unless a specific need is
# identified and reviewed; do not widen this to a blanket 10.0.0.0/8 or
# similar).
#
# The init container needs NET_RAW in addition to NET_ADMIN (both, drop
# ALL otherwise) — arping (used below for the duplicate-address and
# gateway-reachability checks) builds raw ARP frames over an AF_PACKET
# socket, which the kernel gates on CAP_NET_RAW specifically; NET_ADMIN
# alone is not sufficient and arping fails immediately without it. This
# is a hard runtime dependency of this script now, not optional
# hardening — a pod wiring this ConfigMap in without also granting
# NET_RAW will fail closed at the arping step every time.
#
# Corrected from an earlier version after code review: that version
# deleted the eth0 default route and then routed pod/service CIDR
# directly `dev eth0` with no gateway — remote pod addresses (on other
# nodes) are not generally on-link, so those routes would have silently
# failed to actually reach anything once the implicit default route
# (which was the only thing making them reachable) was gone. This
# version captures the real gateway from the default route *before*
# deleting it, and uses that captured gateway explicitly for every eth0
# route added afterward. It also replaced a blind `iptables -F OUTPUT`
# (which would flush any pre-existing OUTPUT rules from other sources,
# not just ours) with a dedicated chain and a single jump, matching the
# same "own chain, don't touch what isn't ours" approach used on
# minisforum's host-level kill switch.
apiVersion: v1
kind: ConfigMap
metadata:
name: vlan50-egress-guard-script
namespace: downloads
data:
guard.sh: |
#!/bin/sh
set -eu
: "${VLAN50_GATEWAY:?required}"
: "${TECHNITIUM_IP:?required}"
: "${POD_CIDR:?required}"
: "${SERVICE_CIDR:?required}"
: "${NODE_IP:?required}"
: "${EXPECTED_VLAN50_IP:?required}"
# Safe to print in full: interface/address/route state only, never
# touches the PIA token, WireGuard keys, or any other credential —
# those live entirely on minisforum's side of the tunnel, not in this
# pod's netns at all.
print_diagnostics() {
echo "[vlan50-egress-guard] diagnostics follow:" >&2
ip -details link show >&2 || true
ip -4 address show >&2 || true
ip -4 route show table all >&2 || true
}
echo "[vlan50-egress-guard] waiting for net1"
ready=0
for _ in $(seq 1 30); do
if ip link show net1 >/dev/null 2>&1; then
ready=1
break
fi
sleep 1
done
if [ "${ready}" -ne 1 ]; then
echo "[vlan50-egress-guard] FATAL: net1 did not appear after 30s — refusing to start without the VLAN 50 attachment (Multus/NAD required, no eth0-only fallback)" >&2
print_diagnostics
exit 1
fi
echo "[vlan50-egress-guard] validating net1's address: expecting ${EXPECTED_VLAN50_IP}/24"
addr_ok=0
for _ in $(seq 1 15); do
if ip -4 -o address show dev net1 | awk '{print $4}' | grep -qx "${EXPECTED_VLAN50_IP}/24"; then
addr_ok=1
break
fi
sleep 1
done
if [ "${addr_ok}" -ne 1 ]; then
echo "[vlan50-egress-guard] FATAL: net1 does not carry the expected ${EXPECTED_VLAN50_IP}/24 after 15s (interface existing is not enough — the static IPAM result itself has to match what this workload is supposed to have). Actual net1 address(es):" >&2
ip -4 -o address show dev net1 >&2 || true
print_diagnostics
exit 1
fi
echo "[vlan50-egress-guard] net1 has ${EXPECTED_VLAN50_IP}/24"
# "Duplicate or tentative" for IPv4: the kernel's IFA_F_TENTATIVE flag
# (what `ip addr show` reports as "tentative") is an IPv6 DAD concept
# only — the static IPAM CNI plugin used here does no IPv4 DAD of its
# own, so there is no tentative *state* to check for IPv4. What
# actually matters — whether some OTHER host on VLAN 50 already holds
# this address — is checked directly with an ARP probe instead.
echo "[vlan50-egress-guard] checking ${EXPECTED_VLAN50_IP} is not already in use on VLAN 50 (ARP duplicate check)"
if ! arping -D -c 2 -w 2 -I net1 "${EXPECTED_VLAN50_IP}" >/tmp/arping-dup.log 2>&1; then
echo "[vlan50-egress-guard] FATAL: ${EXPECTED_VLAN50_IP} appears to already be in use on VLAN 50 — arping -D got a reply from another host, meaning this address is a duplicate. Refusing to proceed with a conflicting static IP. arping output:" >&2
cat /tmp/arping-dup.log >&2 || true
print_diagnostics
exit 1
fi
echo "[vlan50-egress-guard] no duplicate detected for ${EXPECTED_VLAN50_IP}"
# L2 (ARP) reachability, checked before this gateway is trusted with
# the default route below — deliberately not an ICMP ping instead:
# ARP resolution is a hard prerequisite for delivering any IP packet
# over Ethernet at all, so it is a reliable, low-false-positive proof
# the gateway is actually there; an ICMP probe would additionally
# depend on the gateway choosing to answer echo requests, which many
# routers disable for unrelated security reasons while still routing
# traffic completely normally — that would risk failing this check
# closed for a gateway that works fine.
echo "[vlan50-egress-guard] checking VLAN 50 gateway ${VLAN50_GATEWAY} answers ARP on net1 before trusting it"
if ! arping -c 3 -w 3 -I net1 "${VLAN50_GATEWAY}" >/tmp/arping-gw.log 2>&1; then
echo "[vlan50-egress-guard] FATAL: VLAN 50 gateway ${VLAN50_GATEWAY} did not answer ARP on net1 — refusing to install it as the default route. arping output:" >&2
cat /tmp/arping-gw.log >&2 || true
print_diagnostics
exit 1
fi
echo "[vlan50-egress-guard] gateway ${VLAN50_GATEWAY} is reachable"
echo "[vlan50-egress-guard] capturing the original eth0 default gateway"
ETH0_GATEWAY=$(ip route show default dev eth0 2>/dev/null | awk '/^default/ {print $3; exit}')
if [ -z "${ETH0_GATEWAY}" ]; then
echo "[vlan50-egress-guard] FATAL: could not identify eth0's original default gateway — refusing to proceed. Pod/service CIDR routes below need it explicitly (those destinations are not on-link on eth0), and continuing without it would either leave them unreachable or silently do nothing." >&2
print_diagnostics
exit 1
fi
echo "[vlan50-egress-guard] eth0 gateway: ${ETH0_GATEWAY}"
echo "[vlan50-egress-guard] removing the automatic default route on eth0"
ip route del default dev eth0 2>/dev/null || true
ip -6 route del default 2>/dev/null || true
echo "[vlan50-egress-guard] explicit eth0 routes via the captured gateway: pod CIDR, service CIDR, node IP${EXTRA_ETH0_CIDR:+, extra}"
ip route replace "${POD_CIDR}" via "${ETH0_GATEWAY}" dev eth0
ip route replace "${SERVICE_CIDR}" via "${ETH0_GATEWAY}" dev eth0
ip route replace "${NODE_IP}/32" via "${ETH0_GATEWAY}" dev eth0
for cidr in ${EXTRA_ETH0_CIDR:-}; do
ip route replace "${cidr}" via "${ETH0_GATEWAY}" dev eth0
done
echo "[vlan50-egress-guard] net1 routes: Technitium /32, default"
ip route replace "${TECHNITIUM_IP}/32" via "${VLAN50_GATEWAY}" dev net1
ip route replace default via "${VLAN50_GATEWAY}" dev net1
echo "[vlan50-egress-guard] installing IPv4 netns egress guard (dedicated chain, not a blind OUTPUT flush)"
iptables -N VLAN50-GUARD 2>/dev/null || true
iptables -F VLAN50-GUARD
if ! iptables -C OUTPUT -j VLAN50-GUARD 2>/dev/null; then
iptables -I OUTPUT 1 -j VLAN50-GUARD
fi
iptables -A VLAN50-GUARD -o lo -j ACCEPT
iptables -A VLAN50-GUARD -o eth0 -d "${POD_CIDR}" -j ACCEPT
iptables -A VLAN50-GUARD -o eth0 -d "${SERVICE_CIDR}" -j ACCEPT
iptables -A VLAN50-GUARD -o eth0 -d "${NODE_IP}/32" -j ACCEPT
for cidr in ${EXTRA_ETH0_CIDR:-}; do
iptables -A VLAN50-GUARD -o eth0 -d "${cidr}" -j ACCEPT
done
iptables -A VLAN50-GUARD -o net1 -j ACCEPT
iptables -A VLAN50-GUARD -o eth0 -m limit --limit 5/minute -j LOG --log-prefix "VLAN50-EGRESS-GUARD-DROP: "
iptables -A VLAN50-GUARD -o eth0 -j DROP
echo "[vlan50-egress-guard] blocking IPv6 entirely (both interfaces) — IPv4-only design"
# nft (nftables), not ip6tables and not a /proc/sys sysctl write —
# both tried first and both failed live, 2026-08-24:
# - ip6tables: genuinely absent from this image. Alpine 3.18.0
# (nicolaka/netshoot:v0.11's real, tag-pinned base — confirmed
# against that exact Dockerfile, not master's) packages ip6tables
# SEPARATELY from iptables, and this image's Dockerfile only
# installs the latter (confirmed against Alpine's own v3.18
# package index) — "ip6tables: not found", not a PATH issue (the
# iptables calls just above this ran fine from the same image).
# - `echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6`: failed
# with "Read-only file system" despite the [ -w ] check on that
# path reporting it writable and NET_ADMIN already being granted
# — permission bits and the container runtime's own read-only
# mount of /proc/sys are two independent gates, and [ -w ] only
# tests the first. Kubernetes' two supported ways to actually get
# a writable /proc/sys/net/* here are pod-spec
# securityContext.sysctls (kubelet-mediated, pre-container-start
# — but net.ipv6.conf.*.disable_ipv6 isn't on kubelet's own
# default safe-sysctls allowlist, so this would need a
# node-level --allowed-unsafe-sysctls kubelet flag) or
# securityContext.procMount: Unmasked (which itself requires
# pod-level user namespaces, spec.hostUsers: false) — both far
# more cluster-wide blast radius than an IPv6 kill switch on one
# pod justifies.
# - nft is genuinely present: the same Dockerfile that omits
# ip6tables installs the separate "nftables" apk package
# (confirmed against Alpine's own v3.18 package index, same way
# ip6tables' absence was). nftables handles the ip6 address
# family as an ordinary table — no separate ip6-specific binary
# needed — so this needs no image or capability change beyond
# what's already granted (CAP_NET_ADMIN, already present for the
# iptables/ip rules above). Syntax verified against nftables'
# own wiki (base chain + policy, and the meta oifname interface
# matcher), not guessed.
nft add table ip6 vlan50guard6
nft 'add chain ip6 vlan50guard6 output { type filter hook output priority 0; policy drop; }'
nft 'add chain ip6 vlan50guard6 forward { type filter hook forward priority 0; policy drop; }'
nft add rule ip6 vlan50guard6 output meta oifname lo accept
echo "[vlan50-egress-guard] IPv6 blocked (nft table ip6 vlan50guard6, output+forward policy drop, lo excepted)"
echo "[vlan50-egress-guard] final state:"
ip route show
iptables -S OUTPUT
iptables -S VLAN50-GUARD
nft list table ip6 vlan50guard6

2
manifests/network/glances-debian-ingress.yaml
View File

@ -8,7 +8,7 @@ metadata:
namespace: glances
subsets:
- addresses:
- ip: 10.10.40.20
- ip: 192.168.7.183
ports:
- port: 61208
---

2
manifests/network/watch-party-ingress.yaml
View File

@ -8,7 +8,7 @@ metadata:
namespace: default
subsets:
- addresses:
- ip: 10.10.40.30
- ip: 192.168.7.96
ports:
- port: 3000
---

2
values/gitea.yaml
View File

@ -63,7 +63,7 @@ service:
ssh:
type: LoadBalancer
port: 2222
loadBalancerIP: 10.10.40.53
loadBalancerIP: 192.168.7.77
postgresql-ha:
enabled: false

44
values/pihole-debian.yaml
View File

@ -1,7 +1,7 @@
# Apply: helm upgrade --install pihole-debian mojo2600/pihole -f values/pihole-debian.yaml -n pihole --create-namespace
# Description: Helm values for Pi-hole secondary instance on Debian node
replicaCount: 0
replicaCount: 1
nodeSelector:
node-role: storage
@ -15,6 +15,8 @@ serviceWeb:
serviceDns:
type: ClusterIP
port: 53
annotations:
kube-vip.io/loadbalancerIPs: 192.168.7.183
probes:
liveness:
@ -32,26 +34,26 @@ extraEnvVars:
dnsmasq:
customDnsEntries:
- address=/gitea.home.arpa/10.10.40.53
- address=/pihole.home.arpa/10.10.40.53
- address=/home.arpa/10.10.40.53
- address=/nik4nao.home.arpa/10.10.40.20
- address=/dashy.home.arpa/10.10.40.53
- address=/jellyfin.home.arpa/10.10.40.53
- address=/qbittorrent.home.arpa/10.10.40.53
- address=/jdownloader.home.arpa/10.10.40.53
- address=/glances.home.arpa/10.10.40.53
- address=/glances-debian.home.arpa/10.10.40.53
- address=/ca.home.arpa/10.10.40.53
- address=/auth.home.arpa/10.10.40.53
- address=/traefik.home.arpa/10.10.40.53
- address=/gitea.nik4nao.com/10.10.40.53
- address=/ha.home.arpa/10.10.40.53
- address=/argocd.home.arpa/10.10.40.53
- address=/gluetun.home.arpa/10.10.40.53
- address=/alert-bridge.home.arpa/10.10.40.53
- address=/books.home.arpa/10.10.40.53
- address=/books2.home.arpa/10.10.40.53
- address=/gitea.home.arpa/192.168.7.77
- address=/pihole.home.arpa/192.168.7.77
- address=/home.arpa/192.168.7.77
- address=/nik4nao.home.arpa/192.168.7.183
- address=/dashy.home.arpa/192.168.7.77
- address=/jellyfin.home.arpa/192.168.7.77
- address=/qbittorrent.home.arpa/192.168.7.77
- address=/jdownloader.home.arpa/192.168.7.77
- address=/glances.home.arpa/192.168.7.77
- address=/glances-debian.home.arpa/192.168.7.77
- address=/ca.home.arpa/192.168.7.77
- address=/auth.home.arpa/192.168.7.77
- address=/traefik.home.arpa/192.168.7.77
- address=/gitea.nik4nao.com/192.168.7.77
- address=/ha.home.arpa/192.168.7.77
- address=/argocd.home.arpa/192.168.7.77
- address=/gluetun.home.arpa/192.168.7.77
- address=/alert-bridge.home.arpa/192.168.7.77
- address=/books.home.arpa/192.168.7.77
- address=/books2.home.arpa/192.168.7.77
persistentVolumeClaim:
enabled: true

44
values/pihole.yaml
View File

@ -1,13 +1,13 @@
# Apply: helm upgrade --install pihole mojo2600/pihole -f values/pihole.yaml -n pihole --create-namespace
# Description: Helm values for Pi-hole DNS server on Minisforum
replicaCount: 0
replicaCount: 1
image:
tag: "2024.07.0"
serviceDns:
type: ClusterIP
type: LoadBalancer
port: 53
serviceWeb:
@ -65,23 +65,23 @@ extraEnvVars:
dnsmasq:
customDnsEntries:
- address=/gitea.home.arpa/10.10.40.53
- address=/pihole.home.arpa/10.10.40.53
- address=/home.arpa/10.10.40.53
- address=/nik4nao.home.arpa/10.10.40.20
- address=/dashy.home.arpa/10.10.40.53
- address=/jellyfin.home.arpa/10.10.40.53
- address=/qbittorrent.home.arpa/10.10.40.53
- address=/jdownloader.home.arpa/10.10.40.53
- address=/glances.home.arpa/10.10.40.53
- address=/glances-debian.home.arpa/10.10.40.53
- address=/ca.home.arpa/10.10.40.53
- address=/auth.home.arpa/10.10.40.53
- address=/traefik.home.arpa/10.10.40.53
- address=/gitea.nik4nao.com/10.10.40.53
- address=/ha.home.arpa/10.10.40.53
- address=/argocd.home.arpa/10.10.40.53
- address=/gluetun.home.arpa/10.10.40.53
- address=/alert-bridge.home.arpa/10.10.40.53
- address=/books.home.arpa/10.10.40.53
- address=/books2.home.arpa/10.10.40.53
- address=/gitea.home.arpa/192.168.7.77
- address=/pihole.home.arpa/192.168.7.77
- address=/home.arpa/192.168.7.77
- address=/nik4nao.home.arpa/192.168.7.183
- address=/dashy.home.arpa/192.168.7.77
- address=/jellyfin.home.arpa/192.168.7.77
- address=/qbittorrent.home.arpa/192.168.7.77
- address=/jdownloader.home.arpa/192.168.7.77
- address=/glances.home.arpa/192.168.7.77
- address=/glances-debian.home.arpa/192.168.7.77
- address=/ca.home.arpa/192.168.7.77
- address=/auth.home.arpa/192.168.7.77
- address=/traefik.home.arpa/192.168.7.77
- address=/gitea.nik4nao.com/192.168.7.77
- address=/ha.home.arpa/192.168.7.77
- address=/argocd.home.arpa/192.168.7.77
- address=/gluetun.home.arpa/192.168.7.77
- address=/alert-bridge.home.arpa/192.168.7.77
- address=/books.home.arpa/192.168.7.77
- address=/books2.home.arpa/192.168.7.77