Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:
19:38:58 3rd unhealthy check -> restart
19:39:09 recovered - route open, handshake fresh
19:40:03 drift again -> emergency close -> route forced closed,
counter reset to 0
19:41:08 drift again -> emergency close again
VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.
This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.
Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.
Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.
pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ansible
This directory contains host-level automation. It bootstraps machines, installs K3s, prepares storage, and manages services that intentionally run outside the cluster.
Inventory
inventory.yaml defines four groups:
| Group | Host | Purpose |
|---|---|---|
k3s_server |
minisforum |
K3s server at 10.10.40.53 |
k3s_agents |
debian |
K3s agent and NFS storage at 10.10.40.20 |
mac_mini |
mac-mini |
Docker/Ollama host at 10.10.40.30 |
gpu_workstation |
gpu-node |
K3s agent with NVIDIA GPU passthrough at 10.10.40.12 (spot-tainted) |
All hosts use the nik user and the SSH key configured in inventory.yaml.
Collections
Install the third-party collections this repo's roles depend on before running any playbook:
ansible-galaxy collection install -r ansible/requirements.yml
(community.general, ansible.posix, community.docker.)
Common Playbooks
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K
Additional services:
export GITEA_RUNNER_TOKEN=...
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-monitoring.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/homeassistant.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/vlan50-parent.yaml -K
Roles
| Role | Responsibility |
|---|---|
common |
Packages, user setup, firewall, base data directories |
docker |
Docker CE install (Debian and Ubuntu); depended on by homeassistant |
nvidia |
NVIDIA driver, CUDA toolkit, and containerd/Docker GPU runtime config |
k3s-server |
K3s server install, kubeconfig fetch, Helm install, primary node label |
k3s-agent |
K3s agent join and storage/GPU node label |
nfs-server |
Export /mnt/storage from Debian to the K3s server |
monitoring |
Host directories and ownership for Prometheus/Loki |
gitea-runner |
Gitea Actions runner systemd service |
glances |
Host-level Glances service |
ollama |
Ollama service on the Mac Mini and GPU node (branches on OS) |
watch-party |
Watch Party Docker Compose deployment on the Mac Mini |
wireguard |
WireGuard server configuration (inbound home-VPN access — phone/Mac clients) |
pia-gateway |
Minisforum's PIA WireGuard egress gateway for VPN VLAN 50 (policy routing, kill switch, health checks) — see its own README |
vlan50-parent |
nik-debian's tagged VLAN 50 parent interface (enp1s0.50) for Multus — see its own README |
homeassistant |
Standalone Home Assistant deployment (Docker Compose + systemd on minisforum) — this is the only thing serving ha.home.arpa, not legacy/dead |
Notes
- K3s version is defined in three places and must be kept in sync:
roles/k3s-server/defaults/main.yaml,roles/k3s-agent/defaults/main.yaml, and the override inhost_vars/gpu-node.yaml. setup-gitea-runner.yamlreadsGITEA_RUNNER_TOKENfrom the local environment.- The K3s role disables bundled Traefik because Traefik is managed by Argo CD.
- The Debian storage role exports
/mnt/storage; several Kubernetes manifests mount that export directly. - Keep host automation idempotent where practical. These playbooks are meant to be rerunnable during rebuilds.
- To see the real K3s join token (needed once, to populate
vault_k3s_node_token), pass-e k3s_show_token=truetosetup-k3s.yaml; it's suppressed by default. Same pattern for WireGuard client configs via-e wireguard_show_client_configs=trueonwireguard.yaml.