Nik Afiq fb9ecffa9f fix: tolerate content-verified kube-router jumps ahead of the kill switch
Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:

  19:38:58  3rd unhealthy check -> restart
  19:39:09  recovered - route open, handshake fresh
  19:40:03  drift again -> emergency close -> route forced closed,
            counter reset to 0
  19:41:08  drift again -> emergency close again

VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.

This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.

Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.

Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.

pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 19:55:47 +09:00
..

Ansible

This directory contains host-level automation. It bootstraps machines, installs K3s, prepares storage, and manages services that intentionally run outside the cluster.

Inventory

inventory.yaml defines four groups:

Group Host Purpose
k3s_server minisforum K3s server at 10.10.40.53
k3s_agents debian K3s agent and NFS storage at 10.10.40.20
mac_mini mac-mini Docker/Ollama host at 10.10.40.30
gpu_workstation gpu-node K3s agent with NVIDIA GPU passthrough at 10.10.40.12 (spot-tainted)

All hosts use the nik user and the SSH key configured in inventory.yaml.

Collections

Install the third-party collections this repo's roles depend on before running any playbook:

ansible-galaxy collection install -r ansible/requirements.yml

(community.general, ansible.posix, community.docker.)

Common Playbooks

ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K

Additional services:

export GITEA_RUNNER_TOKEN=...
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-monitoring.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/homeassistant.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/vlan50-parent.yaml -K

Roles

Role Responsibility
common Packages, user setup, firewall, base data directories
docker Docker CE install (Debian and Ubuntu); depended on by homeassistant
nvidia NVIDIA driver, CUDA toolkit, and containerd/Docker GPU runtime config
k3s-server K3s server install, kubeconfig fetch, Helm install, primary node label
k3s-agent K3s agent join and storage/GPU node label
nfs-server Export /mnt/storage from Debian to the K3s server
monitoring Host directories and ownership for Prometheus/Loki
gitea-runner Gitea Actions runner systemd service
glances Host-level Glances service
ollama Ollama service on the Mac Mini and GPU node (branches on OS)
watch-party Watch Party Docker Compose deployment on the Mac Mini
wireguard WireGuard server configuration (inbound home-VPN access — phone/Mac clients)
pia-gateway Minisforum's PIA WireGuard egress gateway for VPN VLAN 50 (policy routing, kill switch, health checks) — see its own README
vlan50-parent nik-debian's tagged VLAN 50 parent interface (enp1s0.50) for Multus — see its own README
homeassistant Standalone Home Assistant deployment (Docker Compose + systemd on minisforum) — this is the only thing serving ha.home.arpa, not legacy/dead

Notes

  • K3s version is defined in three places and must be kept in sync: roles/k3s-server/defaults/main.yaml, roles/k3s-agent/defaults/main.yaml, and the override in host_vars/gpu-node.yaml.
  • setup-gitea-runner.yaml reads GITEA_RUNNER_TOKEN from the local environment.
  • The K3s role disables bundled Traefik because Traefik is managed by Argo CD.
  • The Debian storage role exports /mnt/storage; several Kubernetes manifests mount that export directly.
  • Keep host automation idempotent where practical. These playbooks are meant to be rerunnable during rebuilds.
  • To see the real K3s join token (needed once, to populate vault_k3s_node_token), pass -e k3s_show_token=true to setup-k3s.yaml; it's suppressed by default. Same pattern for WireGuard client configs via -e wireguard_show_client_configs=true on wireguard.yaml.