Some checks failed
validate / lint (push) Failing after 1s
Found live on the first deployment of the auto-recovery controller (2026-09-22): pia-gateway register correctly refused with "FORWARD jump to kill-switch chain is not first". Root cause confirmed on minisforum via iptables -S FORWARD/-S KUBE-ROUTER-FORWARD/-S DOCKER-USER/-S DOCKER-FORWARD: the PIA-VLAN50 jump was 4th, behind kube-router's and dockerd's own jumps, which each independently reassert position 1 on their own resyncs/restarts (kube-router had even left a duplicate of its own rule behind). pia-killswitch.sh.j2 only checked whether its jump existed (iptables -C) before deciding not to touch anything — existence isn't proof of position, so once pushed down it stayed down. Confirmed safe to reposition unconditionally by reading the actual content of what's ahead of it rather than assuming: KUBE-ROUTER-FORWARD only matches specific pod /32 addresses, DOCKER-USER was empty, and DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50 arrives via enp1s0) - none can intercept VLAN 50 traffic regardless of order. Fixed the script itself (delete + reinsert at position 1 if the current first rule isn't ours) rather than loosening the Python controller's forward_jump_first safety check, which was working correctly when it refused to register against the drifted state. Verified offline: rendered template passes bash -n, and the exact live topology (rule present but 4th) reproduced against a bash mock of iptables, confirming reposition-when-drifted, idempotency when already first, and insert-when-absent all behave correctly. No real iptables/wg/ip on this dev machine to test further; not yet re-applied or re-verified against minisforum. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ansible
This directory contains host-level automation. It bootstraps machines, installs K3s, prepares storage, and manages services that intentionally run outside the cluster.
Inventory
inventory.yaml defines four groups:
| Group | Host | Purpose |
|---|---|---|
k3s_server |
minisforum |
K3s server at 10.10.40.53 |
k3s_agents |
debian |
K3s agent and NFS storage at 10.10.40.20 |
mac_mini |
mac-mini |
Docker/Ollama host at 10.10.40.30 |
gpu_workstation |
gpu-node |
K3s agent with NVIDIA GPU passthrough at 10.10.40.12 (spot-tainted) |
All hosts use the nik user and the SSH key configured in inventory.yaml.
Collections
Install the third-party collections this repo's roles depend on before running any playbook:
ansible-galaxy collection install -r ansible/requirements.yml
(community.general, ansible.posix, community.docker.)
Common Playbooks
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K
Additional services:
export GITEA_RUNNER_TOKEN=...
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-monitoring.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/homeassistant.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/vlan50-parent.yaml -K
Roles
| Role | Responsibility |
|---|---|
common |
Packages, user setup, firewall, base data directories |
docker |
Docker CE install (Debian and Ubuntu); depended on by homeassistant |
nvidia |
NVIDIA driver, CUDA toolkit, and containerd/Docker GPU runtime config |
k3s-server |
K3s server install, kubeconfig fetch, Helm install, primary node label |
k3s-agent |
K3s agent join and storage/GPU node label |
nfs-server |
Export /mnt/storage from Debian to the K3s server |
monitoring |
Host directories and ownership for Prometheus/Loki |
gitea-runner |
Gitea Actions runner systemd service |
glances |
Host-level Glances service |
ollama |
Ollama service on the Mac Mini and GPU node (branches on OS) |
watch-party |
Watch Party Docker Compose deployment on the Mac Mini |
wireguard |
WireGuard server configuration (inbound home-VPN access — phone/Mac clients) |
pia-gateway |
Minisforum's PIA WireGuard egress gateway for VPN VLAN 50 (policy routing, kill switch, health checks) — see its own README |
vlan50-parent |
nik-debian's tagged VLAN 50 parent interface (enp1s0.50) for Multus — see its own README |
homeassistant |
Standalone Home Assistant deployment (Docker Compose + systemd on minisforum) — this is the only thing serving ha.home.arpa, not legacy/dead |
Notes
- K3s version is defined in three places and must be kept in sync:
roles/k3s-server/defaults/main.yaml,roles/k3s-agent/defaults/main.yaml, and the override inhost_vars/gpu-node.yaml. setup-gitea-runner.yamlreadsGITEA_RUNNER_TOKENfrom the local environment.- The K3s role disables bundled Traefik because Traefik is managed by Argo CD.
- The Debian storage role exports
/mnt/storage; several Kubernetes manifests mount that export directly. - Keep host automation idempotent where practical. These playbooks are meant to be rerunnable during rebuilds.
- To see the real K3s join token (needed once, to populate
vault_k3s_node_token), pass-e k3s_show_token=truetosetup-k3s.yaml; it's suppressed by default. Same pattern for WireGuard client configs via-e wireguard_show_client_configs=trueonwireguard.yaml.