Nik Afiq de5ce7bf52 fix: reposition pia-killswitch's FORWARD jump, not just check it exists
Found live on the first deployment of the auto-recovery controller
(2026-09-22): pia-gateway register correctly refused with "FORWARD
jump to kill-switch chain is not first". Root cause confirmed on
minisforum via iptables -S FORWARD/-S KUBE-ROUTER-FORWARD/-S
DOCKER-USER/-S DOCKER-FORWARD: the PIA-VLAN50 jump was 4th, behind
kube-router's and dockerd's own jumps, which each independently
reassert position 1 on their own resyncs/restarts (kube-router had
even left a duplicate of its own rule behind). pia-killswitch.sh.j2
only checked whether its jump existed (iptables -C) before deciding
not to touch anything — existence isn't proof of position, so once
pushed down it stayed down.

Confirmed safe to reposition unconditionally by reading the actual
content of what's ahead of it rather than assuming: KUBE-ROUTER-FORWARD
only matches specific pod /32 addresses, DOCKER-USER was empty, and
DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50
arrives via enp1s0) - none can intercept VLAN 50 traffic regardless of
order. Fixed the script itself (delete + reinsert at position 1 if the
current first rule isn't ours) rather than loosening the Python
controller's forward_jump_first safety check, which was working
correctly when it refused to register against the drifted state.

Verified offline: rendered template passes bash -n, and the exact
live topology (rule present but 4th) reproduced against a bash mock
of iptables, confirming reposition-when-drifted, idempotency when
already first, and insert-when-absent all behave correctly. No real
iptables/wg/ip on this dev machine to test further; not yet
re-applied or re-verified against minisforum.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 16:46:30 +09:00
..

Ansible

This directory contains host-level automation. It bootstraps machines, installs K3s, prepares storage, and manages services that intentionally run outside the cluster.

Inventory

inventory.yaml defines four groups:

Group Host Purpose
k3s_server minisforum K3s server at 10.10.40.53
k3s_agents debian K3s agent and NFS storage at 10.10.40.20
mac_mini mac-mini Docker/Ollama host at 10.10.40.30
gpu_workstation gpu-node K3s agent with NVIDIA GPU passthrough at 10.10.40.12 (spot-tainted)

All hosts use the nik user and the SSH key configured in inventory.yaml.

Collections

Install the third-party collections this repo's roles depend on before running any playbook:

ansible-galaxy collection install -r ansible/requirements.yml

(community.general, ansible.posix, community.docker.)

Common Playbooks

ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K

Additional services:

export GITEA_RUNNER_TOKEN=...
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-monitoring.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/homeassistant.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/vlan50-parent.yaml -K

Roles

Role Responsibility
common Packages, user setup, firewall, base data directories
docker Docker CE install (Debian and Ubuntu); depended on by homeassistant
nvidia NVIDIA driver, CUDA toolkit, and containerd/Docker GPU runtime config
k3s-server K3s server install, kubeconfig fetch, Helm install, primary node label
k3s-agent K3s agent join and storage/GPU node label
nfs-server Export /mnt/storage from Debian to the K3s server
monitoring Host directories and ownership for Prometheus/Loki
gitea-runner Gitea Actions runner systemd service
glances Host-level Glances service
ollama Ollama service on the Mac Mini and GPU node (branches on OS)
watch-party Watch Party Docker Compose deployment on the Mac Mini
wireguard WireGuard server configuration (inbound home-VPN access — phone/Mac clients)
pia-gateway Minisforum's PIA WireGuard egress gateway for VPN VLAN 50 (policy routing, kill switch, health checks) — see its own README
vlan50-parent nik-debian's tagged VLAN 50 parent interface (enp1s0.50) for Multus — see its own README
homeassistant Standalone Home Assistant deployment (Docker Compose + systemd on minisforum) — this is the only thing serving ha.home.arpa, not legacy/dead

Notes

  • K3s version is defined in three places and must be kept in sync: roles/k3s-server/defaults/main.yaml, roles/k3s-agent/defaults/main.yaml, and the override in host_vars/gpu-node.yaml.
  • setup-gitea-runner.yaml reads GITEA_RUNNER_TOKEN from the local environment.
  • The K3s role disables bundled Traefik because Traefik is managed by Argo CD.
  • The Debian storage role exports /mnt/storage; several Kubernetes manifests mount that export directly.
  • Keep host automation idempotent where practical. These playbooks are meant to be rerunnable during rebuilds.
  • To see the real K3s join token (needed once, to populate vault_k3s_node_token), pass -e k3s_show_token=true to setup-k3s.yaml; it's suppressed by default. Same pattern for WireGuard client configs via -e wireguard_show_client_configs=true on wireguard.yaml.