Nik Afiq
4d613c9c06
Some checks failed
validate / lint (push) Failing after 1s
Enabling MSS clamping exposed a general hazard: systemctl restart wg-quick@pia-wg tears down the live interface using whatever PreDown rules are on disk right now, not what the interface was actually brought up with. If Ansible had already rewritten pia-wg.conf with a changed rule, the live interface (built under the old rules) won't have whatever the new PreDown line tries to delete - iptables -D fails, wg-quick's own set -e aborts the whole down/up sequence, and the interface is left orphaned (never reaches its own built-in ip link delete step), which then makes the following wg-quick up fail too with "already exists". Confirmed against wg-quick's real source (execute_hooks() runs each hook via `(eval "$hook")`) that appending `2>/dev/null || true` to each PreDown iptables -D line makes wg-quick treat it as succeeded regardless of prior state - verified by rendering the template with Jinja2 directly before trusting it against the live host. This prevents the failure mode entirely for any future PostUp/PreDown content change, not just this one. Add INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md covering all four issues from this troubleshooting window: the zombie WireGuard tunnel, a qBittorrent 5.2.0 stale-lock crash loop, the PMTU black hole this fix addresses, and the restart failure hit while deploying it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ansible
This directory contains host-level automation. It bootstraps machines, installs K3s, prepares storage, and manages services that intentionally run outside the cluster.
Inventory
inventory.yaml defines four groups:
| Group | Host | Purpose |
|---|---|---|
k3s_server |
minisforum |
K3s server at 10.10.40.53 |
k3s_agents |
debian |
K3s agent and NFS storage at 10.10.40.20 |
mac_mini |
mac-mini |
Docker/Ollama host at 10.10.40.30 |
gpu_workstation |
gpu-node |
K3s agent with NVIDIA GPU passthrough at 10.10.40.12 (spot-tainted) |
All hosts use the nik user and the SSH key configured in inventory.yaml.
Collections
Install the third-party collections this repo's roles depend on before running any playbook:
ansible-galaxy collection install -r ansible/requirements.yml
(community.general, ansible.posix, community.docker.)
Common Playbooks
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K
Additional services:
export GITEA_RUNNER_TOKEN=...
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-monitoring.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/homeassistant.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/vlan50-parent.yaml -K
Roles
| Role | Responsibility |
|---|---|
common |
Packages, user setup, firewall, base data directories |
docker |
Docker CE install (Debian and Ubuntu); depended on by homeassistant |
nvidia |
NVIDIA driver, CUDA toolkit, and containerd/Docker GPU runtime config |
k3s-server |
K3s server install, kubeconfig fetch, Helm install, primary node label |
k3s-agent |
K3s agent join and storage/GPU node label |
nfs-server |
Export /mnt/storage from Debian to the K3s server |
monitoring |
Host directories and ownership for Prometheus/Loki |
gitea-runner |
Gitea Actions runner systemd service |
glances |
Host-level Glances service |
ollama |
Ollama service on the Mac Mini and GPU node (branches on OS) |
watch-party |
Watch Party Docker Compose deployment on the Mac Mini |
wireguard |
WireGuard server configuration (inbound home-VPN access — phone/Mac clients) |
pia-gateway |
Minisforum's PIA WireGuard egress gateway for VPN VLAN 50 (policy routing, kill switch, health checks) — see its own README |
vlan50-parent |
nik-debian's tagged VLAN 50 parent interface (enp1s0.50) for Multus — see its own README |
homeassistant |
Standalone Home Assistant deployment (Docker Compose + systemd on minisforum) — this is the only thing serving ha.home.arpa, not legacy/dead |
Notes
- K3s version is defined in three places and must be kept in sync:
roles/k3s-server/defaults/main.yaml,roles/k3s-agent/defaults/main.yaml, and the override inhost_vars/gpu-node.yaml. setup-gitea-runner.yamlreadsGITEA_RUNNER_TOKENfrom the local environment.- The K3s role disables bundled Traefik because Traefik is managed by Argo CD.
- The Debian storage role exports
/mnt/storage; several Kubernetes manifests mount that export directly. - Keep host automation idempotent where practical. These playbooks are meant to be rerunnable during rebuilds.
- To see the real K3s join token (needed once, to populate
vault_k3s_node_token), pass-e k3s_show_token=truetosetup-k3s.yaml; it's suppressed by default. Same pattern for WireGuard client configs via-e wireguard_show_client_configs=trueonwireguard.yaml.