Enabling MSS clamping exposed a general hazard: systemctl restart
wg-quick@pia-wg tears down the live interface using whatever PreDown
rules are on disk right now, not what the interface was actually
brought up with. If Ansible had already rewritten pia-wg.conf with a
changed rule, the live interface (built under the old rules) won't
have whatever the new PreDown line tries to delete - iptables -D fails,
wg-quick's own set -e aborts the whole down/up sequence, and the
interface is left orphaned (never reaches its own built-in ip link
delete step), which then makes the following wg-quick up fail too with
"already exists".
Confirmed against wg-quick's real source (execute_hooks() runs each
hook via `(eval "$hook")`) that appending `2>/dev/null || true` to
each PreDown iptables -D line makes wg-quick treat it as succeeded
regardless of prior state - verified by rendering the template with
Jinja2 directly before trusting it against the live host. This
prevents the failure mode entirely for any future PostUp/PreDown
content change, not just this one.
Add INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md covering all four
issues from this troubleshooting window: the zombie WireGuard tunnel,
a qBittorrent 5.2.0 stale-lock crash loop, the PMTU black hole this
fix addresses, and the restart failure hit while deploying it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
UFW allowed 53/udp+tcp from every other internal subnet but not from
10.10.0.0/24 (the WireGuard client subnet), so once the tunnel itself
was reachable, decrypted DNS queries still got dropped at minisforum's
own INPUT chain before reaching Technitium.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
AllowedIPs for the mac road-warrior peer was missing 10.10.40.0/24, so
DNS (10.10.40.53) and other home-LAN hosts were unreachable over the
tunnel. Also fixes the same string-vs-bool `when:` failure already
patched in e757850 (recent ansible-core rejects a `-e ...=true` CLI
string in a boolean conditional) for the client-config display tasks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Creates enp1s0.50 on nik-debian (no L3 address, VLAN 40/node IP/default
route untouched) for Multus to later attach macvlan/ipvlan workloads to.
Verified live: enp1s0.50 is up with no IPv4 address (only the automatic
IPv6 link-local, which is expected and harmless), default route
unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Registers minisforum as a PIA WireGuard peer for VPN VLAN 50, with a
boot-ordered kill switch (dedicated PIA-VLAN50 iptables chain + a
terminal unreachable route in a dedicated routing table), multi-region
addKey fallback (Hong Kong -> Taiwan -> JP Tokyo, each region's full
server list, in order), and an observability-only health check.
Verified live against minisforum: registration succeeds, wg-quick@pia-wg
is up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds explicit UFW routed-allow rules (10.42.0.0/16 pod-to-pod, pod-to-Technitium
DNS) since these nodes default their routed/FORWARD policy to DROP. Also brings
nik-debian's NFS (2049) and SMB (445) ports under Ansible-managed UFW rules via
the existing nfs-server role, scoped to the Lab/Trusted networks that need them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
minisforum, debian, mac-mini, and gpu-node all moved from
192.168.7.0/24 to 10.10.40.0/24. Updates K3s server/agent config and
node IPs (including gpu-node's host_vars override), NFS export
allow-list and exports template, Pi-hole DNS records and
kube-vip/loadBalancerIP pins, WireGuard's pushed DNS/AllowedIPs, and
the NFS server IP baked into Jellyfin/Kavita/gitea-backup PVs and the
Ollama URL used by ai-gateway/Dashy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This reverts commit fd13b8a. The registry-host migration broke image
pulls for home-services/portfolio (internal CA not trusted by
containerd) and has left them Degraded for 11 days; reverting back to
gitea.nik4nao.com pending a decision on how to proceed. Not pushed yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The public domain is unreachable while moving, and the cluster had no
Traefik route to Gitea at all (public or internal), leaving every Argo
CD Application stuck in Unknown sync. Add a gitea.home.arpa
Certificate/IngressRoute, repoint Argo CD's repoURL, Gitea's own
DOMAIN/ROOT_URL/SSH_DOMAIN, the container registry references, the
Gitea Actions runner, and the watch-party clone URL at the internal
hostname.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ansible.cfg sets inject_facts_as_vars: False repo-wide (the ollama role
already follows this via ansible_facts['system']), but these two tasks
referenced the bare ansible_hostname magic var, which doesn't exist under
that setting. Surfaced today as a hard failure re-running setup-gpu-node -
previously latent since these nodes' labels/taints were already correct
from whenever they last applied cleanly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 4 of REFACTOR_PLAN.md.
- Merge ansible/ansible.cfg into root ansible.cfg (single source of truth);
the dual-config setup silently broke documented commands and lost
host_key_checking=False when run from the ansible/ directory.
- Add ansible/requirements.yml pinning community.general, ansible.posix,
community.docker -- previously undocumented deps of the glances/watch-party
roles that would fail a fresh bootstrap.
- Align K3s version to v1.32.4+k3s1 across roles/k3s-server, roles/k3s-agent,
and host_vars/gpu-node.yaml defaults (was skewed: .2 vs .4). This only
changes what a *future* provision installs -- minisforum/debian are still
live on v1.32.2+k3s1 until separately upgraded.
- Fix kubeconfig fetch/replace path mismatch in k3s-server role: the `fetch`
task (flat: true) writes to ~/.kube/config, but `replace` was targeting a
/tmp/k3s-minisforum.yaml nothing creates -- would break a fresh rebuild.
- gitea-runner: only remove /run/docker.sock when it's actually a directory
(task name implied a check that wasn't there); tighten registration-token
systemd unit from 0644 to 0600.
- nvidia: stop unconditionally reporting `changed` (and restarting Docker)
on every run for an idempotent runtime-configure command.
- Gate the K3s join-token debug print and WireGuard client-config/QR display
behind opt-in vars (k3s_show_token, wireguard_show_client_configs), default
off -- both were printing real secrets to console on every run.
- Parameterize the docker role for Debian and Ubuntu; homeassistant now
depends on it (meta/main.yaml) instead of duplicating a Debian-only Docker
install inline.
- FQCN cleanup across wireguard, homeassistant, and ollama roles/handlers
(bare module names -> ansible.builtin.*/community.general.*/ansible.posix.*),
plus a few ansible-lint name-casing/idiom fixes. Handler renames verified
against their `notify:` call sites so notifications still fire.
- Update ansible/README.md and root README.md: add gpu-node/gpu_workstation
(4th host, previously undocumented), docker/nvidia roles, setup-gpu-node.yaml,
homeassistant.yaml, requirements.yml install step; correct the "Legacy"
homeassistant label (it's the only thing serving ha.home.arpa); correct the
Gitea ingress/backup-storage doc-drift; flag the Grafana/Loki static-PV
binding drift discovered via live cluster check.
Verified: all playbooks pass `ansible-playbook --syntax-check`, yamllint clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
feat(gitea): create PersistentVolume and PersistentVolumeClaim for Gitea
feat(gitea): add script to create Gitea runner registration token secret
feat(gitea): deploy Gitea Actions runner with Docker socket access
feat(media): deploy JDownloader with Ingress configuration
feat(media): set up Jellyfin media server with NFS and Ingress
feat(media): configure qBittorrent deployment with Ingress
feat(monitoring): add Grafana Loki datasource ConfigMap
feat(monitoring): create Grafana admin credentials secret
feat(monitoring): define PersistentVolumes for monitoring stack
feat(network): implement DDNS CronJob for Porkbun DNS updates
feat(network): create secret for Porkbun DDNS API credentials
feat(network): set up Glances service and Ingress for Debian node
fix(network): patch Pi-hole DNS services with external IPs
feat(network): configure Traefik dashboard Ingress with Authentik auth
feat(network): set up Watch Party service and Ingress for Mac Mini
refactor(values): update Helm values files for various services
- Created ansible.cfg for configuration settings
- Added inventory.yml for host definitions
- Implemented bootstrap playbook for Minisforum setup
- Developed setup playbook for K3s installation
- Defined common role with user and package management tasks
- Established K3s server role with configuration and installation tasks
- Included Traefik Helm values for ingress management
