41 Commits

Author SHA1 Message Date
02804118d4 fix: reposition pia-killswitch's FORWARD jump, not just check it exists
Some checks failed
validate / lint (push) Failing after 1s
Found live on the first deployment of the auto-recovery controller
(2026-09-22): pia-gateway register correctly refused with "FORWARD
jump to kill-switch chain is not first". Root cause confirmed on
minisforum via iptables -S FORWARD/-S KUBE-ROUTER-FORWARD/-S
DOCKER-USER/-S DOCKER-FORWARD: the PIA-VLAN50 jump was 4th, behind
kube-router's and dockerd's own jumps, which each independently
reassert position 1 on their own resyncs/restarts (kube-router had
even left a duplicate of its own rule behind). pia-killswitch.sh.j2
only checked whether its jump existed (iptables -C) before deciding
not to touch anything — existence isn't proof of position, so once
pushed down it stayed down.

Confirmed safe to reposition unconditionally by reading the actual
content of what's ahead of it rather than assuming: KUBE-ROUTER-FORWARD
only matches specific pod /32 addresses, DOCKER-USER was empty, and
DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50
arrives via enp1s0) - none can intercept VLAN 50 traffic regardless of
order. Fixed the script itself (delete + reinsert at position 1 if the
current first rule isn't ours) rather than loosening the Python
controller's forward_jump_first safety check, which was working
correctly when it refused to register against the drifted state.

Verified offline: rendered template passes bash -n, and the exact
live topology (rule present but 4th) reproduced against a bash mock
of iptables, confirming reposition-when-drifted, idempotency when
already first, and insert-when-absent all behave correctly. No real
iptables/wg/ip on this dev machine to test further; not yet
re-applied or re-verified against minisforum.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 16:44:28 +09:00
87a9303cc3 fix: address code review on pia-gateway auto-recovery controller
Some checks failed
validate / lint (push) Failing after 2s
- Serialize every mutating command (recover, renew, register,
  rewrite-config, pause, resume, state-reset) on the shared lock,
  acquired BEFORE state is read rather than after: reading first and
  locking only around the write let a slower command silently clobber
  whatever a faster one wrote in between — in particular, an operator
  pause could be lost to an in-flight recovery pass that read state
  before the pause ran and only saved its own copy back afterwards.
- Fix emergency_close() to reposition its VLAN 50 DROP to the first
  effective FORWARD position when it exists but has been pushed down
  by something else (e.g. an out-of-band ufw reload) — a bare `-C`
  existence check said nothing about position.
- Tighten wait_for_verified()'s handshake freshness check from >= to >
  since_epoch: wg show and since_epoch are both whole seconds, so a
  handshake at exactly the same second as since_epoch is
  indistinguishable from a stale, pre-activation one and must not
  count as proof of a new one.
- Fix the Ansible config-validation task to fail on any exit code
  outside status's three documented health exits (0/1/2), not just on
  5 — verified live against the real program (no wg/ip/iptables on
  this dev machine reproduces the "nothing configured yet" first-run
  case; status correctly returns 1, and broken/missing config.json
  correctly returns 5 and is now caught).
- Correct the live fault-injection drill in the role README: port
  1337 is only the addKey registration API port, not the tunnel's own
  WireGuard UDP endpoint (derive that from `wg show pia-wg endpoints`
  instead), and make the packet-capture check direction-aware
  (`tcpdump -Q out`) so it can't conflate a real egress leak with
  VLAN 50's normal inbound arrival on enp1s0.

Added 6 tests (86 total, all passing): emergency-close repositioning
past a preceding ACCEPT, pause/resume blocking on an in-flight
lock holder and surviving into a subsequent recovery pass, and the
handshake boundary at exactly/one-second-either-side of since_epoch.

yamllint, ansible-playbook --syntax-check, and ansible-lint (diffed
against main — no new finding categories) all pass. No live changes;
nothing pushed to main.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-22 15:51:02 +09:00
4d613c9c06 fix: make pia-wg PreDown idempotent to prevent restart failures
Some checks failed
validate / lint (push) Failing after 1s
Enabling MSS clamping exposed a general hazard: systemctl restart
wg-quick@pia-wg tears down the live interface using whatever PreDown
rules are on disk right now, not what the interface was actually
brought up with. If Ansible had already rewritten pia-wg.conf with a
changed rule, the live interface (built under the old rules) won't
have whatever the new PreDown line tries to delete - iptables -D fails,
wg-quick's own set -e aborts the whole down/up sequence, and the
interface is left orphaned (never reaches its own built-in ip link
delete step), which then makes the following wg-quick up fail too with
"already exists".

Confirmed against wg-quick's real source (execute_hooks() runs each
hook via `(eval "$hook")`) that appending `2>/dev/null || true` to
each PreDown iptables -D line makes wg-quick treat it as succeeded
regardless of prior state - verified by rendering the template with
Jinja2 directly before trusting it against the live host. This
prevents the failure mode entirely for any future PostUp/PreDown
content change, not just this one.

Add INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md covering all four
issues from this troubleshooting window: the zombie WireGuard tunnel,
a qBittorrent 5.2.0 stale-lock crash loop, the PMTU black hole this
fix addresses, and the restart failure hit while deploying it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-27 20:08:29 +09:00
8629a36278 fix: allow WireGuard road-warrior clients to reach DNS on minisforum
Some checks failed
validate / lint (push) Failing after 2s
UFW allowed 53/udp+tcp from every other internal subnet but not from
10.10.0.0/24 (the WireGuard client subnet), so once the tunnel itself
was reachable, decrypted DNS queries still got dropped at minisforum's
own INPUT chain before reaching Technitium.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-26 20:45:09 +09:00
b37a816b3e fix: route home LAN through mac WireGuard client, fix broken bool conditionals
AllowedIPs for the mac road-warrior peer was missing 10.10.40.0/24, so
DNS (10.10.40.53) and other home-LAN hosts were unreachable over the
tunnel. Also fixes the same string-vs-bool `when:` failure already
patched in e757850 (recent ansible-core rejects a `-e ...=true` CLI
string in a boolean conditional) for the client-config display tasks.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-26 20:45:09 +09:00
663337f818 feat: add workaround for qBittorrent 5.2.0 stale lock issue with init container
Some checks failed
validate / lint (push) Failing after 1s
2026-08-25 22:27:47 +09:00
e757850c3b fix: ensure boolean evaluation for pia_force_reregister and pia_force_key_rotation in registration conditions
Some checks failed
validate / lint (push) Failing after 2s
2026-08-25 20:59:40 +09:00
7eb7072ec5 feat: add vlan50-parent role for nik-debian's tagged VLAN 50 interface
Some checks failed
validate / lint (push) Failing after 1s
Creates enp1s0.50 on nik-debian (no L3 address, VLAN 40/node IP/default
route untouched) for Multus to later attach macvlan/ipvlan workloads to.

Verified live: enp1s0.50 is up with no IPv4 address (only the automatic
IPv6 link-local, which is expected and harmless), default route
unchanged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-24 17:56:59 +09:00
325d3bc5c7 feat: add pia-gateway role for minisforum PIA WireGuard egress
Registers minisforum as a PIA WireGuard peer for VPN VLAN 50, with a
boot-ordered kill switch (dedicated PIA-VLAN50 iptables chain + a
terminal unreachable route in a dedicated routing table), multi-region
addKey fallback (Hong Kong -> Taiwan -> JP Tokyo, each region's full
server list, in order), and an observability-only health check.

Verified live against minisforum: registration succeeds, wg-quick@pia-wg
is up.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-24 17:54:16 +09:00
939a7c6ed1 feat: add routed UFW rules for pod traffic and open NFS/SMB on nik-debian
Some checks failed
validate / lint (push) Failing after 1s
Adds explicit UFW routed-allow rules (10.42.0.0/16 pod-to-pod, pod-to-Technitium
DNS) since these nodes default their routed/FORWARD policy to DROP. Also brings
nik-debian's NFS (2049) and SMB (445) ports under Ansible-managed UFW rules via
the existing nfs-server role, scoped to the Lab/Trusted networks that need them.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-23 15:00:45 +09:00
53d02b7522 feat: configure K3s DNS resolver and update firewall rules for new subnet
Some checks failed
validate / lint (push) Failing after 7s
2026-08-23 14:07:49 +09:00
0824404f96 fix: update host IP addresses to new subnet 10.10.40.0/24 in documentation
Some checks failed
validate / lint (push) Has been cancelled
2026-08-23 11:42:47 +09:00
7a8c73d78a fix: update fleet IPs after subnet migration to 10.10.40.0/24
Some checks failed
validate / lint (push) Has been cancelled
minisforum, debian, mac-mini, and gpu-node all moved from
192.168.7.0/24 to 10.10.40.0/24. Updates K3s server/agent config and
node IPs (including gpu-node's host_vars override), NFS export
allow-list and exports template, Pi-hole DNS records and
kube-vip/loadBalancerIP pins, WireGuard's pushed DNS/AllowedIPs, and
the NFS server IP baked into Jellyfin/Kavita/gitea-backup PVs and the
Ollama URL used by ai-gateway/Dashy.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-23 05:52:41 +09:00
e15911305f Revert "feat: move Gitea traffic from gitea.nik4nao.com to gitea.home.arpa"
This reverts commit fd13b8a. The registry-host migration broke image
pulls for home-services/portfolio (internal CA not trusted by
containerd) and has left them Degraded for 11 days; reverting back to
gitea.nik4nao.com pending a decision on how to proceed. Not pushed yet.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-19 00:22:27 +09:00
fd13b8aedb feat: move Gitea traffic from gitea.nik4nao.com to gitea.home.arpa
Some checks failed
validate / lint (push) Has been cancelled
The public domain is unreachable while moving, and the cluster had no
Traefik route to Gitea at all (public or internal), leaving every Argo
CD Application stuck in Unknown sync. Add a gitea.home.arpa
Certificate/IngressRoute, repoint Argo CD's repoURL, Gitea's own
DOMAIN/ROOT_URL/SSH_DOMAIN, the container registry references, the
Gitea Actions runner, and the watch-party clone URL at the internal
hostname.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-07 23:06:02 +09:00
12718203ac fix: k3s-agent label/taint tasks use undefined ansible_hostname
Some checks failed
validate / lint (push) Failing after 1s
ansible.cfg sets inject_facts_as_vars: False repo-wide (the ollama role
already follows this via ansible_facts['system']), but these two tasks
referenced the bare ansible_hostname magic var, which doesn't exist under
that setting. Surfaced today as a hard failure re-running setup-gpu-node -
previously latent since these nodes' labels/taints were already correct
from whenever they last applied cleanly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-25 00:13:22 +09:00
69d880186a fix: open Flannel/K3s ports in gpu-node's UFW rules
Some checks failed
validate / lint (push) Failing after 1s
host_vars/gpu-node.yaml's ufw_allowed_ports has overridden (not extended)
the common role's default list since the node was added, silently dropping
the Flannel VXLAN (8472/udp), K3s API (6443/tcp), and Kubelet (10250/tcp)
rules every other node gets. Went unnoticed because kubectl logs/exec/stats
tunnel through the agent's outbound connection to the k3s server rather
than needing a direct inbound path - but real pod dataplane traffic (e.g.
tts-gateway on nik-gpu resolving DNS against CoreDNS on nik-debian) needs
actual VXLAN connectivity and was blackholing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-25 00:05:54 +09:00
f2261a2676 feat: add tts-gateway service, pinned to nik-gpu for GPU inference
Some checks failed
validate / lint (push) Failing after 1s
Deploys the Go gRPC gateway and Python/libtorch inference sidecar as one
pod on nik-gpu (nodeSelector/toleration/runtimeClassName: nvidia, sidecar
requesting nvidia.com/gpu: 1), matching the existing GPU device plugin.
Model checkpoint/config are bind-mounted from /data/tts-gateway, added to
gpu-node's Ansible data_dirs for consistency with the other hosts.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-24 23:50:40 +09:00
5a00f5767b fix: Ansible cleanup -- merge dual config, pin collections, align K3s version, fix bugs
Stage 4 of REFACTOR_PLAN.md.

- Merge ansible/ansible.cfg into root ansible.cfg (single source of truth);
  the dual-config setup silently broke documented commands and lost
  host_key_checking=False when run from the ansible/ directory.
- Add ansible/requirements.yml pinning community.general, ansible.posix,
  community.docker -- previously undocumented deps of the glances/watch-party
  roles that would fail a fresh bootstrap.
- Align K3s version to v1.32.4+k3s1 across roles/k3s-server, roles/k3s-agent,
  and host_vars/gpu-node.yaml defaults (was skewed: .2 vs .4). This only
  changes what a *future* provision installs -- minisforum/debian are still
  live on v1.32.2+k3s1 until separately upgraded.
- Fix kubeconfig fetch/replace path mismatch in k3s-server role: the `fetch`
  task (flat: true) writes to ~/.kube/config, but `replace` was targeting a
  /tmp/k3s-minisforum.yaml nothing creates -- would break a fresh rebuild.
- gitea-runner: only remove /run/docker.sock when it's actually a directory
  (task name implied a check that wasn't there); tighten registration-token
  systemd unit from 0644 to 0600.
- nvidia: stop unconditionally reporting `changed` (and restarting Docker)
  on every run for an idempotent runtime-configure command.
- Gate the K3s join-token debug print and WireGuard client-config/QR display
  behind opt-in vars (k3s_show_token, wireguard_show_client_configs), default
  off -- both were printing real secrets to console on every run.
- Parameterize the docker role for Debian and Ubuntu; homeassistant now
  depends on it (meta/main.yaml) instead of duplicating a Debian-only Docker
  install inline.
- FQCN cleanup across wireguard, homeassistant, and ollama roles/handlers
  (bare module names -> ansible.builtin.*/community.general.*/ansible.posix.*),
  plus a few ansible-lint name-casing/idiom fixes. Handler renames verified
  against their `notify:` call sites so notifications still fire.
- Update ansible/README.md and root README.md: add gpu-node/gpu_workstation
  (4th host, previously undocumented), docker/nvidia roles, setup-gpu-node.yaml,
  homeassistant.yaml, requirements.yml install step; correct the "Legacy"
  homeassistant label (it's the only thing serving ha.home.arpa); correct the
  Gitea ingress/backup-storage doc-drift; flag the Grafana/Loki static-PV
  binding drift discovered via live cluster check.

Verified: all playbooks pass `ansible-playbook --syntax-check`, yamllint clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 18:15:12 +09:00
e7718ce356 feat: add GPU node setup with Docker and NVIDIA support, including configuration files and playbook updates 2026-07-23 11:07:18 +09:00
0bf7f0f597 feat: add configuration for GPU node setup and update Ollama role for cross-platform support 2026-05-20 23:17:23 +09:00
d3069eb234 fix: update SSH configuration for security and add inotify limits 2026-05-16 18:50:01 +09:00
4cc3af39d6 feat: update README files to enhance documentation for Ansible, Argo CD, and Kubernetes manifests 2026-05-06 19:42:33 +09:00
f0bef8e4c1 feat: add model checking and conditional pulling for Ollama models 2026-04-21 22:53:25 +09:00
7d2fde96ae feat: add Ollama role with installation, configuration, and service exposure for K3s cluster 2026-04-21 16:43:44 +09:00
34e358ebcc Added mac to the Wireguard peer 2026-03-22 10:22:43 +09:00
863e66577c Added homeassistant to the homelab 2026-03-18 23:45:30 +09:00
7cefa0f387 Add Wireguard for split tunnel VPN via Ansible 2026-03-18 22:27:30 +09:00
26c4234dc6 Migrate gitea to gitea.nik4nao.com 2026-03-17 22:45:03 +09:00
83f46c9748 feat(gitea): add backup CronJob with RBAC and NFS support
feat(gitea): create PersistentVolume and PersistentVolumeClaim for Gitea

feat(gitea): add script to create Gitea runner registration token secret

feat(gitea): deploy Gitea Actions runner with Docker socket access

feat(media): deploy JDownloader with Ingress configuration

feat(media): set up Jellyfin media server with NFS and Ingress

feat(media): configure qBittorrent deployment with Ingress

feat(monitoring): add Grafana Loki datasource ConfigMap

feat(monitoring): create Grafana admin credentials secret

feat(monitoring): define PersistentVolumes for monitoring stack

feat(network): implement DDNS CronJob for Porkbun DNS updates

feat(network): create secret for Porkbun DDNS API credentials

feat(network): set up Glances service and Ingress for Debian node

fix(network): patch Pi-hole DNS services with external IPs

feat(network): configure Traefik dashboard Ingress with Authentik auth

feat(network): set up Watch Party service and Ingress for Mac Mini

refactor(values): update Helm values files for various services
2026-03-12 21:56:32 +09:00
fb09e4198d Add monitoring setup with Grafana, Loki, and Prometheus configurations 2026-03-10 19:40:54 +09:00
9bf86e2355 Add Gitea Actions Runner setup with configuration and deployment manifests 2026-03-10 01:27:48 +09:00
8a90227173 Add NFS server backup directory configuration and Gitea backup CronJob 2026-03-09 22:30:50 +09:00
03cdf9f355 Add Glances deployment and configuration for Debian and Kubernetes 2026-03-09 21:48:07 +09:00
f68bfde849 Add Watch Party deployment playbook and related configurations for Mac Mini 2026-03-06 21:37:11 +09:00
4129089d09 Add K3s agent playbook and related configurations; include JDownloader and qBittorrent manifests 2026-03-06 19:49:18 +09:00
fe4b38e4b1 Add NFS server role with installation, configuration, and restart tasks 2026-03-06 19:12:59 +09:00
5237c03d4b Update kubeconfig paths in setup-k3s.yaml and main.yaml for consistency 2026-03-06 10:48:45 +09:00
208c768b2e Rename YAML file extensions from .yml to .yaml for consistency 2026-03-06 10:37:42 +09:00
feb31a52fb Renamed file extension from yml to yaml 2026-03-05 23:45:59 +09:00
f33fdc4044 Add initial Ansible configuration and playbooks for homelab setup
- Created ansible.cfg for configuration settings
- Added inventory.yml for host definitions
- Implemented bootstrap playbook for Minisforum setup
- Developed setup playbook for K3s installation
- Defined common role with user and package management tasks
- Established K3s server role with configuration and installation tasks
- Included Traefik Helm values for ingress management
2026-03-05 18:55:41 +09:00