- Serialize every mutating command (recover, renew, register, rewrite-config, pause, resume, state-reset) on the shared lock, acquired BEFORE state is read rather than after: reading first and locking only around the write let a slower command silently clobber whatever a faster one wrote in between — in particular, an operator pause could be lost to an in-flight recovery pass that read state before the pause ran and only saved its own copy back afterwards. - Fix emergency_close() to reposition its VLAN 50 DROP to the first effective FORWARD position when it exists but has been pushed down by something else (e.g. an out-of-band ufw reload) — a bare `-C` existence check said nothing about position. - Tighten wait_for_verified()'s handshake freshness check from >= to > since_epoch: wg show and since_epoch are both whole seconds, so a handshake at exactly the same second as since_epoch is indistinguishable from a stale, pre-activation one and must not count as proof of a new one. - Fix the Ansible config-validation task to fail on any exit code outside status's three documented health exits (0/1/2), not just on 5 — verified live against the real program (no wg/ip/iptables on this dev machine reproduces the "nothing configured yet" first-run case; status correctly returns 1, and broken/missing config.json correctly returns 5 and is now caught). - Correct the live fault-injection drill in the role README: port 1337 is only the addKey registration API port, not the tunnel's own WireGuard UDP endpoint (derive that from `wg show pia-wg endpoints` instead), and make the packet-capture check direction-aware (`tcpdump -Q out`) so it can't conflate a real egress leak with VLAN 50's normal inbound arrival on enp1s0. Added 6 tests (86 total, all passing): emergency-close repositioning past a preceding ACCEPT, pause/resume blocking on an in-flight lock holder and surviving into a subsequent recovery pass, and the handshake boundary at exactly/one-second-either-side of since_epoch. yamllint, ansible-playbook --syntax-check, and ansible-lint (diffed against main — no new finding categories) all pass. No live changes; nothing pushed to main. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
22 KiB
homelab
Infrastructure-as-code for a small K3s homelab. Host setup is handled with Ansible, cluster state is reconciled by Argo CD, and service configuration lives in Kubernetes manifests plus Helm values.
Current Architecture
| Host | IP | Role |
|---|---|---|
minisforum |
10.10.40.53 |
K3s server, Traefik entrypoint, primary app node |
debian / nik-debian |
10.10.40.20 |
K3s agent, NFS storage, secondary Pi-hole |
mac-mini |
10.10.40.30 |
Standalone services such as Watch Party and Ollama |
gpu-node / nik-gpu |
10.10.40.12 |
K3s agent with NVIDIA GPU passthrough, spot-tainted; runs Ollama directly on the host |
The cluster uses Traefik instead of the bundled K3s ingress controller. Internal
services are published under home.arpa with certificates from an internal CA.
Public services under nik4nao.com use Let's Encrypt.
Repository Layout
| Path | Purpose |
|---|---|
ansible/ |
Host bootstrap and non-Kubernetes services |
argocd/apps/ |
Argo CD Application objects for Helm charts and manifest directories |
argocd/values/ |
Helm values used to install or reconcile Argo CD itself |
config/ |
App configuration that is injected into manifests, currently Dashy |
manifests/ |
Raw Kubernetes resources grouped by service area |
values/ |
Helm values consumed by Argo CD Applications |
Managed Services
| Area | Services |
|---|---|
| GitOps | Argo CD and app-of-apps |
| Networking | Traefik, Pi-hole primary and secondary, DDNS, CoreDNS custom forwarding |
| TLS | cert-manager, internal CA issuer, Let's Encrypt issuers, CA installer page |
| Identity | Authentik, Traefik forward-auth middleware, OIDC integrations |
| Observability | kube-prometheus-stack, Grafana, Loki, Tempo, OpenTelemetry Collector, Glances |
| Git and CI | Gitea, Gitea Actions runner, registry pull secrets, Gitea backup CronJob |
| Media | Jellyfin, qBittorrent, JDownloader |
| Home services | Home Assistant ingress, HA gateway, AI gateway, Discord bot, Alexa bridge, Alert bridge |
| Public apps | Portfolio, me.nik4nao.com, Watch Party ingress to the Mac Mini |
| Dashboard | Dashy |
Important URLs
| URL | Service | Certificate |
|---|---|---|
https://argocd.home.arpa |
Argo CD | Internal CA |
https://auth.home.arpa |
Authentik internal | Internal CA |
https://auth.nik4nao.com |
Authentik public | Let's Encrypt |
https://traefik.home.arpa |
Traefik dashboard | Internal CA |
https://grafana.nik4nao.com |
Grafana | Let's Encrypt |
https://gitea.nik4nao.com |
Gitea | Let's Encrypt |
https://pihole.home.arpa |
Pi-hole | Internal CA |
https://dashy.home.arpa |
Dashy | Internal CA |
https://jellyfin.home.arpa |
Jellyfin | Internal CA |
https://qbittorrent.home.arpa |
qBittorrent | Internal CA |
https://jdownloader.home.arpa |
JDownloader | Internal CA |
https://ha.home.arpa |
Home Assistant | Internal CA |
https://glances.home.arpa |
Glances on K3s | Internal CA |
https://glances-debian.home.arpa |
Glances on Debian | Internal CA |
http://alert-bridge.home.arpa |
Alert bridge (Discord webhook relay) | Plain HTTP |
https://watch-party.nik4nao.com |
Watch Party on Mac Mini | Let's Encrypt |
https://nik4nao.com |
Portfolio | Let's Encrypt |
https://me.nik4nao.com |
me.nik4nao.com (terminal-style personal site) | Let's Encrypt |
http://ca.home.arpa |
Internal CA installer | Plain HTTP |
home.arpa names are defined explicitly in values/pihole.yaml and
values/pihole-debian.yaml; Pi-hole is not configured as a wildcard DNS server.
Bootstrap
Install workstation tools:
pip install ansible
ansible-galaxy collection install -r ansible/requirements.yml
Also install kubectl, helm, and kubeseal. The inventory expects SSH
access with ~/.ssh/id_ed25519-nik-macbookair.
Bring up hosts and base services:
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K
Install Argo CD once, then hand control to the app-of-apps:
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm upgrade --install argocd argo/argo-cd \
--namespace argocd --create-namespace \
--version 9.4.15 \
--values argocd/values/argocd.yaml
kubectl apply -f manifests/argocd/app-of-apps.yaml
The app-of-apps now brings up cert-manager's ClusterIssuers (via
cert-manager-config) and Authentik's ingress/proxy-outpost/middleware (via
authentik-config) automatically — these used to require untracked manual
kubectl apply steps that weren't documented anywhere. Authentik's Helm
chart itself is still a one-time manual install, since its Argo CD
Application (argocd/apps/authentik.yaml) is deliberately left on manual sync
with a targetRevision placeholder pending a chart-version decision:
helm repo add authentik https://charts.goauthentik.io
helm repo update
helm upgrade --install authentik authentik/authentik \
-f values/authentik.yaml -n authentik --create-namespace
Populate secrets (see "Secrets" below) before Authentik, Gitea, Grafana, or the other services that depend on them will come up healthy. After that, normal changes should flow through Git and Argo CD.
Cold-rebuild order
Rebuilding from nothing, the dependency order that actually matters is:
- Ansible playbooks above (hosts, K3s, NFS).
helm install argocd+kubectl apply -f manifests/argocd/app-of-apps.yaml— this alone now brings up cert-manager, sealed-secrets, and theirClusterIssuers/CA cert via sync-wave ordering.- Runtime secret scripts and Sealed Secret regeneration (see "Secrets") — several Applications (Gitea, Grafana, home-services, Authentik) will sit degraded/crash-looping until their secrets exist.
- Manual Authentik Helm install (above) — every OAuth-gated service (Grafana, Gitea, Argo CD SSO, Traefik dashboard) depends on it.
- Everything else reconciles from Git on its own from here.
Daily Operations
Check cluster and sync state:
kubectl get nodes
kubectl get pods -A
kubectl get applications -n argocd
Apply Dashy config after editing config/dashy/conf.yaml:
bash manifests/core/apply-dashy-config.sh
Patch secondary Pi-hole DNS services after Helm upgrades if needed:
bash manifests/network/pihole-debian-patch.sh
Deploy host-level services:
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
Secrets
Do not commit plaintext secrets. Runtime-only scripts read .env and create
Kubernetes Secrets directly. Sealed Secret scripts regenerate committed
*-sealed.yaml resources.
Start from .env.example and keep the filled .env local.
Runtime secret scripts:
bash manifests/cert-manager/porkbun-secret.sh
bash manifests/authentik/authentik-secret.sh
bash manifests/authentik/authentik-proxy-secret.sh
bash manifests/authentik/authentik-gitea-secret.sh
bash manifests/authentik/authentik-grafana-secret.sh
bash manifests/home-services/registry-secret.sh
bash manifests/monitoring/grafana-secret.sh
bash manifests/network/ddns-secret.sh
bash manifests/portfolio/registry-secret.sh
bash manifests/me-nik4nao-com/registry-secret.sh
Sealed Secret regeneration:
bash manifests/home-services/discord-bot-secret.sh
bash manifests/home-services/ha-gateway-secret.sh
Some sealed secrets are maintained directly in the repo, including Argo CD OIDC,
Gitea admin/OIDC, Grafana admin/OIDC, Pi-hole admin, and home-service secrets.
The host-level Gitea runner reads GITEA_RUNNER_TOKEN from the environment when
running ansible/playbooks/setup-gitea-runner.yaml.
Storage
K3s local-path is used for many app PVCs. Static hostPath PVs are used for
state that must live on known disks:
| Location | Use |
|---|---|
/data/gitea on minisforum |
Gitea shared storage |
/data/prometheus on minisforum |
Prometheus (bound correctly) |
/data/grafana on minisforum |
Static PV defined for Grafana, but not currently bound — see gotcha below |
/data/loki on minisforum |
Static PV defined for Loki, but not currently bound — see gotcha below |
/mnt/storage on debian |
NFS media library |
/home/nik/backups on debian |
NFS export for Gitea's backup CronJob (separate from /mnt/storage) |
The Debian NFS server exports both /mnt/storage and /home/nik/backups to
10.10.40.53.
TLS and Trust
*.home.arpa certificates use internal-ca-issuer from manifests/cert-manager,
which signs off the self-signed root internal-ca-cert (cert-manager namespace).
The CA installer at http://ca.home.arpa serves ca.crt and an iOS/macOS
mobileconfig profile. The ca-sync CronJob updates those files from the
cert-manager/internal-ca-cert secret when the CA changes.
*.nik4nao.com certificates use the Let's Encrypt issuers in
manifests/cert-manager/cluster-issuer-letsencrypt.yaml.
Renewing the internal CA
internal-ca-cert's default privateKey.rotationPolicy is Never, so a
normal renewal reuses the existing private key — every already-issued leaf
cert and every device that already trusts the CA keep working unchanged.
Deleting the internal-ca-cert secret instead of renewing it in place
generates a brand-new key pair, which invalidates every leaf cert and every
device's trust simultaneously (this happened 2026-09-02: leaf certs issued
before the deletion failed openssl verify against the recreated CA
entirely, and Jellyfin/Kavita/etc. all showed as insecure on client devices).
To renew correctly:
cmctl renew certificate internal-ca-cert -n cert-manager
This reissues the CA certificate (new serial/validity) while keeping the same key, so it should not require any downstream action. If the key ever does change (intentional rotation, or the secret was lost and had to be recreated), do both of the following — the CA rotating does not automatically cascade to either:
- Force-reissue every leaf cert, since each
Certificaterenews on its own independent schedule and won't notice the issuing CA changed:
Verify afterward that every leaf chains to the current CA:for c in $(kubectl get certificate -A -o json | \ jq -r '.items[] | select(.spec.issuerRef.name=="internal-ca-issuer") | "\(.metadata.namespace)/\(.metadata.name)"'); do ns="${c%%/*}"; name="${c##*/}" secret=$(kubectl get certificate "$name" -n "$ns" -o jsonpath='{.spec.secretName}') kubectl delete secret "$secret" -n "$ns" donekubectl get secret internal-ca-cert -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/current-ca.crt openssl verify -CAfile /tmp/current-ca.crt <(kubectl get secret <leaf-secret> -n <ns> -o jsonpath='{.data.tls\.crt}' | base64 -d) - Re-trust the CA on every client device — visit
http://ca.home.arpa, reinstall the mobileconfig, and re-enable full trust under Settings → General → About → Certificate Trust Settings (iOS/macOS). The old trust profile does not update itself; nothing pushes the new CA to devices.
VPN VLAN 50 (PIA gateway) — status and runbook
minisforum is a PIA WireGuard egress gateway for a dedicated VPN VLAN
(10.10.50.0/24), giving nik-debian's selected media workloads a second,
PIA-routed network path (net1) alongside their normal cluster networking
(eth0) — full design in plan.md (repo root). Live as of 2026-08-24:
every layer below is applied and independently verified, including the
Phase 5 canary's full validation/failure-test matrix and both media
workloads' own fail-closed kill-switch test (stopping wg-quick@pia-wg
on minisforum visibly stops their egress, no fallback to eth0). Current
PIA endpoint is Hong Kong (pia_region_candidates' first entry —
whichever region actually answers is recorded per-host, see
ansible/roles/pia-gateway/README.md's "Region fallback"). See each
piece's own README for exact commands and rollback:
| Layer | Where | Rollback |
|---|---|---|
| Minisforum PIA gateway (WireGuard, policy routing, kill switch, health check) | ansible/roles/pia-gateway/ |
that role's README |
nik-debian VLAN 50 parent (enp1s0.50) |
ansible/roles/vlan50-parent/ |
that role's README |
| Multus, VLAN 50 NAD, canary | manifests/multus/ (manual-sync Argo CD Application) |
that directory's README |
| qBittorrent VLAN 50 attachment | manifests/media/qbittorrent.yaml |
no file-based copy kept — git show 28d0628:manifests/media/qbittorrent.yaml | kubectl apply -f - restores the last pre-migration (Gluetun-based) version |
| JDownloader VLAN 50 attachment | manifests/media/jdownloader.yaml |
same approach — git show 2f7ec5f:manifests/media/jdownloader.yaml | kubectl apply -f - |
| browser-vpn-proxy (SOCKS5, no auth — see its own header for why) | manifests/multus/30-browser-vpn-proxy.yaml (manual-sync Argo CD Application) |
that directory's README — replaces the standalone manifests/media/gluetun-proxy.yaml (removed 2026-08-24; git show 462a9d9:manifests/media/gluetun-proxy.yaml | kubectl apply -f - restores it, NodePort 30888, if ever needed) |
Boot ordering (minisforum): pia-killswitch.service seeds the closed
routing/firewall state and is ordered Before= wg-quick@pia-wg.service;
both are WantedBy=multi-user.target so a normal boot orders them
correctly with no manual step. pia-gateway-healthcheck.timer runs
independently and only observes (pia-gateway check).
Automatic recovery (ansible/roles/pia-gateway/README.md →
"Automatic recovery" — read that for the state machine, alerts,
residual risk and the live-drill rollout): pia-gateway-recover.timer
on minisforum runs a bounded pass every 60 s — after 3 stale checks it
restarts pia-wg, escalates to a same-key PIA re-registration, then to
one staged key rotation on evidence, verifies a fresh handshake before
calling anything fixed, backs off 5→15→30→60 min on failure, and keeps
VLAN 50 fail-closed throughout. Operator commands on minisforum:
sudo pia-gateway status # class, handshake age, backoff, last verified registration, pause
sudo pia-gateway pause --reason "..." # stop recovery+renewal (persists across reboot); kill switch untouched
sudo pia-gateway resume
sudo pia-gateway retry-now # one pass ignoring threshold/backoff
sudo journalctl -u pia-gateway-recover -n 50
Status as of 2026-09-22: implemented and tested offline (78 unit tests),
not yet deployed or drilled live — see the role README "Rollout".
Renewal (pia_renewal_enabled) stays off until the drills pass.
PIA re-registration / key rotation — from the repo root (the paths
are repo-root-relative — running from ansible/ fails with "playbook
could not be found"):
# re-register with the EXISTING key (what fixed both incidents):
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml \
-e pia_force_reregister=true -K -J
# stage a NEW keypair, register it, promote it only after a verified handshake:
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/pia-gateway.yaml \
-e pia_force_key_rotation=true -K -J
pia_force_reregister reuses the on-disk key (an earlier version of
this paragraph said it regenerated the keypair — it never did; that is
pia_force_key_rotation). Both run the shared on-host transaction
(pia-gateway register), which restarts and verifies a handshake
even when the rendered config is byte-identical. -J is required
because ansible/group_vars/all/vault.yaml is vault-encrypted and
loaded for every host; the PIA credentials themselves are not in
the vault — ansible/host_vars/minisforum.yaml reads PIA_USER/
PIA_PASSWORD from the repo-root .env, and the role now also
provisions those two values to minisforum's root-only
/etc/pia-gateway/credentials.env so recovery works unattended.
Rotating the PIA password: edit .env, re-run the playbook, re-register.
Tunnel silently dead — recurring, known, now auto-recovered: the
symptom is "the proxy doesn't work" (browser through 10.10.40.20:1080
fails, qBittorrent/JDownloader have no egress) while everything in
Kubernetes looks healthy and pia-wg on minisforum also looks up
(interface present, wg-quick@pia-wg active (exited), routes in
place). The kill switch is doing its job — VLAN 50 fails closed rather
than leaking. With the controller deployed, expect an alert and a
recovery within ~10 minutes; check sudo pia-gateway status first. If
it is paused, in backoff, or the controller is not deployed, diagnose
before fixing so the age gets recorded:
ssh minisforum "sudo wg show pia-wg latest-handshakes" # epoch seconds; >180s old = dead
ssh minisforum "sudo journalctl -u pia-gateway-healthcheck -n 5"
then sudo pia-gateway retry-now on minisforum, or the re-registration
command above. A bare systemctl restart wg-quick@pia-wg was not
enough the first time (PIA had stopped recognising the registered key).
Occurrences so far — 2026-08-25 (full write-up in
INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md, Incident 1) and
2026-09-22 (~28 days later; pre-fix handshake age not captured, fix
confirmed by a fresh handshake immediately after). The ~28-day cadence
is the hypothesis behind the 21-day proactive renewal timer.
Maintenance / taking the gateway down deliberately: sudo systemctl stop wg-quick@pia-wg on minisforum closes VLAN 50 Internet egress
immediately (kill switch stays engaged) without touching minisforum's own
networking, k3s, or any other workload — this is the same fail-closed path
exercised by the Phase 5 failure tests in plan.md.
Multus troubleshooting: check kubectl -n kube-system get pods -l app=multus -o wide (should be Running, only on nik-debian), then
/var/lib/rancher/k3s/agent/etc/cni/net.d/ on nik-debian for a generated
00-multus.conf — if it's missing, Multus isn't actually in the CNI
chain regardless of pod status. See manifests/multus/README.md for the
k3s-specific path gotchas that make this different from a stock Multus
install.
Monitoring: currently journald only
(journalctl -u pia-gateway-healthcheck on minisforum) — nothing is wired
into the existing kube-prometheus-stack (values/kube-prometheus-stack .yaml has no custom node-exporter textfile-collector or
additionalPrometheusRulesMap configured, and none was added by this
work). Wiring real alerting would mean enabling node-exporter's textfile
collector for that host and adding a PrometheusRule — not done, flagged
here as the natural next step rather than guessed at.
Rollback (full): work through the table above bottom-to-top — restore
browser-vpn-proxy/qBittorrent/JDownloader from git history first (fastest
to undo, keeps service running — see the table's Rollback column), then
remove the Multus/NAD resources, then ansible/roles/vlan50-parent's
rollback, then ansible/roles/pia-gateway's rollback. None of these
steps touch pia-credentials, PVCs, or the normal k3s/Argo CD control
plane.
Gotchas
manifests/media/qbittorrent.yamlandjdownloader.yamldeploy the moment they're pushed tomain— themediaArgo CD Application already hasselfHeal/automatedsync, unlike the newmultusApplication (deliberately manual-sync). Double-check which gate applies before pushing changes to either file.- Argo CD Applications mostly set
prune: false; removing resources from Git may require manual cleanup. - Gitea uses a manual public
IngressRoute;values/gitea.yamlhas noingress:key at all, so the chart's own ingress is off by chart default, not an explicit setting. - Grafana and Loki's static hostPath PVs (
grafana-pv,loki-pvinmanifests/monitoring/monitoring-pvs.yaml) are currently unbound — their Helm-managed PVCs got dynamically provisioned via thelocal-pathStorageClass instead (confirmed live viakubectl get pv/pvc -n monitoring), unlike Prometheus which bindsprometheus-pvcorrectly. Data is not lost, just not on the disk the docs/manifest imply — needs a deliberate decision (bind properly with a data migration, or drop the orphaned static PVs and document reality) before relying on/data/grafanaor/data/lokifor backups/DR. - Gitea
ROOT_URLchanges can require deleting the generated inline config secret before reconciling. - Pi-hole does not provide wildcard DNS here; add each new internal hostname to both Pi-hole values files.
- Secondary Pi-hole external IPs can be lost during chart upgrades; rerun
manifests/network/pihole-debian-patch.sh. - Authentik forward-auth depends on the
Cookieheader inauthRequestHeaders; removing it causes redirect loops. - CoreDNS custom zone snippets must use the
.serverkey suffix. - Renewing
internal-ca-certby deleting its secret (instead ofcmctl renew) generates a new key pair, silently breaking every leaf cert and every device's trust at once — see "Renewing the internal CA" under TLS and Trust for the recovery/prevention procedure. - The Dashy manifest contains an empty ConfigMap shell; use
manifests/core/apply-dashy-config.shto load the real config. - A newly pushed Argo CD
Application(a new file underargocd/apps/) doesn't create its namespace/resources the instant you push —app-of-appsonly picks it up on its next Git poll. Running a namespace-scopedregistry-secret.sh(or anything else assuming the namespace already exists) right after pushing can fail withnamespaces "<ns>" not found; either wait for the next poll or force one:kubectl patch application app-of-apps -n argocd --type merge -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'.
See the scoped READMEs in ansible/, argocd/, and manifests/ for workflows
specific to those directories.