homelab/README.md

homelab

Infrastructure-as-code for a small K3s homelab. Host setup is handled with Ansible, cluster state is reconciled by Argo CD, and service configuration lives in Kubernetes manifests plus Helm values.

Current Architecture

Host	IP	Role
minisforum	10.10.40.53	K3s server, Traefik entrypoint, primary app node
debian / nik-debian	10.10.40.20	K3s agent, NFS storage, secondary Pi-hole
mac-mini	10.10.40.30	Standalone services such as Watch Party and Ollama
gpu-node / nik-gpu	10.10.40.12	K3s agent with NVIDIA GPU passthrough, spot-tainted; runs Ollama directly on the host

The cluster uses Traefik instead of the bundled K3s ingress controller. Internal services are published under home.arpa with certificates from an internal CA. Public services under nik4nao.com use Let's Encrypt.

Repository Layout

Path	Purpose
ansible/	Host bootstrap and non-Kubernetes services
argocd/apps/	Argo CD Application objects for Helm charts and manifest directories
argocd/values/	Helm values used to install or reconcile Argo CD itself
config/	App configuration that is injected into manifests, currently Dashy
manifests/	Raw Kubernetes resources grouped by service area
values/	Helm values consumed by Argo CD Applications

Managed Services

Area	Services
GitOps	Argo CD and app-of-apps
Networking	Traefik, Pi-hole primary and secondary, DDNS, CoreDNS custom forwarding
TLS	cert-manager, internal CA issuer, Let's Encrypt issuers, CA installer page
Identity	Authentik, Traefik forward-auth middleware, OIDC integrations
Observability	kube-prometheus-stack, Grafana, Loki, Tempo, OpenTelemetry Collector, Glances
Git and CI	Gitea, Gitea Actions runner, registry pull secrets, Gitea backup CronJob
Media	Jellyfin, qBittorrent, JDownloader
Home services	Home Assistant ingress, HA gateway, AI gateway, Discord bot, Alexa bridge, Alert bridge
Public apps	Portfolio, me.nik4nao.com, Watch Party ingress to the Mac Mini
Dashboard	Dashy

Important URLs

URL	Service	Certificate
https://argocd.home.arpa	Argo CD	Internal CA
https://auth.home.arpa	Authentik internal	Internal CA
https://auth.nik4nao.com	Authentik public	Let's Encrypt
https://traefik.home.arpa	Traefik dashboard	Internal CA
https://grafana.nik4nao.com	Grafana	Let's Encrypt
https://gitea.nik4nao.com	Gitea	Let's Encrypt
https://pihole.home.arpa	Pi-hole	Internal CA
https://dashy.home.arpa	Dashy	Internal CA
https://jellyfin.home.arpa	Jellyfin	Internal CA
https://qbittorrent.home.arpa	qBittorrent	Internal CA
https://jdownloader.home.arpa	JDownloader	Internal CA
https://ha.home.arpa	Home Assistant	Internal CA
https://glances.home.arpa	Glances on K3s	Internal CA
https://glances-debian.home.arpa	Glances on Debian	Internal CA
http://alert-bridge.home.arpa	Alert bridge (Discord webhook relay)	Plain HTTP
https://watch-party.nik4nao.com	Watch Party on Mac Mini	Let's Encrypt
https://nik4nao.com	Portfolio	Let's Encrypt
https://me.nik4nao.com	me.nik4nao.com (terminal-style personal site)	Let's Encrypt
http://ca.home.arpa	Internal CA installer	Plain HTTP

home.arpa names are defined explicitly in values/pihole.yaml and values/pihole-debian.yaml; Pi-hole is not configured as a wildcard DNS server.

Bootstrap

Install workstation tools:

pip install ansible
ansible-galaxy collection install -r ansible/requirements.yml

Also install kubectl, helm, and kubeseal. The inventory expects SSH access with ~/.ssh/id_ed25519-nik-macbookair.

Bring up hosts and base services:

ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K

Install Argo CD once, then hand control to the app-of-apps:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm upgrade --install argocd argo/argo-cd \
  --namespace argocd --create-namespace \
  --version 9.4.15 \
  --values argocd/values/argocd.yaml

kubectl apply -f manifests/argocd/app-of-apps.yaml

The app-of-apps now brings up cert-manager's ClusterIssuers (via cert-manager-config) and Authentik's ingress/proxy-outpost/middleware (via authentik-config) automatically — these used to require untracked manual kubectl apply steps that weren't documented anywhere. Authentik's Helm chart itself is still a one-time manual install, since its Argo CD Application (argocd/apps/authentik.yaml) is deliberately left on manual sync with a targetRevision placeholder pending a chart-version decision:

helm repo add authentik https://charts.goauthentik.io
helm repo update
helm upgrade --install authentik authentik/authentik \
  -f values/authentik.yaml -n authentik --create-namespace

Populate secrets (see "Secrets" below) before Authentik, Gitea, Grafana, or the other services that depend on them will come up healthy. After that, normal changes should flow through Git and Argo CD.

Cold-rebuild order

Rebuilding from nothing, the dependency order that actually matters is:

1. Ansible playbooks above (hosts, K3s, NFS).
2. helm install argocd + kubectl apply -f manifests/argocd/app-of-apps.yaml — this alone now brings up cert-manager, sealed-secrets, and their ClusterIssuers/CA cert via sync-wave ordering.
3. Runtime secret scripts and Sealed Secret regeneration (see "Secrets") — several Applications (Gitea, Grafana, home-services, Authentik) will sit degraded/crash-looping until their secrets exist.
4. Manual Authentik Helm install (above) — every OAuth-gated service (Grafana, Gitea, Argo CD SSO, Traefik dashboard) depends on it.
5. Everything else reconciles from Git on its own from here.

Daily Operations

Check cluster and sync state:

kubectl get nodes
kubectl get pods -A
kubectl get applications -n argocd

Apply Dashy config after editing config/dashy/conf.yaml:

bash manifests/core/apply-dashy-config.sh

Patch secondary Pi-hole DNS services after Helm upgrades if needed:

bash manifests/network/pihole-debian-patch.sh

Deploy host-level services:

ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K

Secrets

Do not commit plaintext secrets. Runtime-only scripts read .env and create Kubernetes Secrets directly. Sealed Secret scripts regenerate committed *-sealed.yaml resources.

Start from .env.example and keep the filled .env local.

Runtime secret scripts:

bash manifests/cert-manager/porkbun-secret.sh
bash manifests/authentik/authentik-secret.sh
bash manifests/authentik/authentik-proxy-secret.sh
bash manifests/authentik/authentik-gitea-secret.sh
bash manifests/authentik/authentik-grafana-secret.sh
bash manifests/home-services/registry-secret.sh
bash manifests/monitoring/grafana-secret.sh
bash manifests/network/ddns-secret.sh
bash manifests/portfolio/registry-secret.sh
bash manifests/me-nik4nao-com/registry-secret.sh

Sealed Secret regeneration:

bash manifests/home-services/discord-bot-secret.sh
bash manifests/home-services/ha-gateway-secret.sh

Some sealed secrets are maintained directly in the repo, including Argo CD OIDC, Gitea admin/OIDC, Grafana admin/OIDC, Pi-hole admin, and home-service secrets. The host-level Gitea runner reads GITEA_RUNNER_TOKEN from the environment when running ansible/playbooks/setup-gitea-runner.yaml.

Storage

K3s local-path is used for many app PVCs. Static hostPath PVs are used for state that must live on known disks:

Location	Use
/data/gitea on minisforum	Gitea shared storage
/data/prometheus on minisforum	Prometheus (bound correctly)
/data/grafana on minisforum	Static PV defined for Grafana, but not currently bound — see gotcha below
/data/loki on minisforum	Static PV defined for Loki, but not currently bound — see gotcha below
/mnt/storage on debian	NFS media library
/home/nik/backups on debian	NFS export for Gitea's backup CronJob (separate from /mnt/storage)

The Debian NFS server exports both /mnt/storage and /home/nik/backups to 10.10.40.53.

TLS and Trust

*.home.arpa certificates use internal-ca-issuer from manifests/cert-manager, which signs off the self-signed root internal-ca-cert (cert-manager namespace). The CA installer at http://ca.home.arpa serves ca.crt and an iOS/macOS mobileconfig profile. The ca-sync CronJob updates those files from the cert-manager/internal-ca-cert secret when the CA changes.

*.nik4nao.com certificates use the Let's Encrypt issuers in manifests/cert-manager/cluster-issuer-letsencrypt.yaml.

Renewing the internal CA

internal-ca-cert's default privateKey.rotationPolicy is Never, so a normal renewal reuses the existing private key — every already-issued leaf cert and every device that already trusts the CA keep working unchanged. Deleting the internal-ca-cert secret instead of renewing it in place generates a brand-new key pair, which invalidates every leaf cert and every device's trust simultaneously (this happened 2026-09-02: leaf certs issued before the deletion failed openssl verify against the recreated CA entirely, and Jellyfin/Kavita/etc. all showed as insecure on client devices).

To renew correctly:

cmctl renew certificate internal-ca-cert -n cert-manager

This reissues the CA certificate (new serial/validity) while keeping the same key, so it should not require any downstream action. If the key ever does change (intentional rotation, or the secret was lost and had to be recreated), do both of the following — the CA rotating does not automatically cascade to either:

1. Force-reissue every leaf cert, since each Certificate renews on its own independent schedule and won't notice the issuing CA changed:

   for c in $(kubectl get certificate -A -o json | \
     jq -r '.items[] | select(.spec.issuerRef.name=="internal-ca-issuer") | "\(.metadata.namespace)/\(.metadata.name)"'); do
     ns="${c%%/*}"; name="${c##*/}"
     secret=$(kubectl get certificate "$name" -n "$ns" -o jsonpath='{.spec.secretName}')
     kubectl delete secret "$secret" -n "$ns"
   done
   

   Verify afterward that every leaf chains to the current CA:

   kubectl get secret internal-ca-cert -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/current-ca.crt
   openssl verify -CAfile /tmp/current-ca.crt <(kubectl get secret <leaf-secret> -n <ns> -o jsonpath='{.data.tls\.crt}' | base64 -d)
   

2. Re-trust the CA on every client device — visit http://ca.home.arpa, reinstall the mobileconfig, and re-enable full trust under Settings → General → About → Certificate Trust Settings (iOS/macOS). The old trust profile does not update itself; nothing pushes the new CA to devices.

VPN VLAN 50 (PIA gateway) — status and runbook

minisforum is a PIA WireGuard egress gateway for a dedicated VPN VLAN (10.10.50.0/24), giving nik-debian's selected media workloads a second, PIA-routed network path (net1) alongside their normal cluster networking (eth0) — full design in plan.md (repo root). Live as of 2026-08-24: every layer below is applied and independently verified, including the Phase 5 canary's full validation/failure-test matrix and both media workloads' own fail-closed kill-switch test (stopping wg-quick@pia-wg on minisforum visibly stops their egress, no fallback to eth0). Current PIA endpoint is Hong Kong (pia_region_candidates' first entry — whichever region actually answers is recorded per-host, see ansible/roles/pia-gateway/README.md's "Region fallback"). See each piece's own README for exact commands and rollback:

Layer	Where	Rollback
Minisforum PIA gateway (WireGuard, policy routing, kill switch, health check)	ansible/roles/pia-gateway/	that role's README
nik-debian VLAN 50 parent (enp1s0.50)	ansible/roles/vlan50-parent/	that role's README
Multus, VLAN 50 NAD, canary	manifests/multus/ (manual-sync Argo CD Application)	that directory's README
qBittorrent VLAN 50 attachment	manifests/media/qbittorrent.yaml	no file-based copy kept — git show 28d0628:manifests/media/qbittorrent.yaml | kubectl apply -f - restores the last pre-migration (Gluetun-based) version
JDownloader VLAN 50 attachment	manifests/media/jdownloader.yaml	same approach — git show 2f7ec5f:manifests/media/jdownloader.yaml | kubectl apply -f -
browser-vpn-proxy (SOCKS5, no auth — see its own header for why)	manifests/multus/30-browser-vpn-proxy.yaml (manual-sync Argo CD Application)	that directory's README — replaces the standalone manifests/media/gluetun-proxy.yaml (removed 2026-08-24; git show 462a9d9:manifests/media/gluetun-proxy.yaml | kubectl apply -f - restores it, NodePort 30888, if ever needed)

Boot ordering (minisforum): pia-killswitch.service seeds the closed routing/firewall state and is ordered Before= wg-quick@pia-wg.service; both are WantedBy=multi-user.target so a normal boot orders them correctly with no manual step. pia-gateway-healthcheck.timer runs independently and only observes — see ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2.

PIA endpoint/key rotation: re-run ansible/playbooks/pia-gateway.yaml -e pia_force_reregister=true -K — this regenerates the local keypair and re-registers with PIA, then rewrites pia-wg.conf (handler-triggered restart). Rotating the PIA account password itself just needs ansible-vault edit ansible/group_vars/all/vault.yaml first.

Maintenance / taking the gateway down deliberately: sudo systemctl stop wg-quick@pia-wg on minisforum closes VLAN 50 Internet egress immediately (kill switch stays engaged) without touching minisforum's own networking, k3s, or any other workload — this is the same fail-closed path exercised by the Phase 5 failure tests in plan.md.

Multus troubleshooting: check kubectl -n kube-system get pods -l app=multus -o wide (should be Running, only on nik-debian), then /var/lib/rancher/k3s/agent/etc/cni/net.d/ on nik-debian for a generated 00-multus.conf — if it's missing, Multus isn't actually in the CNI chain regardless of pod status. See manifests/multus/README.md for the k3s-specific path gotchas that make this different from a stock Multus install.

Monitoring: currently journald only (journalctl -u pia-gateway-healthcheck on minisforum) — nothing is wired into the existing kube-prometheus-stack (values/kube-prometheus-stack .yaml has no custom node-exporter textfile-collector or additionalPrometheusRulesMap configured, and none was added by this work). Wiring real alerting would mean enabling node-exporter's textfile collector for that host and adding a PrometheusRule — not done, flagged here as the natural next step rather than guessed at.

Rollback (full): work through the table above bottom-to-top — restore browser-vpn-proxy/qBittorrent/JDownloader from git history first (fastest to undo, keeps service running — see the table's Rollback column), then remove the Multus/NAD resources, then ansible/roles/vlan50-parent's rollback, then ansible/roles/pia-gateway's rollback. None of these steps touch pia-credentials, PVCs, or the normal k3s/Argo CD control plane.

Gotchas

• manifests/media/qbittorrent.yaml and jdownloader.yaml deploy the moment they're pushed to main — the media Argo CD Application already has selfHeal/automated sync, unlike the new multus Application (deliberately manual-sync). Double-check which gate applies before pushing changes to either file.
• Argo CD Applications mostly set prune: false; removing resources from Git may require manual cleanup.
• Gitea uses a manual public IngressRoute; values/gitea.yaml has no ingress: key at all, so the chart's own ingress is off by chart default, not an explicit setting.
• Grafana and Loki's static hostPath PVs (grafana-pv, loki-pv in manifests/monitoring/monitoring-pvs.yaml) are currently unbound — their Helm-managed PVCs got dynamically provisioned via the local-path StorageClass instead (confirmed live via kubectl get pv/pvc -n monitoring), unlike Prometheus which binds prometheus-pv correctly. Data is not lost, just not on the disk the docs/manifest imply — needs a deliberate decision (bind properly with a data migration, or drop the orphaned static PVs and document reality) before relying on /data/grafana or /data/loki for backups/DR.
• Gitea ROOT_URL changes can require deleting the generated inline config secret before reconciling.
• Pi-hole does not provide wildcard DNS here; add each new internal hostname to both Pi-hole values files.
• Secondary Pi-hole external IPs can be lost during chart upgrades; rerun manifests/network/pihole-debian-patch.sh.
• Authentik forward-auth depends on the Cookie header in authRequestHeaders; removing it causes redirect loops.
• CoreDNS custom zone snippets must use the .server key suffix.
• Renewing internal-ca-cert by deleting its secret (instead of cmctl renew) generates a new key pair, silently breaking every leaf cert and every device's trust at once — see "Renewing the internal CA" under TLS and Trust for the recovery/prevention procedure.
• The Dashy manifest contains an empty ConfigMap shell; use manifests/core/apply-dashy-config.sh to load the real config.
• A newly pushed Argo CD Application (a new file under argocd/apps/) doesn't create its namespace/resources the instant you push — app-of-apps only picks it up on its next Git poll. Running a namespace-scoped registry-secret.sh (or anything else assuming the namespace already exists) right after pushing can fail with namespaces "<ns>" not found; either wait for the next poll or force one: kubectl patch application app-of-apps -n argocd --type merge -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'.

See the scoped READMEs in ansible/, argocd/, and manifests/ for workflows specific to those directories.