homelab/README.md

homelab

Infrastructure-as-code for a small K3s homelab. Host setup is handled with Ansible, cluster state is reconciled by Argo CD, and service configuration lives in Kubernetes manifests plus Helm values.

Current Architecture

Host	IP	Role
minisforum	192.168.7.77	K3s server, Traefik entrypoint, primary app node
debian / nik-debian	192.168.7.183	K3s agent, NFS storage, secondary Pi-hole
mac-mini	192.168.7.96	Standalone services such as Watch Party and Ollama
gpu-node / nik-gpu	192.168.7.98	K3s agent with NVIDIA GPU passthrough, spot-tainted; runs Ollama directly on the host

The cluster uses Traefik instead of the bundled K3s ingress controller. Internal services are published under home.arpa with certificates from an internal CA. Public services under nik4nao.com use Let's Encrypt.

Repository Layout

Path	Purpose
ansible/	Host bootstrap and non-Kubernetes services
argocd/apps/	Argo CD Application objects for Helm charts and manifest directories
argocd/values/	Helm values used to install or reconcile Argo CD itself
config/	App configuration that is injected into manifests, currently Dashy
manifests/	Raw Kubernetes resources grouped by service area
values/	Helm values consumed by Argo CD Applications

Managed Services

Area	Services
GitOps	Argo CD and app-of-apps
Networking	Traefik, Pi-hole primary and secondary, DDNS, CoreDNS custom forwarding
TLS	cert-manager, internal CA issuer, Let's Encrypt issuers, CA installer page
Identity	Authentik, Traefik forward-auth middleware, OIDC integrations
Observability	kube-prometheus-stack, Grafana, Loki, Tempo, OpenTelemetry Collector, Glances
Git and CI	Gitea, Gitea Actions runner, registry pull secrets, Gitea backup CronJob
Media	Jellyfin, qBittorrent, JDownloader
Home services	Home Assistant ingress, HA gateway, AI gateway, Discord bot, Alexa bridge, Alert bridge
Public apps	Portfolio, Watch Party ingress to the Mac Mini
Dashboard	Dashy

Important URLs

URL	Service	Certificate
https://argocd.home.arpa	Argo CD	Internal CA
https://auth.home.arpa	Authentik internal	Internal CA
https://auth.nik4nao.com	Authentik public	Let's Encrypt
https://traefik.home.arpa	Traefik dashboard	Internal CA
https://grafana.nik4nao.com	Grafana	Let's Encrypt
https://gitea.home.arpa	Gitea	Internal CA
https://pihole.home.arpa	Pi-hole	Internal CA
https://dashy.home.arpa	Dashy	Internal CA
https://jellyfin.home.arpa	Jellyfin	Internal CA
https://qbittorrent.home.arpa	qBittorrent	Internal CA
https://jdownloader.home.arpa	JDownloader	Internal CA
https://ha.home.arpa	Home Assistant	Internal CA
https://glances.home.arpa	Glances on K3s	Internal CA
https://glances-debian.home.arpa	Glances on Debian	Internal CA
http://alert-bridge.home.arpa	Alert bridge (Discord webhook relay)	Plain HTTP
https://watch-party.nik4nao.com	Watch Party on Mac Mini	Let's Encrypt
https://nik4nao.com	Portfolio	Let's Encrypt
http://ca.home.arpa	Internal CA installer	Plain HTTP

home.arpa names are defined explicitly in values/pihole.yaml and values/pihole-debian.yaml; Pi-hole is not configured as a wildcard DNS server.

Bootstrap

Install workstation tools:

pip install ansible
ansible-galaxy collection install -r ansible/requirements.yml

Also install kubectl, helm, and kubeseal. The inventory expects SSH access with ~/.ssh/id_ed25519-nik-macbookair.

Bring up hosts and base services:

ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K

Install Argo CD once, then hand control to the app-of-apps:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm upgrade --install argocd argo/argo-cd \
  --namespace argocd --create-namespace \
  --version 9.4.15 \
  --values argocd/values/argocd.yaml

kubectl apply -f manifests/argocd/app-of-apps.yaml

The app-of-apps now brings up cert-manager's ClusterIssuers (via cert-manager-config) and Authentik's ingress/proxy-outpost/middleware (via authentik-config) automatically — these used to require untracked manual kubectl apply steps that weren't documented anywhere. Authentik's Helm chart itself is still a one-time manual install, since its Argo CD Application (argocd/apps/authentik.yaml) is deliberately left on manual sync with a targetRevision placeholder pending a chart-version decision:

helm repo add authentik https://charts.goauthentik.io
helm repo update
helm upgrade --install authentik authentik/authentik \
  -f values/authentik.yaml -n authentik --create-namespace

Populate secrets (see "Secrets" below) before Authentik, Gitea, Grafana, or the other services that depend on them will come up healthy. After that, normal changes should flow through Git and Argo CD.

Cold-rebuild order

Rebuilding from nothing, the dependency order that actually matters is:

1. Ansible playbooks above (hosts, K3s, NFS).
2. helm install argocd + kubectl apply -f manifests/argocd/app-of-apps.yaml — this alone now brings up cert-manager, sealed-secrets, and their ClusterIssuers/CA cert via sync-wave ordering.
3. Runtime secret scripts and Sealed Secret regeneration (see "Secrets") — several Applications (Gitea, Grafana, home-services, Authentik) will sit degraded/crash-looping until their secrets exist.
4. Manual Authentik Helm install (above) — every OAuth-gated service (Grafana, Gitea, Argo CD SSO, Traefik dashboard) depends on it.
5. Everything else reconciles from Git on its own from here.

Daily Operations

Check cluster and sync state:

kubectl get nodes
kubectl get pods -A
kubectl get applications -n argocd

Apply Dashy config after editing config/dashy/conf.yaml:

bash manifests/core/apply-dashy-config.sh

Patch secondary Pi-hole DNS services after Helm upgrades if needed:

bash manifests/network/pihole-debian-patch.sh

Deploy host-level services:

ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K

Secrets

Do not commit plaintext secrets. Runtime-only scripts read .env and create Kubernetes Secrets directly. Sealed Secret scripts regenerate committed *-sealed.yaml resources.

Start from .env.example and keep the filled .env local.

Runtime secret scripts:

bash manifests/cert-manager/porkbun-secret.sh
bash manifests/authentik/authentik-secret.sh
bash manifests/authentik/authentik-proxy-secret.sh
bash manifests/authentik/authentik-gitea-secret.sh
bash manifests/authentik/authentik-grafana-secret.sh
bash manifests/home-services/registry-secret.sh
bash manifests/monitoring/grafana-secret.sh
bash manifests/network/ddns-secret.sh
bash manifests/portfolio/registry-secret.sh

Sealed Secret regeneration:

bash manifests/home-services/discord-bot-secret.sh
bash manifests/home-services/ha-gateway-secret.sh

Some sealed secrets are maintained directly in the repo, including Argo CD OIDC, Gitea admin/OIDC, Grafana admin/OIDC, Pi-hole admin, and home-service secrets. The host-level Gitea runner reads GITEA_RUNNER_TOKEN from the environment when running ansible/playbooks/setup-gitea-runner.yaml.

Storage

K3s local-path is used for many app PVCs. Static hostPath PVs are used for state that must live on known disks:

Location	Use
/data/gitea on minisforum	Gitea shared storage
/data/prometheus on minisforum	Prometheus (bound correctly)
/data/grafana on minisforum	Static PV defined for Grafana, but not currently bound — see gotcha below
/data/loki on minisforum	Static PV defined for Loki, but not currently bound — see gotcha below
/mnt/storage on debian	NFS media library
/home/nik/backups on debian	NFS export for Gitea's backup CronJob (separate from /mnt/storage)

The Debian NFS server exports both /mnt/storage and /home/nik/backups to 192.168.7.77.

TLS and Trust

*.home.arpa certificates use internal-ca-issuer from manifests/cert-manager. The CA installer at http://ca.home.arpa serves ca.crt and an iOS/macOS mobileconfig profile. The ca-sync CronJob updates those files from the cert-manager/internal-ca-cert secret when the CA changes.

*.nik4nao.com certificates use the Let's Encrypt issuers in manifests/cert-manager/cluster-issuer-letsencrypt.yaml.

Gotchas

• Argo CD Applications mostly set prune: false; removing resources from Git may require manual cleanup.
• Gitea is routed at gitea.home.arpa via manifests/gitea/gitea-ingress.yaml (Certificate + IngressRoute); values/gitea.yaml has no ingress: key at all, so the chart's own ingress is off by chart default, not an explicit setting.
• Grafana and Loki's static hostPath PVs (grafana-pv, loki-pv in manifests/monitoring/monitoring-pvs.yaml) are currently unbound — their Helm-managed PVCs got dynamically provisioned via the local-path StorageClass instead (confirmed live via kubectl get pv/pvc -n monitoring), unlike Prometheus which binds prometheus-pv correctly. Data is not lost, just not on the disk the docs/manifest imply — needs a deliberate decision (bind properly with a data migration, or drop the orphaned static PVs and document reality) before relying on /data/grafana or /data/loki for backups/DR.
• Gitea ROOT_URL changes can require deleting the generated inline config secret before reconciling.
• Pi-hole does not provide wildcard DNS here; add each new internal hostname to both Pi-hole values files.
• Secondary Pi-hole external IPs can be lost during chart upgrades; rerun manifests/network/pihole-debian-patch.sh.
• Authentik forward-auth depends on the Cookie header in authRequestHeaders; removing it causes redirect loops.
• CoreDNS custom zone snippets must use the .server key suffix.
• The Dashy manifest contains an empty ConfigMap shell; use manifests/core/apply-dashy-config.sh to load the real config.

See the scoped READMEs in ansible/, argocd/, and manifests/ for workflows specific to those directories.