# homelab

Infrastructure-as-code for a small K3s homelab. Host setup is handled with
Ansible, cluster state is reconciled by Argo CD, and service configuration lives
in Kubernetes manifests plus Helm values.

## Current Architecture

| Host | IP | Role |
| --- | --- | --- |
| `minisforum` | `192.168.7.77` | K3s server, Traefik entrypoint, primary app node |
| `debian` / `nik-debian` | `192.168.7.183` | K3s agent, NFS storage, secondary Pi-hole |
| `mac-mini` | `192.168.7.96` | Standalone services such as Watch Party and Ollama |
| `gpu-node` / `nik-gpu` | `192.168.7.98` | K3s agent with NVIDIA GPU passthrough, spot-tainted; runs Ollama directly on the host |

The cluster uses Traefik instead of the bundled K3s ingress controller. Internal
services are published under `home.arpa` with certificates from an internal CA.
Public services under `nik4nao.com` use Let's Encrypt.

## Repository Layout

| Path | Purpose |
| --- | --- |
| `ansible/` | Host bootstrap and non-Kubernetes services |
| `argocd/apps/` | Argo CD `Application` objects for Helm charts and manifest directories |
| `argocd/values/` | Helm values used to install or reconcile Argo CD itself |
| `config/` | App configuration that is injected into manifests, currently Dashy |
| `manifests/` | Raw Kubernetes resources grouped by service area |
| `values/` | Helm values consumed by Argo CD Applications |

## Managed Services

| Area | Services |
| --- | --- |
| GitOps | Argo CD and app-of-apps |
| Networking | Traefik, Pi-hole primary and secondary, DDNS, CoreDNS custom forwarding |
| TLS | cert-manager, internal CA issuer, Let's Encrypt issuers, CA installer page |
| Identity | Authentik, Traefik forward-auth middleware, OIDC integrations |
| Observability | kube-prometheus-stack, Grafana, Loki, Tempo, OpenTelemetry Collector, Glances |
| Git and CI | Gitea, Gitea Actions runner, registry pull secrets, Gitea backup CronJob |
| Media | Jellyfin, qBittorrent, JDownloader |
| Home services | Home Assistant ingress, HA gateway, AI gateway, Discord bot, Alexa bridge, Alert bridge |
| Public apps | Portfolio, Watch Party ingress to the Mac Mini |
| Dashboard | Dashy |

## Important URLs

| URL | Service | Certificate |
| --- | --- | --- |
| `https://argocd.home.arpa` | Argo CD | Internal CA |
| `https://auth.home.arpa` | Authentik internal | Internal CA |
| `https://auth.nik4nao.com` | Authentik public | Let's Encrypt |
| `https://traefik.home.arpa` | Traefik dashboard | Internal CA |
| `https://grafana.nik4nao.com` | Grafana | Let's Encrypt |
| `https://gitea.nik4nao.com` | Gitea | Let's Encrypt |
| `https://pihole.home.arpa` | Pi-hole | Internal CA |
| `https://dashy.home.arpa` | Dashy | Internal CA |
| `https://jellyfin.home.arpa` | Jellyfin | Internal CA |
| `https://qbittorrent.home.arpa` | qBittorrent | Internal CA |
| `https://jdownloader.home.arpa` | JDownloader | Internal CA |
| `https://ha.home.arpa` | Home Assistant | Internal CA |
| `https://glances.home.arpa` | Glances on K3s | Internal CA |
| `https://glances-debian.home.arpa` | Glances on Debian | Internal CA |
| `http://alert-bridge.home.arpa` | Alert bridge (Discord webhook relay) | Plain HTTP |
| `https://watch-party.nik4nao.com` | Watch Party on Mac Mini | Let's Encrypt |
| `https://nik4nao.com` | Portfolio | Let's Encrypt |
| `http://ca.home.arpa` | Internal CA installer | Plain HTTP |

`home.arpa` names are defined explicitly in `values/pihole.yaml` and
`values/pihole-debian.yaml`; Pi-hole is not configured as a wildcard DNS server.

## Bootstrap

Install workstation tools:

```bash
pip install ansible
ansible-galaxy collection install -r ansible/requirements.yml
```

Also install `kubectl`, `helm`, and `kubeseal`. The inventory expects SSH
access with `~/.ssh/id_ed25519-nik-macbookair`.

Bring up hosts and base services:

```bash
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K
```

Install Argo CD once, then hand control to the app-of-apps:

```bash
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm upgrade --install argocd argo/argo-cd \
  --namespace argocd --create-namespace \
  --version 9.4.15 \
  --values argocd/values/argocd.yaml

kubectl apply -f manifests/argocd/app-of-apps.yaml
```

The app-of-apps now brings up cert-manager's `ClusterIssuer`s (via
`cert-manager-config`) and Authentik's ingress/proxy-outpost/middleware (via
`authentik-config`) automatically — these used to require untracked manual
`kubectl apply` steps that weren't documented anywhere. **Authentik's Helm
chart itself is still a one-time manual install**, since its Argo CD
Application (`argocd/apps/authentik.yaml`) is deliberately left on manual sync
with a `targetRevision` placeholder pending a chart-version decision:

```bash
helm repo add authentik https://charts.goauthentik.io
helm repo update
helm upgrade --install authentik authentik/authentik \
  -f values/authentik.yaml -n authentik --create-namespace
```

Populate secrets (see "Secrets" below) before Authentik, Gitea, Grafana, or
the other services that depend on them will come up healthy. After that,
normal changes should flow through Git and Argo CD.

### Cold-rebuild order

Rebuilding from nothing, the dependency order that actually matters is:

1. Ansible playbooks above (hosts, K3s, NFS).
2. `helm install argocd` + `kubectl apply -f manifests/argocd/app-of-apps.yaml`
   — this alone now brings up cert-manager, sealed-secrets, and their
   `ClusterIssuer`s/CA cert via sync-wave ordering.
3. Runtime secret scripts and Sealed Secret regeneration (see "Secrets") —
   several Applications (Gitea, Grafana, home-services, Authentik) will sit
   degraded/crash-looping until their secrets exist.
4. Manual Authentik Helm install (above) — every OAuth-gated service
   (Grafana, Gitea, Argo CD SSO, Traefik dashboard) depends on it.
5. Everything else reconciles from Git on its own from here.

## Daily Operations

Check cluster and sync state:

```bash
kubectl get nodes
kubectl get pods -A
kubectl get applications -n argocd
```

Apply Dashy config after editing `config/dashy/conf.yaml`:

```bash
bash manifests/core/apply-dashy-config.sh
```

Patch secondary Pi-hole DNS services after Helm upgrades if needed:

```bash
bash manifests/network/pihole-debian-patch.sh
```

Deploy host-level services:

```bash
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
```

## Secrets

Do not commit plaintext secrets. Runtime-only scripts read `.env` and create
Kubernetes Secrets directly. Sealed Secret scripts regenerate committed
`*-sealed.yaml` resources.

Start from `.env.example` and keep the filled `.env` local.

Runtime secret scripts:

```bash
bash manifests/cert-manager/porkbun-secret.sh
bash manifests/authentik/authentik-secret.sh
bash manifests/authentik/authentik-proxy-secret.sh
bash manifests/authentik/authentik-gitea-secret.sh
bash manifests/authentik/authentik-grafana-secret.sh
bash manifests/home-services/registry-secret.sh
bash manifests/monitoring/grafana-secret.sh
bash manifests/network/ddns-secret.sh
bash manifests/portfolio/registry-secret.sh
```

Sealed Secret regeneration:

```bash
bash manifests/home-services/discord-bot-secret.sh
bash manifests/home-services/ha-gateway-secret.sh
```

Some sealed secrets are maintained directly in the repo, including Argo CD OIDC,
Gitea admin/OIDC, Grafana admin/OIDC, Pi-hole admin, and home-service secrets.
The host-level Gitea runner reads `GITEA_RUNNER_TOKEN` from the environment when
running `ansible/playbooks/setup-gitea-runner.yaml`.

## Storage

K3s `local-path` is used for many app PVCs. Static hostPath PVs are used for
state that must live on known disks:

| Location | Use |
| --- | --- |
| `/data/gitea` on `minisforum` | Gitea shared storage |
| `/data/prometheus` on `minisforum` | Prometheus (bound correctly) |
| `/data/grafana` on `minisforum` | Static PV defined for Grafana, but **not currently bound** — see gotcha below |
| `/data/loki` on `minisforum` | Static PV defined for Loki, but **not currently bound** — see gotcha below |
| `/mnt/storage` on `debian` | NFS media library |
| `/home/nik/backups` on `debian` | NFS export for Gitea's backup CronJob (separate from `/mnt/storage`) |

The Debian NFS server exports both `/mnt/storage` and `/home/nik/backups` to
`192.168.7.77`.

## TLS and Trust

`*.home.arpa` certificates use `internal-ca-issuer` from `manifests/cert-manager`.
The CA installer at `http://ca.home.arpa` serves `ca.crt` and an iOS/macOS
mobileconfig profile. The `ca-sync` CronJob updates those files from the
`cert-manager/internal-ca-cert` secret when the CA changes.

`*.nik4nao.com` certificates use the Let's Encrypt issuers in
`manifests/cert-manager/cluster-issuer-letsencrypt.yaml`.

## Gotchas

- Argo CD Applications mostly set `prune: false`; removing resources from Git may
  require manual cleanup.
- Gitea uses a manual public `IngressRoute`; `values/gitea.yaml` has no
  `ingress:` key at all, so the chart's own ingress is off by chart default,
  not an explicit setting.
- Grafana and Loki's static hostPath PVs (`grafana-pv`, `loki-pv` in
  `manifests/monitoring/monitoring-pvs.yaml`) are currently unbound — their
  Helm-managed PVCs got dynamically provisioned via the `local-path`
  StorageClass instead (confirmed live via `kubectl get pv/pvc -n
  monitoring`), unlike Prometheus which binds `prometheus-pv` correctly. Data
  is not lost, just not on the disk the docs/manifest imply — needs a
  deliberate decision (bind properly with a data migration, or drop the
  orphaned static PVs and document reality) before relying on `/data/grafana`
  or `/data/loki` for backups/DR.
- Gitea `ROOT_URL` changes can require deleting the generated inline config
  secret before reconciling.
- Pi-hole does not provide wildcard DNS here; add each new internal hostname to
  both Pi-hole values files.
- Secondary Pi-hole external IPs can be lost during chart upgrades; rerun
  `manifests/network/pihole-debian-patch.sh`.
- Authentik forward-auth depends on the `Cookie` header in
  `authRequestHeaders`; removing it causes redirect loops.
- CoreDNS custom zone snippets must use the `.server` key suffix.
- The Dashy manifest contains an empty ConfigMap shell; use
  `manifests/core/apply-dashy-config.sh` to load the real config.

See the scoped READMEs in `ansible/`, `argocd/`, and `manifests/` for workflows
specific to those directories.