homelab/README.md

# homelab

Infrastructure-as-code for a small K3s homelab. Host setup is handled with
Ansible, cluster state is reconciled by Argo CD, and service configuration lives
in Kubernetes manifests plus Helm values.

## Current Architecture

| Host | IP | Role |
| --- | --- | --- |
| `minisforum` | `10.10.40.53` | K3s server, Traefik entrypoint, primary app node |
| `debian` / `nik-debian` | `10.10.40.20` | K3s agent, NFS storage, secondary Pi-hole |
| `mac-mini` | `10.10.40.30` | Standalone services such as Watch Party and Ollama |
| `gpu-node` / `nik-gpu` | `10.10.40.12` | K3s agent with NVIDIA GPU passthrough, spot-tainted; runs Ollama directly on the host |

The cluster uses Traefik instead of the bundled K3s ingress controller. Internal
services are published under `home.arpa` with certificates from an internal CA.
Public services under `nik4nao.com` use Let's Encrypt.

## Repository Layout

| Path | Purpose |
| --- | --- |
| `ansible/` | Host bootstrap and non-Kubernetes services |
| `argocd/apps/` | Argo CD `Application` objects for Helm charts and manifest directories |
| `argocd/values/` | Helm values used to install or reconcile Argo CD itself |
| `config/` | App configuration that is injected into manifests, currently Dashy |
| `manifests/` | Raw Kubernetes resources grouped by service area |
| `values/` | Helm values consumed by Argo CD Applications |

## Managed Services

| Area | Services |
| --- | --- |
| GitOps | Argo CD and app-of-apps |
| Networking | Traefik, Pi-hole primary and secondary, DDNS, CoreDNS custom forwarding |
| TLS | cert-manager, internal CA issuer, Let's Encrypt issuers, CA installer page |
| Identity | Authentik, Traefik forward-auth middleware, OIDC integrations |
| Observability | kube-prometheus-stack, Grafana, Loki, Tempo, OpenTelemetry Collector, Glances |
| Git and CI | Gitea, Gitea Actions runner, registry pull secrets, Gitea backup CronJob |
| Media | Jellyfin, qBittorrent, JDownloader |
| Home services | Home Assistant ingress, HA gateway, AI gateway, Discord bot, Alexa bridge, Alert bridge |
| Public apps | Portfolio, Watch Party ingress to the Mac Mini |
| Dashboard | Dashy |

## Important URLs

| URL | Service | Certificate |
| --- | --- | --- |
| `https://argocd.home.arpa` | Argo CD | Internal CA |
| `https://auth.home.arpa` | Authentik internal | Internal CA |
| `https://auth.nik4nao.com` | Authentik public | Let's Encrypt |
| `https://traefik.home.arpa` | Traefik dashboard | Internal CA |
| `https://grafana.nik4nao.com` | Grafana | Let's Encrypt |
| `https://gitea.nik4nao.com` | Gitea | Let's Encrypt |
| `https://pihole.home.arpa` | Pi-hole | Internal CA |
| `https://dashy.home.arpa` | Dashy | Internal CA |
| `https://jellyfin.home.arpa` | Jellyfin | Internal CA |
| `https://qbittorrent.home.arpa` | qBittorrent | Internal CA |
| `https://jdownloader.home.arpa` | JDownloader | Internal CA |
| `https://ha.home.arpa` | Home Assistant | Internal CA |
| `https://glances.home.arpa` | Glances on K3s | Internal CA |
| `https://glances-debian.home.arpa` | Glances on Debian | Internal CA |
| `http://alert-bridge.home.arpa` | Alert bridge (Discord webhook relay) | Plain HTTP |
| `https://watch-party.nik4nao.com` | Watch Party on Mac Mini | Let's Encrypt |
| `https://nik4nao.com` | Portfolio | Let's Encrypt |
| `http://ca.home.arpa` | Internal CA installer | Plain HTTP |

`home.arpa` names are defined explicitly in `values/pihole.yaml` and
`values/pihole-debian.yaml`; Pi-hole is not configured as a wildcard DNS server.

## Bootstrap

Install workstation tools:

```bash
pip install ansible
ansible-galaxy collection install -r ansible/requirements.yml
```

Also install `kubectl`, `helm`, and `kubeseal`. The inventory expects SSH
access with `~/.ssh/id_ed25519-nik-macbookair`.

Bring up hosts and base services:

```bash
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K
```

Install Argo CD once, then hand control to the app-of-apps:

```bash
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm upgrade --install argocd argo/argo-cd \
  --namespace argocd --create-namespace \
  --version 9.4.15 \
  --values argocd/values/argocd.yaml

kubectl apply -f manifests/argocd/app-of-apps.yaml
```

The app-of-apps now brings up cert-manager's `ClusterIssuer`s (via
`cert-manager-config`) and Authentik's ingress/proxy-outpost/middleware (via
`authentik-config`) automatically — these used to require untracked manual
`kubectl apply` steps that weren't documented anywhere. **Authentik's Helm
chart itself is still a one-time manual install**, since its Argo CD
Application (`argocd/apps/authentik.yaml`) is deliberately left on manual sync
with a `targetRevision` placeholder pending a chart-version decision:

```bash
helm repo add authentik https://charts.goauthentik.io
helm repo update
helm upgrade --install authentik authentik/authentik \
  -f values/authentik.yaml -n authentik --create-namespace
```

Populate secrets (see "Secrets" below) before Authentik, Gitea, Grafana, or
the other services that depend on them will come up healthy. After that,
normal changes should flow through Git and Argo CD.

### Cold-rebuild order

Rebuilding from nothing, the dependency order that actually matters is:

1. Ansible playbooks above (hosts, K3s, NFS).
2. `helm install argocd` + `kubectl apply -f manifests/argocd/app-of-apps.yaml`
   — this alone now brings up cert-manager, sealed-secrets, and their
   `ClusterIssuer`s/CA cert via sync-wave ordering.
3. Runtime secret scripts and Sealed Secret regeneration (see "Secrets") —
   several Applications (Gitea, Grafana, home-services, Authentik) will sit
   degraded/crash-looping until their secrets exist.
4. Manual Authentik Helm install (above) — every OAuth-gated service
   (Grafana, Gitea, Argo CD SSO, Traefik dashboard) depends on it.
5. Everything else reconciles from Git on its own from here.

## Daily Operations

Check cluster and sync state:

```bash
kubectl get nodes
kubectl get pods -A
kubectl get applications -n argocd
```

Apply Dashy config after editing `config/dashy/conf.yaml`:

```bash
bash manifests/core/apply-dashy-config.sh
```

Patch secondary Pi-hole DNS services after Helm upgrades if needed:

```bash
bash manifests/network/pihole-debian-patch.sh
```

Deploy host-level services:

```bash
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
```

## Secrets

Do not commit plaintext secrets. Runtime-only scripts read `.env` and create
Kubernetes Secrets directly. Sealed Secret scripts regenerate committed
`*-sealed.yaml` resources.

Start from `.env.example` and keep the filled `.env` local.

Runtime secret scripts:

```bash
bash manifests/cert-manager/porkbun-secret.sh
bash manifests/authentik/authentik-secret.sh
bash manifests/authentik/authentik-proxy-secret.sh
bash manifests/authentik/authentik-gitea-secret.sh
bash manifests/authentik/authentik-grafana-secret.sh
bash manifests/home-services/registry-secret.sh
bash manifests/monitoring/grafana-secret.sh
bash manifests/network/ddns-secret.sh
bash manifests/portfolio/registry-secret.sh
```

Sealed Secret regeneration:

```bash
bash manifests/home-services/discord-bot-secret.sh
bash manifests/home-services/ha-gateway-secret.sh
```

Some sealed secrets are maintained directly in the repo, including Argo CD OIDC,
Gitea admin/OIDC, Grafana admin/OIDC, Pi-hole admin, and home-service secrets.
The host-level Gitea runner reads `GITEA_RUNNER_TOKEN` from the environment when
running `ansible/playbooks/setup-gitea-runner.yaml`.

## Storage

K3s `local-path` is used for many app PVCs. Static hostPath PVs are used for
state that must live on known disks:

| Location | Use |
| --- | --- |
| `/data/gitea` on `minisforum` | Gitea shared storage |
| `/data/prometheus` on `minisforum` | Prometheus (bound correctly) |
| `/data/grafana` on `minisforum` | Static PV defined for Grafana, but **not currently bound** — see gotcha below |
| `/data/loki` on `minisforum` | Static PV defined for Loki, but **not currently bound** — see gotcha below |
| `/mnt/storage` on `debian` | NFS media library |
| `/home/nik/backups` on `debian` | NFS export for Gitea's backup CronJob (separate from `/mnt/storage`) |

The Debian NFS server exports both `/mnt/storage` and `/home/nik/backups` to
`10.10.40.53`.

## TLS and Trust

`*.home.arpa` certificates use `internal-ca-issuer` from `manifests/cert-manager`.
The CA installer at `http://ca.home.arpa` serves `ca.crt` and an iOS/macOS
mobileconfig profile. The `ca-sync` CronJob updates those files from the
`cert-manager/internal-ca-cert` secret when the CA changes.

`*.nik4nao.com` certificates use the Let's Encrypt issuers in
`manifests/cert-manager/cluster-issuer-letsencrypt.yaml`.

## VPN VLAN 50 (PIA gateway) — status and runbook

`minisforum` is a PIA WireGuard egress gateway for a dedicated VPN VLAN
(`10.10.50.0/24`), giving `nik-debian`'s selected media workloads a second,
PIA-routed network path (`net1`) alongside their normal cluster networking
(`eth0`) — full design in `plan.md` (repo root). **Live as of 2026-08-24**:
every layer below is applied and independently verified, including the
Phase 5 canary's full validation/failure-test matrix and both media
workloads' own fail-closed kill-switch test (stopping `wg-quick@pia-wg`
on minisforum visibly stops their egress, no fallback to `eth0`). Current
PIA endpoint is Hong Kong (`pia_region_candidates`' first entry —
whichever region actually answers is recorded per-host, see
`ansible/roles/pia-gateway/README.md`'s "Region fallback"). See each
piece's own README for exact commands and rollback:

| Layer | Where | Rollback |
| --- | --- | --- |
| Minisforum PIA gateway (WireGuard, policy routing, kill switch, health check) | `ansible/roles/pia-gateway/` | that role's README |
| nik-debian VLAN 50 parent (`enp1s0.50`) | `ansible/roles/vlan50-parent/` | that role's README |
| Multus, VLAN 50 NAD, canary | `manifests/multus/` (manual-sync Argo CD Application) | that directory's README |
| qBittorrent VLAN 50 attachment | `manifests/media/qbittorrent.yaml` | no file-based copy kept — `git show 28d0628:manifests/media/qbittorrent.yaml \| kubectl apply -f -` restores the last pre-migration (Gluetun-based) version |
| JDownloader VLAN 50 attachment | `manifests/media/jdownloader.yaml` | same approach — `git show 2f7ec5f:manifests/media/jdownloader.yaml \| kubectl apply -f -` |
| browser-vpn-proxy (SOCKS5, no auth — see its own header for why) | `manifests/multus/30-browser-vpn-proxy.yaml` (manual-sync Argo CD Application) | that directory's README — replaces the standalone `manifests/media/gluetun-proxy.yaml` (removed 2026-08-24; `git show 462a9d9:manifests/media/gluetun-proxy.yaml \| kubectl apply -f -` restores it, NodePort `30888`, if ever needed) |

**Boot ordering** (minisforum): `pia-killswitch.service` seeds the closed
routing/firewall state and is ordered `Before=` `wg-quick@pia-wg.service`;
both are `WantedBy=multi-user.target` so a normal boot orders them
correctly with no manual step. `pia-gateway-healthcheck.timer` runs
independently and only observes — see
`ansible/roles/pia-gateway/templates/pia-gateway-healthcheck.sh.j2`.

**PIA endpoint/key rotation**: re-run `ansible/playbooks/pia-gateway.yaml
-e pia_force_reregister=true -K` — this regenerates the local keypair and
re-registers with PIA, then rewrites `pia-wg.conf` (handler-triggered
restart). Rotating the PIA account password itself just needs
`ansible-vault edit ansible/group_vars/all/vault.yaml` first.

**Maintenance / taking the gateway down deliberately**: `sudo systemctl
stop wg-quick@pia-wg` on minisforum closes VLAN 50 Internet egress
immediately (kill switch stays engaged) without touching minisforum's own
networking, k3s, or any other workload — this is the same fail-closed path
exercised by the Phase 5 failure tests in `plan.md`.

**Multus troubleshooting**: check `kubectl -n kube-system get pods -l
app=multus -o wide` (should be Running, only on `nik-debian`), then
`/var/lib/rancher/k3s/agent/etc/cni/net.d/` on nik-debian for a generated
`00-multus.conf` — if it's missing, Multus isn't actually in the CNI
chain regardless of pod status. See `manifests/multus/README.md` for the
k3s-specific path gotchas that make this different from a stock Multus
install.

**Monitoring**: currently journald only
(`journalctl -u pia-gateway-healthcheck` on minisforum) — nothing is wired
into the existing kube-prometheus-stack (`values/kube-prometheus-stack
.yaml` has no custom node-exporter textfile-collector or
`additionalPrometheusRulesMap` configured, and none was added by this
work). Wiring real alerting would mean enabling node-exporter's textfile
collector for that host and adding a `PrometheusRule` — not done, flagged
here as the natural next step rather than guessed at.

**Rollback (full)**: work through the table above bottom-to-top — restore
browser-vpn-proxy/qBittorrent/JDownloader from git history first (fastest
to undo, keeps service running — see the table's Rollback column), then
remove the Multus/NAD resources, then `ansible/roles/vlan50-parent`'s
rollback, then `ansible/roles/pia-gateway`'s rollback. None of these
steps touch `pia-credentials`, PVCs, or the normal k3s/Argo CD control
plane.

## Gotchas

- **`manifests/media/qbittorrent.yaml` and `jdownloader.yaml` deploy the
  moment they're pushed to `main`** — the `media` Argo CD Application
  already has `selfHeal`/`automated` sync, unlike the new `multus`
  Application (deliberately manual-sync). Double-check which gate applies
  before pushing changes to either file.
- Argo CD Applications mostly set `prune: false`; removing resources from Git may
  require manual cleanup.
- Gitea uses a manual public `IngressRoute`; `values/gitea.yaml` has no
  `ingress:` key at all, so the chart's own ingress is off by chart default,
  not an explicit setting.
- Grafana and Loki's static hostPath PVs (`grafana-pv`, `loki-pv` in
  `manifests/monitoring/monitoring-pvs.yaml`) are currently unbound — their
  Helm-managed PVCs got dynamically provisioned via the `local-path`
  StorageClass instead (confirmed live via `kubectl get pv/pvc -n
  monitoring`), unlike Prometheus which binds `prometheus-pv` correctly. Data
  is not lost, just not on the disk the docs/manifest imply — needs a
  deliberate decision (bind properly with a data migration, or drop the
  orphaned static PVs and document reality) before relying on `/data/grafana`
  or `/data/loki` for backups/DR.
- Gitea `ROOT_URL` changes can require deleting the generated inline config
  secret before reconciling.
- Pi-hole does not provide wildcard DNS here; add each new internal hostname to
  both Pi-hole values files.
- Secondary Pi-hole external IPs can be lost during chart upgrades; rerun
  `manifests/network/pihole-debian-patch.sh`.
- Authentik forward-auth depends on the `Cookie` header in
  `authRequestHeaders`; removing it causes redirect loops.
- CoreDNS custom zone snippets must use the `.server` key suffix.
- The Dashy manifest contains an empty ConfigMap shell; use
  `manifests/core/apply-dashy-config.sh` to load the real config.

See the scoped READMEs in `ansible/`, `argocd/`, and `manifests/` for workflows
specific to those directories.