264 lines
11 KiB
Markdown
264 lines
11 KiB
Markdown
# homelab
|
|
|
|
Infrastructure-as-code for a small K3s homelab. Host setup is handled with
|
|
Ansible, cluster state is reconciled by Argo CD, and service configuration lives
|
|
in Kubernetes manifests plus Helm values.
|
|
|
|
## Current Architecture
|
|
|
|
| Host | IP | Role |
|
|
| --- | --- | --- |
|
|
| `minisforum` | `10.10.40.53` | K3s server, Traefik entrypoint, primary app node |
|
|
| `debian` / `nik-debian` | `10.10.40.20` | K3s agent, NFS storage, secondary Pi-hole |
|
|
| `mac-mini` | `10.10.40.30` | Standalone services such as Watch Party and Ollama |
|
|
| `gpu-node` / `nik-gpu` | `10.10.40.12` | K3s agent with NVIDIA GPU passthrough, spot-tainted; runs Ollama directly on the host |
|
|
|
|
The cluster uses Traefik instead of the bundled K3s ingress controller. Internal
|
|
services are published under `home.arpa` with certificates from an internal CA.
|
|
Public services under `nik4nao.com` use Let's Encrypt.
|
|
|
|
## Repository Layout
|
|
|
|
| Path | Purpose |
|
|
| --- | --- |
|
|
| `ansible/` | Host bootstrap and non-Kubernetes services |
|
|
| `argocd/apps/` | Argo CD `Application` objects for Helm charts and manifest directories |
|
|
| `argocd/values/` | Helm values used to install or reconcile Argo CD itself |
|
|
| `config/` | App configuration that is injected into manifests, currently Dashy |
|
|
| `manifests/` | Raw Kubernetes resources grouped by service area |
|
|
| `values/` | Helm values consumed by Argo CD Applications |
|
|
|
|
## Managed Services
|
|
|
|
| Area | Services |
|
|
| --- | --- |
|
|
| GitOps | Argo CD and app-of-apps |
|
|
| Networking | Traefik, Pi-hole primary and secondary, DDNS, CoreDNS custom forwarding |
|
|
| TLS | cert-manager, internal CA issuer, Let's Encrypt issuers, CA installer page |
|
|
| Identity | Authentik, Traefik forward-auth middleware, OIDC integrations |
|
|
| Observability | kube-prometheus-stack, Grafana, Loki, Tempo, OpenTelemetry Collector, Glances |
|
|
| Git and CI | Gitea, Gitea Actions runner, registry pull secrets, Gitea backup CronJob |
|
|
| Media | Jellyfin, qBittorrent, JDownloader |
|
|
| Home services | Home Assistant ingress, HA gateway, AI gateway, Discord bot, Alexa bridge, Alert bridge |
|
|
| Public apps | Portfolio, Watch Party ingress to the Mac Mini |
|
|
| Dashboard | Dashy |
|
|
|
|
## Important URLs
|
|
|
|
| URL | Service | Certificate |
|
|
| --- | --- | --- |
|
|
| `https://argocd.home.arpa` | Argo CD | Internal CA |
|
|
| `https://auth.home.arpa` | Authentik internal | Internal CA |
|
|
| `https://auth.nik4nao.com` | Authentik public | Let's Encrypt |
|
|
| `https://traefik.home.arpa` | Traefik dashboard | Internal CA |
|
|
| `https://grafana.nik4nao.com` | Grafana | Let's Encrypt |
|
|
| `https://gitea.nik4nao.com` | Gitea | Let's Encrypt |
|
|
| `https://pihole.home.arpa` | Pi-hole | Internal CA |
|
|
| `https://dashy.home.arpa` | Dashy | Internal CA |
|
|
| `https://jellyfin.home.arpa` | Jellyfin | Internal CA |
|
|
| `https://qbittorrent.home.arpa` | qBittorrent | Internal CA |
|
|
| `https://jdownloader.home.arpa` | JDownloader | Internal CA |
|
|
| `https://ha.home.arpa` | Home Assistant | Internal CA |
|
|
| `https://glances.home.arpa` | Glances on K3s | Internal CA |
|
|
| `https://glances-debian.home.arpa` | Glances on Debian | Internal CA |
|
|
| `http://alert-bridge.home.arpa` | Alert bridge (Discord webhook relay) | Plain HTTP |
|
|
| `https://watch-party.nik4nao.com` | Watch Party on Mac Mini | Let's Encrypt |
|
|
| `https://nik4nao.com` | Portfolio | Let's Encrypt |
|
|
| `http://ca.home.arpa` | Internal CA installer | Plain HTTP |
|
|
|
|
`home.arpa` names are defined explicitly in `values/pihole.yaml` and
|
|
`values/pihole-debian.yaml`; Pi-hole is not configured as a wildcard DNS server.
|
|
|
|
## Bootstrap
|
|
|
|
Install workstation tools:
|
|
|
|
```bash
|
|
pip install ansible
|
|
ansible-galaxy collection install -r ansible/requirements.yml
|
|
```
|
|
|
|
Also install `kubectl`, `helm`, and `kubeseal`. The inventory expects SSH
|
|
access with `~/.ssh/id_ed25519-nik-macbookair`.
|
|
|
|
Bring up hosts and base services:
|
|
|
|
```bash
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K
|
|
```
|
|
|
|
Install Argo CD once, then hand control to the app-of-apps:
|
|
|
|
```bash
|
|
helm repo add argo https://argoproj.github.io/argo-helm
|
|
helm repo update
|
|
helm upgrade --install argocd argo/argo-cd \
|
|
--namespace argocd --create-namespace \
|
|
--version 9.4.15 \
|
|
--values argocd/values/argocd.yaml
|
|
|
|
kubectl apply -f manifests/argocd/app-of-apps.yaml
|
|
```
|
|
|
|
The app-of-apps now brings up cert-manager's `ClusterIssuer`s (via
|
|
`cert-manager-config`) and Authentik's ingress/proxy-outpost/middleware (via
|
|
`authentik-config`) automatically — these used to require untracked manual
|
|
`kubectl apply` steps that weren't documented anywhere. **Authentik's Helm
|
|
chart itself is still a one-time manual install**, since its Argo CD
|
|
Application (`argocd/apps/authentik.yaml`) is deliberately left on manual sync
|
|
with a `targetRevision` placeholder pending a chart-version decision:
|
|
|
|
```bash
|
|
helm repo add authentik https://charts.goauthentik.io
|
|
helm repo update
|
|
helm upgrade --install authentik authentik/authentik \
|
|
-f values/authentik.yaml -n authentik --create-namespace
|
|
```
|
|
|
|
Populate secrets (see "Secrets" below) before Authentik, Gitea, Grafana, or
|
|
the other services that depend on them will come up healthy. After that,
|
|
normal changes should flow through Git and Argo CD.
|
|
|
|
### Cold-rebuild order
|
|
|
|
Rebuilding from nothing, the dependency order that actually matters is:
|
|
|
|
1. Ansible playbooks above (hosts, K3s, NFS).
|
|
2. `helm install argocd` + `kubectl apply -f manifests/argocd/app-of-apps.yaml`
|
|
— this alone now brings up cert-manager, sealed-secrets, and their
|
|
`ClusterIssuer`s/CA cert via sync-wave ordering.
|
|
3. Runtime secret scripts and Sealed Secret regeneration (see "Secrets") —
|
|
several Applications (Gitea, Grafana, home-services, Authentik) will sit
|
|
degraded/crash-looping until their secrets exist.
|
|
4. Manual Authentik Helm install (above) — every OAuth-gated service
|
|
(Grafana, Gitea, Argo CD SSO, Traefik dashboard) depends on it.
|
|
5. Everything else reconciles from Git on its own from here.
|
|
|
|
## Daily Operations
|
|
|
|
Check cluster and sync state:
|
|
|
|
```bash
|
|
kubectl get nodes
|
|
kubectl get pods -A
|
|
kubectl get applications -n argocd
|
|
```
|
|
|
|
Apply Dashy config after editing `config/dashy/conf.yaml`:
|
|
|
|
```bash
|
|
bash manifests/core/apply-dashy-config.sh
|
|
```
|
|
|
|
Patch secondary Pi-hole DNS services after Helm upgrades if needed:
|
|
|
|
```bash
|
|
bash manifests/network/pihole-debian-patch.sh
|
|
```
|
|
|
|
Deploy host-level services:
|
|
|
|
```bash
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
|
|
```
|
|
|
|
## Secrets
|
|
|
|
Do not commit plaintext secrets. Runtime-only scripts read `.env` and create
|
|
Kubernetes Secrets directly. Sealed Secret scripts regenerate committed
|
|
`*-sealed.yaml` resources.
|
|
|
|
Start from `.env.example` and keep the filled `.env` local.
|
|
|
|
Runtime secret scripts:
|
|
|
|
```bash
|
|
bash manifests/cert-manager/porkbun-secret.sh
|
|
bash manifests/authentik/authentik-secret.sh
|
|
bash manifests/authentik/authentik-proxy-secret.sh
|
|
bash manifests/authentik/authentik-gitea-secret.sh
|
|
bash manifests/authentik/authentik-grafana-secret.sh
|
|
bash manifests/home-services/registry-secret.sh
|
|
bash manifests/monitoring/grafana-secret.sh
|
|
bash manifests/network/ddns-secret.sh
|
|
bash manifests/portfolio/registry-secret.sh
|
|
```
|
|
|
|
Sealed Secret regeneration:
|
|
|
|
```bash
|
|
bash manifests/home-services/discord-bot-secret.sh
|
|
bash manifests/home-services/ha-gateway-secret.sh
|
|
```
|
|
|
|
Some sealed secrets are maintained directly in the repo, including Argo CD OIDC,
|
|
Gitea admin/OIDC, Grafana admin/OIDC, Pi-hole admin, and home-service secrets.
|
|
The host-level Gitea runner reads `GITEA_RUNNER_TOKEN` from the environment when
|
|
running `ansible/playbooks/setup-gitea-runner.yaml`.
|
|
|
|
## Storage
|
|
|
|
K3s `local-path` is used for many app PVCs. Static hostPath PVs are used for
|
|
state that must live on known disks:
|
|
|
|
| Location | Use |
|
|
| --- | --- |
|
|
| `/data/gitea` on `minisforum` | Gitea shared storage |
|
|
| `/data/prometheus` on `minisforum` | Prometheus (bound correctly) |
|
|
| `/data/grafana` on `minisforum` | Static PV defined for Grafana, but **not currently bound** — see gotcha below |
|
|
| `/data/loki` on `minisforum` | Static PV defined for Loki, but **not currently bound** — see gotcha below |
|
|
| `/mnt/storage` on `debian` | NFS media library |
|
|
| `/home/nik/backups` on `debian` | NFS export for Gitea's backup CronJob (separate from `/mnt/storage`) |
|
|
|
|
The Debian NFS server exports both `/mnt/storage` and `/home/nik/backups` to
|
|
`10.10.40.53`.
|
|
|
|
## TLS and Trust
|
|
|
|
`*.home.arpa` certificates use `internal-ca-issuer` from `manifests/cert-manager`.
|
|
The CA installer at `http://ca.home.arpa` serves `ca.crt` and an iOS/macOS
|
|
mobileconfig profile. The `ca-sync` CronJob updates those files from the
|
|
`cert-manager/internal-ca-cert` secret when the CA changes.
|
|
|
|
`*.nik4nao.com` certificates use the Let's Encrypt issuers in
|
|
`manifests/cert-manager/cluster-issuer-letsencrypt.yaml`.
|
|
|
|
## Gotchas
|
|
|
|
- Argo CD Applications mostly set `prune: false`; removing resources from Git may
|
|
require manual cleanup.
|
|
- Gitea uses a manual public `IngressRoute`; `values/gitea.yaml` has no
|
|
`ingress:` key at all, so the chart's own ingress is off by chart default,
|
|
not an explicit setting.
|
|
- Grafana and Loki's static hostPath PVs (`grafana-pv`, `loki-pv` in
|
|
`manifests/monitoring/monitoring-pvs.yaml`) are currently unbound — their
|
|
Helm-managed PVCs got dynamically provisioned via the `local-path`
|
|
StorageClass instead (confirmed live via `kubectl get pv/pvc -n
|
|
monitoring`), unlike Prometheus which binds `prometheus-pv` correctly. Data
|
|
is not lost, just not on the disk the docs/manifest imply — needs a
|
|
deliberate decision (bind properly with a data migration, or drop the
|
|
orphaned static PVs and document reality) before relying on `/data/grafana`
|
|
or `/data/loki` for backups/DR.
|
|
- Gitea `ROOT_URL` changes can require deleting the generated inline config
|
|
secret before reconciling.
|
|
- Pi-hole does not provide wildcard DNS here; add each new internal hostname to
|
|
both Pi-hole values files.
|
|
- Secondary Pi-hole external IPs can be lost during chart upgrades; rerun
|
|
`manifests/network/pihole-debian-patch.sh`.
|
|
- Authentik forward-auth depends on the `Cookie` header in
|
|
`authRequestHeaders`; removing it causes redirect loops.
|
|
- CoreDNS custom zone snippets must use the `.server` key suffix.
|
|
- The Dashy manifest contains an empty ConfigMap shell; use
|
|
`manifests/core/apply-dashy-config.sh` to load the real config.
|
|
|
|
See the scoped READMEs in `ansible/`, `argocd/`, and `manifests/` for workflows
|
|
specific to those directories.
|