220 lines
8.2 KiB
Markdown
220 lines
8.2 KiB
Markdown
# homelab
|
|
|
|
Infrastructure-as-code for a small K3s homelab. Host setup is handled with
|
|
Ansible, cluster state is reconciled by Argo CD, and service configuration lives
|
|
in Kubernetes manifests plus Helm values.
|
|
|
|
## Current Architecture
|
|
|
|
| Host | IP | Role |
|
|
| --- | --- | --- |
|
|
| `minisforum` | `192.168.7.77` | K3s server, Traefik entrypoint, primary app node |
|
|
| `debian` / `nik-debian` | `192.168.7.183` | K3s agent, NFS storage, secondary Pi-hole |
|
|
| `mac-mini` | `192.168.7.96` | Standalone services such as Watch Party and Ollama |
|
|
|
|
The cluster uses Traefik instead of the bundled K3s ingress controller. Internal
|
|
services are published under `home.arpa` with certificates from an internal CA.
|
|
Public services under `nik4nao.com` use Let's Encrypt.
|
|
|
|
## Repository Layout
|
|
|
|
| Path | Purpose |
|
|
| --- | --- |
|
|
| `ansible/` | Host bootstrap and non-Kubernetes services |
|
|
| `argocd/apps/` | Argo CD `Application` objects for Helm charts and manifest directories |
|
|
| `argocd/values/` | Helm values used to install or reconcile Argo CD itself |
|
|
| `config/` | App configuration that is injected into manifests, currently Dashy |
|
|
| `manifests/` | Raw Kubernetes resources grouped by service area |
|
|
| `values/` | Helm values consumed by Argo CD Applications |
|
|
|
|
## Managed Services
|
|
|
|
| Area | Services |
|
|
| --- | --- |
|
|
| GitOps | Argo CD and app-of-apps |
|
|
| Networking | Traefik, Pi-hole primary and secondary, DDNS, CoreDNS custom forwarding |
|
|
| TLS | cert-manager, internal CA issuer, Let's Encrypt issuers, CA installer page |
|
|
| Identity | Authentik, Traefik forward-auth middleware, OIDC integrations |
|
|
| Observability | kube-prometheus-stack, Grafana, Loki, Tempo, OpenTelemetry Collector, Glances |
|
|
| Git and CI | Gitea, Gitea Actions runner, registry pull secrets, Gitea backup CronJob |
|
|
| Media | Jellyfin, qBittorrent, JDownloader, Photoview, Immich |
|
|
| Home services | Home Assistant ingress, HA gateway, AI gateway, Discord bot |
|
|
| Public apps | Portfolio, Watch Party ingress to the Mac Mini |
|
|
| Dashboard | Dashy |
|
|
|
|
## Important URLs
|
|
|
|
| URL | Service | Certificate |
|
|
| --- | --- | --- |
|
|
| `https://argocd.home.arpa` | Argo CD | Internal CA |
|
|
| `https://auth.home.arpa` | Authentik internal | Internal CA |
|
|
| `https://auth.nik4nao.com` | Authentik public | Let's Encrypt |
|
|
| `https://traefik.home.arpa` | Traefik dashboard | Internal CA |
|
|
| `https://grafana.nik4nao.com` | Grafana | Let's Encrypt |
|
|
| `https://gitea.nik4nao.com` | Gitea | Let's Encrypt |
|
|
| `https://pihole.home.arpa` | Pi-hole | Internal CA |
|
|
| `https://dashy.home.arpa` | Dashy | Internal CA |
|
|
| `https://jellyfin.home.arpa` | Jellyfin | Internal CA |
|
|
| `https://qbittorrent.home.arpa` | qBittorrent | Internal CA |
|
|
| `https://jdownloader.home.arpa` | JDownloader | Internal CA |
|
|
| `https://photoview.home.arpa` | Photoview | Internal CA |
|
|
| `https://immich.home.arpa` | Immich | Internal CA |
|
|
| `https://ha.home.arpa` | Home Assistant | Internal CA |
|
|
| `https://glances.home.arpa` | Glances on K3s | Internal CA |
|
|
| `https://glances-debian.home.arpa` | Glances on Debian | Internal CA |
|
|
| `https://watch-party.nik4nao.com` | Watch Party on Mac Mini | Let's Encrypt |
|
|
| `https://nik4nao.com` | Portfolio | Let's Encrypt |
|
|
| `http://ca.home.arpa` | Internal CA installer | Plain HTTP |
|
|
|
|
`home.arpa` names are defined explicitly in `values/pihole.yaml` and
|
|
`values/pihole-debian.yaml`; Pi-hole is not configured as a wildcard DNS server.
|
|
|
|
## Bootstrap
|
|
|
|
Install workstation tools:
|
|
|
|
```bash
|
|
pip install ansible
|
|
ansible-galaxy collection install community.general ansible.posix
|
|
```
|
|
|
|
Also install `kubectl`, `helm`, and `kubeseal`. The inventory expects SSH
|
|
access with `~/.ssh/id_ed25519-nik-macbookair`.
|
|
|
|
Bring up hosts and base services:
|
|
|
|
```bash
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
|
|
```
|
|
|
|
Install Argo CD once, then hand control to the app-of-apps:
|
|
|
|
```bash
|
|
helm repo add argo https://argoproj.github.io/argo-helm
|
|
helm repo update
|
|
helm upgrade --install argocd argo/argo-cd \
|
|
--namespace argocd --create-namespace \
|
|
--version 9.4.15 \
|
|
--values argocd/values/argocd.yaml
|
|
|
|
kubectl apply -f manifests/argocd/app-of-apps.yaml
|
|
```
|
|
|
|
After that, normal changes should flow through Git and Argo CD.
|
|
|
|
## Daily Operations
|
|
|
|
Check cluster and sync state:
|
|
|
|
```bash
|
|
kubectl get nodes
|
|
kubectl get pods -A
|
|
kubectl get applications -n argocd
|
|
```
|
|
|
|
Apply Dashy config after editing `config/dashy/conf.yaml`:
|
|
|
|
```bash
|
|
bash manifests/core/apply-dashy-config.sh
|
|
```
|
|
|
|
Patch secondary Pi-hole DNS services after Helm upgrades if needed:
|
|
|
|
```bash
|
|
bash manifests/network/pihole-debian-patch.sh
|
|
```
|
|
|
|
Deploy host-level services:
|
|
|
|
```bash
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
|
|
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
|
|
```
|
|
|
|
## Secrets
|
|
|
|
Do not commit plaintext secrets. Runtime-only scripts read `.env` and create
|
|
Kubernetes Secrets directly. Sealed Secret scripts regenerate committed
|
|
`*-sealed.yaml` resources.
|
|
|
|
Start from `.env.example` and keep the filled `.env` local.
|
|
|
|
Runtime secret scripts:
|
|
|
|
```bash
|
|
bash manifests/cert-manager/porkbun-secret.sh
|
|
bash manifests/authentik/authentik-secret.sh
|
|
bash manifests/authentik/authentik-proxy-secret.sh
|
|
bash manifests/authentik/authentik-gitea-secret.sh
|
|
bash manifests/authentik/authentik-grafana-secret.sh
|
|
bash manifests/home-services/registry-secret.sh
|
|
bash manifests/monitoring/grafana-secret.sh
|
|
bash manifests/network/ddns-secret.sh
|
|
bash manifests/portfolio/registry-secret.sh
|
|
```
|
|
|
|
Sealed Secret regeneration:
|
|
|
|
```bash
|
|
bash manifests/home-services/discord-bot-secret.sh
|
|
bash manifests/home-services/ha-gateway-secret.sh
|
|
bash manifests/media/immich-postgres-secret.sh
|
|
```
|
|
|
|
Some sealed secrets are maintained directly in the repo, including Argo CD OIDC,
|
|
Gitea admin/OIDC, Grafana admin/OIDC, Pi-hole admin, and home-service secrets.
|
|
The host-level Gitea runner reads `GITEA_RUNNER_TOKEN` from the environment when
|
|
running `ansible/playbooks/setup-gitea-runner.yaml`.
|
|
|
|
## Storage
|
|
|
|
K3s `local-path` is used for many app PVCs. Static hostPath PVs are used for
|
|
state that must live on known disks:
|
|
|
|
| Location | Use |
|
|
| --- | --- |
|
|
| `/data/gitea` on `minisforum` | Gitea shared storage |
|
|
| `/data/prometheus` on `minisforum` | Prometheus |
|
|
| `/data/grafana` on `minisforum` | Grafana |
|
|
| `/data/loki` on `minisforum` | Loki |
|
|
| `/mnt/storage` on `debian` | NFS media library and backups |
|
|
|
|
The Debian NFS server exports `/mnt/storage` to `192.168.7.77`.
|
|
|
|
## TLS and Trust
|
|
|
|
`*.home.arpa` certificates use `internal-ca-issuer` from `manifests/cert-manager`.
|
|
The CA installer at `http://ca.home.arpa` serves `ca.crt` and an iOS/macOS
|
|
mobileconfig profile. The `ca-sync` CronJob updates those files from the
|
|
`cert-manager/internal-ca-cert` secret when the CA changes.
|
|
|
|
`*.nik4nao.com` certificates use the Let's Encrypt issuers in
|
|
`manifests/cert-manager/cluster-issuer-letsencrypt.yaml`.
|
|
|
|
## Gotchas
|
|
|
|
- Argo CD Applications mostly set `prune: false`; removing resources from Git may
|
|
require manual cleanup.
|
|
- Gitea uses a manual public `IngressRoute`; the chart ingress is disabled in
|
|
`values/gitea.yaml`.
|
|
- Gitea `ROOT_URL` changes can require deleting the generated inline config
|
|
secret before reconciling.
|
|
- Pi-hole does not provide wildcard DNS here; add each new internal hostname to
|
|
both Pi-hole values files.
|
|
- Secondary Pi-hole external IPs can be lost during chart upgrades; rerun
|
|
`manifests/network/pihole-debian-patch.sh`.
|
|
- Authentik forward-auth depends on the `Cookie` header in
|
|
`authRequestHeaders`; removing it causes redirect loops.
|
|
- CoreDNS custom zone snippets must use the `.server` key suffix.
|
|
- The Dashy manifest contains an empty ConfigMap shell; use
|
|
`manifests/core/apply-dashy-config.sh` to load the real config.
|
|
|
|
See the scoped READMEs in `ansible/`, `argocd/`, and `manifests/` for workflows
|
|
specific to those directories.
|