nik/homelab
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
homelab/README.md
Nik Afiq 5a00f5767b fix: Ansible cleanup -- merge dual config, pin collections, align K3s version, fix bugs 
Stage 4 of REFACTOR_PLAN.md.

- Merge ansible/ansible.cfg into root ansible.cfg (single source of truth);
  the dual-config setup silently broke documented commands and lost
  host_key_checking=False when run from the ansible/ directory.
- Add ansible/requirements.yml pinning community.general, ansible.posix,
  community.docker -- previously undocumented deps of the glances/watch-party
  roles that would fail a fresh bootstrap.
- Align K3s version to v1.32.4+k3s1 across roles/k3s-server, roles/k3s-agent,
  and host_vars/gpu-node.yaml defaults (was skewed: .2 vs .4). This only
  changes what a *future* provision installs -- minisforum/debian are still
  live on v1.32.2+k3s1 until separately upgraded.
- Fix kubeconfig fetch/replace path mismatch in k3s-server role: the `fetch`
  task (flat: true) writes to ~/.kube/config, but `replace` was targeting a
  /tmp/k3s-minisforum.yaml nothing creates -- would break a fresh rebuild.
- gitea-runner: only remove /run/docker.sock when it's actually a directory
  (task name implied a check that wasn't there); tighten registration-token
  systemd unit from 0644 to 0600.
- nvidia: stop unconditionally reporting `changed` (and restarting Docker)
  on every run for an idempotent runtime-configure command.
- Gate the K3s join-token debug print and WireGuard client-config/QR display
  behind opt-in vars (k3s_show_token, wireguard_show_client_configs), default
  off -- both were printing real secrets to console on every run.
- Parameterize the docker role for Debian and Ubuntu; homeassistant now
  depends on it (meta/main.yaml) instead of duplicating a Debian-only Docker
  install inline.
- FQCN cleanup across wireguard, homeassistant, and ollama roles/handlers
  (bare module names -> ansible.builtin.*/community.general.*/ansible.posix.*),
  plus a few ansible-lint name-casing/idiom fixes. Handler renames verified
  against their `notify:` call sites so notifications still fire.
- Update ansible/README.md and root README.md: add gpu-node/gpu_workstation
  (4th host, previously undocumented), docker/nvidia roles, setup-gpu-node.yaml,
  homeassistant.yaml, requirements.yml install step; correct the "Legacy"
  homeassistant label (it's the only thing serving ha.home.arpa); correct the
  Gitea ingress/backup-storage doc-drift; flag the Grafana/Loki static-PV
  binding drift discovered via live cluster check.

Verified: all playbooks pass `ansible-playbook --syntax-check`, yamllint clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 18:15:12 +09:00

232 lines
9.2 KiB
Markdown
Raw Blame History

# homelab
Infrastructure-as-code for a small K3s homelab. Host setup is handled with
Ansible, cluster state is reconciled by Argo CD, and service configuration lives
in Kubernetes manifests plus Helm values.
## Current Architecture
| Host | IP | Role |
| --- | --- | --- |
| `minisforum` | `192.168.7.77` | K3s server, Traefik entrypoint, primary app node |
| `debian` / `nik-debian` | `192.168.7.183` | K3s agent, NFS storage, secondary Pi-hole |
| `mac-mini` | `192.168.7.96` | Standalone services such as Watch Party and Ollama |
| `gpu-node` / `nik-gpu` | `192.168.7.98` | K3s agent with NVIDIA GPU passthrough, spot-tainted; runs Ollama directly on the host |
The cluster uses Traefik instead of the bundled K3s ingress controller. Internal
services are published under `home.arpa` with certificates from an internal CA.
Public services under `nik4nao.com` use Let's Encrypt.
## Repository Layout
| Path | Purpose |
| --- | --- |
| `ansible/` | Host bootstrap and non-Kubernetes services |
| `argocd/apps/` | Argo CD `Application` objects for Helm charts and manifest directories |
| `argocd/values/` | Helm values used to install or reconcile Argo CD itself |
| `config/` | App configuration that is injected into manifests, currently Dashy |
| `manifests/` | Raw Kubernetes resources grouped by service area |
| `values/` | Helm values consumed by Argo CD Applications |
## Managed Services
| Area | Services |
| --- | --- |
| GitOps | Argo CD and app-of-apps |
| Networking | Traefik, Pi-hole primary and secondary, DDNS, CoreDNS custom forwarding |
| TLS | cert-manager, internal CA issuer, Let's Encrypt issuers, CA installer page |
| Identity | Authentik, Traefik forward-auth middleware, OIDC integrations |
| Observability | kube-prometheus-stack, Grafana, Loki, Tempo, OpenTelemetry Collector, Glances |
| Git and CI | Gitea, Gitea Actions runner, registry pull secrets, Gitea backup CronJob |
| Media | Jellyfin, qBittorrent, JDownloader, Immich |
| Home services | Home Assistant ingress, HA gateway, AI gateway, Discord bot |
| Public apps | Portfolio, Watch Party ingress to the Mac Mini |
| Dashboard | Dashy |
## Important URLs
| URL | Service | Certificate |
| --- | --- | --- |
| `https://argocd.home.arpa` | Argo CD | Internal CA |
| `https://auth.home.arpa` | Authentik internal | Internal CA |
| `https://auth.nik4nao.com` | Authentik public | Let's Encrypt |
| `https://traefik.home.arpa` | Traefik dashboard | Internal CA |
| `https://grafana.nik4nao.com` | Grafana | Let's Encrypt |
| `https://gitea.nik4nao.com` | Gitea | Let's Encrypt |
| `https://pihole.home.arpa` | Pi-hole | Internal CA |
| `https://dashy.home.arpa` | Dashy | Internal CA |
| `https://jellyfin.home.arpa` | Jellyfin | Internal CA |
| `https://qbittorrent.home.arpa` | qBittorrent | Internal CA |
| `https://jdownloader.home.arpa` | JDownloader | Internal CA |
| `https://immich.home.arpa` | Immich | Internal CA |
| `https://ha.home.arpa` | Home Assistant | Internal CA |
| `https://glances.home.arpa` | Glances on K3s | Internal CA |
| `https://glances-debian.home.arpa` | Glances on Debian | Internal CA |
| `https://watch-party.nik4nao.com` | Watch Party on Mac Mini | Let's Encrypt |
| `https://nik4nao.com` | Portfolio | Let's Encrypt |
| `http://ca.home.arpa` | Internal CA installer | Plain HTTP |
`home.arpa` names are defined explicitly in `values/pihole.yaml` and
`values/pihole-debian.yaml`; Pi-hole is not configured as a wildcard DNS server.
## Bootstrap
Install workstation tools:
```bash
pip install ansible
ansible-galaxy collection install community.general ansible.posix
```
Also install `kubectl`, `helm`, and `kubeseal`. The inventory expects SSH
access with `~/.ssh/id_ed25519-nik-macbookair`.
Bring up hosts and base services:
```bash
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
```
Install Argo CD once, then hand control to the app-of-apps:
```bash
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm upgrade --install argocd argo/argo-cd \
--namespace argocd --create-namespace \
--version 9.4.15 \
--values argocd/values/argocd.yaml
kubectl apply -f manifests/argocd/app-of-apps.yaml
```
After that, normal changes should flow through Git and Argo CD.
## Daily Operations
Check cluster and sync state:
```bash
kubectl get nodes
kubectl get pods -A
kubectl get applications -n argocd
```
Apply Dashy config after editing `config/dashy/conf.yaml`:
```bash
bash manifests/core/apply-dashy-config.sh
```
Patch secondary Pi-hole DNS services after Helm upgrades if needed:
```bash
bash manifests/network/pihole-debian-patch.sh
```
Deploy host-level services:
```bash
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
```
## Secrets
Do not commit plaintext secrets. Runtime-only scripts read `.env` and create
Kubernetes Secrets directly. Sealed Secret scripts regenerate committed
`*-sealed.yaml` resources.
Start from `.env.example` and keep the filled `.env` local.
Runtime secret scripts:
```bash
bash manifests/cert-manager/porkbun-secret.sh
bash manifests/authentik/authentik-secret.sh
bash manifests/authentik/authentik-proxy-secret.sh
bash manifests/authentik/authentik-gitea-secret.sh
bash manifests/authentik/authentik-grafana-secret.sh
bash manifests/home-services/registry-secret.sh
bash manifests/monitoring/grafana-secret.sh
bash manifests/network/ddns-secret.sh
bash manifests/portfolio/registry-secret.sh
```
Sealed Secret regeneration:
```bash
bash manifests/home-services/discord-bot-secret.sh
bash manifests/home-services/ha-gateway-secret.sh
bash manifests/media/immich-postgres-secret.sh
```
Some sealed secrets are maintained directly in the repo, including Argo CD OIDC,
Gitea admin/OIDC, Grafana admin/OIDC, Pi-hole admin, and home-service secrets.
The host-level Gitea runner reads `GITEA_RUNNER_TOKEN` from the environment when
running `ansible/playbooks/setup-gitea-runner.yaml`.
## Storage
K3s `local-path` is used for many app PVCs. Static hostPath PVs are used for
state that must live on known disks:
| Location | Use |
| --- | --- |
| `/data/gitea` on `minisforum` | Gitea shared storage |
| `/data/prometheus` on `minisforum` | Prometheus (bound correctly) |
| `/data/grafana` on `minisforum` | Static PV defined for Grafana, but **not currently bound** — see gotcha below |
| `/data/loki` on `minisforum` | Static PV defined for Loki, but **not currently bound** — see gotcha below |
| `/mnt/storage` on `debian` | NFS media library |
| `/home/nik/backups` on `debian` | NFS export for Gitea's backup CronJob (separate from `/mnt/storage`) |
The Debian NFS server exports both `/mnt/storage` and `/home/nik/backups` to
`192.168.7.77`.
## TLS and Trust
`*.home.arpa` certificates use `internal-ca-issuer` from `manifests/cert-manager`.
The CA installer at `http://ca.home.arpa` serves `ca.crt` and an iOS/macOS
mobileconfig profile. The `ca-sync` CronJob updates those files from the
`cert-manager/internal-ca-cert` secret when the CA changes.
`*.nik4nao.com` certificates use the Let's Encrypt issuers in
`manifests/cert-manager/cluster-issuer-letsencrypt.yaml`.
## Gotchas
- Argo CD Applications mostly set `prune: false`; removing resources from Git may
require manual cleanup.
- Gitea uses a manual public `IngressRoute`; `values/gitea.yaml` has no
`ingress:` key at all, so the chart's own ingress is off by chart default,
not an explicit setting.
- Grafana and Loki's static hostPath PVs (`grafana-pv`, `loki-pv` in
`manifests/monitoring/monitoring-pvs.yaml`) are currently unbound — their
Helm-managed PVCs got dynamically provisioned via the `local-path`
StorageClass instead (confirmed live via `kubectl get pv/pvc -n
monitoring`), unlike Prometheus which binds `prometheus-pv` correctly. Data
is not lost, just not on the disk the docs/manifest imply — needs a
deliberate decision (bind properly with a data migration, or drop the
orphaned static PVs and document reality) before relying on `/data/grafana`
or `/data/loki` for backups/DR.
- Gitea `ROOT_URL` changes can require deleting the generated inline config
secret before reconciling.
- Pi-hole does not provide wildcard DNS here; add each new internal hostname to
both Pi-hole values files.
- Secondary Pi-hole external IPs can be lost during chart upgrades; rerun
`manifests/network/pihole-debian-patch.sh`.
- Authentik forward-auth depends on the `Cookie` header in
`authRequestHeaders`; removing it causes redirect loops.
- CoreDNS custom zone snippets must use the `.server` key suffix.
- The Dashy manifest contains an empty ConfigMap shell; use
`manifests/core/apply-dashy-config.sh` to load the real config.
See the scoped READMEs in `ansible/`, `argocd/`, and `manifests/` for workflows
specific to those directories.
Reference in New Issue View Git Blame Copy Permalink