# homelab

Infrastructure-as-code for a small K3s homelab. Host setup is handled with
Ansible, cluster state is reconciled by Argo CD, and service configuration lives
in Kubernetes manifests plus Helm values.

## Current Architecture

| Host | IP | Role |
| --- | --- | --- |
| `minisforum` | `192.168.7.77` | K3s server, Traefik entrypoint, primary app node |
| `debian` / `nik-debian` | `192.168.7.183` | K3s agent, NFS storage, secondary Pi-hole |
| `mac-mini` | `192.168.7.96` | Standalone services such as Watch Party and Ollama |

The cluster uses Traefik instead of the bundled K3s ingress controller. Internal
services are published under `home.arpa` with certificates from an internal CA.
Public services under `nik4nao.com` use Let's Encrypt.

## Repository Layout

| Path | Purpose |
| --- | --- |
| `ansible/` | Host bootstrap and non-Kubernetes services |
| `argocd/apps/` | Argo CD `Application` objects for Helm charts and manifest directories |
| `argocd/values/` | Helm values used to install or reconcile Argo CD itself |
| `config/` | App configuration that is injected into manifests, currently Dashy |
| `manifests/` | Raw Kubernetes resources grouped by service area |
| `values/` | Helm values consumed by Argo CD Applications |

## Managed Services

| Area | Services |
| --- | --- |
| GitOps | Argo CD and app-of-apps |
| Networking | Traefik, Pi-hole primary and secondary, DDNS, CoreDNS custom forwarding |
| TLS | cert-manager, internal CA issuer, Let's Encrypt issuers, CA installer page |
| Identity | Authentik, Traefik forward-auth middleware, OIDC integrations |
| Observability | kube-prometheus-stack, Grafana, Loki, Tempo, OpenTelemetry Collector, Glances |
| Git and CI | Gitea, Gitea Actions runner, registry pull secrets, Gitea backup CronJob |
| Media | Jellyfin, qBittorrent, JDownloader, Immich |
| Home services | Home Assistant ingress, HA gateway, AI gateway, Discord bot |
| Public apps | Portfolio, Watch Party ingress to the Mac Mini |
| Dashboard | Dashy |

## Important URLs

| URL | Service | Certificate |
| --- | --- | --- |
| `https://argocd.home.arpa` | Argo CD | Internal CA |
| `https://auth.home.arpa` | Authentik internal | Internal CA |
| `https://auth.nik4nao.com` | Authentik public | Let's Encrypt |
| `https://traefik.home.arpa` | Traefik dashboard | Internal CA |
| `https://grafana.nik4nao.com` | Grafana | Let's Encrypt |
| `https://gitea.nik4nao.com` | Gitea | Let's Encrypt |
| `https://pihole.home.arpa` | Pi-hole | Internal CA |
| `https://dashy.home.arpa` | Dashy | Internal CA |
| `https://jellyfin.home.arpa` | Jellyfin | Internal CA |
| `https://qbittorrent.home.arpa` | qBittorrent | Internal CA |
| `https://jdownloader.home.arpa` | JDownloader | Internal CA |
| `https://immich.home.arpa` | Immich | Internal CA |
| `https://ha.home.arpa` | Home Assistant | Internal CA |
| `https://glances.home.arpa` | Glances on K3s | Internal CA |
| `https://glances-debian.home.arpa` | Glances on Debian | Internal CA |
| `https://watch-party.nik4nao.com` | Watch Party on Mac Mini | Let's Encrypt |
| `https://nik4nao.com` | Portfolio | Let's Encrypt |
| `http://ca.home.arpa` | Internal CA installer | Plain HTTP |

`home.arpa` names are defined explicitly in `values/pihole.yaml` and
`values/pihole-debian.yaml`; Pi-hole is not configured as a wildcard DNS server.

## Bootstrap

Install workstation tools:

```bash
pip install ansible
ansible-galaxy collection install community.general ansible.posix
```

Also install `kubectl`, `helm`, and `kubeseal`. The inventory expects SSH
access with `~/.ssh/id_ed25519-nik-macbookair`.

Bring up hosts and base services:

```bash
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
```

Install Argo CD once, then hand control to the app-of-apps:

```bash
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm upgrade --install argocd argo/argo-cd \
  --namespace argocd --create-namespace \
  --version 9.4.15 \
  --values argocd/values/argocd.yaml

kubectl apply -f manifests/argocd/app-of-apps.yaml
```

After that, normal changes should flow through Git and Argo CD.

## Daily Operations

Check cluster and sync state:

```bash
kubectl get nodes
kubectl get pods -A
kubectl get applications -n argocd
```

Apply Dashy config after editing `config/dashy/conf.yaml`:

```bash
bash manifests/core/apply-dashy-config.sh
```

Patch secondary Pi-hole DNS services after Helm upgrades if needed:

```bash
bash manifests/network/pihole-debian-patch.sh
```

Deploy host-level services:

```bash
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
```

## Secrets

Do not commit plaintext secrets. Runtime-only scripts read `.env` and create
Kubernetes Secrets directly. Sealed Secret scripts regenerate committed
`*-sealed.yaml` resources.

Start from `.env.example` and keep the filled `.env` local.

Runtime secret scripts:

```bash
bash manifests/cert-manager/porkbun-secret.sh
bash manifests/authentik/authentik-secret.sh
bash manifests/authentik/authentik-proxy-secret.sh
bash manifests/authentik/authentik-gitea-secret.sh
bash manifests/authentik/authentik-grafana-secret.sh
bash manifests/home-services/registry-secret.sh
bash manifests/monitoring/grafana-secret.sh
bash manifests/network/ddns-secret.sh
bash manifests/portfolio/registry-secret.sh
```

Sealed Secret regeneration:

```bash
bash manifests/home-services/discord-bot-secret.sh
bash manifests/home-services/ha-gateway-secret.sh
bash manifests/media/immich-postgres-secret.sh
```

Some sealed secrets are maintained directly in the repo, including Argo CD OIDC,
Gitea admin/OIDC, Grafana admin/OIDC, Pi-hole admin, and home-service secrets.
The host-level Gitea runner reads `GITEA_RUNNER_TOKEN` from the environment when
running `ansible/playbooks/setup-gitea-runner.yaml`.

## Storage

K3s `local-path` is used for many app PVCs. Static hostPath PVs are used for
state that must live on known disks:

| Location | Use |
| --- | --- |
| `/data/gitea` on `minisforum` | Gitea shared storage |
| `/data/prometheus` on `minisforum` | Prometheus |
| `/data/grafana` on `minisforum` | Grafana |
| `/data/loki` on `minisforum` | Loki |
| `/mnt/storage` on `debian` | NFS media library and backups |

The Debian NFS server exports `/mnt/storage` to `192.168.7.77`.

## TLS and Trust

`*.home.arpa` certificates use `internal-ca-issuer` from `manifests/cert-manager`.
The CA installer at `http://ca.home.arpa` serves `ca.crt` and an iOS/macOS
mobileconfig profile. The `ca-sync` CronJob updates those files from the
`cert-manager/internal-ca-cert` secret when the CA changes.

`*.nik4nao.com` certificates use the Let's Encrypt issuers in
`manifests/cert-manager/cluster-issuer-letsencrypt.yaml`.

## Gotchas

- Argo CD Applications mostly set `prune: false`; removing resources from Git may
  require manual cleanup.
- Gitea uses a manual public `IngressRoute`; the chart ingress is disabled in
  `values/gitea.yaml`.
- Gitea `ROOT_URL` changes can require deleting the generated inline config
  secret before reconciling.
- Pi-hole does not provide wildcard DNS here; add each new internal hostname to
  both Pi-hole values files.
- Secondary Pi-hole external IPs can be lost during chart upgrades; rerun
  `manifests/network/pihole-debian-patch.sh`.
- Authentik forward-auth depends on the `Cookie` header in
  `authRequestHeaders`; removing it causes redirect loops.
- CoreDNS custom zone snippets must use the `.server` key suffix.
- The Dashy manifest contains an empty ConfigMap shell; use
  `manifests/core/apply-dashy-config.sh` to load the real config.

See the scoped READMEs in `ansible/`, `argocd/`, and `manifests/` for workflows
specific to those directories.