# homelab

Infrastructure-as-Code for a 3-machine homelab running K3s.

## Architecture

| Machine | IP | Role |
|---|---|---|
| Minisforum UM780 XTX | `192.168.7.77` | K3s control-plane |
| nik-debian (HP ProDesk) | `192.168.7.183` | K3s storage agent |
| Mac Mini M2 | `192.168.7.96` | Standalone Docker (outside cluster) |

## Roadmap

| Phase | Description | Status |
|---|---|---|
| P0 | CA web installer (`ca.home.arpa`) | ✅ Done |
| P1 | Prometheus + Grafana + Loki | ✅ Done |
| P2 | Authentik SSO | ✅ Done |
| P3 | Photoview | ✅ Done |
| P4 | Home Assistant | 🔜 Planned |
| P5 | Portfolio site (`nik4nao.com`) | ✅ Done |
| P6 | WireGuard split tunnel | 🔜 Planned |

## Live Services

| Service | URL | TLS | Notes |
|---|---|---|---|
| Traefik dashboard | `https://traefik.home.arpa` | Internal CA | Protected by Authentik |
| Authentik | `https://auth.home.arpa` / `https://auth.nik4nao.com` | Internal CA / Let's Encrypt | SSO for all services |
| Gitea | `https://gitea.nik4nao.com` | Let's Encrypt | Git + Docker registry |
| Pi-hole (primary) | `https://pihole.home.arpa` | Internal CA | DNS, runs on Minisforum |
| Pi-hole (secondary) | — | — | externalIPs on Debian (`192.168.7.183`) |
| Grafana | `https://grafana.nik4nao.com` | Let's Encrypt | Protected by Authentik OIDC |
| Prometheus | internal | — | kube-prometheus-stack |
| Loki + Promtail | internal | — | Log aggregation |
| Jellyfin | `https://jellyfin.home.arpa` | Internal CA | Media server, NFS storage |
| qBittorrent | `https://qbittorrent.home.arpa` | Internal CA | `/mnt/storage/torrents` |
| JDownloader | `https://jdownloader.home.arpa` | Internal CA | `/mnt/storage/dl` |
| Dashy | `https://dashy.home.arpa` | Internal CA | Config via ConfigMap |
| Glances | `https://glances.home.arpa` | Internal CA | DaemonSet + Debian Docker |
| Photoview | `https://photoview.home.arpa` | Internal CA | NFS photo gallery |
| Watch Party | `https://watch-party.nik4nao.com` | Let's Encrypt | Mac Mini, CI/CD deployed |
| Portfolio | `https://nik4nao.com` | Let's Encrypt | Hugo + terminal theme |
| CA installer | `http://ca.home.arpa` | — | Internal CA cert download page |

## Auth

- **SSO:** Authentik at `auth.home.arpa` (internal) / `auth.nik4nao.com` (public)
- **Protected services:** Traefik dashboard, Grafana (OIDC), Gitea (OIDC)
- **MFA:** TOTP enforced, 8hr sessions
- **Users:** `nik` (admin), `akadmin` (break-glass)
- **Gitea:** local login disabled, OIDC only

## TLS

- **`*.home.arpa`** — internal self-signed CA (cert-manager). Install CA cert via `http://ca.home.arpa`
- **`*.nik4nao.com`** — Let's Encrypt via HTTP-01 (Traefik)

## Repo Structure

```
ansible/
  inventory.yaml
  playbooks/
    bootstrap-minisforum.yaml   # OS hardening, packages, UFW, /data dirs
    deploy-watch-party.yaml     # deploy watch-party app on Mac Mini
    join-debian-agent.yaml      # join Debian as K3s agent
    setup-gitea-runner.yaml     # Gitea Actions runner (act_runner systemd)
    setup-glances-debian.yaml   # Glances on Debian host
    setup-k3s.yaml              # K3s server install, Helm, kubeconfig
    setup-monitoring.yaml       # Prometheus + Grafana + Loki stack
    setup-nfs-debian.yaml       # NFS server on Debian
  roles/
    common/                     # user, SSH hardening, UFW, base packages
    gitea-runner/               # act_runner v0.2.11 systemd service
    glances/                    # Glances system monitor
    k3s-agent/                  # K3s agent node join
    k3s-server/                 # K3s server install + Helm
    monitoring/                 # Prometheus/Grafana stack
    nfs-server/                 # NFS server configuration
    watch-party/                # Watch Party Docker Compose on Mac Mini
config/
  dashy/conf.yaml               # Dashy dashboard config (applied via ConfigMap)
manifests/
  authentik/                    # Ingress, middleware, proxy outpost, secrets
  cert-manager/                 # ClusterIssuers, porkbun-secret.sh
  core/                         # Dashy, Glances, CA installer, CoreDNS config
  gitea/                        # PV, runner, backup CronJob, public ingress
  media/                        # Jellyfin, qBittorrent, JDownloader, Photoview
  monitoring/                   # Grafana/Loki datasource ConfigMap, PVs, grafana-secret.sh
  network/                      # DDNS CronJob, Traefik dashboard, pihole-debian-patch.sh
  portfolio/                    # Portfolio deployment, registry pull secret
values/
  authentik.yaml
  cert-manager.yaml
  gitea.yaml
  kube-prometheus-stack.yaml
  loki-stack.yaml
  pihole.yaml
  pihole-debian.yaml
  traefik.yaml
```

## Prerequisites

- Ansible on workstation: `pip install ansible`
- Ansible collections: `ansible-galaxy collection install community.general ansible.posix`
- SSH key: `~/.ssh/id_ed25519-nik-macbookair`
- kubectl + helm installed

## Connecting

```bash
# SSH
ssh minisforum        # port 430, via ~/.ssh/config
ssh nik-debian        # port 22

# Kubectl
export KUBECONFIG=/tmp/k3s-minisforum.yaml
kubectl get nodes
kubectl get pods -A
```

## Deploying / Re-deploying

### Ansible (host-level)

```bash
# Bootstrap Minisforum
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforum.yaml

# K3s server
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml

# Join Debian as agent
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml

# NFS on Debian
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml

# Gitea Actions runner
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gitea-runner.yaml

# Glances on Debian
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debian.yaml

# Watch Party on Mac Mini
ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
```

### Helm (cluster services)

```bash
# Traefik
helm repo add traefik https://helm.traefik.io/traefik && helm repo update
helm upgrade --install traefik traefik/traefik \
  --namespace traefik --create-namespace -f values/traefik.yaml

# cert-manager
helm repo add jetstack https://charts.jetstack.io && helm repo update
helm upgrade --install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace -f values/cert-manager.yaml

# Gitea
helm repo add gitea-charts https://dl.gitea.com/charts/ && helm repo update
helm upgrade --install gitea gitea-charts/gitea \
  --namespace gitea --create-namespace -f values/gitea.yaml

# Pi-hole (Minisforum)
helm repo add mojo2600 https://mojo2600.github.io/pihole-kubernetes/ && helm repo update
helm upgrade --install pihole mojo2600/pihole \
  --namespace pihole --create-namespace -f values/pihole.yaml
# Note: re-run manifests/network/pihole-debian-patch.sh after every Pi-hole upgrade
#       (externalIPs for Debian secondary are lost on upgrade)

# Authentik
helm repo add authentik https://charts.goauthentik.io && helm repo update
helm upgrade --install authentik authentik/authentik \
  --namespace authentik --create-namespace -f values/authentik.yaml

# kube-prometheus-stack
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && helm repo update
helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace -f values/kube-prometheus-stack.yaml

# Loki
helm repo add grafana https://grafana.github.io/helm-charts && helm repo update
helm upgrade --install loki grafana/loki-stack \
  --namespace monitoring --create-namespace -f values/loki-stack.yaml
```

### Secrets (create before applying manifests)

```bash
# Porkbun API (cert-manager DNS-01)
bash manifests/cert-manager/porkbun-secret.sh

# Grafana admin password
bash manifests/monitoring/grafana-secret.sh

# DDNS credentials
bash manifests/network/ddns-secret.sh

# Gitea Actions runner token
bash manifests/gitea/runner-secret.sh
```

## Known Gotchas

- **Gitea ROOT_URL:** changing `ROOT_URL` in `values/gitea.yaml` is not enough — must also delete the `gitea-inline-config` secret and re-run `helm upgrade`. Disable the built-in ingress (`ingress.enabled=false`) and use the manual IngressRoute in `manifests/gitea/`.
- **Pihole secondary externalIPs:** lost on every Helm upgrade — re-run `manifests/network/pihole-debian-patch.sh` after each upgrade.
- **Prometheus hostPath:** `/data/prometheus` requires `chmod -R 777` (owned by UID 65534).
- **Grafana PVC:** use `local-path` dynamic provisioning — do not use a static hostPath PV, K3s overrides `storageClassName: ""`.
- **Loki datasource:** provisioned via labeled ConfigMap (`grafana_datasource`), not the Grafana UI — the bundled plugin in loki-stack v2.9.3 is incompatible with Grafana 12.
- **Authentik forwardAuth:** `Cookie` header must be in `authRequestHeaders` in the Traefik middleware or you get an infinite redirect loop after login.
- **Traefik v3 `api@internal`:** requires both `traefik` and `websecure` entrypoints in the IngressRoute, otherwise 404.
- **CoreDNS custom config:** use `.server` suffix for zone blocks. `.override` suffix cannot contain `zone {}` syntax — crashes CoreDNS.
- **Photoview video:** `PHOTOVIEW_DISABLE_VIDEO=true` only takes effect on a fresh scan — delete the SQLite DB and restart before rescanning.
- **CI/CD Buildkit CA:** internal CA must be injected into the `buildx_buildkit_multiarch0` container on every CI run (does not persist across restarts).
- **Pihole DNS:** no wildcard support — every new `home.arpa` subdomain needs an explicit entry in `values/pihole.yaml`.