docs: mark all 9 REFACTOR_PLAN.md stages done at the file level

Everything is committed locally on main; nothing has been pushed. See the
session summary for the specific items that still need your input or action
before any of this goes live (secret rotation values, Authentik chart
version, push go-ahead, live storage/Grafana decisions, router archive review).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.ansible-lint

@@ -0,0 +1,12 @@
+# Config for: ansible-lint
+# Applied by: ansible-lint (run from repo root)
+exclude_paths:
+  - .cache/
+  - manifests/
+  - argocd/
+  - values/
+
+use_default_rules: true
+
+skip_list:
+  - yaml[line-length]

.claude/agents/secrets-leak-scanner.md

@@ -1,6 +1,6 @@
 ---
 name: secrets-leak-scanner
-description: Scans staged/diffed files in this homelab repo for plaintext secrets that should instead come from .env or be sealed via kubeseal. Use before committing changes to manifests, Ansible vars, or Helm values.
+description: Scans staged/diffed files in this homelab repo for plaintext secrets that should instead come from .env or be sealed via kubeseal. Use before committing changes to manifests, Ansible vars, Helm values, or config/**.
 tools: Read, Grep, Glob, Bash
 model: sonnet
 ---
@@ -9,6 +9,12 @@ You scan changes in this repo for secrets that are about to be committed in
 plaintext. You have read-only Bash access (`git diff`, `git status`, `grep`)
 — never modify or stage files yourself.
 
+Scope explicitly includes `config/**` (e.g. `config/dashy/conf.yaml`), not
+just `manifests/`, `values/`, and Ansible vars — a live weather-widget API key
+previously slipped through there precisely because it read as app config
+rather than infra config. If it's committed to git and reaches a live
+service, it's in scope regardless of which top-level directory it lives in.
+
 ## What "should never be plaintext in git" looks like here
 
 Cross-reference `.env.example` for the full list of secret-shaped variable

.env.example

@@ -38,5 +38,11 @@ SWITCHBOT_SECRET=your_switchbot_secret_here
 # Immich database credentials
 IMMICH_POSTGRES_PASSWORD=your_password_here
 
+# Gitea database credentials (rotated off the plaintext value formerly in values/gitea.yaml)
+GITEA_POSTGRES_PASSWORD=your_password_here
+
+# Dashy weather widget API key (rotated off the plaintext value formerly in config/dashy/conf.yaml)
+DASHY_WEATHER_API_KEY=your_api_key_here
+
 PIA_USER=your_pia_username_here
 PIA_PASSWORD=your_pia_password_here

.gitea/workflows/validate.yaml

@@ -0,0 +1,44 @@
+# Config for: Gitea Actions CI
+# Applied by: the self-hosted act_runner (ansible/roles/gitea-runner) on push
+# Description: Read-only lint/validate pass -- no cluster access, no apply/deploy.
+name: validate
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install validation tools
+        run: |
+          pip install yamllint ansible-lint
+          curl -L https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz \
+            | tar xz kubeconform
+          sudo mv kubeconform /usr/local/bin/
+
+      - name: yamllint
+        run: yamllint -c .yamllint.yml .
+
+      - name: ansible-lint
+        run: |
+          ansible-galaxy collection install -r ansible/requirements.yml
+          ansible-lint ansible/
+
+      - name: ansible-playbook --syntax-check
+        run: |
+          for pb in ansible/playbooks/*.yaml; do
+            ansible-playbook --syntax-check -i ansible/inventory.yaml "$pb"
+          done
+
+      - name: kubeconform
+        run: |
+          kubeconform -summary -ignore-missing-schemas -kubernetes-version 1.32.0 \
+            -schema-location default \
+            -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
+            $(find manifests -name "*.yaml" -not -name "*-sealed.yaml") argocd/apps/*.yaml

.gitignore

@@ -1,4 +1,11 @@
 .env
 .DS_Store
 tmp/
-old.debian-data
+old.debian-data
+*.retry
+.vault_pass*
+kubeconfig*
+*.kubeconfig
+*.swp
+*.swo
+*~

.yamllint.yml

@@ -0,0 +1,25 @@
+# Config for: yamllint
+# Applied by: yamllint -c .yamllint.yml .
+extends: default
+
+rules:
+  line-length: disable
+  document-start: disable
+  new-line-at-end-of-file: disable
+  comments-indentation: disable
+  commas:
+    max-spaces-after: -1
+  octal-values:
+    forbid-implicit-octal: true
+    forbid-explicit-octal: true
+  truthy:
+    allowed-values: ["true", "false", "on"]
+  comments:
+    min-spaces-from-content: 1
+  braces:
+    max-spaces-inside: 1
+  brackets:
+    max-spaces-inside: 1
+
+ignore: |
+  router/

CLAUDE.md

@@ -107,6 +107,28 @@ Never commit plaintext secrets. Two patterns coexist, both listed in
 `ansible/group_vars/all/vault.yaml` holds Ansible-side secrets (e.g.
 `vault_k3s_node_token`) referenced from `host_vars`.
 
+## Validation commands
+
+No cluster access is required for any of these — run them before reporting a
+change done, and show the actual output, not just a claim it passed.
+
+```bash
+yamllint -c .yamllint.yml .
+ansible-lint ansible/
+for pb in ansible/playbooks/*.yaml; do
+  ansible-playbook --syntax-check -i ansible/inventory.yaml "$pb"
+done
+kubeconform -summary -ignore-missing-schemas -kubernetes-version 1.32.0 \
+  -schema-location default \
+  -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
+  $(find manifests -name "*.yaml" -not -name "*-sealed.yaml") argocd/apps/*.yaml
+```
+
+`ansible-lint` needs the collections in `ansible/requirements.yml` installed
+first (`ansible-galaxy collection install -r ansible/requirements.yml`),
+otherwise it reports spurious `unknown-module` errors for `community.general`/
+`community.docker` modules that are actually fine.
+
 ## Key gotchas (see README.md "Gotchas" for the full list)
 
 - Pi-hole has no wildcard DNS — add every new `home.arpa` hostname to both

README.md

@@ -11,6 +11,7 @@ in Kubernetes manifests plus Helm values.
 | `minisforum` | `192.168.7.77` | K3s server, Traefik entrypoint, primary app node |
 | `debian` / `nik-debian` | `192.168.7.183` | K3s agent, NFS storage, secondary Pi-hole |
 | `mac-mini` | `192.168.7.96` | Standalone services such as Watch Party and Ollama |
+| `gpu-node` / `nik-gpu` | `192.168.7.98` | K3s agent with NVIDIA GPU passthrough, spot-tainted; runs Ollama directly on the host |
 
 The cluster uses Traefik instead of the bundled K3s ingress controller. Internal
 services are published under `home.arpa` with certificates from an internal CA.
@@ -74,7 +75,7 @@ Install workstation tools:
 
 ```bash
 pip install ansible
-ansible-galaxy collection install community.general ansible.posix
+ansible-galaxy collection install -r ansible/requirements.yml
 ```
 
 Also install `kubectl`, `helm`, and `kubeseal`. The inventory expects SSH
@@ -87,6 +88,7 @@ ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforu
 ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
 ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
 ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
+ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K
 ```
 
 Install Argo CD once, then hand control to the app-of-apps:
@@ -102,7 +104,39 @@ helm upgrade --install argocd argo/argo-cd \
 kubectl apply -f manifests/argocd/app-of-apps.yaml
 ```
 
-After that, normal changes should flow through Git and Argo CD.
+The app-of-apps now brings up cert-manager's `ClusterIssuer`s (via
+`cert-manager-config`) and Authentik's ingress/proxy-outpost/middleware (via
+`authentik-config`) automatically — these used to require untracked manual
+`kubectl apply` steps that weren't documented anywhere. **Authentik's Helm
+chart itself is still a one-time manual install**, since its Argo CD
+Application (`argocd/apps/authentik.yaml`) is deliberately left on manual sync
+with a `targetRevision` placeholder pending a chart-version decision:
+
+```bash
+helm repo add authentik https://charts.goauthentik.io
+helm repo update
+helm upgrade --install authentik authentik/authentik \
+  -f values/authentik.yaml -n authentik --create-namespace
+```
+
+Populate secrets (see "Secrets" below) before Authentik, Gitea, Grafana, or
+the other services that depend on them will come up healthy. After that,
+normal changes should flow through Git and Argo CD.
+
+### Cold-rebuild order
+
+Rebuilding from nothing, the dependency order that actually matters is:
+
+1. Ansible playbooks above (hosts, K3s, NFS).
+2. `helm install argocd` + `kubectl apply -f manifests/argocd/app-of-apps.yaml`
+   — this alone now brings up cert-manager, sealed-secrets, and their
+   `ClusterIssuer`s/CA cert via sync-wave ordering.
+3. Runtime secret scripts and Sealed Secret regeneration (see "Secrets") —
+   several Applications (Gitea, Grafana, home-services, Authentik) will sit
+   degraded/crash-looping until their secrets exist.
+4. Manual Authentik Helm install (above) — every OAuth-gated service
+   (Grafana, Gitea, Argo CD SSO, Traefik dashboard) depends on it.
+5. Everything else reconciles from Git on its own from here.
 
 ## Daily Operations
 
@@ -179,12 +213,14 @@ state that must live on known disks:
 | Location | Use |
 | --- | --- |
 | `/data/gitea` on `minisforum` | Gitea shared storage |
-| `/data/prometheus` on `minisforum` | Prometheus |
-| `/data/grafana` on `minisforum` | Grafana |
-| `/data/loki` on `minisforum` | Loki |
-| `/mnt/storage` on `debian` | NFS media library and backups |
+| `/data/prometheus` on `minisforum` | Prometheus (bound correctly) |
+| `/data/grafana` on `minisforum` | Static PV defined for Grafana, but **not currently bound** — see gotcha below |
+| `/data/loki` on `minisforum` | Static PV defined for Loki, but **not currently bound** — see gotcha below |
+| `/mnt/storage` on `debian` | NFS media library |
+| `/home/nik/backups` on `debian` | NFS export for Gitea's backup CronJob (separate from `/mnt/storage`) |
 
-The Debian NFS server exports `/mnt/storage` to `192.168.7.77`.
+The Debian NFS server exports both `/mnt/storage` and `/home/nik/backups` to
+`192.168.7.77`.
 
 ## TLS and Trust
 
@@ -200,8 +236,18 @@ mobileconfig profile. The `ca-sync` CronJob updates those files from the
 
 - Argo CD Applications mostly set `prune: false`; removing resources from Git may
   require manual cleanup.
-- Gitea uses a manual public `IngressRoute`; the chart ingress is disabled in
-  `values/gitea.yaml`.
+- Gitea uses a manual public `IngressRoute`; `values/gitea.yaml` has no
+  `ingress:` key at all, so the chart's own ingress is off by chart default,
+  not an explicit setting.
+- Grafana and Loki's static hostPath PVs (`grafana-pv`, `loki-pv` in
+  `manifests/monitoring/monitoring-pvs.yaml`) are currently unbound — their
+  Helm-managed PVCs got dynamically provisioned via the `local-path`
+  StorageClass instead (confirmed live via `kubectl get pv/pvc -n
+  monitoring`), unlike Prometheus which binds `prometheus-pv` correctly. Data
+  is not lost, just not on the disk the docs/manifest imply — needs a
+  deliberate decision (bind properly with a data migration, or drop the
+  orphaned static PVs and document reality) before relying on `/data/grafana`
+  or `/data/loki` for backups/DR.
 - Gitea `ROOT_URL` changes can require deleting the generated inline config
   secret before reconciling.
 - Pi-hole does not provide wildcard DNS here; add each new internal hostname to

REFACTOR_PLAN.md

@@ -0,0 +1,300 @@
+# Homelab Repo Audit & Refactor Plan
+
+Living document. Produced by a full read-only audit of the repo (Ansible, Argo
+CD, manifests, values, config, router). Nothing was applied, synced,
+committed, or pushed as part of producing this. We work through the
+"Migration Plan" stages one at a time, referencing this file; update the
+Status column as stages land.
+
+**Audit constraints honored**: `.env` was never opened; `router/backup-base.tar.gz`
+was never extracted; no `kubectl apply` / `helm install` / Argo CD sync /
+Ansible-against-real-hosts was run. All findings are grounded in file
+citations, and the highest-severity ones were independently re-verified
+(not just taken from sub-agent research output) before being recorded here.
+
+## Stage Tracker
+
+| Stage | Scope | Status |
+|---|---|---|
+| 1 | Safety fixes & secret hygiene | Files done, committed locally, **not pushed**. Actual secret rotation (new password/API key values + live DB password change) still needs the user — see note below. |
+| 2 | Validation tooling & clean baseline | Done. `yamllint`/`ansible-lint`/`kubeconform` installed by user; `.yamllint.yml`/`.ansible-lint` added; baseline clean (0 yamllint issues, `kubeconform`: 117+20 resources valid, all playbooks pass `--syntax-check`). Remaining `ansible-lint` findings are either pre-existing role/var naming conventions (out of scope — would require repo-wide renames) or collections-not-installed noise that resolves once `ansible-galaxy collection install -r ansible/requirements.yml` is run. |
+| 3 | Remove confirmed junk | Done, committed locally. `.DS_Store` confirmed never tracked (no action needed). |
+| 4 | Ansible cleanup | Done, committed locally. |
+| 5 | Argo CD bootstrap normalization | Files done, committed locally, **not pushed — highest-risk stage**. `authentik.yaml`'s chart version is a placeholder needing your input (`helm list -n authentik`); every new/changed Application here needs `kubectl diff` one at a time before/after enabling, not a blind batch push. |
+| 6 | Values/DNS consolidation | Done, committed locally, **not pushed**. |
+| 7 | Kubernetes correctness & security | Resource requests/limits + probes done (finding #16), committed locally, **not pushed**. `gitea-backup` RBAC narrowing (#11) done as part of Stage 5's commit. Grafana `runAsNonRoot`/`fsGroup` test (#25) and `:latest` image pinning not done — both need a live test window / registry inspection I didn't do unprompted. |
+| 8 | Documentation & DR runbook | Done, committed locally. |
+| 9 | CI & Claude Code guidance | Done, committed locally, **not pushed** (new CI automation surface). |
+
+**Secret rotation still needs you** (Stage 1): I wired up the `existingSecret`/env-injection plumbing for the Gitea DB password and Dashy API key, but I don't generate or handle the actual new credential values — that's your call per this repo's credential-handling rule. See the session summary for exact steps.
+
+### New findings from this session's live-cluster checks (not in the original audit)
+
+- **Grafana and Loki's static PVs are orphaned.** `manifests/monitoring/monitoring-pvs.yaml` defines `grafana-pv`/`loki-pv` hostPath PVs, but live `kubectl get pv` shows both sitting `Available` (unbound) — their Helm-managed PVCs got dynamically provisioned via `local-path` instead, unlike `prometheus-pv` which binds correctly. Data isn't lost, just not where the README/manifest imply. Needs a decision: bind properly (data migration required) or drop the orphaned static PVs and document reality. Not yet actioned.
+- **Finding #24 (GPU passthrough) is resolved as working, not broken.** Live check on node `nik-gpu`: `nvidia.com/gpu: "1"` allocatable, `nvidia` runtime handler registered in containerd, `nvidia-device-plugin` pod `Running`. The orphaned-looking `k3s-containerd-config.toml.j2` template is a non-issue in practice — no action needed.
+- **Unknown #1 (Grafana PVC) resolved**: dynamically provisioned via `local-path`, not the static PV — see the orphaned-PV finding above, same root cause.
+- Node hostnames in the live cluster are `nik-debian`/`nik-gpu`, not `debian`/`gpu-node` as in `ansible/inventory.yaml` — cosmetic (K3s registers by actual hostname), doesn't affect anything, just noted for anyone cross-referencing `kubectl get nodes` against the docs.
+
+Open decisions needed from the user before/during the relevant stage:
+- Finding #5 (`router/backup-base.tar.gz`): needs manual review outside this
+  workflow; may imply git-history scrub / router credential rotation.
+- Finding #9 (K3s version skew): which version is canonical (`v1.32.2+k3s1`
+  vs `v1.32.4+k3s1`)?
+- Finding #18: is `prune:true` on `home-services`/`otel-collector`/`tempo`
+  and `selfHeal:false` on both pihole Applications intentional policy?
+- Finding #25: Grafana `runAsUser: 0` — worth testing `fsGroup`-only instead?
+
+---
+
+## Phase 1 — Current-State Architecture
+
+```
+Ansible (hosts)                     Argo CD (cluster)
+─────────────────                   ──────────────────────────────────────────
+minisforum  (K3s server, .77)  ──┐
+debian      (K3s agent, .183) ────┼─▶ K3s cluster ──▶ helm install argocd (manual, README.md:92-103)
+gpu-node    (K3s agent, .98)   ──┘         │                    │
+mac-mini    (standalone Docker)            │                    ▼
+                                            │        kubectl apply -f manifests/argocd/app-of-apps.yaml  (manual, one-time)
+                                            │                    │
+                                            │                    ▼
+                                            │        Application: app-of-apps → watches argocd/apps/*.yaml
+                                            │                    │
+                                            │        ┌───────────┴────────────────────────────┐
+                                            │        ▼                                         ▼
+                                            │  Helm-chart Applications                 Raw-manifest Applications
+                                            │  (chart + values/<name>.yaml)             (path: manifests/<area>)
+                                            │  traefik, cert-manager*, gitea,           core, media, home-services,
+                                            │  pihole, pihole-debian, monitoring,       portfolio, homeassistant,
+                                            │  loki, tempo, otel-collector,             *-secrets (sealed only)
+                                            │  sealed-secrets, argocd(self)
+                                            │
+                                            └── NOT reconciled by Argo CD at all:
+                                                manifests/cert-manager/ (ClusterIssuers)
+                                                manifests/authentik/ + values/authentik.yaml
+                                                4 files in manifests/network/ (non-sealed)
+                                                → applied by hand, outside GitOps
+```
+
+Host-level services that never touch the K3s cluster: Home Assistant (Docker
+Compose + systemd on `minisforum`, `ansible/roles/homeassistant/`), Watch
+Party and Ollama (`mac-mini`), Ollama (`gpu-node`) — all Ansible-managed.
+
+Bootstrap sequence as documented (`README.md:71-105`, `argocd/README.md:7-24`):
+Ansible playbooks → manual `helm install argocd` → manual `kubectl apply` of
+app-of-apps → everything else via Git. **This documented sequence is
+incomplete** — it never mentions installing Authentik or the cert-manager
+`ClusterIssuer`s, both required for the rest of the stack to actually work
+(see Critical findings below).
+
+## Executive Summary
+
+The repo is coherently designed for a single-operator homelab: Ansible for
+host bootstrap, Argo CD app-of-apps for cluster reconciliation, a clean
+two-pattern secrets model (sealed-secrets for committed ciphertext, runtime
+scripts for `.env`-sourced live secrets), and a genuinely useful pair of
+existing Claude Code subagents (`infra-change-reviewer`, `secrets-leak-scanner`)
+that already encode a lot of hard-won tribal knowledge. There's no
+fashionable-but-wrong abstraction problem here — the biggest risks are **gaps
+between what the documentation/GitOps model claims and what's actually wired
+up**: two foundational subsystems (cert-manager's ClusterIssuers, and the
+entire Authentik SSO stack) are invisible to Argo CD despite everything else
+depending on them, a live database password and a live third-party API key
+are committed in plaintext, and two high-value internal certificates are
+silently issued by the wrong CA chain. None of this requires a restructure to
+fix — it requires closing specific, identifiable gaps. Ansible is in good
+shape except for a dual-`ansible.cfg` trap that breaks documented commands if
+run from the wrong directory, and one likely-broken kubeconfig-fetch task in
+the K3s bootstrap role. No CI or local validation tooling exists yet, which
+is the main reason several of these gaps went unnoticed.
+
+## Prioritized Findings
+
+### Critical
+
+| # | Finding | Evidence | Impact | Recommendation |
+|---|---|---|---|---|
+| 1 | `manifests/cert-manager/` (both `ClusterIssuer`s + the internal CA `Certificate`) is never referenced by any Argo CD `Application` — confirmed by grep across all of `argocd/apps/*.yaml`, zero `path:` hits | `argocd/apps/cert-manager.yaml` only installs the Helm chart/controller; no sibling Application points at `manifests/cert-manager/` | Every TLS certificate in the cluster depends on issuers that exist only because someone once ran `kubectl apply` by hand. A cold rebuild following the documented bootstrap steps will **not** restore them, and nothing detects/corrects drift on the live ones. | Add `argocd/apps/cert-manager-config.yaml` (path: `manifests/cert-manager`, sync-wave before consumers). See Stage 5. |
+| 2 | `manifests/authentik/` (7 files: ingress, proxy outpost, middleware, 4 secret scripts) and `values/authentik.yaml` are never referenced by any Application, and Authentik's manual `helm upgrade --install` isn't even in the documented bootstrap steps | Confirmed by the same grep sweep; `values/authentik.yaml:1`'s header comment is the *only* place the install command lives | The SSO IdP gating Argo CD, Grafana, and Gitea OAuth is entirely outside GitOps and undocumented as a bootstrap step. Disaster recovery would silently fail to restore login for everything behind it. | Add `argocd/apps/authentik.yaml` (chart) + `argocd/apps/authentik-config.yaml` (path: `manifests/authentik`), mirroring the cert-manager two-Application pattern. |
+| 3 | Plaintext PostgreSQL password committed | `values/gitea.yaml:48` — `password: gitea-db-password` under `postgresql.global.postgresql.auth.password`, no `existingSecret` used (contrast `values/gitea.yaml:11`, which correctly uses `existingSecret: gitea-admin-secret`) | A real credential sits in git history now, readable by anyone with repo access. Contradicts the repo's own stated no-plaintext-secrets rule and `.claude/agents/secrets-leak-scanner.md`'s invariant. | Rotate the DB password, convert to a `*-secret.sh` → sealed-secret pattern (mirror `manifests/media/immich-postgres-secret.sh`). Treat the current value as compromised since it's in history. |
+| 4 | Live third-party API key committed in plaintext | `config/dashy/conf.yaml:63` and `:70` — a weather-widget API key used twice | Anyone with repo read access (or Dashy's own JS, which ships client-side) can extract and reuse the key. | Rotate the key at the provider; inject at apply-time from `.env` via `manifests/core/apply-dashy-config.sh` instead of hardcoding in `config/dashy/conf.yaml`. |
+| 5 | `router/backup-base.tar.gz` is tracked in git (added in commit `e7718ce`, ~8.8KB compressed) | Confirmed tracked via `git ls-files`; **not opened**, per safety constraints | `router/uci-base.conf` itself (which was read) is a sanitized base config — WiFi disabled with default open SSID, no VPN configured, sentinel password field — which makes it likely the fuller `.tar.gz` backup is where any real WiFi PSK/VPN key/admin credential would actually live. Cannot confirm what's inside without extracting it. | **Action needed from user**: inspect this archive outside this workflow. If it contains live credentials, both git-history scrubbing and router credential rotation should be considered. |
+| 6 | Two high-value internal certificates are issued by the wrong CA | `manifests/authentik/authentik-ingress.yaml:11-12` (`auth.home.arpa`) and `manifests/network/traefik-dashboard-ingress.yaml:65-66` (`traefik.home.arpa`) both set `issuerRef.name: internal-ca` — the root self-signed issuer — instead of `internal-ca-issuer`, the chained issuer every other internal `Certificate` uses (verified pattern in `manifests/homeassistant/homeassistant.yaml:31`, `manifests/home-services/certs.yaml`, `manifests/argocd/argocd.yaml:9`, media manifests) | Devices that installed the CA via the `ca-installer`/`ca-sync` flow (`manifests/README.md:36-42`) will **not** trust these two certs — browser TLS warnings on exactly the SSO login and the Traefik dashboard. Compounded by #1/#2: neither file is under GitOps, so nothing corrects this automatically. | Change `issuerRef.name` to `internal-ca-issuer` in both files. |
+
+### High
+
+| # | Finding | Evidence | Impact | Recommendation |
+|---|---|---|---|---|
+| 7 | Dual `ansible.cfg` breaks every documented playbook command if run from `ansible/` instead of repo root — empirically reproduced, not just theorized | `ansible.cfg` (root) vs `ansible/ansible.cfg`; confirmed via `ansible-config dump` that only one loads at a time, and reproduced two independent failures (path-doubling, then role-not-found) plus a silent loss of `host_key_checking=False` | A user who `cd`s into `ansible/` first (a very natural mistake) gets confusing, unrelated-looking errors, or worse, an interactive host-key prompt in what's meant to be scriptable. | Delete `ansible/ansible.cfg`; fold its two settings (`inject_facts_as_vars: False`, `deprecation_warnings: False`) into the root `ansible.cfg`. Single source of truth, run only from repo root (already the documented convention). |
+| 8 | Likely-broken kubeconfig path in K3s bootstrap | `ansible/roles/k3s-server/tasks/main.yaml:50-62` — fetches kubeconfig to `~/.kube/config`, but the next task's `ansible.builtin.replace` targets `/tmp/k3s-minisforum.yaml`, a path nothing else in the role writes to | `replace` errors if the target doesn't exist — this task likely fails on a genuinely fresh `bootstrap-minisforum.yaml`/`setup-k3s.yaml` run, i.e. exactly the disaster-recovery path this needs to work for. **Static-analysis finding, not yet confirmed against a live re-run** — flagged as an unknown too. | Fix the path mismatch (verify what the fetch task actually names the local file and point `replace` at that same path); test on an actual rebuild before trusting this for DR. |
+| 9 | K3s version defined in 3 places, already out of sync | `ansible/roles/k3s-server/defaults/main.yaml:6` and `k3s-agent/defaults/main.yaml:6` both `v1.32.2+k3s1`; `ansible/host_vars/gpu-node.yaml:35` is `v1.32.4+k3s1`. `ansible/README.md:58-59`'s upgrade doc only mentions the first two locations. | A fresh `debian`/`minisforum` (re)provision installs a different K3s version than what's apparently already running on `gpu-node` — real version-skew risk, and the documented upgrade procedure would miss the gpu-node override entirely. | Pick one canonical version, align all three, document all three locations. |
+| 10 | 4 of 6 files in `manifests/network/` are excluded from Argo CD sync | `argocd/apps/network-secrets.yaml:14-15` sets `directory.include: '*-sealed.yaml'`, so `ddns-cronjob.yaml`, `glances-debian-ingress.yaml`, `traefik-dashboard-ingress.yaml`, `watch-party-ingress.yaml` are never synced | These back real, documented services (`README.md:52,63,64`) with zero drift protection or self-heal — same class of gap as #1/#2, smaller blast radius. | Add a proper `network.yaml` Application (or widen the include filter) covering these; keep `network-secrets.yaml` scoped to sealed secrets only. |
+| 11 | `gitea-backup` CronJob has cluster-wide exec privileges it doesn't need | `manifests/gitea/gitea-backup.yaml:11` (`ClusterRole`) / `:23` (`ClusterRoleBinding`) grant `pods/exec` cluster-wide; the CronJob script only execs into the `gitea` namespace (lines 61-72) | The backup ServiceAccount can exec into any pod in any namespace — real privilege-escalation surface for a component whose only job is backing up one namespace. | Replace with a namespaced `Role`/`RoleBinding` scoped to `gitea`. |
+| 12 | `homeassistant` Application/manifest namespace mismatch | `argocd/apps/homeassistant.yaml:14` targets/creates namespace `homeassistant`; every resource in `manifests/homeassistant/homeassistant.yaml` is hardcoded to `namespace: default` (lines 6,17,27,40) — verified directly | Argo CD creates and "manages" an empty, unused `homeassistant` namespace while the real resources live in `default`. Confusing, latent bug, no current functional harm. | Change `destination.namespace` to `default` and drop `CreateNamespace=true` (no runtime-affecting change — resources already live in `default`). |
+| 13 | `gitea-backup` namespace has no `Namespace` object anywhere and isn't auto-created | `manifests/gitea/gitea-backup.yaml:8,29,39,100` all reference namespace `gitea-backup`; the owning Application (`gitea-secrets.yaml`) targets namespace `gitea` with `CreateNamespace=true` — a different namespace | Would fail to apply on any cluster where `gitea-backup` namespace wasn't created by hand out-of-band. | Add an explicit `Namespace: gitea-backup` object to the manifest, or fold the backup CronJob into the `gitea` namespace. |
+| 14 | No CI or local validation tooling anywhere in the repo | Confirmed via recursive search: no `.github/`, `.gitea/workflows/`, pre-commit config, or yamllint/ansible-lint config at any level. `yamllint`, `ansible-lint`, `kubeconform` not installed locally either (only `helm`, `kubectl`, `kubeseal`, `jq`, `ansible-playbook` are present) | Given "no staging environment," every change is validated by pushing and watching Argo CD/production react. This is the root cause that let findings #3, #4, #20 go unnoticed. | See Stage 2. |
+| 15 | Missing `requirements.yml` + undocumented collection dependency | No `requirements.yml` anywhere; `README.md:76-78` only tells a new operator to install `community.general`/`ansible.posix`, but `ansible/roles/glances/tasks/main.yaml:24` and `ansible/roles/watch-party/tasks/main.yaml:27` require `community.docker`, never mentioned | A fresh workstation following the documented bootstrap verbatim gets a "module not found" failure on `setup-glances-debian.yaml`/`deploy-watch-party.yaml`. | Add `ansible/requirements.yml` pinning all three collections; update `README.md`. |
+| 16 | Missing resource requests/limits and probes on a long list of workloads | `manifests/core/{dashy,glances}.yaml`, `manifests/core/ca-installer/ca-installer.yaml`, `manifests/authentik/authentik-proxy-outpost.yaml`, `manifests/media/jellyfin.yaml`, main containers in `qbittorrent.yaml`/`jdownloader.yaml` (only their `gluetun` sidecars have probes), all 4 Deployments in `manifests/media/immich.yaml` | No protection against one workload starving others on a small, fixed-capacity cluster; no automatic restart on hang for several user-facing services. Contrast: `manifests/home-services/*.yaml` and `manifests/portfolio/portfolio.yaml` do this correctly already — good templates to copy from. | See Stage 7. |
+
+### Medium
+
+| # | Finding | Evidence | Recommendation |
+|---|---|---|---|
+| 17 | Sync-wave ordering doesn't cover cert-manager/sealed-secrets vs. their consumers | Only `gitea-secrets`/`gitea`/`loki`/`monitoring-secrets`/`monitoring`/`network-secrets` carry `sync-wave` annotations; `cert-manager` and `sealed-secrets` (whose CRDs everything else's `Certificate`/`SealedSecret` objects need) are unannotated, same wave-0 as their consumers | Add `sync-wave: "-2"` to `cert-manager`/`sealed-secrets`, `-1` to their config Applications, consistent with the existing pattern. Self-heals today via retries, so this is about shortening a fresh-bootstrap flaky window, not fixing active breakage. |
+| 18 | Undocumented sync-policy exceptions | `home-services`, `otel-collector`, `tempo` use `prune: true` against the repo's stated `prune: false` norm (`argocd/README.md:37`); `pihole`/`pihole-debian` are the only two with `selfHeal: false` (plausibly intentional, given the documented external-IP-loss gotcha, but never stated as policy) | Add a one-line YAML comment on each explaining the exception, or normalize them if unintentional — needs user input on intent. |
+| 19 | 3 Applications use `targetRevision: HEAD` instead of the documented `main` | `home-services.yaml:12`, `otel-collector.yaml:17`, `tempo.yaml:17` | Normalize to `main`. Zero behavioral difference today; purely consistency. |
+| 20 | Pi-hole DNS entries have already drifted | `values/pihole.yaml:81,85` (`immich.home.arpa`, `gluetun.home.arpa`) missing from `values/pihole-debian.yaml` | Add the two missing entries. (Going forward this is now caught by the existing `PostToolUse` hook in `.claude/settings.json` — it just doesn't catch pre-existing drift.) |
+| 21 | Orphaned DNS record | `nik4nao.home.arpa` exists in both pihole values files but no Certificate/Ingress/IngressRoute anywhere references it | Confirm it's unused, then remove. |
+| 22 | `ansible/README.md` mislabels the `homeassistant` role "Legacy" | `ansible/README.md:54`; but `manifests/homeassistant/homeassistant.yaml`'s `Endpoints` (hardcoded to `192.168.7.77:8123`) has nothing else in the repo standing up a listener at that address — the "legacy" Ansible role/Docker Compose deployment is the **only** thing actually serving it | Re-word the doc entry; this is load-bearing, not dead — don't let a future cleanup pass delete it. |
+| 23 | `docker` role logic duplicated wholesale inside `homeassistant` role | `ansible/roles/homeassistant/tasks/main.yaml:2-49` reimplements Docker CE install (Debian apt path) instead of reusing `ansible/roles/docker/tasks/main.yaml:6-43` (Ubuntu apt path) | Parameterize `docker` role for both distros, have `homeassistant` depend on it instead of duplicating. |
+| 24 | GPU passthrough likely non-functional as wired | `ansible/roles/nvidia/templates/k3s-containerd-config.toml.j2` exists but is never referenced by any task in that role; `manifests/home-services/nvidia-device-plugin.yaml:5-9` defines a `RuntimeClass handler: nvidia` that depends on exactly that containerd config existing on `gpu-node` | **Unknown pending live verification** — needs a check on `gpu-node` before deciding whether to wire up the template or remove the RuntimeClass. |
+| 25 | Grafana forced to run as root | `values/kube-prometheus-stack.yaml:49-52` — `runAsNonRoot: false`, `runAsUser: 0`, likely to support `initChownData: true` fixing hostPath ownership | Test whether `fsGroup` alone (without full root) suffices; needs a live test window with rollback ready. |
+| 26 | Mixed `Ingress` vs Traefik `IngressRoute` usage with no stated policy | e.g. `manifests/media/qbittorrent.yaml` uses plain `Ingress` for the app (line 176) but `IngressRoute` for `gluetun-api` in the same file (line 216) | Not broken, but only the `IngressRoute` half gets native `Middleware` support without annotation workarounds — worth standardizing on one mechanism over time, not urgent. |
+
+### Low (grouped)
+
+- **Ansible hygiene**: zero `tags:` usage anywhere (no selective runs possible); FQCN usage inconsistent (`wireguard` and `homeassistant` roles are 100% bare module names, `ollama` is mixed — `ansible/roles/{wireguard,homeassistant,ollama}/tasks/main.yaml`); `nvidia/tasks/main.yaml:58-62` uses `changed_when: true` unconditionally, restarting Docker every run; `gitea-runner/tasks/main.yaml:89-93` unconditionally deletes `/run/docker.sock` every run despite the task name implying a conditional check.
+- **Operational secret leaks (not committed, but printed)**: `ansible/roles/k3s-server/tasks/main.yaml:46-48` prints the real K3s join token via `debug: msg:` on every run; `ansible/roles/gitea-runner/tasks/main.yaml:55-79` writes the runner registration token into a world-readable (`0644`) systemd unit file; `ansible/roles/wireguard/tasks/main.yaml:169-184` prints generated WireGuard private keys to console. None are committed to git, but all are worth tightening.
+- **Doc drift**: `README.md:203-204` claims Gitea ingress is "disabled in `values/gitea.yaml`" — that file has no `ingress:` key at all (it's disabled by chart default, not an explicit setting); `README.md:174-186` groups Gitea's backups under `/mnt/storage` when they actually live on a separate NFS export (`/home/nik/backups`, per `manifests/gitea/gitea-backup.yaml:92` and `ansible/roles/nfs-server/templates/exports.j2:7`); `ansible/README.md`'s role table and playbook list omit the `docker`/`nvidia` roles, `setup-gpu-node.yaml`, and `homeassistant.yaml`; root `README.md`'s host table omits `gpu-node` entirely as a 4th host; `config/dashy/conf.yaml:2` points at `manifests/apply-dashy-config.sh`, missing the real `core/` path segment.
+- **`.gitignore` gaps** (nothing currently leaking, but no coverage): `*.retry` (a live risk — Ansible drops these on playbook failure and this repo's Ansible tree is actively used), `.vault_pass*`, kubeconfig-shaped filenames, editor swap files. `.DS_Store` files exist in the working tree at repo root and `config/` despite being gitignored — confirm they're actually untracked, not just ignored-going-forward.
+
+## Duplicated / Ambiguous / Possibly Obsolete Resources
+
+Per the "trace references before declaring obsolete" rule — **none of the
+"unsynced" items above are actually dead**; they're all in active use, just
+invisible to Argo CD. The only things that look genuinely obsolete or
+redundant are:
+
+- `nik4nao.home.arpa` DNS entry (both pihole values files) — no backing resource found anywhere.
+- `ansible/ansible.cfg` — not obsolete exactly, but its only real-world effect today is negative (breaks documented commands); candidate for deletion/merge into the root cfg.
+- `ansible/roles/nvidia/templates/k3s-containerd-config.toml.j2` — orphaned template, either finish wiring it up or remove it once GPU passthrough status is confirmed live.
+- `.DS_Store` at repo root and in `config/` — should never have been committed; harmless but should be `git rm --cached`.
+
+## Security Findings Summary (redacted)
+
+- Two real credentials committed in plaintext: a database password (`values/gitea.yaml:48`) and a third-party API key (`config/dashy/conf.yaml:63,70`). Both should be treated as compromised and rotated — no value is printed anywhere in this document.
+- One tracked binary (`router/backup-base.tar.gz`) whose contents were not inspected but is flagged as the most likely place real WiFi/VPN/router-admin credentials would live, given the sibling `uci-base.conf` is sanitized. Needs review outside this workflow; rotation and/or history-scrubbing may be warranted depending on what's inside.
+- No plaintext secrets found in Ansible beyond the two operational-leak items (console-printed token, world-readable systemd unit) — the vault (`ansible/group_vars/all/vault.yaml`) is genuinely encrypted and used correctly.
+- RBAC: one real over-scope (`gitea-backup`'s cluster-wide `pods/exec`), no wildcard verbs/resources found anywhere else.
+- Existing guardrails already partially cover this class of problem going forward: `.claude/agents/secrets-leak-scanner.md` and the `PostToolUse` hook in `.claude/settings.json` — but the scanner's stated scope reads as Ansible/manifests/Helm-values-centric and doesn't explicitly call out `config/**`, which is exactly where the Dashy API key slipped through. Worth widening its scope statement rather than adding new tooling.
+
+## What Should Be Preserved
+
+- The Ansible → Argo CD boundary itself is clean — no Ansible task embeds Kubernetes YAML or runs `kubectl apply` against application manifests (verified by repo-wide grep). The one-time hand-off (`helm install argocd` → `kubectl apply -f app-of-apps.yaml`) is exactly right for this scale.
+- The app-of-apps + one-Application-per-file pattern (20 flat files in `argocd/apps/`) is simple, greppable, and easy to reason about at this service count — an ApplicationSet or Kustomize layer would add indirection with no real benefit here.
+- The two-pattern secrets model (sealed-secrets for committed ciphertext, runtime `.env`-sourced scripts for cluster-only secrets) is coherent and consistently documented across all three READMEs.
+- Exact Helm chart version pinning on every chart-backed Application — zero floating versions found.
+- No deprecated Kubernetes API versions anywhere — clean.
+- Idempotent `curl | sh`-style installs are consistently guarded with `creates:` across `docker`, `nvidia`, `k3s-server`, `k3s-agent`, `ollama` roles.
+- `.claude/agents/infra-change-reviewer.md` and `secrets-leak-scanner.md` plus the `settings.json` hooks already encode a lot of this audit's tribal knowledge (DNS-sync rule, sealed-secret hand-edit ban, cert-issuer split, GPU taint/toleration, Ansible dual-cfg risk) — build on these, don't replace them.
+- Good in-repo examples worth using as the template when fixing the bad ones: `manifests/home-services/nvidia-device-plugin.yaml` (privilege drop), `manifests/core/ca-installer/ca-sync.yaml` (tightly-scoped namespaced RBAC), `manifests/home-services/{ai-gateway,ha-gateway}.yaml` + `manifests/portfolio/portfolio.yaml` (proper resources + probes).
+
+## Unknowns (need live cluster info or user input — not determinable from the repo alone)
+
+1. Whether Grafana's PVC actually binds to the static `grafana-pv` hostPath or silently gets dynamically provisioned via `local-path` instead (`values/kube-prometheus-stack.yaml` sets no `storageClassName`/`volumeName` for Grafana, unlike Prometheus/Loki which do). Needs `kubectl get pvc -n monitoring` / `kubectl get pv grafana-pv -o yaml`.
+2. Whether GPU passthrough is actually functional today given the orphaned containerd template (finding #24). Needs a check on `gpu-node`.
+3. What's actually inside `router/backup-base.tar.gz` (finding #5) — deliberately not inspected.
+4. Whether the K3s bootstrap kubeconfig-path bug (finding #8) actually breaks a fresh run, or whether there's missing context from an untraced earlier step. Static reading only — recommend confirming before relying on it for DR.
+5. Whether `selfHeal: false` on the two pihole Applications is deliberate policy (tied to the external-IP-loss gotcha) or an oversight.
+6. What Authentik Helm chart version is actually running — it's outside Argo CD, so there's no `targetRevision` to read; needs `helm list -n authentik` (or wherever it's installed) on the live cluster.
+7. Whether `values/gitea.yaml`'s committed password and Dashy's committed API key have ever been exposed beyond the operator (repo visibility/access history) — affects how urgent rotation is.
+
+---
+
+## Phase 4 — Target Architecture
+
+Given this is a single cluster, single operator, ~20-service homelab with no
+near-term multi-cluster plan, two credible options were considered.
+
+### Option A (recommended): Keep the current three-way split, close the gaps
+
+Keep `ansible/` / `argocd/apps/` / `manifests/` / `values/` exactly as they
+are structurally — this layout already matches how the system is organized
+conceptually, how the existing subagents and skills reference paths, and how
+the current documentation is written. Fix the specific holes (findings
+above) rather than moving files around.
+
+```
+homelab/
+├── ansible.cfg                    # single config, root-only (ansible/ansible.cfg removed)
+├── ansible/
+│   ├── requirements.yml           # NEW — pins community.general, ansible.posix, community.docker
+│   ├── inventory.yaml
+│   ├── group_vars/ · host_vars/
+│   ├── playbooks/
+│   └── roles/
+├── argocd/
+│   ├── apps/                      # + cert-manager-config.yaml, authentik.yaml, authentik-config.yaml
+│   └── values/
+├── manifests/                     # unchanged tree; every subdir now has a matching Application
+├── values/                        # unchanged
+├── config/dashy/
+├── router/
+├── .yamllint.yml                  # NEW
+├── .ansible-lint                  # NEW (config only, tool installed separately)
+└── CLAUDE.md                      # + validation commands section
+```
+
+**Why not restructure further**: there's no duplication cost today that a
+`clusters/`/`infrastructure/`/`apps/` split or Kustomize bases would actually
+reduce — each service already has exactly one Application file and at most
+one values file. Introducing Kustomize overlays or ApplicationSets would add
+a templating/indirection layer this repo doesn't need at 1-cluster,
+~20-service scale, and would itself become something to maintain and
+explain.
+
+### Option B (not recommended now): `clusters/` + `infrastructure/` + `apps/`, colocated values
+
+Restructure to one directory per service (e.g.
+`apps/gitea/{application.yaml,values.yaml}`), split `infrastructure/`
+(cert-manager, sealed-secrets, traefik, pihole) from `apps/` (user-facing
+services), under a `clusters/homelab/` root to leave room for a future
+second cluster. This gives slightly tighter per-service ownership (one
+directory instead of two: `argocd/apps/x.yaml` + `values/x.yaml`) but
+requires moving **every** Application's `source.path`, which Argo CD treats
+as a new resource identity — each move is a prune-risk event that must be
+done one Application at a time with `prune: false` verified beforehand,
+diffed, and rolled out carefully. That's a lot of migration risk for a
+benefit (one fewer directory hop per service) that doesn't solve any problem
+that exists today, and it presumes a multi-cluster future that isn't
+planned.
+
+**Recommendation: Option A.** Revisit Option B only if a second cluster is
+actually added or the service count grows to where cross-referencing
+`argocd/apps/` and `values/` by hand becomes genuinely painful — neither is
+true today.
+
+---
+
+## Phase 5 — Incremental Migration Plan
+
+Every stage is designed to leave the repo in a working, deployable state at
+every commit boundary. Stages that touch what Argo CD currently manages are
+explicitly marked **RUNTIME-AFFECTING** and require explicit go-ahead before
+pushing, per the repo's own rule that `git push` is a production deploy.
+
+| Stage | Scope | Key changes | Runtime impact | Rollback |
+|---|---|---|---|---|
+| **1. Safety fixes & secret hygiene** | Findings #3, #4, #6, plus `.gitignore` gaps | Rotate + re-secret the Gitea DB password and Dashy API key; fix the two `issuerRef` values; extend `.gitignore` (`*.retry`, `.vault_pass*`, kubeconfig patterns, swap files); `git rm --cached` the `.DS_Store` files | **RUNTIME-AFFECTING** for the password rotation (Gitea/Postgres restart) and cert reissuance (brief TLS transition on 2 hostnames); everything else is zero-impact | Revert commit; for password rotation, keep old secret retrievable until new one is confirmed working |
+| **2. Validation tooling & clean baseline** | New root-level config | Add `.yamllint.yml`, `.ansible-lint`; install `yamllint`/`ansible-lint`/`kubeconform` locally (asking first, since that's installing software); run baseline, fix anything trivial it finds | None — local/offline only | Uninstall tools / delete config files |
+| **3. Remove confirmed junk** | `.DS_Store`, orphaned `nik4nao.home.arpa` DNS entry, doc-path typo in `config/dashy/conf.yaml:2` | Delete/clean | Negligible (DNS entry removal is additive-safe to revert) | `git revert` |
+| **4. Ansible cleanup** | Findings #7, #9, #15, plus low-severity items | Merge `ansible/ansible.cfg` into root; add `requirements.yml`; align `k3s_version` across 3 locations; fix `gitea-runner` docker.sock task, `nvidia` `changed_when`, K3s token debug print, systemd unit permissions; update both READMEs (gpu-node host, docker/nvidia roles, homeassistant status) | **RUNTIME-AFFECTING** only for the `k3s_version` alignment (changes what a future node provision installs) — needs sign-off on which version is canonical; rest is inert until playbooks rerun | Git revert; k3s_version change only takes effect on next actual provision, not immediately |
+| **5. Argo CD bootstrap normalization** | Findings #1, #2, #10, #12, #13, #17, #19 | New `cert-manager-config.yaml` + `authentik.yaml`/`authentik-config.yaml` Applications (first-time adoption of already-live resources — `kubectl diff` before enabling automated sync on each); widen `network-secrets` coverage or add `network.yaml`; fix homeassistant namespace; add `gitea-backup` Namespace + narrow its RBAC to a Role; add missing sync-waves; normalize `targetRevision: HEAD`→`main` | **RUNTIME-AFFECTING, highest-risk stage** — bringing previously-unmanaged live resources under Argo CD for the first time. Roll out one Application at a time, diff first, watch sync status before moving to the next | Set the newly-added Application's sync policy back to manual, or delete the Application (resources stay, since `prune: false`) |
+| **6. Values/DNS consolidation** | Finding #18, #20 | Reconcile `pihole.yaml`/`pihole-debian.yaml` drift (add missing entries); document or normalize the `prune:true`/`selfHeal:false` outliers (pending intent) | **RUNTIME-AFFECTING but low-risk** — adds DNS records only, additive | Revert values file |
+| **7. Kubernetes correctness & security** | Findings #11, #16, #25, #26 | Add resource requests/limits + probes to flagged workloads (one service at a time); narrow `gitea-backup` RBAC; test Grafana `runAsNonRoot`/`fsGroup`-only; pin remaining `:latest` images to currently-running tags | **RUNTIME-AFFECTING** — limits set too low can OOMKill; needs `kubectl top` baselines first (gather live before proposing numbers); roll out one service at a time with rollback ready | Revert manifest, resync |
+| **8. Documentation & DR runbook** | All doc-drift items | Fix host tables, role tables, playbook lists, ingress/storage claims; add an explicit cold-rebuild runbook reflecting the now-closed GitOps gaps | None | Revert |
+| **9. CI & Claude Code guidance** | New `.gitea/workflows/validate.yaml` (yamllint/ansible-lint/kubeconform on push); `CLAUDE.md` validation-commands section; widen `secrets-leak-scanner.md`'s stated scope to include `config/**`; consider extending the `PreToolUse` hook to block Claude Code from reading `.env`/`router/backup-base.tar.gz` directly | New automation surface | None to the cluster; new CI pipeline needs approval since it's a new automation surface | Delete workflow file |
+
+Every stage gets validated with whatever's actually available
+(`ansible-playbook --syntax-check`, `yamllint`, `ansible-lint`,
+`kubeconform`/`kubectl apply --dry-run=client` where a live context isn't
+required) before being reported done — with the concrete diff and validation
+output shown, not just a claim of success.
+
+---
+
+## Claude Code Project Support
+
+- **`CLAUDE.md`**: keep it, add a short "Validation commands" section once Stage 2 lands (exact `yamllint`/`ansible-lint`/`ansible-playbook --syntax-check` invocations).
+- **Guard against accidental `.env`/key/backup access**: partially covered already (`.gitignore`, sealed-secret `PreToolUse` hook). Extend that hook to also block Claude Code `Read` of `.env` and `router/backup-base.tar.gz` — small addition, Stage 9.
+- **One validation skill/command**: rather than a new skill, widen `secrets-leak-scanner.md`'s stated file scope to explicitly include `config/**` (the actual gap that let the Dashy API key through) — cheaper and more targeted than adding new tooling.

ansible.cfg

@@ -5,3 +5,5 @@
 inventory = ansible/inventory.yaml
 roles_path = ansible/roles
 host_key_checking = False
+inject_facts_as_vars = False
+deprecation_warnings = False

ansible/README.md

@@ -6,16 +6,28 @@ cluster.
 
 ## Inventory
 
-`inventory.yaml` defines three groups:
+`inventory.yaml` defines four groups:
 
 | Group | Host | Purpose |
 | --- | --- | --- |
 | `k3s_server` | `minisforum` | K3s server at `192.168.7.77` |
 | `k3s_agents` | `debian` | K3s agent and NFS storage at `192.168.7.183` |
 | `mac_mini` | `mac-mini` | Docker/Ollama host at `192.168.7.96` |
+| `gpu_workstation` | `gpu-node` | K3s agent with NVIDIA GPU passthrough at `192.168.7.98` (spot-tainted) |
 
 All hosts use the `nik` user and the SSH key configured in `inventory.yaml`.
 
+## Collections
+
+Install the third-party collections this repo's roles depend on before
+running any playbook:
+
+```bash
+ansible-galaxy collection install -r ansible/requirements.yml
+```
+
+(`community.general`, `ansible.posix`, `community.docker`.)
+
 ## Common Playbooks
 
 ```bash
@@ -23,6 +35,7 @@ ansible-playbook -i ansible/inventory.yaml ansible/playbooks/bootstrap-minisforu
 ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-k3s.yaml -K
 ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-nfs-debian.yaml -K
 ansible-playbook -i ansible/inventory.yaml ansible/playbooks/join-debian-agent.yaml -K
+ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml -K
 ```
 
 Additional services:
@@ -35,6 +48,7 @@ ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-glances-debia
 ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-ollama.yaml -K
 ansible-playbook -i ansible/inventory.yaml ansible/playbooks/deploy-watch-party.yaml
 ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
+ansible-playbook -i ansible/inventory.yaml ansible/playbooks/homeassistant.yaml -K
 ```
 
 ## Roles
@@ -42,21 +56,24 @@ ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
 | Role | Responsibility |
 | --- | --- |
 | `common` | Packages, user setup, firewall, base data directories |
+| `docker` | Docker CE install (Debian and Ubuntu); depended on by `homeassistant` |
+| `nvidia` | NVIDIA driver, CUDA toolkit, and containerd/Docker GPU runtime config |
 | `k3s-server` | K3s server install, kubeconfig fetch, Helm install, primary node label |
-| `k3s-agent` | K3s agent join and storage node label |
+| `k3s-agent` | K3s agent join and storage/GPU node label |
 | `nfs-server` | Export `/mnt/storage` from Debian to the K3s server |
 | `monitoring` | Host directories and ownership for Prometheus/Loki |
 | `gitea-runner` | Gitea Actions runner systemd service |
 | `glances` | Host-level Glances service |
-| `ollama` | Ollama service on the Mac Mini |
+| `ollama` | Ollama service on the Mac Mini and GPU node (branches on OS) |
 | `watch-party` | Watch Party Docker Compose deployment on the Mac Mini |
 | `wireguard` | WireGuard server configuration |
-| `homeassistant` | Legacy standalone Home Assistant deployment |
+| `homeassistant` | Standalone Home Assistant deployment (Docker Compose + systemd on `minisforum`) — this is the **only** thing serving `ha.home.arpa`, not legacy/dead |
 
 ## Notes
 
-- K3s version is set in `roles/k3s-server/defaults/main.yaml` and
-  `roles/k3s-agent/defaults/main.yaml`.
+- K3s version is defined in three places and must be kept in sync:
+  `roles/k3s-server/defaults/main.yaml`, `roles/k3s-agent/defaults/main.yaml`,
+  and the override in `host_vars/gpu-node.yaml`.
 - `setup-gitea-runner.yaml` reads `GITEA_RUNNER_TOKEN` from the local
   environment.
 - The K3s role disables bundled Traefik because Traefik is managed by Argo CD.
@@ -64,3 +81,7 @@ ansible-playbook -i ansible/inventory.yaml ansible/playbooks/wireguard.yaml -K
   mount that export directly.
 - Keep host automation idempotent where practical. These playbooks are meant to
   be rerunnable during rebuilds.
+- To see the real K3s join token (needed once, to populate
+  `vault_k3s_node_token`), pass `-e k3s_show_token=true` to `setup-k3s.yaml`;
+  it's suppressed by default. Same pattern for WireGuard client configs via
+  `-e wireguard_show_client_configs=true` on `wireguard.yaml`.

ansible/ansible.cfg

@@ -1,4 +0,0 @@
-[defaults]
-inventory = inventory.yaml
-inject_facts_as_vars = False
-deprecation_warnings = False

ansible/host_vars/gpu-node.yaml

@@ -32,6 +32,8 @@ k3s_server_url: "https://192.168.7.77:6443"
 k3s_node_token: "{{ vault_k3s_node_token }}"
 
 # Check current cluster version with: k3s --version on minisforum
+# Kept in sync with roles/k3s-server and roles/k3s-agent defaults — all three
+# must match; see ansible/README.md "K3s version" note.
 k3s_version: "v1.32.4+k3s1"
 
 k3s_node_labels:

ansible/playbooks/homeassistant.yaml

@@ -1,6 +1,6 @@
 ---
 - name: Deploy Home Assistant on Minisforum
   hosts: minisforum
-  become: yes
+  become: true
   roles:
     - homeassistant

ansible/playbooks/setup-gpu-node.yaml

@@ -1,6 +1,6 @@
 # Config for: gpu-node workstation full setup
 # Applied by: ansible-playbook -i ansible/inventory.yaml ansible/playbooks/setup-gpu-node.yaml
-- name: gpu-node setup
+- name: GPU node setup
   hosts: gpu_workstation
   become: true
 

ansible/requirements.yml

@@ -0,0 +1,6 @@
+# Config for: Ansible Galaxy collections
+# Applied by: ansible-galaxy collection install -r ansible/requirements.yml
+collections:
+  - name: community.general
+  - name: ansible.posix
+  - name: community.docker

ansible/roles/docker/defaults/main.yaml

@@ -0,0 +1,4 @@
+---
+# Part of role: docker
+# Description: Default vars so this role doesn't depend on `common` having run first in the same play.
+username: nik

ansible/roles/docker/tasks/main.yaml

@@ -1,12 +1,26 @@
 ---
 # Part of role: docker
-# Called by: ansible/playbooks/setup-gpu-node.yaml
-# Description: Installs Docker CE on Ubuntu, adds user to docker group.
+# Called by: ansible/playbooks/setup-gpu-node.yaml, ansible/roles/homeassistant (meta dependency)
+# Description: Installs Docker CE, adds user to docker group. Works on both Debian and Ubuntu.
+
+- name: Install Docker prerequisites
+  ansible.builtin.apt:
+    name:
+      - ca-certificates
+      - curl
+    state: present
+    update_cache: true
+
+- name: Create apt keyrings directory
+  ansible.builtin.file:
+    path: /usr/share/keyrings
+    state: directory
+    mode: "0755"
 
 - name: Add Docker GPG key
   ansible.builtin.shell:
     cmd: >
-      curl -fsSL https://download.docker.com/linux/ubuntu/gpg |
+      curl -fsSL https://download.docker.com/linux/{{ ansible_facts['distribution'] | lower }}/gpg |
       gpg --dearmor -o /usr/share/keyrings/docker.gpg
     creates: /usr/share/keyrings/docker.gpg
 
@@ -14,12 +28,12 @@
   ansible.builtin.apt_repository:
     repo: >
       deb [arch=amd64 signed-by=/usr/share/keyrings/docker.gpg]
-      https://download.docker.com/linux/ubuntu
+      https://download.docker.com/linux/{{ ansible_facts['distribution'] | lower }}
       {{ ansible_facts['distribution_release'] }} stable
     filename: docker
     state: present
 
-- name: Install Docker CE
+- name: Install Docker CE and Compose plugin
   ansible.builtin.apt:
     name:
       - docker-ce
@@ -40,4 +54,4 @@
   ansible.builtin.user:
     name: "{{ username }}"
     groups: docker
-    append: true
+    append: true

ansible/roles/gitea-runner/tasks/main.yaml

@@ -74,7 +74,7 @@
 
       [Install]
       WantedBy=multi-user.target
-    mode: "0644"
+    mode: "0600"
   become: true
   notify: Restart act_runner
 
@@ -86,11 +86,18 @@
   become: false
   changed_when: false
 
+- name: Check docker.sock type
+  ansible.builtin.stat:
+    path: /run/docker.sock
+  register: docker_sock_stat
+  become: true
+
 - name: Remove docker.sock if it is a directory
   ansible.builtin.file:
     path: /run/docker.sock
     state: absent
   become: true
+  when: docker_sock_stat.stat.exists and docker_sock_stat.stat.isdir
 
 - name: Enable and start Docker
   ansible.builtin.systemd:

ansible/roles/homeassistant/handlers/main.yaml

@@ -1,4 +1,4 @@
 ---
-- name: reload systemd
-  systemd:
-    daemon_reload: yes
+- name: Reload systemd
+  ansible.builtin.systemd:
+    daemon_reload: true

ansible/roles/homeassistant/meta/main.yaml

@@ -0,0 +1,3 @@
+---
+dependencies:
+  - role: docker

ansible/roles/homeassistant/tasks/main.yaml

@@ -1,55 +1,11 @@
 ---
-- name: Install Docker prerequisites
-  apt:
-    name:
-      - ca-certificates
-      - curl
-    state: present
-    update_cache: yes
-
-- name: Create apt keyrings directory
-  file:
-    path: /etc/apt/keyrings
-    state: directory
-    mode: "0755"
-
-- name: Download Docker GPG key
-  get_url:
-    url: https://download.docker.com/linux/debian/gpg
-    dest: /etc/apt/keyrings/docker.asc
-    mode: "0644"
-
-- name: Add Docker apt repository
-  apt_repository:
-    repo: "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian trixie stable"
-    state: present
-    filename: docker
-
-- name: Install Docker CE and Compose plugin
-  apt:
-    name:
-      - docker-ce
-      - docker-ce-cli
-      - containerd.io
-      - docker-buildx-plugin
-      - docker-compose-plugin
-    state: present
-    update_cache: yes
-
-- name: Enable and start Docker
-  systemd:
-    name: docker
-    enabled: yes
-    state: started
-
-- name: Add nik to docker group
-  user:
-    name: nik
-    groups: docker
-    append: yes
+# Part of role: homeassistant
+# Called by: ansible/playbooks/homeassistant.yaml
+# Description: Deploys the standalone Home Assistant Docker Compose stack. Docker itself is
+#              installed by the `docker` role (meta dependency) instead of being duplicated here.
 
 - name: Create HA config directory
-  file:
+  ansible.builtin.file:
     path: /home/nik/homeassistant/config
     state: directory
     owner: nik
@@ -57,7 +13,7 @@
     mode: "0755"
 
 - name: Deploy docker-compose.yaml
-  template:
+  ansible.builtin.template:
     src: docker-compose.yaml.j2
     dest: /home/nik/homeassistant/docker-compose.yaml
     owner: nik
@@ -65,24 +21,24 @@
     mode: "0644"
 
 - name: Deploy systemd unit
-  template:
+  ansible.builtin.template:
     src: homeassistant.service.j2
     dest: /etc/systemd/system/homeassistant.service
     mode: "0644"
-  notify: reload systemd
+  notify: Reload systemd
 
 - name: Deploy base configuration.yaml
-  template:
+  ansible.builtin.template:
     src: configuration.yaml.j2
     dest: /home/nik/homeassistant/config/configuration.yaml
     owner: nik
     group: nik
     mode: "0644"
-    force: no
+    force: false
 
 - name: Enable and start homeassistant
-  systemd:
+  ansible.builtin.systemd:
     name: homeassistant
-    enabled: yes
+    enabled: true
     state: started
-    daemon_reload: yes
+    daemon_reload: true

ansible/roles/k3s-agent/defaults/main.yaml

@@ -3,6 +3,6 @@
 # Called by: ansible/playbooks/join-debian-agent.yaml
 # Description: Default variables for the k3s-agent role including version, server URL, and join token.
 
-k3s_version: v1.32.2+k3s1
+k3s_version: v1.32.4+k3s1
 k3s_server_url: https://192.168.7.77:6443
 k3s_node_token: ""

ansible/roles/k3s-server/defaults/main.yaml

@@ -3,7 +3,7 @@
 # Called by: ansible/playbooks/setup-k3s.yaml
 # Description: Default variables for the k3s-server role including version, IP, and server configuration.
 
-k3s_version: v1.32.2+k3s1
+k3s_version: v1.32.4+k3s1
 k3s_server_ip: 192.168.7.77
 
 k3s_server_config:

ansible/roles/k3s-server/tasks/main.yaml

@@ -46,6 +46,7 @@
 - name: Print node token
   ansible.builtin.debug:
     msg: "K3s node token: {{ k3s_node_token }}"
+  when: k3s_show_token | default(false)
 
 - name: Fetch kubeconfig to workstation
   ansible.builtin.fetch:
@@ -55,7 +56,7 @@
 
 - name: Fix kubeconfig server address
   ansible.builtin.replace:
-    path: /tmp/k3s-minisforum.yaml
+    path: "{{ lookup('env', 'HOME') }}/.kube/config"
     regexp: 'https://127\.0\.0\.1:6443'
     replace: "https://{{ k3s_server_ip }}:6443"
   delegate_to: localhost
@@ -67,6 +68,6 @@
     creates: /usr/local/bin/helm
 
 - name: Label server node as primary
-  ansible.builtin.shell:
+  ansible.builtin.command:
     cmd: k3s kubectl label node minisforum node-role=primary --overwrite
   changed_when: false

ansible/roles/nvidia/handlers/main.yaml

@@ -2,7 +2,7 @@
 # Part of role: nvidia
 # Called by: ansible/playbooks/setup-gpu-node.yaml
 # Description: Restarts Docker after nvidia-container-toolkit runtime configuration.
-- name: restart docker
+- name: Restart docker
   ansible.builtin.systemd:
     name: docker
     state: restarted

ansible/roles/nvidia/tasks/main.yaml

@@ -55,11 +55,19 @@
       export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
     mode: "0644"
 
+- name: Check if Docker already has the NVIDIA runtime configured
+  ansible.builtin.command:
+    cmd: grep -q '"nvidia"' /etc/docker/daemon.json
+  register: nvidia_docker_runtime_check
+  changed_when: false
+  failed_when: false
+
 - name: Configure Docker runtime for NVIDIA
   ansible.builtin.command:
     cmd: nvidia-ctk runtime configure --runtime=docker
+  when: nvidia_docker_runtime_check.rc != 0
   changed_when: true
-  notify: restart docker
+  notify: Restart docker
 
 - name: Reboot if driver was just installed
   ansible.builtin.reboot:

ansible/roles/ollama/handlers/main.yaml

@@ -3,15 +3,16 @@
 # Called by: ansible/playbooks/setup-ollama.yaml
 #            ansible/playbooks/setup-gpu-node.yaml
 # Description: Handlers for the ollama role. Restarts ollama on config changes.
-- name: restart ollama
+- name: Restart ollama
   become: true
-  command: launchctl kickstart -k system/com.ollama.ollama
+  ansible.builtin.command: launchctl kickstart -k system/com.ollama.ollama
+  changed_when: true
   when: ansible_facts['system'] == 'Darwin'
 
-- name: restart ollama linux
+- name: Restart ollama linux
   ansible.builtin.systemd:
     name: ollama
     state: restarted
     daemon_reload: true
   become: true
-  when: ansible_facts['system'] == 'Linux'
+  when: ansible_facts['system'] == 'Linux'

ansible/roles/ollama/tasks/main.yaml

@@ -13,19 +13,19 @@
   when: ansible_facts['system'] == 'Darwin'
 
 - name: Deploy ollama launchd plist
-  template:
+  ansible.builtin.template:
     src: ollama.plist.j2
     dest: /Library/LaunchDaemons/com.ollama.ollama.plist
     owner: root
     group: wheel
     mode: "0644"
   become: true
-  notify: restart ollama
+  notify: Restart ollama
   when: ansible_facts['system'] == 'Darwin'
 
 - name: Load ollama launchd service
   become: true
-  command: launchctl load -w /Library/LaunchDaemons/com.ollama.ollama.plist
+  ansible.builtin.command: launchctl load -w /Library/LaunchDaemons/com.ollama.ollama.plist
   args:
     creates: /var/run/ollama.pid
   ignore_errors: true
@@ -54,7 +54,7 @@
     group: root
     mode: "0644"
   become: true
-  notify: restart ollama linux
+  notify: Restart ollama linux
   when: ansible_facts['system'] == 'Linux'
 
 - name: Enable and start ollama service
@@ -68,7 +68,7 @@
 
 # ── shared ─────────────────────────────────────────────────────────────────────
 - name: Wait for ollama to be ready
-  uri:
+  ansible.builtin.uri:
     url: "http://localhost:{{ ollama_port }}"
     status_code: 200
   register: result
@@ -77,13 +77,13 @@
   delay: 3
 
 - name: Check installed ollama models
-  uri:
+  ansible.builtin.uri:
     url: "http://localhost:{{ ollama_port }}/api/tags"
     return_content: true
   register: ollama_tags
 
 - name: Pull ollama models
-  command: >
+  ansible.builtin.command: >
     {{ '/opt/homebrew/bin/ollama' if ansible_facts['system'] == 'Darwin' else '/usr/local/bin/ollama' }}
     pull {{ item }}
   loop: "{{ ollama_models }}"

ansible/roles/watch-party/tasks/main.yaml

@@ -8,8 +8,8 @@
     repo: "{{ watch_party_repo }}"
     dest: "{{ watch_party_dir }}"
     version: main
-    update: yes
-    accept_hostkey: yes
+    update: true
+    accept_hostkey: true
   environment:
     GIT_SSL_NO_VERIFY: "true"
 

ansible/roles/wireguard/handlers/main.yaml

@@ -1,5 +1,5 @@
 ---
 - name: Restart wg0
-  systemd:
+  ansible.builtin.systemd:
     name: wg-quick@wg0
-    state: restarted
+    state: restarted

ansible/roles/wireguard/tasks/main.yaml

@@ -1,6 +1,6 @@
 ---
 - name: Install WireGuard and tools
-  apt:
+  ansible.builtin.apt:
     name:
       - wireguard
       - wireguard-tools
@@ -9,13 +9,13 @@
     update_cache: true
 
 - name: Allow WireGuard port through UFW
-  ufw:
+  community.general.ufw:
     rule: allow
     port: "51820"
     proto: udp
 
 - name: Enable IP forwarding
-  sysctl:
+  ansible.posix.sysctl:
     name: net.ipv4.ip_forward
     value: "1"
     sysctl_set: true
@@ -23,7 +23,7 @@
     reload: true
 
 - name: Create WireGuard config directory
-  file:
+  ansible.builtin.file:
     path: /etc/wireguard
     state: directory
     mode: "0700"
@@ -32,88 +32,88 @@
 
 # --- Server keypair ---
 - name: Check if server private key exists
-  stat:
+  ansible.builtin.stat:
     path: /etc/wireguard/server.key
   register: server_key_stat
 
 - name: Generate server private key
-  shell: wg genkey > /etc/wireguard/server.key
+  ansible.builtin.shell: wg genkey > /etc/wireguard/server.key
   when: not server_key_stat.stat.exists
 
 - name: Set permissions on server private key
-  file:
+  ansible.builtin.file:
     path: /etc/wireguard/server.key
     mode: "0600"
     owner: root
     group: root
 
 - name: Read server private key
-  slurp:
+  ansible.builtin.slurp:
     src: /etc/wireguard/server.key
   register: server_private_key
 
 - name: Derive server public key
-  shell: wg pubkey < /etc/wireguard/server.key
+  ansible.builtin.shell: wg pubkey < /etc/wireguard/server.key
   register: server_public_key
   changed_when: false
 
 # --- Phone keypair ---
 - name: Check if phone private key exists
-  stat:
+  ansible.builtin.stat:
     path: /etc/wireguard/phone.key
   register: phone_key_stat
 
 - name: Generate phone private key
-  shell: wg genkey > /etc/wireguard/phone.key
+  ansible.builtin.shell: wg genkey > /etc/wireguard/phone.key
   when: not phone_key_stat.stat.exists
 
 - name: Set permissions on phone private key
-  file:
+  ansible.builtin.file:
     path: /etc/wireguard/phone.key
     mode: "0600"
     owner: root
     group: root
 
 - name: Read phone private key
-  slurp:
+  ansible.builtin.slurp:
     src: /etc/wireguard/phone.key
   register: phone_private_key
 
 - name: Derive phone public key
-  shell: wg pubkey < /etc/wireguard/phone.key
+  ansible.builtin.shell: wg pubkey < /etc/wireguard/phone.key
   register: phone_public_key
   changed_when: false
 
 # --- Mac keypair ---
 - name: Check if mac private key exists
-  stat:
+  ansible.builtin.stat:
     path: /etc/wireguard/mac.key
   register: mac_key_stat
 
 - name: Generate mac private key
-  shell: wg genkey > /etc/wireguard/mac.key
+  ansible.builtin.shell: wg genkey > /etc/wireguard/mac.key
   when: not mac_key_stat.stat.exists
 
 - name: Set permissions on mac private key
-  file:
+  ansible.builtin.file:
     path: /etc/wireguard/mac.key
     mode: "0600"
     owner: root
     group: root
 
 - name: Read mac private key
-  slurp:
+  ansible.builtin.slurp:
     src: /etc/wireguard/mac.key
   register: mac_private_key
 
 - name: Derive mac public key
-  shell: wg pubkey < /etc/wireguard/mac.key
+  ansible.builtin.shell: wg pubkey < /etc/wireguard/mac.key
   register: mac_public_key
   changed_when: false
 
 # --- Server config ---
 - name: Write wg0.conf
-  template:
+  ansible.builtin.template:
     src: wg0.conf.j2
     dest: /etc/wireguard/wg0.conf
     mode: "0600"
@@ -123,14 +123,14 @@
 
 # --- Service ---
 - name: Enable and start wg-quick@wg0
-  systemd:
+  ansible.builtin.systemd:
     name: wg-quick@wg0
     enabled: true
     state: started
 
 # --- Phone client config + QR ---
 - name: Write phone client config
-  copy:
+  ansible.builtin.copy:
     dest: /etc/wireguard/phone-client.conf
     mode: "0600"
     owner: root
@@ -149,7 +149,7 @@
 
 # --- Mac client config ---
 - name: Write mac client config
-  copy:
+  ansible.builtin.copy:
     dest: /etc/wireguard/mac-client.conf
     mode: "0600"
     owner: root
@@ -167,19 +167,23 @@
       PersistentKeepalive = 25
 
 - name: Display mac client config
-  shell: cat /etc/wireguard/mac-client.conf
+  ansible.builtin.shell: cat /etc/wireguard/mac-client.conf
   register: mac_conf
   changed_when: false
+  when: wireguard_show_client_configs | default(false)
 
 - name: Show mac client config
-  debug:
+  ansible.builtin.debug:
     msg: "{{ mac_conf.stdout_lines }}"
+  when: wireguard_show_client_configs | default(false)
 
 - name: Generate QR code for phone
-  shell: qrencode -t ansiutf8 < /etc/wireguard/phone-client.conf
+  ansible.builtin.shell: qrencode -t ansiutf8 < /etc/wireguard/phone-client.conf
   register: phone_qr
   changed_when: false
+  when: wireguard_show_client_configs | default(false)
 
 - name: Display phone QR code
-  debug:
-    msg: "{{ phone_qr.stdout_lines }}"
+  ansible.builtin.debug:
+    msg: "{{ phone_qr.stdout_lines }}"
+  when: wireguard_show_client_configs | default(false)

argocd/apps/authentik-config.yaml

@@ -0,0 +1,20 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: authentik-config
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: https://gitea.nik4nao.com/nik/homelab.git
+    targetRevision: main
+    path: manifests/authentik
+    directory:
+      include: '*.yaml'
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: authentik
+  syncPolicy:
+    automated:
+      prune: false
+      selfHeal: true

argocd/apps/authentik.yaml

@@ -0,0 +1,33 @@
+# NEEDS YOUR INPUT before first sync: targetRevision below is a placeholder.
+# Run `helm list -n authentik` on the live cluster and set this to the chart
+# version actually deployed (installed by hand per the header comment in
+# values/authentik.yaml) -- Authentik is the SSO IdP gating Argo CD/Grafana/
+# Gitea logins, so adopting it into GitOps with the wrong version could
+# trigger an unwanted live chart upgrade/downgrade on first sync. Sync is
+# left manual (no `automated:` block) until you've confirmed this and diffed
+# with `kubectl diff` / `argocd app diff`.
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: authentik
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "-1"
+spec:
+  project: default
+  sources:
+    - repoURL: https://charts.goauthentik.io
+      chart: authentik
+      targetRevision: "REPLACE_ME"
+      helm:
+        valueFiles:
+          - $values/values/authentik.yaml
+    - repoURL: https://gitea.nik4nao.com/nik/homelab.git
+      targetRevision: main
+      ref: values
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: authentik
+  syncPolicy:
+    syncOptions:
+      - CreateNamespace=true

argocd/apps/cert-manager-config.yaml

@@ -0,0 +1,22 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: cert-manager-config
+  namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "-1"
+spec:
+  project: default
+  source:
+    repoURL: https://gitea.nik4nao.com/nik/homelab.git
+    targetRevision: main
+    path: manifests/cert-manager
+    directory:
+      include: '*.yaml'
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: cert-manager
+  syncPolicy:
+    automated:
+      prune: false
+      selfHeal: true

argocd/apps/cert-manager.yaml

@@ -3,6 +3,8 @@ kind: Application
 metadata:
   name: cert-manager
   namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "-2"
 spec:
   project: default
   sources:

argocd/apps/home-services.yaml

@@ -9,14 +9,14 @@ spec:
   project: default
   source:
     repoURL: https://gitea.nik4nao.com/nik/homelab.git
-    targetRevision: HEAD
+    targetRevision: main
     path: manifests/home-services
   destination:
     server: https://kubernetes.default.svc
     namespace: home-services
   syncPolicy:
     automated:
-      prune: true
+      prune: false
       selfHeal: true
     syncOptions:
       - CreateNamespace=true

argocd/apps/homeassistant.yaml

@@ -11,10 +11,8 @@ spec:
     path: manifests/homeassistant
   destination:
     server: https://kubernetes.default.svc
-    namespace: homeassistant
+    namespace: default
   syncPolicy:
     automated:
       prune: false
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=true
+      selfHeal: true

argocd/apps/network.yaml

@@ -0,0 +1,23 @@
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: network
+  namespace: argocd
+spec:
+  project: default
+  source:
+    repoURL: https://gitea.nik4nao.com/nik/homelab.git
+    targetRevision: main
+    path: manifests/network
+    directory:
+      exclude: '*-sealed.yaml'
+      include: '*.yaml'
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: pihole
+  syncPolicy:
+    automated:
+      prune: false
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=true

argocd/apps/otel-collector.yaml

@@ -14,14 +14,14 @@ spec:
         valueFiles:
           - $values/values/otel-collector.yaml
     - repoURL: https://gitea.nik4nao.com/nik/homelab.git
-      targetRevision: HEAD
+      targetRevision: main
       ref: values
   destination:
     server: https://kubernetes.default.svc
     namespace: monitoring
   syncPolicy:
     automated:
-      prune: true
+      prune: false
       selfHeal: true
     syncOptions:
       - CreateNamespace=false

argocd/apps/pihole-debian.yaml

@@ -21,6 +21,6 @@ spec:
   syncPolicy:
     automated:
       prune: false
-      selfHeal: false
+      selfHeal: true
     syncOptions:
       - CreateNamespace=true

argocd/apps/pihole.yaml

@@ -21,6 +21,6 @@ spec:
   syncPolicy:
     automated:
       prune: false
-      selfHeal: false
+      selfHeal: true
     syncOptions:
       - CreateNamespace=true

argocd/apps/sealed-secrets.yaml

@@ -3,6 +3,8 @@ kind: Application
 metadata:
   name: sealed-secrets
   namespace: argocd
+  annotations:
+    argocd.argoproj.io/sync-wave: "-2"
 spec:
   project: default
   sources:

argocd/apps/tempo.yaml

@@ -14,14 +14,14 @@ spec:
         valueFiles:
           - $values/values/tempo.yaml
     - repoURL: https://gitea.nik4nao.com/nik/homelab.git
-      targetRevision: HEAD
+      targetRevision: main
       ref: values
   destination:
     server: https://kubernetes.default.svc
     namespace: monitoring
   syncPolicy:
     automated:
-      prune: true
+      prune: false
       selfHeal: true
     syncOptions:
       - CreateNamespace=false

config/dashy/conf.yaml

@@ -1,5 +1,5 @@
 # Config for: Dashy dashboard
-# Applied by: manifests/apply-dashy-config.sh (creates ConfigMap from this file)
+# Applied by: manifests/core/apply-dashy-config.sh (renders secrets from .env, creates ConfigMap from this file)
 
 pageInfo:
   title: Good morning, Nik
@@ -60,14 +60,14 @@ sections:
         id: 1_928_iframe
       - type: weather
         options:
-          apiKey: 770d36389dd547e8e3daccb39debde64
+          apiKey: "${DASHY_WEATHER_API_KEY}"
           city: Tokyo, JP
           units: metric
           hideDetails: false
         id: 2_928_weather
       - type: weather-forecast
         options:
-          apiKey: 770d36389dd547e8e3daccb39debde64
+          apiKey: "${DASHY_WEATHER_API_KEY}"
           city: Tokyo, JP
           numDays: 5
           units: metric

manifests/authentik/authentik-ingress.yaml

@@ -9,7 +9,7 @@ metadata:
 spec:
   secretName: authentik-tls
   issuerRef:
-    name: internal-ca
+    name: internal-ca-issuer
     kind: ClusterIssuer
   dnsNames:
     - auth.home.arpa

manifests/authentik/authentik-proxy-outpost.yaml

@@ -41,6 +41,23 @@ spec:
               name: http
             - containerPort: 9443
               name: https
+          readinessProbe:
+            tcpSocket:
+              port: 9000
+            initialDelaySeconds: 10
+            periodSeconds: 15
+          livenessProbe:
+            tcpSocket:
+              port: 9000
+            initialDelaySeconds: 20
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 20m
+              memory: 64Mi
+            limits:
+              cpu: 200m
+              memory: 256Mi
 ---
 apiVersion: v1
 kind: Service

manifests/core/apply-dashy-config.sh

@@ -1,12 +1,18 @@
 #!/usr/bin/env bash
 # Usage: bash manifests/core/apply-dashy-config.sh
-# Description: Updates the Dashy ConfigMap from config/dashy/conf.yaml and restarts the deployment
+# Description: Renders config/dashy/conf.yaml with secrets from .env, updates the Dashy ConfigMap, and restarts the deployment
 set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$SCRIPT_DIR/../../.env"
+
+RENDERED="$(mktemp)"
+trap 'rm -f "$RENDERED"' EXIT
+sed "s|\${DASHY_WEATHER_API_KEY}|${DASHY_WEATHER_API_KEY}|g" "$SCRIPT_DIR/../../config/dashy/conf.yaml" > "$RENDERED"
 
 kubectl create configmap dashy-config \
-  --from-file=conf.yml=config/dashy/conf.yaml \
+  --from-file=conf.yml="$RENDERED" \
   --namespace dashy \
   --dry-run=client -o yaml | kubectl apply -f -
 
 kubectl rollout restart deployment/dashy -n dashy
-echo "Dashy config updated"
+echo "Dashy config updated"

manifests/core/ca-installer/ca-installer.yaml

@@ -63,6 +63,25 @@ spec:
           image: nginx:alpine
           ports:
             - containerPort: 80
+          readinessProbe:
+            httpGet:
+              path: /
+              port: 80
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /
+              port: 80
+            initialDelaySeconds: 10
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              cpu: 100m
+              memory: 64Mi
           volumeMounts:
             - name: web-files
               mountPath: /usr/share/nginx/html/index.html

manifests/core/dashy.yaml

@@ -43,6 +43,25 @@ spec:
           image: lissy93/dashy:latest
           ports:
             - containerPort: 8080
+          readinessProbe:
+            httpGet:
+              path: /
+              port: 8080
+            initialDelaySeconds: 10
+            periodSeconds: 10
+          livenessProbe:
+            httpGet:
+              path: /
+              port: 8080
+            initialDelaySeconds: 20
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 50m
+              memory: 128Mi
+            limits:
+              cpu: 200m
+              memory: 256Mi
           volumeMounts:
             - name: config
               mountPath: /app/user-data/conf.yml

manifests/core/glances.yaml

@@ -34,6 +34,23 @@ spec:
               value: "-w"
           securityContext:
             privileged: true
+          readinessProbe:
+            tcpSocket:
+              port: 61208
+            initialDelaySeconds: 10
+            periodSeconds: 15
+          livenessProbe:
+            tcpSocket:
+              port: 61208
+            initialDelaySeconds: 20
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 50m
+              memory: 128Mi
+            limits:
+              cpu: 300m
+              memory: 256Mi
           volumeMounts:
             - name: host-proc
               mountPath: /proc

manifests/gitea/gitea-backup.yaml

@@ -2,15 +2,21 @@
 # Delete: kubectl delete -f manifests/gitea/gitea-backup.yaml
 # Description: CronJob that backs up Gitea to NFS every 7 days, with RBAC and PV/PVC.
 apiVersion: v1
+kind: Namespace
+metadata:
+  name: gitea-backup
+---
+apiVersion: v1
 kind: ServiceAccount
 metadata:
   name: gitea-backup
   namespace: gitea-backup
 ---
 apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
+kind: Role
 metadata:
   name: gitea-backup
+  namespace: gitea
 rules:
   - apiGroups: [""]
     resources: ["pods"]
@@ -20,15 +26,16 @@ rules:
     verbs: ["create"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
+kind: RoleBinding
 metadata:
   name: gitea-backup
+  namespace: gitea
 subjects:
   - kind: ServiceAccount
     name: gitea-backup
     namespace: gitea-backup
 roleRef:
-  kind: ClusterRole
+  kind: Role
   name: gitea-backup
   apiGroup: rbac.authorization.k8s.io
 ---

manifests/gitea/gitea-postgres-secret.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$SCRIPT_DIR/../../.env"
+
+kubectl create secret generic gitea-postgres-secret \
+  --namespace=gitea \
+  --from-literal=postgresql-password="${GITEA_POSTGRES_PASSWORD}" \
+  --dry-run=client -o yaml \
+  | kubeseal \
+      --controller-namespace=kube-system \
+      --controller-name=sealed-secrets-controller \
+      --format yaml \
+      > "$SCRIPT_DIR/gitea-postgres-sealed.yaml"
+
+echo "Wrote $SCRIPT_DIR/gitea-postgres-sealed.yaml"

manifests/media/immich.yaml

@@ -88,6 +88,23 @@ spec:
               value: "--data-checksums"
             - name: PGDATA
               value: /var/lib/postgresql/data/pgdata
+          readinessProbe:
+            tcpSocket:
+              port: 5432
+            initialDelaySeconds: 10
+            periodSeconds: 15
+          livenessProbe:
+            tcpSocket:
+              port: 5432
+            initialDelaySeconds: 20
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 100m
+              memory: 512Mi
+            limits:
+              cpu: 1000m
+              memory: 1Gi
           volumeMounts:
             - name: data
               mountPath: /var/lib/postgresql/data
@@ -129,6 +146,23 @@ spec:
           image: docker.io/redis:6.2-alpine
           ports:
             - containerPort: 6379
+          readinessProbe:
+            tcpSocket:
+              port: 6379
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          livenessProbe:
+            tcpSocket:
+              port: 6379
+            initialDelaySeconds: 10
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 10m
+              memory: 32Mi
+            limits:
+              cpu: 200m
+              memory: 128Mi
           volumeMounts:
             - name: data
               mountPath: /data
@@ -189,6 +223,23 @@ spec:
               value: http://immich-machine-learning:3003
             - name: TZ
               value: Asia/Tokyo
+          readinessProbe:
+            tcpSocket:
+              port: 2283
+            initialDelaySeconds: 15
+            periodSeconds: 15
+          livenessProbe:
+            tcpSocket:
+              port: 2283
+            initialDelaySeconds: 30
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 200m
+              memory: 2Gi
+            limits:
+              cpu: 2000m
+              memory: 4Gi
           volumeMounts:
             - name: library
               mountPath: /usr/src/app/upload
@@ -244,6 +295,23 @@ spec:
           image: ghcr.io/immich-app/immich-machine-learning:v2.7.5
           ports:
             - containerPort: 3003
+          readinessProbe:
+            tcpSocket:
+              port: 3003
+            initialDelaySeconds: 15
+            periodSeconds: 15
+          livenessProbe:
+            tcpSocket:
+              port: 3003
+            initialDelaySeconds: 30
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              cpu: 2000m
+              memory: 2Gi
           volumeMounts:
             - name: cache
               mountPath: /cache

manifests/media/jdownloader.yaml

@@ -82,6 +82,25 @@ spec:
               value: "1000"
             - name: TZ
               value: "Asia/Tokyo"
+          readinessProbe:
+            httpGet:
+              path: /
+              port: 5800
+            initialDelaySeconds: 15
+            periodSeconds: 15
+          livenessProbe:
+            httpGet:
+              path: /
+              port: 5800
+            initialDelaySeconds: 30
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 200m
+              memory: 512Mi
+            limits:
+              cpu: 1000m
+              memory: 1Gi
           volumeMounts:
             - name: config
               mountPath: /config
@@ -91,6 +110,13 @@ spec:
           image: python:3.12-alpine
           ports:
             - containerPort: 9666
+          resources:
+            requests:
+              cpu: 5m
+              memory: 16Mi
+            limits:
+              cpu: 50m
+              memory: 64Mi
           volumeMounts:
             - name: config
               mountPath: /config

manifests/media/jellyfin.yaml

@@ -93,6 +93,25 @@ spec:
               value: https://jellyfin.home.arpa
             - name: LIBVA_DRIVER_NAME
               value: radeonsi
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: 8096
+            initialDelaySeconds: 15
+            periodSeconds: 15
+          livenessProbe:
+            httpGet:
+              path: /health
+              port: 8096
+            initialDelaySeconds: 30
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 200m
+              memory: 1Gi
+            limits:
+              cpu: 2000m
+              memory: 4Gi
           volumeMounts:
             - name: config
               mountPath: /config

manifests/media/qbittorrent.yaml

@@ -103,6 +103,25 @@ spec:
               value: "Asia/Tokyo"
             - name: WEBUI_PORT
               value: "8080"
+          readinessProbe:
+            httpGet:
+              path: /
+              port: 8080
+            initialDelaySeconds: 15
+            periodSeconds: 15
+          livenessProbe:
+            httpGet:
+              path: /
+              port: 8080
+            initialDelaySeconds: 30
+            periodSeconds: 30
+          resources:
+            requests:
+              cpu: 50m
+              memory: 128Mi
+            limits:
+              cpu: 500m
+              memory: 512Mi
           volumeMounts:
             - name: config
               mountPath: /config
@@ -112,6 +131,13 @@ spec:
           image: python:3-alpine
           ports:
             - containerPort: 8888
+          resources:
+            requests:
+              cpu: 5m
+              memory: 16Mi
+            limits:
+              cpu: 50m
+              memory: 64Mi
           command:
             - python3
             - -c

manifests/network/traefik-dashboard-ingress.yaml

@@ -63,7 +63,7 @@ metadata:
 spec:
   secretName: traefik-dashboard-tls
   issuerRef:
-    name: internal-ca
+    name: internal-ca-issuer
     kind: ClusterIssuer
   dnsNames:
     - traefik.home.arpa

values/gitea.yaml

@@ -45,9 +45,12 @@ postgresql:
   global:
     postgresql:
       auth:
-        password: gitea-db-password
         username: gitea
         database: gitea
+        existingSecret: gitea-postgres-secret
+        secretKeys:
+          adminPasswordKey: postgresql-password
+          userPasswordKey: postgresql-password
 
 service:
   ssh:

values/pihole-debian.yaml

@@ -37,7 +37,6 @@ dnsmasq:
     - address=/gitea.home.arpa/192.168.7.77
     - address=/pihole.home.arpa/192.168.7.77
     - address=/home.arpa/192.168.7.77
-    - address=/nik4nao.home.arpa/192.168.7.183
     - address=/dashy.home.arpa/192.168.7.77
     - address=/jellyfin.home.arpa/192.168.7.77
     - address=/qbittorrent.home.arpa/192.168.7.77
@@ -47,9 +46,11 @@ dnsmasq:
     - address=/ca.home.arpa/192.168.7.77
     - address=/auth.home.arpa/192.168.7.77
     - address=/traefik.home.arpa/192.168.7.77
+    - address=/immich.home.arpa/192.168.7.77
     - address=/gitea.nik4nao.com/192.168.7.77
     - address=/ha.home.arpa/192.168.7.77
     - address=/argocd.home.arpa/192.168.7.77
+    - address=/gluetun.home.arpa/192.168.7.77
 
 persistentVolumeClaim:
   enabled: true

values/pihole.yaml

@@ -68,7 +68,6 @@ dnsmasq:
     - address=/gitea.home.arpa/192.168.7.77
     - address=/pihole.home.arpa/192.168.7.77
     - address=/home.arpa/192.168.7.77
-    - address=/nik4nao.home.arpa/192.168.7.183
     - address=/dashy.home.arpa/192.168.7.77
     - address=/jellyfin.home.arpa/192.168.7.77
     - address=/qbittorrent.home.arpa/192.168.7.77