diff --git a/REFACTOR_PLAN.md b/REFACTOR_PLAN.md deleted file mode 100644 index 14da434..0000000 --- a/REFACTOR_PLAN.md +++ /dev/null @@ -1,318 +0,0 @@ -# Homelab Repo Audit & Refactor Plan - -Living document. Produced by a full read-only audit of the repo (Ansible, Argo -CD, manifests, values, config, router). Nothing was applied, synced, -committed, or pushed as part of producing this. We work through the -"Migration Plan" stages one at a time, referencing this file; update the -Status column as stages land. - -**Audit constraints honored**: `.env` was never opened; `router/backup-base.tar.gz` -was never extracted; no `kubectl apply` / `helm install` / Argo CD sync / -Ansible-against-real-hosts was run. All findings are grounded in file -citations, and the highest-severity ones were independently re-verified -(not just taken from sub-agent research output) before being recorded here. - -**Immich removed entirely, 2026-07-23**: not used enough to justify chasing -its stale-NFS-handle issue. `manifests/media/immich.yaml` (+ its secret -script/sealed secret), `immich.home.arpa` DNS entries, its Dashy tile, and all -other repo references are gone. Findings #16 and #20 that mentioned Immich -are now moot for it specifically (kept as historical record, not rewritten). -Live namespace/PVCs need a manual `kubectl delete namespace immich` to -actually free the data -- Argo CD's `media` Application won't do this itself -(`prune: false`). - -## Stage Tracker - -| Stage | Scope | Status | -|---|---|---| -| 1 | Safety fixes & secret hygiene | Files done, committed locally, **not pushed**. Actual secret rotation (new password/API key values + live DB password change) still needs the user — see note below. | -| 2 | Validation tooling & clean baseline | Done. `yamllint`/`ansible-lint`/`kubeconform` installed by user; `.yamllint.yml`/`.ansible-lint` added; baseline clean (0 yamllint issues, `kubeconform`: 117+20 resources valid, all playbooks pass `--syntax-check`). Remaining `ansible-lint` findings are either pre-existing role/var naming conventions (out of scope — would require repo-wide renames) or collections-not-installed noise that resolves once `ansible-galaxy collection install -r ansible/requirements.yml` is run. | -| 3 | Remove confirmed junk | Done, committed locally. `.DS_Store` confirmed never tracked (no action needed). | -| 4 | Ansible cleanup | Done, committed locally. | -| 5 | Argo CD bootstrap normalization | Files done, committed locally, **not pushed — highest-risk stage**. `authentik.yaml`'s chart version is a placeholder needing your input (`helm list -n authentik`); every new/changed Application here needs `kubectl diff` one at a time before/after enabling, not a blind batch push. | -| 6 | Values/DNS consolidation | Done, committed locally, **not pushed**. | -| 7 | Kubernetes correctness & security | Resource requests/limits + probes done (finding #16), committed locally, **not pushed**. `gitea-backup` RBAC narrowing (#11) done as part of Stage 5's commit. Grafana `runAsNonRoot`/`fsGroup` test (#25) and `:latest` image pinning not done — both need a live test window / registry inspection I didn't do unprompted. | -| 8 | Documentation & DR runbook | Done, committed locally. | -| 9 | CI & Claude Code guidance | Done, committed locally, **not pushed** (new CI automation surface). | - -**Correction, 2026-07-23**: the original `existingSecret`-based fix for Finding #3 was wrong and caused a real Gitea outage. Confirmed directly from the chart source (`gitea/helm-gitea` `templates/gitea/config.yaml`): `{{ set .Values.gitea.config.database "PASSWD" .Values.postgresql.global.postgresql.auth.password }}` — Gitea's own `[database]` config is *always* built from this plaintext values field, regardless of `existingSecret` (which only affects the bundled postgresql subchart's own credential provisioning, a separate concern). Removing the plaintext field made it silently fall back to the chart's built-in default password, breaking every fresh Gitea pod's DB connection. `values/gitea.yaml` is back to a plaintext (rotated) password with a comment explaining why, and `manifests/gitea/gitea-postgres-secret.sh` was removed. Getting this fully off plaintext would need a Config Management Plugin or similar — not attempted here; flagging as a real open item if it matters enough to invest in. - -Dashy's API key fix (env-injection via `manifests/core/apply-dashy-config.sh`) is unaffected by this and still needs you to rotate the actual key value. - -### New findings from this session's live-cluster checks (not in the original audit) - -- **Grafana and Loki's static PVs are orphaned.** `manifests/monitoring/monitoring-pvs.yaml` defines `grafana-pv`/`loki-pv` hostPath PVs, but live `kubectl get pv` shows both sitting `Available` (unbound) — their Helm-managed PVCs got dynamically provisioned via `local-path` instead, unlike `prometheus-pv` which binds correctly. Data isn't lost, just not where the README/manifest imply. Needs a decision: bind properly (data migration required) or drop the orphaned static PVs and document reality. Not yet actioned. -- **Finding #24 (GPU passthrough) is resolved as working, not broken.** Live check on node `nik-gpu`: `nvidia.com/gpu: "1"` allocatable, `nvidia` runtime handler registered in containerd, `nvidia-device-plugin` pod `Running`. The orphaned-looking `k3s-containerd-config.toml.j2` template is a non-issue in practice — no action needed. -- **Unknown #1 (Grafana PVC) resolved**: dynamically provisioned via `local-path`, not the static PV — see the orphaned-PV finding above, same root cause. -- Node hostnames in the live cluster are `nik-debian`/`nik-gpu`, not `debian`/`gpu-node` as in `ansible/inventory.yaml` — cosmetic (K3s registers by actual hostname), doesn't affect anything, just noted for anyone cross-referencing `kubectl get nodes` against the docs. - -Open decisions needed from the user before/during the relevant stage: -- Finding #5 (`router/backup-base.tar.gz`): needs manual review outside this - workflow; may imply git-history scrub / router credential rotation. -- Finding #9 (K3s version skew): which version is canonical (`v1.32.2+k3s1` - vs `v1.32.4+k3s1`)? -- Finding #18: is `prune:true` on `home-services`/`otel-collector`/`tempo` - and `selfHeal:false` on both pihole Applications intentional policy? -- Finding #25: Grafana `runAsUser: 0` — worth testing `fsGroup`-only instead? - ---- - -## Phase 1 — Current-State Architecture - -``` -Ansible (hosts) Argo CD (cluster) -───────────────── ────────────────────────────────────────── -minisforum (K3s server, .77) ──┐ -debian (K3s agent, .183) ────┼─▶ K3s cluster ──▶ helm install argocd (manual, README.md:92-103) -gpu-node (K3s agent, .98) ──┘ │ │ -mac-mini (standalone Docker) │ ▼ - │ kubectl apply -f manifests/argocd/app-of-apps.yaml (manual, one-time) - │ │ - │ ▼ - │ Application: app-of-apps → watches argocd/apps/*.yaml - │ │ - │ ┌───────────┴────────────────────────────┐ - │ ▼ ▼ - │ Helm-chart Applications Raw-manifest Applications - │ (chart + values/.yaml) (path: manifests/) - │ traefik, cert-manager*, gitea, core, media, home-services, - │ pihole, pihole-debian, monitoring, portfolio, homeassistant, - │ loki, tempo, otel-collector, *-secrets (sealed only) - │ sealed-secrets, argocd(self) - │ - └── NOT reconciled by Argo CD at all: - manifests/cert-manager/ (ClusterIssuers) - manifests/authentik/ + values/authentik.yaml - 4 files in manifests/network/ (non-sealed) - → applied by hand, outside GitOps -``` - -Host-level services that never touch the K3s cluster: Home Assistant (Docker -Compose + systemd on `minisforum`, `ansible/roles/homeassistant/`), Watch -Party and Ollama (`mac-mini`), Ollama (`gpu-node`) — all Ansible-managed. - -Bootstrap sequence as documented (`README.md:71-105`, `argocd/README.md:7-24`): -Ansible playbooks → manual `helm install argocd` → manual `kubectl apply` of -app-of-apps → everything else via Git. **This documented sequence is -incomplete** — it never mentions installing Authentik or the cert-manager -`ClusterIssuer`s, both required for the rest of the stack to actually work -(see Critical findings below). - -## Executive Summary - -The repo is coherently designed for a single-operator homelab: Ansible for -host bootstrap, Argo CD app-of-apps for cluster reconciliation, a clean -two-pattern secrets model (sealed-secrets for committed ciphertext, runtime -scripts for `.env`-sourced live secrets), and a genuinely useful pair of -existing Claude Code subagents (`infra-change-reviewer`, `secrets-leak-scanner`) -that already encode a lot of hard-won tribal knowledge. There's no -fashionable-but-wrong abstraction problem here — the biggest risks are **gaps -between what the documentation/GitOps model claims and what's actually wired -up**: two foundational subsystems (cert-manager's ClusterIssuers, and the -entire Authentik SSO stack) are invisible to Argo CD despite everything else -depending on them, a live database password and a live third-party API key -are committed in plaintext, and two high-value internal certificates are -silently issued by the wrong CA chain. None of this requires a restructure to -fix — it requires closing specific, identifiable gaps. Ansible is in good -shape except for a dual-`ansible.cfg` trap that breaks documented commands if -run from the wrong directory, and one likely-broken kubeconfig-fetch task in -the K3s bootstrap role. No CI or local validation tooling exists yet, which -is the main reason several of these gaps went unnoticed. - -## Prioritized Findings - -### Critical - -| # | Finding | Evidence | Impact | Recommendation | -|---|---|---|---|---| -| 1 | `manifests/cert-manager/` (both `ClusterIssuer`s + the internal CA `Certificate`) is never referenced by any Argo CD `Application` — confirmed by grep across all of `argocd/apps/*.yaml`, zero `path:` hits | `argocd/apps/cert-manager.yaml` only installs the Helm chart/controller; no sibling Application points at `manifests/cert-manager/` | Every TLS certificate in the cluster depends on issuers that exist only because someone once ran `kubectl apply` by hand. A cold rebuild following the documented bootstrap steps will **not** restore them, and nothing detects/corrects drift on the live ones. | Add `argocd/apps/cert-manager-config.yaml` (path: `manifests/cert-manager`, sync-wave before consumers). See Stage 5. | -| 2 | `manifests/authentik/` (7 files: ingress, proxy outpost, middleware, 4 secret scripts) and `values/authentik.yaml` are never referenced by any Application, and Authentik's manual `helm upgrade --install` isn't even in the documented bootstrap steps | Confirmed by the same grep sweep; `values/authentik.yaml:1`'s header comment is the *only* place the install command lives | The SSO IdP gating Argo CD, Grafana, and Gitea OAuth is entirely outside GitOps and undocumented as a bootstrap step. Disaster recovery would silently fail to restore login for everything behind it. | Add `argocd/apps/authentik.yaml` (chart) + `argocd/apps/authentik-config.yaml` (path: `manifests/authentik`), mirroring the cert-manager two-Application pattern. | -| 3 | Plaintext PostgreSQL password committed | `values/gitea.yaml:48` — `password: gitea-db-password` under `postgresql.global.postgresql.auth.password`, no `existingSecret` used (contrast `values/gitea.yaml:11`, which correctly uses `existingSecret: gitea-admin-secret`) | A real credential sits in git history now, readable by anyone with repo access. Contradicts the repo's own stated no-plaintext-secrets rule and `.claude/agents/secrets-leak-scanner.md`'s invariant. | Rotate the DB password, convert to a `*-secret.sh` → sealed-secret pattern (mirror `manifests/media/immich-postgres-secret.sh`). Treat the current value as compromised since it's in history. | -| 4 | Live third-party API key committed in plaintext | `config/dashy/conf.yaml:63` and `:70` — a weather-widget API key used twice | Anyone with repo read access (or Dashy's own JS, which ships client-side) can extract and reuse the key. | Rotate the key at the provider; inject at apply-time from `.env` via `manifests/core/apply-dashy-config.sh` instead of hardcoding in `config/dashy/conf.yaml`. | -| 5 | `router/backup-base.tar.gz` is tracked in git (added in commit `e7718ce`, ~8.8KB compressed) | Confirmed tracked via `git ls-files`; **not opened**, per safety constraints | `router/uci-base.conf` itself (which was read) is a sanitized base config — WiFi disabled with default open SSID, no VPN configured, sentinel password field — which makes it likely the fuller `.tar.gz` backup is where any real WiFi PSK/VPN key/admin credential would actually live. Cannot confirm what's inside without extracting it. | **Action needed from user**: inspect this archive outside this workflow. If it contains live credentials, both git-history scrubbing and router credential rotation should be considered. | -| 6 | Two high-value internal certificates are issued by the wrong CA | `manifests/authentik/authentik-ingress.yaml:11-12` (`auth.home.arpa`) and `manifests/network/traefik-dashboard-ingress.yaml:65-66` (`traefik.home.arpa`) both set `issuerRef.name: internal-ca` — the root self-signed issuer — instead of `internal-ca-issuer`, the chained issuer every other internal `Certificate` uses (verified pattern in `manifests/homeassistant/homeassistant.yaml:31`, `manifests/home-services/certs.yaml`, `manifests/argocd/argocd.yaml:9`, media manifests) | Devices that installed the CA via the `ca-installer`/`ca-sync` flow (`manifests/README.md:36-42`) will **not** trust these two certs — browser TLS warnings on exactly the SSO login and the Traefik dashboard. Compounded by #1/#2: neither file is under GitOps, so nothing corrects this automatically. | Change `issuerRef.name` to `internal-ca-issuer` in both files. | - -### High - -| # | Finding | Evidence | Impact | Recommendation | -|---|---|---|---|---| -| 7 | Dual `ansible.cfg` breaks every documented playbook command if run from `ansible/` instead of repo root — empirically reproduced, not just theorized | `ansible.cfg` (root) vs `ansible/ansible.cfg`; confirmed via `ansible-config dump` that only one loads at a time, and reproduced two independent failures (path-doubling, then role-not-found) plus a silent loss of `host_key_checking=False` | A user who `cd`s into `ansible/` first (a very natural mistake) gets confusing, unrelated-looking errors, or worse, an interactive host-key prompt in what's meant to be scriptable. | Delete `ansible/ansible.cfg`; fold its two settings (`inject_facts_as_vars: False`, `deprecation_warnings: False`) into the root `ansible.cfg`. Single source of truth, run only from repo root (already the documented convention). | -| 8 | Likely-broken kubeconfig path in K3s bootstrap | `ansible/roles/k3s-server/tasks/main.yaml:50-62` — fetches kubeconfig to `~/.kube/config`, but the next task's `ansible.builtin.replace` targets `/tmp/k3s-minisforum.yaml`, a path nothing else in the role writes to | `replace` errors if the target doesn't exist — this task likely fails on a genuinely fresh `bootstrap-minisforum.yaml`/`setup-k3s.yaml` run, i.e. exactly the disaster-recovery path this needs to work for. **Static-analysis finding, not yet confirmed against a live re-run** — flagged as an unknown too. | Fix the path mismatch (verify what the fetch task actually names the local file and point `replace` at that same path); test on an actual rebuild before trusting this for DR. | -| 9 | K3s version defined in 3 places, already out of sync | `ansible/roles/k3s-server/defaults/main.yaml:6` and `k3s-agent/defaults/main.yaml:6` both `v1.32.2+k3s1`; `ansible/host_vars/gpu-node.yaml:35` is `v1.32.4+k3s1`. `ansible/README.md:58-59`'s upgrade doc only mentions the first two locations. | A fresh `debian`/`minisforum` (re)provision installs a different K3s version than what's apparently already running on `gpu-node` — real version-skew risk, and the documented upgrade procedure would miss the gpu-node override entirely. | Pick one canonical version, align all three, document all three locations. | -| 10 | 4 of 6 files in `manifests/network/` are excluded from Argo CD sync | `argocd/apps/network-secrets.yaml:14-15` sets `directory.include: '*-sealed.yaml'`, so `ddns-cronjob.yaml`, `glances-debian-ingress.yaml`, `traefik-dashboard-ingress.yaml`, `watch-party-ingress.yaml` are never synced | These back real, documented services (`README.md:52,63,64`) with zero drift protection or self-heal — same class of gap as #1/#2, smaller blast radius. | Add a proper `network.yaml` Application (or widen the include filter) covering these; keep `network-secrets.yaml` scoped to sealed secrets only. | -| 11 | `gitea-backup` CronJob has cluster-wide exec privileges it doesn't need | `manifests/gitea/gitea-backup.yaml:11` (`ClusterRole`) / `:23` (`ClusterRoleBinding`) grant `pods/exec` cluster-wide; the CronJob script only execs into the `gitea` namespace (lines 61-72) | The backup ServiceAccount can exec into any pod in any namespace — real privilege-escalation surface for a component whose only job is backing up one namespace. | Replace with a namespaced `Role`/`RoleBinding` scoped to `gitea`. | -| 12 | `homeassistant` Application/manifest namespace mismatch | `argocd/apps/homeassistant.yaml:14` targets/creates namespace `homeassistant`; every resource in `manifests/homeassistant/homeassistant.yaml` is hardcoded to `namespace: default` (lines 6,17,27,40) — verified directly | Argo CD creates and "manages" an empty, unused `homeassistant` namespace while the real resources live in `default`. Confusing, latent bug, no current functional harm. | Change `destination.namespace` to `default` and drop `CreateNamespace=true` (no runtime-affecting change — resources already live in `default`). | -| 13 | `gitea-backup` namespace has no `Namespace` object anywhere and isn't auto-created | `manifests/gitea/gitea-backup.yaml:8,29,39,100` all reference namespace `gitea-backup`; the owning Application (`gitea-secrets.yaml`) targets namespace `gitea` with `CreateNamespace=true` — a different namespace | Would fail to apply on any cluster where `gitea-backup` namespace wasn't created by hand out-of-band. | Add an explicit `Namespace: gitea-backup` object to the manifest, or fold the backup CronJob into the `gitea` namespace. | -| 14 | No CI or local validation tooling anywhere in the repo | Confirmed via recursive search: no `.github/`, `.gitea/workflows/`, pre-commit config, or yamllint/ansible-lint config at any level. `yamllint`, `ansible-lint`, `kubeconform` not installed locally either (only `helm`, `kubectl`, `kubeseal`, `jq`, `ansible-playbook` are present) | Given "no staging environment," every change is validated by pushing and watching Argo CD/production react. This is the root cause that let findings #3, #4, #20 go unnoticed. | See Stage 2. | -| 15 | Missing `requirements.yml` + undocumented collection dependency | No `requirements.yml` anywhere; `README.md:76-78` only tells a new operator to install `community.general`/`ansible.posix`, but `ansible/roles/glances/tasks/main.yaml:24` and `ansible/roles/watch-party/tasks/main.yaml:27` require `community.docker`, never mentioned | A fresh workstation following the documented bootstrap verbatim gets a "module not found" failure on `setup-glances-debian.yaml`/`deploy-watch-party.yaml`. | Add `ansible/requirements.yml` pinning all three collections; update `README.md`. | -| 16 | Missing resource requests/limits and probes on a long list of workloads | `manifests/core/{dashy,glances}.yaml`, `manifests/core/ca-installer/ca-installer.yaml`, `manifests/authentik/authentik-proxy-outpost.yaml`, `manifests/media/jellyfin.yaml`, main containers in `qbittorrent.yaml`/`jdownloader.yaml` (only their `gluetun` sidecars have probes), all 4 Deployments in `manifests/media/immich.yaml` | No protection against one workload starving others on a small, fixed-capacity cluster; no automatic restart on hang for several user-facing services. Contrast: `manifests/home-services/*.yaml` and `manifests/portfolio/portfolio.yaml` do this correctly already — good templates to copy from. | See Stage 7. | - -### Medium - -| # | Finding | Evidence | Recommendation | -|---|---|---|---| -| 17 | Sync-wave ordering doesn't cover cert-manager/sealed-secrets vs. their consumers | Only `gitea-secrets`/`gitea`/`loki`/`monitoring-secrets`/`monitoring`/`network-secrets` carry `sync-wave` annotations; `cert-manager` and `sealed-secrets` (whose CRDs everything else's `Certificate`/`SealedSecret` objects need) are unannotated, same wave-0 as their consumers | Add `sync-wave: "-2"` to `cert-manager`/`sealed-secrets`, `-1` to their config Applications, consistent with the existing pattern. Self-heals today via retries, so this is about shortening a fresh-bootstrap flaky window, not fixing active breakage. | -| 18 | Undocumented sync-policy exceptions | `home-services`, `otel-collector`, `tempo` use `prune: true` against the repo's stated `prune: false` norm (`argocd/README.md:37`); `pihole`/`pihole-debian` are the only two with `selfHeal: false` (plausibly intentional, given the documented external-IP-loss gotcha, but never stated as policy) | Add a one-line YAML comment on each explaining the exception, or normalize them if unintentional — needs user input on intent. | -| 19 | 3 Applications use `targetRevision: HEAD` instead of the documented `main` | `home-services.yaml:12`, `otel-collector.yaml:17`, `tempo.yaml:17` | Normalize to `main`. Zero behavioral difference today; purely consistency. | -| 20 | Pi-hole DNS entries have already drifted | `values/pihole.yaml:81,85` (`immich.home.arpa`, `gluetun.home.arpa`) missing from `values/pihole-debian.yaml` | Add the two missing entries. (Going forward this is now caught by the existing `PostToolUse` hook in `.claude/settings.json` — it just doesn't catch pre-existing drift.) | -| 21 | Orphaned DNS record | `nik4nao.home.arpa` exists in both pihole values files but no Certificate/Ingress/IngressRoute anywhere references it | Confirm it's unused, then remove. | -| 22 | `ansible/README.md` mislabels the `homeassistant` role "Legacy" | `ansible/README.md:54`; but `manifests/homeassistant/homeassistant.yaml`'s `Endpoints` (hardcoded to `192.168.7.77:8123`) has nothing else in the repo standing up a listener at that address — the "legacy" Ansible role/Docker Compose deployment is the **only** thing actually serving it | Re-word the doc entry; this is load-bearing, not dead — don't let a future cleanup pass delete it. | -| 23 | `docker` role logic duplicated wholesale inside `homeassistant` role | `ansible/roles/homeassistant/tasks/main.yaml:2-49` reimplements Docker CE install (Debian apt path) instead of reusing `ansible/roles/docker/tasks/main.yaml:6-43` (Ubuntu apt path) | Parameterize `docker` role for both distros, have `homeassistant` depend on it instead of duplicating. | -| 24 | GPU passthrough likely non-functional as wired | `ansible/roles/nvidia/templates/k3s-containerd-config.toml.j2` exists but is never referenced by any task in that role; `manifests/home-services/nvidia-device-plugin.yaml:5-9` defines a `RuntimeClass handler: nvidia` that depends on exactly that containerd config existing on `gpu-node` | **Unknown pending live verification** — needs a check on `gpu-node` before deciding whether to wire up the template or remove the RuntimeClass. | -| 25 | Grafana forced to run as root | `values/kube-prometheus-stack.yaml:49-52` — `runAsNonRoot: false`, `runAsUser: 0`, likely to support `initChownData: true` fixing hostPath ownership | Test whether `fsGroup` alone (without full root) suffices; needs a live test window with rollback ready. | -| 26 | Mixed `Ingress` vs Traefik `IngressRoute` usage with no stated policy | e.g. `manifests/media/qbittorrent.yaml` uses plain `Ingress` for the app (line 176) but `IngressRoute` for `gluetun-api` in the same file (line 216) | Not broken, but only the `IngressRoute` half gets native `Middleware` support without annotation workarounds — worth standardizing on one mechanism over time, not urgent. | - -### Low (grouped) - -- **Ansible hygiene**: zero `tags:` usage anywhere (no selective runs possible); FQCN usage inconsistent (`wireguard` and `homeassistant` roles are 100% bare module names, `ollama` is mixed — `ansible/roles/{wireguard,homeassistant,ollama}/tasks/main.yaml`); `nvidia/tasks/main.yaml:58-62` uses `changed_when: true` unconditionally, restarting Docker every run; `gitea-runner/tasks/main.yaml:89-93` unconditionally deletes `/run/docker.sock` every run despite the task name implying a conditional check. -- **Operational secret leaks (not committed, but printed)**: `ansible/roles/k3s-server/tasks/main.yaml:46-48` prints the real K3s join token via `debug: msg:` on every run; `ansible/roles/gitea-runner/tasks/main.yaml:55-79` writes the runner registration token into a world-readable (`0644`) systemd unit file; `ansible/roles/wireguard/tasks/main.yaml:169-184` prints generated WireGuard private keys to console. None are committed to git, but all are worth tightening. -- **Doc drift**: `README.md:203-204` claims Gitea ingress is "disabled in `values/gitea.yaml`" — that file has no `ingress:` key at all (it's disabled by chart default, not an explicit setting); `README.md:174-186` groups Gitea's backups under `/mnt/storage` when they actually live on a separate NFS export (`/home/nik/backups`, per `manifests/gitea/gitea-backup.yaml:92` and `ansible/roles/nfs-server/templates/exports.j2:7`); `ansible/README.md`'s role table and playbook list omit the `docker`/`nvidia` roles, `setup-gpu-node.yaml`, and `homeassistant.yaml`; root `README.md`'s host table omits `gpu-node` entirely as a 4th host; `config/dashy/conf.yaml:2` points at `manifests/apply-dashy-config.sh`, missing the real `core/` path segment. -- **`.gitignore` gaps** (nothing currently leaking, but no coverage): `*.retry` (a live risk — Ansible drops these on playbook failure and this repo's Ansible tree is actively used), `.vault_pass*`, kubeconfig-shaped filenames, editor swap files. `.DS_Store` files exist in the working tree at repo root and `config/` despite being gitignored — confirm they're actually untracked, not just ignored-going-forward. - -## Duplicated / Ambiguous / Possibly Obsolete Resources - -Per the "trace references before declaring obsolete" rule — **none of the -"unsynced" items above are actually dead**; they're all in active use, just -invisible to Argo CD. The only things that look genuinely obsolete or -redundant are: - -- ~~`nik4nao.home.arpa` DNS entry~~ — **WRONG, corrected 2026-07-23**: this - points at `192.168.7.183` (nik-debian's raw IP) for a host-level SMB/Samba - share, not a Kubernetes Service — there was never going to be a - Certificate/Ingress/IngressRoute for it, so "no K8s resource references it" - was the wrong test entirely. It was removed from both pihole values files - and broke real SMB access; restored. Lesson: that specific check only - proves a hostname isn't routed through Traefik, not that it's unused — - doesn't apply to raw-IP entries for non-HTTP/non-K8s services. -- `ansible/ansible.cfg` — not obsolete exactly, but its only real-world effect today is negative (breaks documented commands); candidate for deletion/merge into the root cfg. -- `ansible/roles/nvidia/templates/k3s-containerd-config.toml.j2` — orphaned template, either finish wiring it up or remove it once GPU passthrough status is confirmed live. -- `.DS_Store` at repo root and in `config/` — should never have been committed; harmless but should be `git rm --cached`. - -## Security Findings Summary (redacted) - -- Two real credentials committed in plaintext: a database password (`values/gitea.yaml:48`) and a third-party API key (`config/dashy/conf.yaml:63,70`). Both should be treated as compromised and rotated — no value is printed anywhere in this document. -- One tracked binary (`router/backup-base.tar.gz`) whose contents were not inspected but is flagged as the most likely place real WiFi/VPN/router-admin credentials would live, given the sibling `uci-base.conf` is sanitized. Needs review outside this workflow; rotation and/or history-scrubbing may be warranted depending on what's inside. -- No plaintext secrets found in Ansible beyond the two operational-leak items (console-printed token, world-readable systemd unit) — the vault (`ansible/group_vars/all/vault.yaml`) is genuinely encrypted and used correctly. -- RBAC: one real over-scope (`gitea-backup`'s cluster-wide `pods/exec`), no wildcard verbs/resources found anywhere else. -- Existing guardrails already partially cover this class of problem going forward: `.claude/agents/secrets-leak-scanner.md` and the `PostToolUse` hook in `.claude/settings.json` — but the scanner's stated scope reads as Ansible/manifests/Helm-values-centric and doesn't explicitly call out `config/**`, which is exactly where the Dashy API key slipped through. Worth widening its scope statement rather than adding new tooling. - -## What Should Be Preserved - -- The Ansible → Argo CD boundary itself is clean — no Ansible task embeds Kubernetes YAML or runs `kubectl apply` against application manifests (verified by repo-wide grep). The one-time hand-off (`helm install argocd` → `kubectl apply -f app-of-apps.yaml`) is exactly right for this scale. -- The app-of-apps + one-Application-per-file pattern (20 flat files in `argocd/apps/`) is simple, greppable, and easy to reason about at this service count — an ApplicationSet or Kustomize layer would add indirection with no real benefit here. -- The two-pattern secrets model (sealed-secrets for committed ciphertext, runtime `.env`-sourced scripts for cluster-only secrets) is coherent and consistently documented across all three READMEs. -- Exact Helm chart version pinning on every chart-backed Application — zero floating versions found. -- No deprecated Kubernetes API versions anywhere — clean. -- Idempotent `curl | sh`-style installs are consistently guarded with `creates:` across `docker`, `nvidia`, `k3s-server`, `k3s-agent`, `ollama` roles. -- `.claude/agents/infra-change-reviewer.md` and `secrets-leak-scanner.md` plus the `settings.json` hooks already encode a lot of this audit's tribal knowledge (DNS-sync rule, sealed-secret hand-edit ban, cert-issuer split, GPU taint/toleration, Ansible dual-cfg risk) — build on these, don't replace them. -- Good in-repo examples worth using as the template when fixing the bad ones: `manifests/home-services/nvidia-device-plugin.yaml` (privilege drop), `manifests/core/ca-installer/ca-sync.yaml` (tightly-scoped namespaced RBAC), `manifests/home-services/{ai-gateway,ha-gateway}.yaml` + `manifests/portfolio/portfolio.yaml` (proper resources + probes). - -## Unknowns (need live cluster info or user input — not determinable from the repo alone) - -1. Whether Grafana's PVC actually binds to the static `grafana-pv` hostPath or silently gets dynamically provisioned via `local-path` instead (`values/kube-prometheus-stack.yaml` sets no `storageClassName`/`volumeName` for Grafana, unlike Prometheus/Loki which do). Needs `kubectl get pvc -n monitoring` / `kubectl get pv grafana-pv -o yaml`. -2. Whether GPU passthrough is actually functional today given the orphaned containerd template (finding #24). Needs a check on `gpu-node`. -3. What's actually inside `router/backup-base.tar.gz` (finding #5) — deliberately not inspected. -4. Whether the K3s bootstrap kubeconfig-path bug (finding #8) actually breaks a fresh run, or whether there's missing context from an untraced earlier step. Static reading only — recommend confirming before relying on it for DR. -5. Whether `selfHeal: false` on the two pihole Applications is deliberate policy (tied to the external-IP-loss gotcha) or an oversight. -6. What Authentik Helm chart version is actually running — it's outside Argo CD, so there's no `targetRevision` to read; needs `helm list -n authentik` (or wherever it's installed) on the live cluster. -7. Whether `values/gitea.yaml`'s committed password and Dashy's committed API key have ever been exposed beyond the operator (repo visibility/access history) — affects how urgent rotation is. - ---- - -## Phase 4 — Target Architecture - -Given this is a single cluster, single operator, ~20-service homelab with no -near-term multi-cluster plan, two credible options were considered. - -### Option A (recommended): Keep the current three-way split, close the gaps - -Keep `ansible/` / `argocd/apps/` / `manifests/` / `values/` exactly as they -are structurally — this layout already matches how the system is organized -conceptually, how the existing subagents and skills reference paths, and how -the current documentation is written. Fix the specific holes (findings -above) rather than moving files around. - -``` -homelab/ -├── ansible.cfg # single config, root-only (ansible/ansible.cfg removed) -├── ansible/ -│ ├── requirements.yml # NEW — pins community.general, ansible.posix, community.docker -│ ├── inventory.yaml -│ ├── group_vars/ · host_vars/ -│ ├── playbooks/ -│ └── roles/ -├── argocd/ -│ ├── apps/ # + cert-manager-config.yaml, authentik.yaml, authentik-config.yaml -│ └── values/ -├── manifests/ # unchanged tree; every subdir now has a matching Application -├── values/ # unchanged -├── config/dashy/ -├── router/ -├── .yamllint.yml # NEW -├── .ansible-lint # NEW (config only, tool installed separately) -└── CLAUDE.md # + validation commands section -``` - -**Why not restructure further**: there's no duplication cost today that a -`clusters/`/`infrastructure/`/`apps/` split or Kustomize bases would actually -reduce — each service already has exactly one Application file and at most -one values file. Introducing Kustomize overlays or ApplicationSets would add -a templating/indirection layer this repo doesn't need at 1-cluster, -~20-service scale, and would itself become something to maintain and -explain. - -### Option B (not recommended now): `clusters/` + `infrastructure/` + `apps/`, colocated values - -Restructure to one directory per service (e.g. -`apps/gitea/{application.yaml,values.yaml}`), split `infrastructure/` -(cert-manager, sealed-secrets, traefik, pihole) from `apps/` (user-facing -services), under a `clusters/homelab/` root to leave room for a future -second cluster. This gives slightly tighter per-service ownership (one -directory instead of two: `argocd/apps/x.yaml` + `values/x.yaml`) but -requires moving **every** Application's `source.path`, which Argo CD treats -as a new resource identity — each move is a prune-risk event that must be -done one Application at a time with `prune: false` verified beforehand, -diffed, and rolled out carefully. That's a lot of migration risk for a -benefit (one fewer directory hop per service) that doesn't solve any problem -that exists today, and it presumes a multi-cluster future that isn't -planned. - -**Recommendation: Option A.** Revisit Option B only if a second cluster is -actually added or the service count grows to where cross-referencing -`argocd/apps/` and `values/` by hand becomes genuinely painful — neither is -true today. - ---- - -## Phase 5 — Incremental Migration Plan - -Every stage is designed to leave the repo in a working, deployable state at -every commit boundary. Stages that touch what Argo CD currently manages are -explicitly marked **RUNTIME-AFFECTING** and require explicit go-ahead before -pushing, per the repo's own rule that `git push` is a production deploy. - -| Stage | Scope | Key changes | Runtime impact | Rollback | -|---|---|---|---|---| -| **1. Safety fixes & secret hygiene** | Findings #3, #4, #6, plus `.gitignore` gaps | Rotate + re-secret the Gitea DB password and Dashy API key; fix the two `issuerRef` values; extend `.gitignore` (`*.retry`, `.vault_pass*`, kubeconfig patterns, swap files); `git rm --cached` the `.DS_Store` files | **RUNTIME-AFFECTING** for the password rotation (Gitea/Postgres restart) and cert reissuance (brief TLS transition on 2 hostnames); everything else is zero-impact | Revert commit; for password rotation, keep old secret retrievable until new one is confirmed working | -| **2. Validation tooling & clean baseline** | New root-level config | Add `.yamllint.yml`, `.ansible-lint`; install `yamllint`/`ansible-lint`/`kubeconform` locally (asking first, since that's installing software); run baseline, fix anything trivial it finds | None — local/offline only | Uninstall tools / delete config files | -| **3. Remove confirmed junk** | `.DS_Store`, orphaned `nik4nao.home.arpa` DNS entry, doc-path typo in `config/dashy/conf.yaml:2` | Delete/clean | Negligible (DNS entry removal is additive-safe to revert) | `git revert` | -| **4. Ansible cleanup** | Findings #7, #9, #15, plus low-severity items | Merge `ansible/ansible.cfg` into root; add `requirements.yml`; align `k3s_version` across 3 locations; fix `gitea-runner` docker.sock task, `nvidia` `changed_when`, K3s token debug print, systemd unit permissions; update both READMEs (gpu-node host, docker/nvidia roles, homeassistant status) | **RUNTIME-AFFECTING** only for the `k3s_version` alignment (changes what a future node provision installs) — needs sign-off on which version is canonical; rest is inert until playbooks rerun | Git revert; k3s_version change only takes effect on next actual provision, not immediately | -| **5. Argo CD bootstrap normalization** | Findings #1, #2, #10, #12, #13, #17, #19 | New `cert-manager-config.yaml` + `authentik.yaml`/`authentik-config.yaml` Applications (first-time adoption of already-live resources — `kubectl diff` before enabling automated sync on each); widen `network-secrets` coverage or add `network.yaml`; fix homeassistant namespace; add `gitea-backup` Namespace + narrow its RBAC to a Role; add missing sync-waves; normalize `targetRevision: HEAD`→`main` | **RUNTIME-AFFECTING, highest-risk stage** — bringing previously-unmanaged live resources under Argo CD for the first time. Roll out one Application at a time, diff first, watch sync status before moving to the next | Set the newly-added Application's sync policy back to manual, or delete the Application (resources stay, since `prune: false`) | -| **6. Values/DNS consolidation** | Finding #18, #20 | Reconcile `pihole.yaml`/`pihole-debian.yaml` drift (add missing entries); document or normalize the `prune:true`/`selfHeal:false` outliers (pending intent) | **RUNTIME-AFFECTING but low-risk** — adds DNS records only, additive | Revert values file | -| **7. Kubernetes correctness & security** | Findings #11, #16, #25, #26 | Add resource requests/limits + probes to flagged workloads (one service at a time); narrow `gitea-backup` RBAC; test Grafana `runAsNonRoot`/`fsGroup`-only; pin remaining `:latest` images to currently-running tags | **RUNTIME-AFFECTING** — limits set too low can OOMKill; needs `kubectl top` baselines first (gather live before proposing numbers); roll out one service at a time with rollback ready | Revert manifest, resync | -| **8. Documentation & DR runbook** | All doc-drift items | Fix host tables, role tables, playbook lists, ingress/storage claims; add an explicit cold-rebuild runbook reflecting the now-closed GitOps gaps | None | Revert | -| **9. CI & Claude Code guidance** | New `.gitea/workflows/validate.yaml` (yamllint/ansible-lint/kubeconform on push); `CLAUDE.md` validation-commands section; widen `secrets-leak-scanner.md`'s stated scope to include `config/**`; consider extending the `PreToolUse` hook to block Claude Code from reading `.env`/`router/backup-base.tar.gz` directly | New automation surface | None to the cluster; new CI pipeline needs approval since it's a new automation surface | Delete workflow file | - -Every stage gets validated with whatever's actually available -(`ansible-playbook --syntax-check`, `yamllint`, `ansible-lint`, -`kubeconform`/`kubectl apply --dry-run=client` where a live context isn't -required) before being reported done — with the concrete diff and validation -output shown, not just a claim of success. - ---- - -## Claude Code Project Support - -- **`CLAUDE.md`**: keep it, add a short "Validation commands" section once Stage 2 lands (exact `yamllint`/`ansible-lint`/`ansible-playbook --syntax-check` invocations). -- **Guard against accidental `.env`/key/backup access**: partially covered already (`.gitignore`, sealed-secret `PreToolUse` hook). Extend that hook to also block Claude Code `Read` of `.env` and `router/backup-base.tar.gz` — small addition, Stage 9. -- **One validation skill/command**: rather than a new skill, widen `secrets-leak-scanner.md`'s stated file scope to explicitly include `config/**` (the actual gap that let the Dashy API key through) — cheaper and more targeted than adding new tooling.