# Homelab Repo Audit & Refactor Plan Living document. Produced by a full read-only audit of the repo (Ansible, Argo CD, manifests, values, config, router). Nothing was applied, synced, committed, or pushed as part of producing this. We work through the "Migration Plan" stages one at a time, referencing this file; update the Status column as stages land. **Audit constraints honored**: `.env` was never opened; `router/backup-base.tar.gz` was never extracted; no `kubectl apply` / `helm install` / Argo CD sync / Ansible-against-real-hosts was run. All findings are grounded in file citations, and the highest-severity ones were independently re-verified (not just taken from sub-agent research output) before being recorded here. **Immich removed entirely, 2026-07-23**: not used enough to justify chasing its stale-NFS-handle issue. `manifests/media/immich.yaml` (+ its secret script/sealed secret), `immich.home.arpa` DNS entries, its Dashy tile, and all other repo references are gone. Findings #16 and #20 that mentioned Immich are now moot for it specifically (kept as historical record, not rewritten). Live namespace/PVCs need a manual `kubectl delete namespace immich` to actually free the data -- Argo CD's `media` Application won't do this itself (`prune: false`). ## Stage Tracker | Stage | Scope | Status | |---|---|---| | 1 | Safety fixes & secret hygiene | Files done, committed locally, **not pushed**. Actual secret rotation (new password/API key values + live DB password change) still needs the user — see note below. | | 2 | Validation tooling & clean baseline | Done. `yamllint`/`ansible-lint`/`kubeconform` installed by user; `.yamllint.yml`/`.ansible-lint` added; baseline clean (0 yamllint issues, `kubeconform`: 117+20 resources valid, all playbooks pass `--syntax-check`). Remaining `ansible-lint` findings are either pre-existing role/var naming conventions (out of scope — would require repo-wide renames) or collections-not-installed noise that resolves once `ansible-galaxy collection install -r ansible/requirements.yml` is run. | | 3 | Remove confirmed junk | Done, committed locally. `.DS_Store` confirmed never tracked (no action needed). | | 4 | Ansible cleanup | Done, committed locally. | | 5 | Argo CD bootstrap normalization | Files done, committed locally, **not pushed — highest-risk stage**. `authentik.yaml`'s chart version is a placeholder needing your input (`helm list -n authentik`); every new/changed Application here needs `kubectl diff` one at a time before/after enabling, not a blind batch push. | | 6 | Values/DNS consolidation | Done, committed locally, **not pushed**. | | 7 | Kubernetes correctness & security | Resource requests/limits + probes done (finding #16), committed locally, **not pushed**. `gitea-backup` RBAC narrowing (#11) done as part of Stage 5's commit. Grafana `runAsNonRoot`/`fsGroup` test (#25) and `:latest` image pinning not done — both need a live test window / registry inspection I didn't do unprompted. | | 8 | Documentation & DR runbook | Done, committed locally. | | 9 | CI & Claude Code guidance | Done, committed locally, **not pushed** (new CI automation surface). | **Correction, 2026-07-23**: the original `existingSecret`-based fix for Finding #3 was wrong and caused a real Gitea outage. Confirmed directly from the chart source (`gitea/helm-gitea` `templates/gitea/config.yaml`): `{{ set .Values.gitea.config.database "PASSWD" .Values.postgresql.global.postgresql.auth.password }}` — Gitea's own `[database]` config is *always* built from this plaintext values field, regardless of `existingSecret` (which only affects the bundled postgresql subchart's own credential provisioning, a separate concern). Removing the plaintext field made it silently fall back to the chart's built-in default password, breaking every fresh Gitea pod's DB connection. `values/gitea.yaml` is back to a plaintext (rotated) password with a comment explaining why, and `manifests/gitea/gitea-postgres-secret.sh` was removed. Getting this fully off plaintext would need a Config Management Plugin or similar — not attempted here; flagging as a real open item if it matters enough to invest in. Dashy's API key fix (env-injection via `manifests/core/apply-dashy-config.sh`) is unaffected by this and still needs you to rotate the actual key value. ### New findings from this session's live-cluster checks (not in the original audit) - **Grafana and Loki's static PVs are orphaned.** `manifests/monitoring/monitoring-pvs.yaml` defines `grafana-pv`/`loki-pv` hostPath PVs, but live `kubectl get pv` shows both sitting `Available` (unbound) — their Helm-managed PVCs got dynamically provisioned via `local-path` instead, unlike `prometheus-pv` which binds correctly. Data isn't lost, just not where the README/manifest imply. Needs a decision: bind properly (data migration required) or drop the orphaned static PVs and document reality. Not yet actioned. - **Finding #24 (GPU passthrough) is resolved as working, not broken.** Live check on node `nik-gpu`: `nvidia.com/gpu: "1"` allocatable, `nvidia` runtime handler registered in containerd, `nvidia-device-plugin` pod `Running`. The orphaned-looking `k3s-containerd-config.toml.j2` template is a non-issue in practice — no action needed. - **Unknown #1 (Grafana PVC) resolved**: dynamically provisioned via `local-path`, not the static PV — see the orphaned-PV finding above, same root cause. - Node hostnames in the live cluster are `nik-debian`/`nik-gpu`, not `debian`/`gpu-node` as in `ansible/inventory.yaml` — cosmetic (K3s registers by actual hostname), doesn't affect anything, just noted for anyone cross-referencing `kubectl get nodes` against the docs. Open decisions needed from the user before/during the relevant stage: - Finding #5 (`router/backup-base.tar.gz`): needs manual review outside this workflow; may imply git-history scrub / router credential rotation. - Finding #9 (K3s version skew): which version is canonical (`v1.32.2+k3s1` vs `v1.32.4+k3s1`)? - Finding #18: is `prune:true` on `home-services`/`otel-collector`/`tempo` and `selfHeal:false` on both pihole Applications intentional policy? - Finding #25: Grafana `runAsUser: 0` — worth testing `fsGroup`-only instead? --- ## Phase 1 — Current-State Architecture ``` Ansible (hosts) Argo CD (cluster) ───────────────── ────────────────────────────────────────── minisforum (K3s server, .77) ──┐ debian (K3s agent, .183) ────┼─▶ K3s cluster ──▶ helm install argocd (manual, README.md:92-103) gpu-node (K3s agent, .98) ──┘ │ │ mac-mini (standalone Docker) │ ▼ │ kubectl apply -f manifests/argocd/app-of-apps.yaml (manual, one-time) │ │ │ ▼ │ Application: app-of-apps → watches argocd/apps/*.yaml │ │ │ ┌───────────┴────────────────────────────┐ │ ▼ ▼ │ Helm-chart Applications Raw-manifest Applications │ (chart + values/.yaml) (path: manifests/) │ traefik, cert-manager*, gitea, core, media, home-services, │ pihole, pihole-debian, monitoring, portfolio, homeassistant, │ loki, tempo, otel-collector, *-secrets (sealed only) │ sealed-secrets, argocd(self) │ └── NOT reconciled by Argo CD at all: manifests/cert-manager/ (ClusterIssuers) manifests/authentik/ + values/authentik.yaml 4 files in manifests/network/ (non-sealed) → applied by hand, outside GitOps ``` Host-level services that never touch the K3s cluster: Home Assistant (Docker Compose + systemd on `minisforum`, `ansible/roles/homeassistant/`), Watch Party and Ollama (`mac-mini`), Ollama (`gpu-node`) — all Ansible-managed. Bootstrap sequence as documented (`README.md:71-105`, `argocd/README.md:7-24`): Ansible playbooks → manual `helm install argocd` → manual `kubectl apply` of app-of-apps → everything else via Git. **This documented sequence is incomplete** — it never mentions installing Authentik or the cert-manager `ClusterIssuer`s, both required for the rest of the stack to actually work (see Critical findings below). ## Executive Summary The repo is coherently designed for a single-operator homelab: Ansible for host bootstrap, Argo CD app-of-apps for cluster reconciliation, a clean two-pattern secrets model (sealed-secrets for committed ciphertext, runtime scripts for `.env`-sourced live secrets), and a genuinely useful pair of existing Claude Code subagents (`infra-change-reviewer`, `secrets-leak-scanner`) that already encode a lot of hard-won tribal knowledge. There's no fashionable-but-wrong abstraction problem here — the biggest risks are **gaps between what the documentation/GitOps model claims and what's actually wired up**: two foundational subsystems (cert-manager's ClusterIssuers, and the entire Authentik SSO stack) are invisible to Argo CD despite everything else depending on them, a live database password and a live third-party API key are committed in plaintext, and two high-value internal certificates are silently issued by the wrong CA chain. None of this requires a restructure to fix — it requires closing specific, identifiable gaps. Ansible is in good shape except for a dual-`ansible.cfg` trap that breaks documented commands if run from the wrong directory, and one likely-broken kubeconfig-fetch task in the K3s bootstrap role. No CI or local validation tooling exists yet, which is the main reason several of these gaps went unnoticed. ## Prioritized Findings ### Critical | # | Finding | Evidence | Impact | Recommendation | |---|---|---|---|---| | 1 | `manifests/cert-manager/` (both `ClusterIssuer`s + the internal CA `Certificate`) is never referenced by any Argo CD `Application` — confirmed by grep across all of `argocd/apps/*.yaml`, zero `path:` hits | `argocd/apps/cert-manager.yaml` only installs the Helm chart/controller; no sibling Application points at `manifests/cert-manager/` | Every TLS certificate in the cluster depends on issuers that exist only because someone once ran `kubectl apply` by hand. A cold rebuild following the documented bootstrap steps will **not** restore them, and nothing detects/corrects drift on the live ones. | Add `argocd/apps/cert-manager-config.yaml` (path: `manifests/cert-manager`, sync-wave before consumers). See Stage 5. | | 2 | `manifests/authentik/` (7 files: ingress, proxy outpost, middleware, 4 secret scripts) and `values/authentik.yaml` are never referenced by any Application, and Authentik's manual `helm upgrade --install` isn't even in the documented bootstrap steps | Confirmed by the same grep sweep; `values/authentik.yaml:1`'s header comment is the *only* place the install command lives | The SSO IdP gating Argo CD, Grafana, and Gitea OAuth is entirely outside GitOps and undocumented as a bootstrap step. Disaster recovery would silently fail to restore login for everything behind it. | Add `argocd/apps/authentik.yaml` (chart) + `argocd/apps/authentik-config.yaml` (path: `manifests/authentik`), mirroring the cert-manager two-Application pattern. | | 3 | Plaintext PostgreSQL password committed | `values/gitea.yaml:48` — `password: gitea-db-password` under `postgresql.global.postgresql.auth.password`, no `existingSecret` used (contrast `values/gitea.yaml:11`, which correctly uses `existingSecret: gitea-admin-secret`) | A real credential sits in git history now, readable by anyone with repo access. Contradicts the repo's own stated no-plaintext-secrets rule and `.claude/agents/secrets-leak-scanner.md`'s invariant. | Rotate the DB password, convert to a `*-secret.sh` → sealed-secret pattern (mirror `manifests/media/immich-postgres-secret.sh`). Treat the current value as compromised since it's in history. | | 4 | Live third-party API key committed in plaintext | `config/dashy/conf.yaml:63` and `:70` — a weather-widget API key used twice | Anyone with repo read access (or Dashy's own JS, which ships client-side) can extract and reuse the key. | Rotate the key at the provider; inject at apply-time from `.env` via `manifests/core/apply-dashy-config.sh` instead of hardcoding in `config/dashy/conf.yaml`. | | 5 | `router/backup-base.tar.gz` is tracked in git (added in commit `e7718ce`, ~8.8KB compressed) | Confirmed tracked via `git ls-files`; **not opened**, per safety constraints | `router/uci-base.conf` itself (which was read) is a sanitized base config — WiFi disabled with default open SSID, no VPN configured, sentinel password field — which makes it likely the fuller `.tar.gz` backup is where any real WiFi PSK/VPN key/admin credential would actually live. Cannot confirm what's inside without extracting it. | **Action needed from user**: inspect this archive outside this workflow. If it contains live credentials, both git-history scrubbing and router credential rotation should be considered. | | 6 | Two high-value internal certificates are issued by the wrong CA | `manifests/authentik/authentik-ingress.yaml:11-12` (`auth.home.arpa`) and `manifests/network/traefik-dashboard-ingress.yaml:65-66` (`traefik.home.arpa`) both set `issuerRef.name: internal-ca` — the root self-signed issuer — instead of `internal-ca-issuer`, the chained issuer every other internal `Certificate` uses (verified pattern in `manifests/homeassistant/homeassistant.yaml:31`, `manifests/home-services/certs.yaml`, `manifests/argocd/argocd.yaml:9`, media manifests) | Devices that installed the CA via the `ca-installer`/`ca-sync` flow (`manifests/README.md:36-42`) will **not** trust these two certs — browser TLS warnings on exactly the SSO login and the Traefik dashboard. Compounded by #1/#2: neither file is under GitOps, so nothing corrects this automatically. | Change `issuerRef.name` to `internal-ca-issuer` in both files. | ### High | # | Finding | Evidence | Impact | Recommendation | |---|---|---|---|---| | 7 | Dual `ansible.cfg` breaks every documented playbook command if run from `ansible/` instead of repo root — empirically reproduced, not just theorized | `ansible.cfg` (root) vs `ansible/ansible.cfg`; confirmed via `ansible-config dump` that only one loads at a time, and reproduced two independent failures (path-doubling, then role-not-found) plus a silent loss of `host_key_checking=False` | A user who `cd`s into `ansible/` first (a very natural mistake) gets confusing, unrelated-looking errors, or worse, an interactive host-key prompt in what's meant to be scriptable. | Delete `ansible/ansible.cfg`; fold its two settings (`inject_facts_as_vars: False`, `deprecation_warnings: False`) into the root `ansible.cfg`. Single source of truth, run only from repo root (already the documented convention). | | 8 | Likely-broken kubeconfig path in K3s bootstrap | `ansible/roles/k3s-server/tasks/main.yaml:50-62` — fetches kubeconfig to `~/.kube/config`, but the next task's `ansible.builtin.replace` targets `/tmp/k3s-minisforum.yaml`, a path nothing else in the role writes to | `replace` errors if the target doesn't exist — this task likely fails on a genuinely fresh `bootstrap-minisforum.yaml`/`setup-k3s.yaml` run, i.e. exactly the disaster-recovery path this needs to work for. **Static-analysis finding, not yet confirmed against a live re-run** — flagged as an unknown too. | Fix the path mismatch (verify what the fetch task actually names the local file and point `replace` at that same path); test on an actual rebuild before trusting this for DR. | | 9 | K3s version defined in 3 places, already out of sync | `ansible/roles/k3s-server/defaults/main.yaml:6` and `k3s-agent/defaults/main.yaml:6` both `v1.32.2+k3s1`; `ansible/host_vars/gpu-node.yaml:35` is `v1.32.4+k3s1`. `ansible/README.md:58-59`'s upgrade doc only mentions the first two locations. | A fresh `debian`/`minisforum` (re)provision installs a different K3s version than what's apparently already running on `gpu-node` — real version-skew risk, and the documented upgrade procedure would miss the gpu-node override entirely. | Pick one canonical version, align all three, document all three locations. | | 10 | 4 of 6 files in `manifests/network/` are excluded from Argo CD sync | `argocd/apps/network-secrets.yaml:14-15` sets `directory.include: '*-sealed.yaml'`, so `ddns-cronjob.yaml`, `glances-debian-ingress.yaml`, `traefik-dashboard-ingress.yaml`, `watch-party-ingress.yaml` are never synced | These back real, documented services (`README.md:52,63,64`) with zero drift protection or self-heal — same class of gap as #1/#2, smaller blast radius. | Add a proper `network.yaml` Application (or widen the include filter) covering these; keep `network-secrets.yaml` scoped to sealed secrets only. | | 11 | `gitea-backup` CronJob has cluster-wide exec privileges it doesn't need | `manifests/gitea/gitea-backup.yaml:11` (`ClusterRole`) / `:23` (`ClusterRoleBinding`) grant `pods/exec` cluster-wide; the CronJob script only execs into the `gitea` namespace (lines 61-72) | The backup ServiceAccount can exec into any pod in any namespace — real privilege-escalation surface for a component whose only job is backing up one namespace. | Replace with a namespaced `Role`/`RoleBinding` scoped to `gitea`. | | 12 | `homeassistant` Application/manifest namespace mismatch | `argocd/apps/homeassistant.yaml:14` targets/creates namespace `homeassistant`; every resource in `manifests/homeassistant/homeassistant.yaml` is hardcoded to `namespace: default` (lines 6,17,27,40) — verified directly | Argo CD creates and "manages" an empty, unused `homeassistant` namespace while the real resources live in `default`. Confusing, latent bug, no current functional harm. | Change `destination.namespace` to `default` and drop `CreateNamespace=true` (no runtime-affecting change — resources already live in `default`). | | 13 | `gitea-backup` namespace has no `Namespace` object anywhere and isn't auto-created | `manifests/gitea/gitea-backup.yaml:8,29,39,100` all reference namespace `gitea-backup`; the owning Application (`gitea-secrets.yaml`) targets namespace `gitea` with `CreateNamespace=true` — a different namespace | Would fail to apply on any cluster where `gitea-backup` namespace wasn't created by hand out-of-band. | Add an explicit `Namespace: gitea-backup` object to the manifest, or fold the backup CronJob into the `gitea` namespace. | | 14 | No CI or local validation tooling anywhere in the repo | Confirmed via recursive search: no `.github/`, `.gitea/workflows/`, pre-commit config, or yamllint/ansible-lint config at any level. `yamllint`, `ansible-lint`, `kubeconform` not installed locally either (only `helm`, `kubectl`, `kubeseal`, `jq`, `ansible-playbook` are present) | Given "no staging environment," every change is validated by pushing and watching Argo CD/production react. This is the root cause that let findings #3, #4, #20 go unnoticed. | See Stage 2. | | 15 | Missing `requirements.yml` + undocumented collection dependency | No `requirements.yml` anywhere; `README.md:76-78` only tells a new operator to install `community.general`/`ansible.posix`, but `ansible/roles/glances/tasks/main.yaml:24` and `ansible/roles/watch-party/tasks/main.yaml:27` require `community.docker`, never mentioned | A fresh workstation following the documented bootstrap verbatim gets a "module not found" failure on `setup-glances-debian.yaml`/`deploy-watch-party.yaml`. | Add `ansible/requirements.yml` pinning all three collections; update `README.md`. | | 16 | Missing resource requests/limits and probes on a long list of workloads | `manifests/core/{dashy,glances}.yaml`, `manifests/core/ca-installer/ca-installer.yaml`, `manifests/authentik/authentik-proxy-outpost.yaml`, `manifests/media/jellyfin.yaml`, main containers in `qbittorrent.yaml`/`jdownloader.yaml` (only their `gluetun` sidecars have probes), all 4 Deployments in `manifests/media/immich.yaml` | No protection against one workload starving others on a small, fixed-capacity cluster; no automatic restart on hang for several user-facing services. Contrast: `manifests/home-services/*.yaml` and `manifests/portfolio/portfolio.yaml` do this correctly already — good templates to copy from. | See Stage 7. | ### Medium | # | Finding | Evidence | Recommendation | |---|---|---|---| | 17 | Sync-wave ordering doesn't cover cert-manager/sealed-secrets vs. their consumers | Only `gitea-secrets`/`gitea`/`loki`/`monitoring-secrets`/`monitoring`/`network-secrets` carry `sync-wave` annotations; `cert-manager` and `sealed-secrets` (whose CRDs everything else's `Certificate`/`SealedSecret` objects need) are unannotated, same wave-0 as their consumers | Add `sync-wave: "-2"` to `cert-manager`/`sealed-secrets`, `-1` to their config Applications, consistent with the existing pattern. Self-heals today via retries, so this is about shortening a fresh-bootstrap flaky window, not fixing active breakage. | | 18 | Undocumented sync-policy exceptions | `home-services`, `otel-collector`, `tempo` use `prune: true` against the repo's stated `prune: false` norm (`argocd/README.md:37`); `pihole`/`pihole-debian` are the only two with `selfHeal: false` (plausibly intentional, given the documented external-IP-loss gotcha, but never stated as policy) | Add a one-line YAML comment on each explaining the exception, or normalize them if unintentional — needs user input on intent. | | 19 | 3 Applications use `targetRevision: HEAD` instead of the documented `main` | `home-services.yaml:12`, `otel-collector.yaml:17`, `tempo.yaml:17` | Normalize to `main`. Zero behavioral difference today; purely consistency. | | 20 | Pi-hole DNS entries have already drifted | `values/pihole.yaml:81,85` (`immich.home.arpa`, `gluetun.home.arpa`) missing from `values/pihole-debian.yaml` | Add the two missing entries. (Going forward this is now caught by the existing `PostToolUse` hook in `.claude/settings.json` — it just doesn't catch pre-existing drift.) | | 21 | Orphaned DNS record | `nik4nao.home.arpa` exists in both pihole values files but no Certificate/Ingress/IngressRoute anywhere references it | Confirm it's unused, then remove. | | 22 | `ansible/README.md` mislabels the `homeassistant` role "Legacy" | `ansible/README.md:54`; but `manifests/homeassistant/homeassistant.yaml`'s `Endpoints` (hardcoded to `192.168.7.77:8123`) has nothing else in the repo standing up a listener at that address — the "legacy" Ansible role/Docker Compose deployment is the **only** thing actually serving it | Re-word the doc entry; this is load-bearing, not dead — don't let a future cleanup pass delete it. | | 23 | `docker` role logic duplicated wholesale inside `homeassistant` role | `ansible/roles/homeassistant/tasks/main.yaml:2-49` reimplements Docker CE install (Debian apt path) instead of reusing `ansible/roles/docker/tasks/main.yaml:6-43` (Ubuntu apt path) | Parameterize `docker` role for both distros, have `homeassistant` depend on it instead of duplicating. | | 24 | GPU passthrough likely non-functional as wired | `ansible/roles/nvidia/templates/k3s-containerd-config.toml.j2` exists but is never referenced by any task in that role; `manifests/home-services/nvidia-device-plugin.yaml:5-9` defines a `RuntimeClass handler: nvidia` that depends on exactly that containerd config existing on `gpu-node` | **Unknown pending live verification** — needs a check on `gpu-node` before deciding whether to wire up the template or remove the RuntimeClass. | | 25 | Grafana forced to run as root | `values/kube-prometheus-stack.yaml:49-52` — `runAsNonRoot: false`, `runAsUser: 0`, likely to support `initChownData: true` fixing hostPath ownership | Test whether `fsGroup` alone (without full root) suffices; needs a live test window with rollback ready. | | 26 | Mixed `Ingress` vs Traefik `IngressRoute` usage with no stated policy | e.g. `manifests/media/qbittorrent.yaml` uses plain `Ingress` for the app (line 176) but `IngressRoute` for `gluetun-api` in the same file (line 216) | Not broken, but only the `IngressRoute` half gets native `Middleware` support without annotation workarounds — worth standardizing on one mechanism over time, not urgent. | ### Low (grouped) - **Ansible hygiene**: zero `tags:` usage anywhere (no selective runs possible); FQCN usage inconsistent (`wireguard` and `homeassistant` roles are 100% bare module names, `ollama` is mixed — `ansible/roles/{wireguard,homeassistant,ollama}/tasks/main.yaml`); `nvidia/tasks/main.yaml:58-62` uses `changed_when: true` unconditionally, restarting Docker every run; `gitea-runner/tasks/main.yaml:89-93` unconditionally deletes `/run/docker.sock` every run despite the task name implying a conditional check. - **Operational secret leaks (not committed, but printed)**: `ansible/roles/k3s-server/tasks/main.yaml:46-48` prints the real K3s join token via `debug: msg:` on every run; `ansible/roles/gitea-runner/tasks/main.yaml:55-79` writes the runner registration token into a world-readable (`0644`) systemd unit file; `ansible/roles/wireguard/tasks/main.yaml:169-184` prints generated WireGuard private keys to console. None are committed to git, but all are worth tightening. - **Doc drift**: `README.md:203-204` claims Gitea ingress is "disabled in `values/gitea.yaml`" — that file has no `ingress:` key at all (it's disabled by chart default, not an explicit setting); `README.md:174-186` groups Gitea's backups under `/mnt/storage` when they actually live on a separate NFS export (`/home/nik/backups`, per `manifests/gitea/gitea-backup.yaml:92` and `ansible/roles/nfs-server/templates/exports.j2:7`); `ansible/README.md`'s role table and playbook list omit the `docker`/`nvidia` roles, `setup-gpu-node.yaml`, and `homeassistant.yaml`; root `README.md`'s host table omits `gpu-node` entirely as a 4th host; `config/dashy/conf.yaml:2` points at `manifests/apply-dashy-config.sh`, missing the real `core/` path segment. - **`.gitignore` gaps** (nothing currently leaking, but no coverage): `*.retry` (a live risk — Ansible drops these on playbook failure and this repo's Ansible tree is actively used), `.vault_pass*`, kubeconfig-shaped filenames, editor swap files. `.DS_Store` files exist in the working tree at repo root and `config/` despite being gitignored — confirm they're actually untracked, not just ignored-going-forward. ## Duplicated / Ambiguous / Possibly Obsolete Resources Per the "trace references before declaring obsolete" rule — **none of the "unsynced" items above are actually dead**; they're all in active use, just invisible to Argo CD. The only things that look genuinely obsolete or redundant are: - `nik4nao.home.arpa` DNS entry (both pihole values files) — no backing resource found anywhere. - `ansible/ansible.cfg` — not obsolete exactly, but its only real-world effect today is negative (breaks documented commands); candidate for deletion/merge into the root cfg. - `ansible/roles/nvidia/templates/k3s-containerd-config.toml.j2` — orphaned template, either finish wiring it up or remove it once GPU passthrough status is confirmed live. - `.DS_Store` at repo root and in `config/` — should never have been committed; harmless but should be `git rm --cached`. ## Security Findings Summary (redacted) - Two real credentials committed in plaintext: a database password (`values/gitea.yaml:48`) and a third-party API key (`config/dashy/conf.yaml:63,70`). Both should be treated as compromised and rotated — no value is printed anywhere in this document. - One tracked binary (`router/backup-base.tar.gz`) whose contents were not inspected but is flagged as the most likely place real WiFi/VPN/router-admin credentials would live, given the sibling `uci-base.conf` is sanitized. Needs review outside this workflow; rotation and/or history-scrubbing may be warranted depending on what's inside. - No plaintext secrets found in Ansible beyond the two operational-leak items (console-printed token, world-readable systemd unit) — the vault (`ansible/group_vars/all/vault.yaml`) is genuinely encrypted and used correctly. - RBAC: one real over-scope (`gitea-backup`'s cluster-wide `pods/exec`), no wildcard verbs/resources found anywhere else. - Existing guardrails already partially cover this class of problem going forward: `.claude/agents/secrets-leak-scanner.md` and the `PostToolUse` hook in `.claude/settings.json` — but the scanner's stated scope reads as Ansible/manifests/Helm-values-centric and doesn't explicitly call out `config/**`, which is exactly where the Dashy API key slipped through. Worth widening its scope statement rather than adding new tooling. ## What Should Be Preserved - The Ansible → Argo CD boundary itself is clean — no Ansible task embeds Kubernetes YAML or runs `kubectl apply` against application manifests (verified by repo-wide grep). The one-time hand-off (`helm install argocd` → `kubectl apply -f app-of-apps.yaml`) is exactly right for this scale. - The app-of-apps + one-Application-per-file pattern (20 flat files in `argocd/apps/`) is simple, greppable, and easy to reason about at this service count — an ApplicationSet or Kustomize layer would add indirection with no real benefit here. - The two-pattern secrets model (sealed-secrets for committed ciphertext, runtime `.env`-sourced scripts for cluster-only secrets) is coherent and consistently documented across all three READMEs. - Exact Helm chart version pinning on every chart-backed Application — zero floating versions found. - No deprecated Kubernetes API versions anywhere — clean. - Idempotent `curl | sh`-style installs are consistently guarded with `creates:` across `docker`, `nvidia`, `k3s-server`, `k3s-agent`, `ollama` roles. - `.claude/agents/infra-change-reviewer.md` and `secrets-leak-scanner.md` plus the `settings.json` hooks already encode a lot of this audit's tribal knowledge (DNS-sync rule, sealed-secret hand-edit ban, cert-issuer split, GPU taint/toleration, Ansible dual-cfg risk) — build on these, don't replace them. - Good in-repo examples worth using as the template when fixing the bad ones: `manifests/home-services/nvidia-device-plugin.yaml` (privilege drop), `manifests/core/ca-installer/ca-sync.yaml` (tightly-scoped namespaced RBAC), `manifests/home-services/{ai-gateway,ha-gateway}.yaml` + `manifests/portfolio/portfolio.yaml` (proper resources + probes). ## Unknowns (need live cluster info or user input — not determinable from the repo alone) 1. Whether Grafana's PVC actually binds to the static `grafana-pv` hostPath or silently gets dynamically provisioned via `local-path` instead (`values/kube-prometheus-stack.yaml` sets no `storageClassName`/`volumeName` for Grafana, unlike Prometheus/Loki which do). Needs `kubectl get pvc -n monitoring` / `kubectl get pv grafana-pv -o yaml`. 2. Whether GPU passthrough is actually functional today given the orphaned containerd template (finding #24). Needs a check on `gpu-node`. 3. What's actually inside `router/backup-base.tar.gz` (finding #5) — deliberately not inspected. 4. Whether the K3s bootstrap kubeconfig-path bug (finding #8) actually breaks a fresh run, or whether there's missing context from an untraced earlier step. Static reading only — recommend confirming before relying on it for DR. 5. Whether `selfHeal: false` on the two pihole Applications is deliberate policy (tied to the external-IP-loss gotcha) or an oversight. 6. What Authentik Helm chart version is actually running — it's outside Argo CD, so there's no `targetRevision` to read; needs `helm list -n authentik` (or wherever it's installed) on the live cluster. 7. Whether `values/gitea.yaml`'s committed password and Dashy's committed API key have ever been exposed beyond the operator (repo visibility/access history) — affects how urgent rotation is. --- ## Phase 4 — Target Architecture Given this is a single cluster, single operator, ~20-service homelab with no near-term multi-cluster plan, two credible options were considered. ### Option A (recommended): Keep the current three-way split, close the gaps Keep `ansible/` / `argocd/apps/` / `manifests/` / `values/` exactly as they are structurally — this layout already matches how the system is organized conceptually, how the existing subagents and skills reference paths, and how the current documentation is written. Fix the specific holes (findings above) rather than moving files around. ``` homelab/ ├── ansible.cfg # single config, root-only (ansible/ansible.cfg removed) ├── ansible/ │ ├── requirements.yml # NEW — pins community.general, ansible.posix, community.docker │ ├── inventory.yaml │ ├── group_vars/ · host_vars/ │ ├── playbooks/ │ └── roles/ ├── argocd/ │ ├── apps/ # + cert-manager-config.yaml, authentik.yaml, authentik-config.yaml │ └── values/ ├── manifests/ # unchanged tree; every subdir now has a matching Application ├── values/ # unchanged ├── config/dashy/ ├── router/ ├── .yamllint.yml # NEW ├── .ansible-lint # NEW (config only, tool installed separately) └── CLAUDE.md # + validation commands section ``` **Why not restructure further**: there's no duplication cost today that a `clusters/`/`infrastructure/`/`apps/` split or Kustomize bases would actually reduce — each service already has exactly one Application file and at most one values file. Introducing Kustomize overlays or ApplicationSets would add a templating/indirection layer this repo doesn't need at 1-cluster, ~20-service scale, and would itself become something to maintain and explain. ### Option B (not recommended now): `clusters/` + `infrastructure/` + `apps/`, colocated values Restructure to one directory per service (e.g. `apps/gitea/{application.yaml,values.yaml}`), split `infrastructure/` (cert-manager, sealed-secrets, traefik, pihole) from `apps/` (user-facing services), under a `clusters/homelab/` root to leave room for a future second cluster. This gives slightly tighter per-service ownership (one directory instead of two: `argocd/apps/x.yaml` + `values/x.yaml`) but requires moving **every** Application's `source.path`, which Argo CD treats as a new resource identity — each move is a prune-risk event that must be done one Application at a time with `prune: false` verified beforehand, diffed, and rolled out carefully. That's a lot of migration risk for a benefit (one fewer directory hop per service) that doesn't solve any problem that exists today, and it presumes a multi-cluster future that isn't planned. **Recommendation: Option A.** Revisit Option B only if a second cluster is actually added or the service count grows to where cross-referencing `argocd/apps/` and `values/` by hand becomes genuinely painful — neither is true today. --- ## Phase 5 — Incremental Migration Plan Every stage is designed to leave the repo in a working, deployable state at every commit boundary. Stages that touch what Argo CD currently manages are explicitly marked **RUNTIME-AFFECTING** and require explicit go-ahead before pushing, per the repo's own rule that `git push` is a production deploy. | Stage | Scope | Key changes | Runtime impact | Rollback | |---|---|---|---|---| | **1. Safety fixes & secret hygiene** | Findings #3, #4, #6, plus `.gitignore` gaps | Rotate + re-secret the Gitea DB password and Dashy API key; fix the two `issuerRef` values; extend `.gitignore` (`*.retry`, `.vault_pass*`, kubeconfig patterns, swap files); `git rm --cached` the `.DS_Store` files | **RUNTIME-AFFECTING** for the password rotation (Gitea/Postgres restart) and cert reissuance (brief TLS transition on 2 hostnames); everything else is zero-impact | Revert commit; for password rotation, keep old secret retrievable until new one is confirmed working | | **2. Validation tooling & clean baseline** | New root-level config | Add `.yamllint.yml`, `.ansible-lint`; install `yamllint`/`ansible-lint`/`kubeconform` locally (asking first, since that's installing software); run baseline, fix anything trivial it finds | None — local/offline only | Uninstall tools / delete config files | | **3. Remove confirmed junk** | `.DS_Store`, orphaned `nik4nao.home.arpa` DNS entry, doc-path typo in `config/dashy/conf.yaml:2` | Delete/clean | Negligible (DNS entry removal is additive-safe to revert) | `git revert` | | **4. Ansible cleanup** | Findings #7, #9, #15, plus low-severity items | Merge `ansible/ansible.cfg` into root; add `requirements.yml`; align `k3s_version` across 3 locations; fix `gitea-runner` docker.sock task, `nvidia` `changed_when`, K3s token debug print, systemd unit permissions; update both READMEs (gpu-node host, docker/nvidia roles, homeassistant status) | **RUNTIME-AFFECTING** only for the `k3s_version` alignment (changes what a future node provision installs) — needs sign-off on which version is canonical; rest is inert until playbooks rerun | Git revert; k3s_version change only takes effect on next actual provision, not immediately | | **5. Argo CD bootstrap normalization** | Findings #1, #2, #10, #12, #13, #17, #19 | New `cert-manager-config.yaml` + `authentik.yaml`/`authentik-config.yaml` Applications (first-time adoption of already-live resources — `kubectl diff` before enabling automated sync on each); widen `network-secrets` coverage or add `network.yaml`; fix homeassistant namespace; add `gitea-backup` Namespace + narrow its RBAC to a Role; add missing sync-waves; normalize `targetRevision: HEAD`→`main` | **RUNTIME-AFFECTING, highest-risk stage** — bringing previously-unmanaged live resources under Argo CD for the first time. Roll out one Application at a time, diff first, watch sync status before moving to the next | Set the newly-added Application's sync policy back to manual, or delete the Application (resources stay, since `prune: false`) | | **6. Values/DNS consolidation** | Finding #18, #20 | Reconcile `pihole.yaml`/`pihole-debian.yaml` drift (add missing entries); document or normalize the `prune:true`/`selfHeal:false` outliers (pending intent) | **RUNTIME-AFFECTING but low-risk** — adds DNS records only, additive | Revert values file | | **7. Kubernetes correctness & security** | Findings #11, #16, #25, #26 | Add resource requests/limits + probes to flagged workloads (one service at a time); narrow `gitea-backup` RBAC; test Grafana `runAsNonRoot`/`fsGroup`-only; pin remaining `:latest` images to currently-running tags | **RUNTIME-AFFECTING** — limits set too low can OOMKill; needs `kubectl top` baselines first (gather live before proposing numbers); roll out one service at a time with rollback ready | Revert manifest, resync | | **8. Documentation & DR runbook** | All doc-drift items | Fix host tables, role tables, playbook lists, ingress/storage claims; add an explicit cold-rebuild runbook reflecting the now-closed GitOps gaps | None | Revert | | **9. CI & Claude Code guidance** | New `.gitea/workflows/validate.yaml` (yamllint/ansible-lint/kubeconform on push); `CLAUDE.md` validation-commands section; widen `secrets-leak-scanner.md`'s stated scope to include `config/**`; consider extending the `PreToolUse` hook to block Claude Code from reading `.env`/`router/backup-base.tar.gz` directly | New automation surface | None to the cluster; new CI pipeline needs approval since it's a new automation surface | Delete workflow file | Every stage gets validated with whatever's actually available (`ansible-playbook --syntax-check`, `yamllint`, `ansible-lint`, `kubeconform`/`kubectl apply --dry-run=client` where a live context isn't required) before being reported done — with the concrete diff and validation output shown, not just a claim of success. --- ## Claude Code Project Support - **`CLAUDE.md`**: keep it, add a short "Validation commands" section once Stage 2 lands (exact `yamllint`/`ansible-lint`/`ansible-playbook --syntax-check` invocations). - **Guard against accidental `.env`/key/backup access**: partially covered already (`.gitignore`, sealed-secret `PreToolUse` hook). Extend that hook to also block Claude Code `Read` of `.env` and `router/backup-base.tar.gz` — small addition, Stage 9. - **One validation skill/command**: rather than a new skill, widen `secrets-leak-scanner.md`'s stated file scope to explicitly include `config/**` (the actual gap that let the Dashy API key through) — cheaper and more targeted than adding new tooling.