homelab/REFACTOR_PLAN.md

Homelab Repo Audit & Refactor Plan

Living document. Produced by a full read-only audit of the repo (Ansible, Argo CD, manifests, values, config, router). Nothing was applied, synced, committed, or pushed as part of producing this. We work through the "Migration Plan" stages one at a time, referencing this file; update the Status column as stages land.

Audit constraints honored: .env was never opened; router/backup-base.tar.gz was never extracted; no kubectl apply / helm install / Argo CD sync / Ansible-against-real-hosts was run. All findings are grounded in file citations, and the highest-severity ones were independently re-verified (not just taken from sub-agent research output) before being recorded here.

Stage Tracker

Stage	Scope	Status
1	Safety fixes & secret hygiene	Files done, committed locally, not pushed. Actual secret rotation (new password/API key values + live DB password change) still needs the user — see note below.
2	Validation tooling & clean baseline	Done. yamllint/ansible-lint/kubeconform installed by user; .yamllint.yml/.ansible-lint added; baseline clean (0 yamllint issues, kubeconform: 117+20 resources valid, all playbooks pass --syntax-check). Remaining ansible-lint findings are either pre-existing role/var naming conventions (out of scope — would require repo-wide renames) or collections-not-installed noise that resolves once ansible-galaxy collection install -r ansible/requirements.yml is run.
3	Remove confirmed junk	Done, committed locally. .DS_Store confirmed never tracked (no action needed).
4	Ansible cleanup	Done, committed locally.
5	Argo CD bootstrap normalization	Files done, committed locally, not pushed — highest-risk stage. authentik.yaml's chart version is a placeholder needing your input (helm list -n authentik); every new/changed Application here needs kubectl diff one at a time before/after enabling, not a blind batch push.
6	Values/DNS consolidation	Done, committed locally, not pushed.
7	Kubernetes correctness & security	Resource requests/limits + probes done (finding #16), committed locally, not pushed. gitea-backup RBAC narrowing (#11) done as part of Stage 5's commit. Grafana runAsNonRoot/fsGroup test (#25) and :latest image pinning not done — both need a live test window / registry inspection I didn't do unprompted.
8	Documentation & DR runbook	Done, committed locally.
9	CI & Claude Code guidance	Done, committed locally, not pushed (new CI automation surface).

Correction, 2026-07-23: the original existingSecret-based fix for Finding #3 was wrong and caused a real Gitea outage. Confirmed directly from the chart source (gitea/helm-gitea templates/gitea/config.yaml): {{ set .Values.gitea.config.database "PASSWD" .Values.postgresql.global.postgresql.auth.password }} — Gitea's own [database] config is always built from this plaintext values field, regardless of existingSecret (which only affects the bundled postgresql subchart's own credential provisioning, a separate concern). Removing the plaintext field made it silently fall back to the chart's built-in default password, breaking every fresh Gitea pod's DB connection. values/gitea.yaml is back to a plaintext (rotated) password with a comment explaining why, and manifests/gitea/gitea-postgres-secret.sh was removed. Getting this fully off plaintext would need a Config Management Plugin or similar — not attempted here; flagging as a real open item if it matters enough to invest in.

Dashy's API key fix (env-injection via manifests/core/apply-dashy-config.sh) is unaffected by this and still needs you to rotate the actual key value.

New findings from this session's live-cluster checks (not in the original audit)

• Grafana and Loki's static PVs are orphaned. manifests/monitoring/monitoring-pvs.yaml defines grafana-pv/loki-pv hostPath PVs, but live kubectl get pv shows both sitting Available (unbound) — their Helm-managed PVCs got dynamically provisioned via local-path instead, unlike prometheus-pv which binds correctly. Data isn't lost, just not where the README/manifest imply. Needs a decision: bind properly (data migration required) or drop the orphaned static PVs and document reality. Not yet actioned.
• Finding #24 (GPU passthrough) is resolved as working, not broken. Live check on node nik-gpu: nvidia.com/gpu: "1" allocatable, nvidia runtime handler registered in containerd, nvidia-device-plugin pod Running. The orphaned-looking k3s-containerd-config.toml.j2 template is a non-issue in practice — no action needed.
• Unknown #1 (Grafana PVC) resolved: dynamically provisioned via local-path, not the static PV — see the orphaned-PV finding above, same root cause.
• Node hostnames in the live cluster are nik-debian/nik-gpu, not debian/gpu-node as in ansible/inventory.yaml — cosmetic (K3s registers by actual hostname), doesn't affect anything, just noted for anyone cross-referencing kubectl get nodes against the docs.

Open decisions needed from the user before/during the relevant stage:

• Finding #5 (router/backup-base.tar.gz): needs manual review outside this workflow; may imply git-history scrub / router credential rotation.
• Finding #9 (K3s version skew): which version is canonical (v1.32.2+k3s1 vs v1.32.4+k3s1)?
• Finding #18: is prune:true on home-services/otel-collector/tempo and selfHeal:false on both pihole Applications intentional policy?
• Finding #25: Grafana runAsUser: 0 — worth testing fsGroup-only instead?

---

Phase 1 — Current-State Architecture

Ansible (hosts)                     Argo CD (cluster)
─────────────────                   ──────────────────────────────────────────
minisforum  (K3s server, .77)  ──┐
debian      (K3s agent, .183) ────┼─▶ K3s cluster ──▶ helm install argocd (manual, README.md:92-103)
gpu-node    (K3s agent, .98)   ──┘         │                    │
mac-mini    (standalone Docker)            │                    ▼
                                            │        kubectl apply -f manifests/argocd/app-of-apps.yaml  (manual, one-time)
                                            │                    │
                                            │                    ▼
                                            │        Application: app-of-apps → watches argocd/apps/*.yaml
                                            │                    │
                                            │        ┌───────────┴────────────────────────────┐
                                            │        ▼                                         ▼
                                            │  Helm-chart Applications                 Raw-manifest Applications
                                            │  (chart + values/<name>.yaml)             (path: manifests/<area>)
                                            │  traefik, cert-manager*, gitea,           core, media, home-services,
                                            │  pihole, pihole-debian, monitoring,       portfolio, homeassistant,
                                            │  loki, tempo, otel-collector,             *-secrets (sealed only)
                                            │  sealed-secrets, argocd(self)
                                            │
                                            └── NOT reconciled by Argo CD at all:
                                                manifests/cert-manager/ (ClusterIssuers)
                                                manifests/authentik/ + values/authentik.yaml
                                                4 files in manifests/network/ (non-sealed)
                                                → applied by hand, outside GitOps

Host-level services that never touch the K3s cluster: Home Assistant (Docker Compose + systemd on minisforum, ansible/roles/homeassistant/), Watch Party and Ollama (mac-mini), Ollama (gpu-node) — all Ansible-managed.

Bootstrap sequence as documented (README.md:71-105, argocd/README.md:7-24): Ansible playbooks → manual helm install argocd → manual kubectl apply of app-of-apps → everything else via Git. This documented sequence is incomplete — it never mentions installing Authentik or the cert-manager ClusterIssuers, both required for the rest of the stack to actually work (see Critical findings below).

Executive Summary

The repo is coherently designed for a single-operator homelab: Ansible for host bootstrap, Argo CD app-of-apps for cluster reconciliation, a clean two-pattern secrets model (sealed-secrets for committed ciphertext, runtime scripts for .env-sourced live secrets), and a genuinely useful pair of existing Claude Code subagents (infra-change-reviewer, secrets-leak-scanner) that already encode a lot of hard-won tribal knowledge. There's no fashionable-but-wrong abstraction problem here — the biggest risks are gaps between what the documentation/GitOps model claims and what's actually wired up: two foundational subsystems (cert-manager's ClusterIssuers, and the entire Authentik SSO stack) are invisible to Argo CD despite everything else depending on them, a live database password and a live third-party API key are committed in plaintext, and two high-value internal certificates are silently issued by the wrong CA chain. None of this requires a restructure to fix — it requires closing specific, identifiable gaps. Ansible is in good shape except for a dual-ansible.cfg trap that breaks documented commands if run from the wrong directory, and one likely-broken kubeconfig-fetch task in the K3s bootstrap role. No CI or local validation tooling exists yet, which is the main reason several of these gaps went unnoticed.

Prioritized Findings

Critical

#	Finding	Evidence	Impact	Recommendation
1	manifests/cert-manager/ (both ClusterIssuers + the internal CA Certificate) is never referenced by any Argo CD Application — confirmed by grep across all of argocd/apps/*.yaml, zero path: hits	argocd/apps/cert-manager.yaml only installs the Helm chart/controller; no sibling Application points at manifests/cert-manager/	Every TLS certificate in the cluster depends on issuers that exist only because someone once ran kubectl apply by hand. A cold rebuild following the documented bootstrap steps will not restore them, and nothing detects/corrects drift on the live ones.	Add argocd/apps/cert-manager-config.yaml (path: manifests/cert-manager, sync-wave before consumers). See Stage 5.
2	manifests/authentik/ (7 files: ingress, proxy outpost, middleware, 4 secret scripts) and values/authentik.yaml are never referenced by any Application, and Authentik's manual helm upgrade --install isn't even in the documented bootstrap steps	Confirmed by the same grep sweep; values/authentik.yaml:1's header comment is the only place the install command lives	The SSO IdP gating Argo CD, Grafana, and Gitea OAuth is entirely outside GitOps and undocumented as a bootstrap step. Disaster recovery would silently fail to restore login for everything behind it.	Add argocd/apps/authentik.yaml (chart) + argocd/apps/authentik-config.yaml (path: manifests/authentik), mirroring the cert-manager two-Application pattern.
3	Plaintext PostgreSQL password committed	values/gitea.yaml:48 — password: gitea-db-password under postgresql.global.postgresql.auth.password, no existingSecret used (contrast values/gitea.yaml:11, which correctly uses existingSecret: gitea-admin-secret)	A real credential sits in git history now, readable by anyone with repo access. Contradicts the repo's own stated no-plaintext-secrets rule and .claude/agents/secrets-leak-scanner.md's invariant.	Rotate the DB password, convert to a *-secret.sh → sealed-secret pattern (mirror manifests/media/immich-postgres-secret.sh). Treat the current value as compromised since it's in history.
4	Live third-party API key committed in plaintext	config/dashy/conf.yaml:63 and :70 — a weather-widget API key used twice	Anyone with repo read access (or Dashy's own JS, which ships client-side) can extract and reuse the key.	Rotate the key at the provider; inject at apply-time from .env via manifests/core/apply-dashy-config.sh instead of hardcoding in config/dashy/conf.yaml.
5	router/backup-base.tar.gz is tracked in git (added in commit e7718ce, ~8.8KB compressed)	Confirmed tracked via git ls-files; not opened, per safety constraints	router/uci-base.conf itself (which was read) is a sanitized base config — WiFi disabled with default open SSID, no VPN configured, sentinel password field — which makes it likely the fuller .tar.gz backup is where any real WiFi PSK/VPN key/admin credential would actually live. Cannot confirm what's inside without extracting it.	Action needed from user: inspect this archive outside this workflow. If it contains live credentials, both git-history scrubbing and router credential rotation should be considered.
6	Two high-value internal certificates are issued by the wrong CA	manifests/authentik/authentik-ingress.yaml:11-12 (auth.home.arpa) and manifests/network/traefik-dashboard-ingress.yaml:65-66 (traefik.home.arpa) both set issuerRef.name: internal-ca — the root self-signed issuer — instead of internal-ca-issuer, the chained issuer every other internal Certificate uses (verified pattern in manifests/homeassistant/homeassistant.yaml:31, manifests/home-services/certs.yaml, manifests/argocd/argocd.yaml:9, media manifests)	Devices that installed the CA via the ca-installer/ca-sync flow (manifests/README.md:36-42) will not trust these two certs — browser TLS warnings on exactly the SSO login and the Traefik dashboard. Compounded by #1/#2: neither file is under GitOps, so nothing corrects this automatically.	Change issuerRef.name to internal-ca-issuer in both files.

High

#	Finding	Evidence	Impact	Recommendation
7	Dual ansible.cfg breaks every documented playbook command if run from ansible/ instead of repo root — empirically reproduced, not just theorized	ansible.cfg (root) vs ansible/ansible.cfg; confirmed via ansible-config dump that only one loads at a time, and reproduced two independent failures (path-doubling, then role-not-found) plus a silent loss of host_key_checking=False	A user who cds into ansible/ first (a very natural mistake) gets confusing, unrelated-looking errors, or worse, an interactive host-key prompt in what's meant to be scriptable.	Delete ansible/ansible.cfg; fold its two settings (inject_facts_as_vars: False, deprecation_warnings: False) into the root ansible.cfg. Single source of truth, run only from repo root (already the documented convention).
8	Likely-broken kubeconfig path in K3s bootstrap	ansible/roles/k3s-server/tasks/main.yaml:50-62 — fetches kubeconfig to ~/.kube/config, but the next task's ansible.builtin.replace targets /tmp/k3s-minisforum.yaml, a path nothing else in the role writes to	replace errors if the target doesn't exist — this task likely fails on a genuinely fresh bootstrap-minisforum.yaml/setup-k3s.yaml run, i.e. exactly the disaster-recovery path this needs to work for. Static-analysis finding, not yet confirmed against a live re-run — flagged as an unknown too.	Fix the path mismatch (verify what the fetch task actually names the local file and point replace at that same path); test on an actual rebuild before trusting this for DR.
9	K3s version defined in 3 places, already out of sync	ansible/roles/k3s-server/defaults/main.yaml:6 and k3s-agent/defaults/main.yaml:6 both v1.32.2+k3s1; ansible/host_vars/gpu-node.yaml:35 is v1.32.4+k3s1. ansible/README.md:58-59's upgrade doc only mentions the first two locations.	A fresh debian/minisforum (re)provision installs a different K3s version than what's apparently already running on gpu-node — real version-skew risk, and the documented upgrade procedure would miss the gpu-node override entirely.	Pick one canonical version, align all three, document all three locations.
10	4 of 6 files in manifests/network/ are excluded from Argo CD sync	argocd/apps/network-secrets.yaml:14-15 sets directory.include: '*-sealed.yaml', so ddns-cronjob.yaml, glances-debian-ingress.yaml, traefik-dashboard-ingress.yaml, watch-party-ingress.yaml are never synced	These back real, documented services (README.md:52,63,64) with zero drift protection or self-heal — same class of gap as #1/#2, smaller blast radius.	Add a proper network.yaml Application (or widen the include filter) covering these; keep network-secrets.yaml scoped to sealed secrets only.
11	gitea-backup CronJob has cluster-wide exec privileges it doesn't need	manifests/gitea/gitea-backup.yaml:11 (ClusterRole) / :23 (ClusterRoleBinding) grant pods/exec cluster-wide; the CronJob script only execs into the gitea namespace (lines 61-72)	The backup ServiceAccount can exec into any pod in any namespace — real privilege-escalation surface for a component whose only job is backing up one namespace.	Replace with a namespaced Role/RoleBinding scoped to gitea.
12	homeassistant Application/manifest namespace mismatch	argocd/apps/homeassistant.yaml:14 targets/creates namespace homeassistant; every resource in manifests/homeassistant/homeassistant.yaml is hardcoded to namespace: default (lines 6,17,27,40) — verified directly	Argo CD creates and "manages" an empty, unused homeassistant namespace while the real resources live in default. Confusing, latent bug, no current functional harm.	Change destination.namespace to default and drop CreateNamespace=true (no runtime-affecting change — resources already live in default).
13	gitea-backup namespace has no Namespace object anywhere and isn't auto-created	manifests/gitea/gitea-backup.yaml:8,29,39,100 all reference namespace gitea-backup; the owning Application (gitea-secrets.yaml) targets namespace gitea with CreateNamespace=true — a different namespace	Would fail to apply on any cluster where gitea-backup namespace wasn't created by hand out-of-band.	Add an explicit Namespace: gitea-backup object to the manifest, or fold the backup CronJob into the gitea namespace.
14	No CI or local validation tooling anywhere in the repo	Confirmed via recursive search: no .github/, .gitea/workflows/, pre-commit config, or yamllint/ansible-lint config at any level. yamllint, ansible-lint, kubeconform not installed locally either (only helm, kubectl, kubeseal, jq, ansible-playbook are present)	Given "no staging environment," every change is validated by pushing and watching Argo CD/production react. This is the root cause that let findings #3, #4, #20 go unnoticed.	See Stage 2.
15	Missing requirements.yml + undocumented collection dependency	No requirements.yml anywhere; README.md:76-78 only tells a new operator to install community.general/ansible.posix, but ansible/roles/glances/tasks/main.yaml:24 and ansible/roles/watch-party/tasks/main.yaml:27 require community.docker, never mentioned	A fresh workstation following the documented bootstrap verbatim gets a "module not found" failure on setup-glances-debian.yaml/deploy-watch-party.yaml.	Add ansible/requirements.yml pinning all three collections; update README.md.
16	Missing resource requests/limits and probes on a long list of workloads	manifests/core/{dashy,glances}.yaml, manifests/core/ca-installer/ca-installer.yaml, manifests/authentik/authentik-proxy-outpost.yaml, manifests/media/jellyfin.yaml, main containers in qbittorrent.yaml/jdownloader.yaml (only their gluetun sidecars have probes), all 4 Deployments in manifests/media/immich.yaml	No protection against one workload starving others on a small, fixed-capacity cluster; no automatic restart on hang for several user-facing services. Contrast: manifests/home-services/*.yaml and manifests/portfolio/portfolio.yaml do this correctly already — good templates to copy from.	See Stage 7.

Medium

#	Finding	Evidence	Recommendation
17	Sync-wave ordering doesn't cover cert-manager/sealed-secrets vs. their consumers	Only gitea-secrets/gitea/loki/monitoring-secrets/monitoring/network-secrets carry sync-wave annotations; cert-manager and sealed-secrets (whose CRDs everything else's Certificate/SealedSecret objects need) are unannotated, same wave-0 as their consumers	Add sync-wave: "-2" to cert-manager/sealed-secrets, -1 to their config Applications, consistent with the existing pattern. Self-heals today via retries, so this is about shortening a fresh-bootstrap flaky window, not fixing active breakage.
18	Undocumented sync-policy exceptions	home-services, otel-collector, tempo use prune: true against the repo's stated prune: false norm (argocd/README.md:37); pihole/pihole-debian are the only two with selfHeal: false (plausibly intentional, given the documented external-IP-loss gotcha, but never stated as policy)	Add a one-line YAML comment on each explaining the exception, or normalize them if unintentional — needs user input on intent.
19	3 Applications use targetRevision: HEAD instead of the documented main	home-services.yaml:12, otel-collector.yaml:17, tempo.yaml:17	Normalize to main. Zero behavioral difference today; purely consistency.
20	Pi-hole DNS entries have already drifted	values/pihole.yaml:81,85 (immich.home.arpa, gluetun.home.arpa) missing from values/pihole-debian.yaml	Add the two missing entries. (Going forward this is now caught by the existing PostToolUse hook in .claude/settings.json — it just doesn't catch pre-existing drift.)
21	Orphaned DNS record	nik4nao.home.arpa exists in both pihole values files but no Certificate/Ingress/IngressRoute anywhere references it	Confirm it's unused, then remove.
22	ansible/README.md mislabels the homeassistant role "Legacy"	ansible/README.md:54; but manifests/homeassistant/homeassistant.yaml's Endpoints (hardcoded to 192.168.7.77:8123) has nothing else in the repo standing up a listener at that address — the "legacy" Ansible role/Docker Compose deployment is the only thing actually serving it	Re-word the doc entry; this is load-bearing, not dead — don't let a future cleanup pass delete it.
23	docker role logic duplicated wholesale inside homeassistant role	ansible/roles/homeassistant/tasks/main.yaml:2-49 reimplements Docker CE install (Debian apt path) instead of reusing ansible/roles/docker/tasks/main.yaml:6-43 (Ubuntu apt path)	Parameterize docker role for both distros, have homeassistant depend on it instead of duplicating.
24	GPU passthrough likely non-functional as wired	ansible/roles/nvidia/templates/k3s-containerd-config.toml.j2 exists but is never referenced by any task in that role; manifests/home-services/nvidia-device-plugin.yaml:5-9 defines a RuntimeClass handler: nvidia that depends on exactly that containerd config existing on gpu-node	Unknown pending live verification — needs a check on gpu-node before deciding whether to wire up the template or remove the RuntimeClass.
25	Grafana forced to run as root	values/kube-prometheus-stack.yaml:49-52 — runAsNonRoot: false, runAsUser: 0, likely to support initChownData: true fixing hostPath ownership	Test whether fsGroup alone (without full root) suffices; needs a live test window with rollback ready.
26	Mixed Ingress vs Traefik IngressRoute usage with no stated policy	e.g. manifests/media/qbittorrent.yaml uses plain Ingress for the app (line 176) but IngressRoute for gluetun-api in the same file (line 216)	Not broken, but only the IngressRoute half gets native Middleware support without annotation workarounds — worth standardizing on one mechanism over time, not urgent.

Low (grouped)

• Ansible hygiene: zero tags: usage anywhere (no selective runs possible); FQCN usage inconsistent (wireguard and homeassistant roles are 100% bare module names, ollama is mixed — ansible/roles/{wireguard,homeassistant,ollama}/tasks/main.yaml); nvidia/tasks/main.yaml:58-62 uses changed_when: true unconditionally, restarting Docker every run; gitea-runner/tasks/main.yaml:89-93 unconditionally deletes /run/docker.sock every run despite the task name implying a conditional check.
• Operational secret leaks (not committed, but printed): ansible/roles/k3s-server/tasks/main.yaml:46-48 prints the real K3s join token via debug: msg: on every run; ansible/roles/gitea-runner/tasks/main.yaml:55-79 writes the runner registration token into a world-readable (0644) systemd unit file; ansible/roles/wireguard/tasks/main.yaml:169-184 prints generated WireGuard private keys to console. None are committed to git, but all are worth tightening.
• Doc drift: README.md:203-204 claims Gitea ingress is "disabled in values/gitea.yaml" — that file has no ingress: key at all (it's disabled by chart default, not an explicit setting); README.md:174-186 groups Gitea's backups under /mnt/storage when they actually live on a separate NFS export (/home/nik/backups, per manifests/gitea/gitea-backup.yaml:92 and ansible/roles/nfs-server/templates/exports.j2:7); ansible/README.md's role table and playbook list omit the docker/nvidia roles, setup-gpu-node.yaml, and homeassistant.yaml; root README.md's host table omits gpu-node entirely as a 4th host; config/dashy/conf.yaml:2 points at manifests/apply-dashy-config.sh, missing the real core/ path segment.
• .gitignore gaps (nothing currently leaking, but no coverage): *.retry (a live risk — Ansible drops these on playbook failure and this repo's Ansible tree is actively used), .vault_pass*, kubeconfig-shaped filenames, editor swap files. .DS_Store files exist in the working tree at repo root and config/ despite being gitignored — confirm they're actually untracked, not just ignored-going-forward.

Duplicated / Ambiguous / Possibly Obsolete Resources

Per the "trace references before declaring obsolete" rule — none of the "unsynced" items above are actually dead; they're all in active use, just invisible to Argo CD. The only things that look genuinely obsolete or redundant are:

• nik4nao.home.arpa DNS entry (both pihole values files) — no backing resource found anywhere.
• ansible/ansible.cfg — not obsolete exactly, but its only real-world effect today is negative (breaks documented commands); candidate for deletion/merge into the root cfg.
• ansible/roles/nvidia/templates/k3s-containerd-config.toml.j2 — orphaned template, either finish wiring it up or remove it once GPU passthrough status is confirmed live.
• .DS_Store at repo root and in config/ — should never have been committed; harmless but should be git rm --cached.

Security Findings Summary (redacted)

• Two real credentials committed in plaintext: a database password (values/gitea.yaml:48) and a third-party API key (config/dashy/conf.yaml:63,70). Both should be treated as compromised and rotated — no value is printed anywhere in this document.
• One tracked binary (router/backup-base.tar.gz) whose contents were not inspected but is flagged as the most likely place real WiFi/VPN/router-admin credentials would live, given the sibling uci-base.conf is sanitized. Needs review outside this workflow; rotation and/or history-scrubbing may be warranted depending on what's inside.
• No plaintext secrets found in Ansible beyond the two operational-leak items (console-printed token, world-readable systemd unit) — the vault (ansible/group_vars/all/vault.yaml) is genuinely encrypted and used correctly.
• RBAC: one real over-scope (gitea-backup's cluster-wide pods/exec), no wildcard verbs/resources found anywhere else.
• Existing guardrails already partially cover this class of problem going forward: .claude/agents/secrets-leak-scanner.md and the PostToolUse hook in .claude/settings.json — but the scanner's stated scope reads as Ansible/manifests/Helm-values-centric and doesn't explicitly call out config/**, which is exactly where the Dashy API key slipped through. Worth widening its scope statement rather than adding new tooling.

What Should Be Preserved

• The Ansible → Argo CD boundary itself is clean — no Ansible task embeds Kubernetes YAML or runs kubectl apply against application manifests (verified by repo-wide grep). The one-time hand-off (helm install argocd → kubectl apply -f app-of-apps.yaml) is exactly right for this scale.
• The app-of-apps + one-Application-per-file pattern (20 flat files in argocd/apps/) is simple, greppable, and easy to reason about at this service count — an ApplicationSet or Kustomize layer would add indirection with no real benefit here.
• The two-pattern secrets model (sealed-secrets for committed ciphertext, runtime .env-sourced scripts for cluster-only secrets) is coherent and consistently documented across all three READMEs.
• Exact Helm chart version pinning on every chart-backed Application — zero floating versions found.
• No deprecated Kubernetes API versions anywhere — clean.
• Idempotent curl | sh-style installs are consistently guarded with creates: across docker, nvidia, k3s-server, k3s-agent, ollama roles.
• .claude/agents/infra-change-reviewer.md and secrets-leak-scanner.md plus the settings.json hooks already encode a lot of this audit's tribal knowledge (DNS-sync rule, sealed-secret hand-edit ban, cert-issuer split, GPU taint/toleration, Ansible dual-cfg risk) — build on these, don't replace them.
• Good in-repo examples worth using as the template when fixing the bad ones: manifests/home-services/nvidia-device-plugin.yaml (privilege drop), manifests/core/ca-installer/ca-sync.yaml (tightly-scoped namespaced RBAC), manifests/home-services/{ai-gateway,ha-gateway}.yaml + manifests/portfolio/portfolio.yaml (proper resources + probes).

Unknowns (need live cluster info or user input — not determinable from the repo alone)

1. Whether Grafana's PVC actually binds to the static grafana-pv hostPath or silently gets dynamically provisioned via local-path instead (values/kube-prometheus-stack.yaml sets no storageClassName/volumeName for Grafana, unlike Prometheus/Loki which do). Needs kubectl get pvc -n monitoring / kubectl get pv grafana-pv -o yaml.
2. Whether GPU passthrough is actually functional today given the orphaned containerd template (finding #24). Needs a check on gpu-node.
3. What's actually inside router/backup-base.tar.gz (finding #5) — deliberately not inspected.
4. Whether the K3s bootstrap kubeconfig-path bug (finding #8) actually breaks a fresh run, or whether there's missing context from an untraced earlier step. Static reading only — recommend confirming before relying on it for DR.
5. Whether selfHeal: false on the two pihole Applications is deliberate policy (tied to the external-IP-loss gotcha) or an oversight.
6. What Authentik Helm chart version is actually running — it's outside Argo CD, so there's no targetRevision to read; needs helm list -n authentik (or wherever it's installed) on the live cluster.
7. Whether values/gitea.yaml's committed password and Dashy's committed API key have ever been exposed beyond the operator (repo visibility/access history) — affects how urgent rotation is.

---

Phase 4 — Target Architecture

Given this is a single cluster, single operator, ~20-service homelab with no near-term multi-cluster plan, two credible options were considered.

Option A (recommended): Keep the current three-way split, close the gaps

Keep ansible/ / argocd/apps/ / manifests/ / values/ exactly as they are structurally — this layout already matches how the system is organized conceptually, how the existing subagents and skills reference paths, and how the current documentation is written. Fix the specific holes (findings above) rather than moving files around.

homelab/
├── ansible.cfg                    # single config, root-only (ansible/ansible.cfg removed)
├── ansible/
│   ├── requirements.yml           # NEW — pins community.general, ansible.posix, community.docker
│   ├── inventory.yaml
│   ├── group_vars/ · host_vars/
│   ├── playbooks/
│   └── roles/
├── argocd/
│   ├── apps/                      # + cert-manager-config.yaml, authentik.yaml, authentik-config.yaml
│   └── values/
├── manifests/                     # unchanged tree; every subdir now has a matching Application
├── values/                        # unchanged
├── config/dashy/
├── router/
├── .yamllint.yml                  # NEW
├── .ansible-lint                  # NEW (config only, tool installed separately)
└── CLAUDE.md                      # + validation commands section

Why not restructure further: there's no duplication cost today that a clusters//infrastructure//apps/ split or Kustomize bases would actually reduce — each service already has exactly one Application file and at most one values file. Introducing Kustomize overlays or ApplicationSets would add a templating/indirection layer this repo doesn't need at 1-cluster, ~20-service scale, and would itself become something to maintain and explain.

Option B (not recommended now): clusters/ + infrastructure/ + apps/, colocated values

Restructure to one directory per service (e.g. apps/gitea/{application.yaml,values.yaml}), split infrastructure/ (cert-manager, sealed-secrets, traefik, pihole) from apps/ (user-facing services), under a clusters/homelab/ root to leave room for a future second cluster. This gives slightly tighter per-service ownership (one directory instead of two: argocd/apps/x.yaml + values/x.yaml) but requires moving every Application's source.path, which Argo CD treats as a new resource identity — each move is a prune-risk event that must be done one Application at a time with prune: false verified beforehand, diffed, and rolled out carefully. That's a lot of migration risk for a benefit (one fewer directory hop per service) that doesn't solve any problem that exists today, and it presumes a multi-cluster future that isn't planned.

Recommendation: Option A. Revisit Option B only if a second cluster is actually added or the service count grows to where cross-referencing argocd/apps/ and values/ by hand becomes genuinely painful — neither is true today.

---

Phase 5 — Incremental Migration Plan

Every stage is designed to leave the repo in a working, deployable state at every commit boundary. Stages that touch what Argo CD currently manages are explicitly marked RUNTIME-AFFECTING and require explicit go-ahead before pushing, per the repo's own rule that git push is a production deploy.

Stage	Scope	Key changes	Runtime impact	Rollback
1. Safety fixes & secret hygiene	Findings #3, #4, #6, plus .gitignore gaps	Rotate + re-secret the Gitea DB password and Dashy API key; fix the two issuerRef values; extend .gitignore (*.retry, .vault_pass*, kubeconfig patterns, swap files); git rm --cached the .DS_Store files	RUNTIME-AFFECTING for the password rotation (Gitea/Postgres restart) and cert reissuance (brief TLS transition on 2 hostnames); everything else is zero-impact	Revert commit; for password rotation, keep old secret retrievable until new one is confirmed working
2. Validation tooling & clean baseline	New root-level config	Add .yamllint.yml, .ansible-lint; install yamllint/ansible-lint/kubeconform locally (asking first, since that's installing software); run baseline, fix anything trivial it finds	None — local/offline only	Uninstall tools / delete config files
3. Remove confirmed junk	.DS_Store, orphaned nik4nao.home.arpa DNS entry, doc-path typo in config/dashy/conf.yaml:2	Delete/clean	Negligible (DNS entry removal is additive-safe to revert)	git revert
4. Ansible cleanup	Findings #7, #9, #15, plus low-severity items	Merge ansible/ansible.cfg into root; add requirements.yml; align k3s_version across 3 locations; fix gitea-runner docker.sock task, nvidia changed_when, K3s token debug print, systemd unit permissions; update both READMEs (gpu-node host, docker/nvidia roles, homeassistant status)	RUNTIME-AFFECTING only for the k3s_version alignment (changes what a future node provision installs) — needs sign-off on which version is canonical; rest is inert until playbooks rerun	Git revert; k3s_version change only takes effect on next actual provision, not immediately
5. Argo CD bootstrap normalization	Findings #1, #2, #10, #12, #13, #17, #19	New cert-manager-config.yaml + authentik.yaml/authentik-config.yaml Applications (first-time adoption of already-live resources — kubectl diff before enabling automated sync on each); widen network-secrets coverage or add network.yaml; fix homeassistant namespace; add gitea-backup Namespace + narrow its RBAC to a Role; add missing sync-waves; normalize targetRevision: HEAD→main	RUNTIME-AFFECTING, highest-risk stage — bringing previously-unmanaged live resources under Argo CD for the first time. Roll out one Application at a time, diff first, watch sync status before moving to the next	Set the newly-added Application's sync policy back to manual, or delete the Application (resources stay, since prune: false)
6. Values/DNS consolidation	Finding #18, #20	Reconcile pihole.yaml/pihole-debian.yaml drift (add missing entries); document or normalize the prune:true/selfHeal:false outliers (pending intent)	RUNTIME-AFFECTING but low-risk — adds DNS records only, additive	Revert values file
7. Kubernetes correctness & security	Findings #11, #16, #25, #26	Add resource requests/limits + probes to flagged workloads (one service at a time); narrow gitea-backup RBAC; test Grafana runAsNonRoot/fsGroup-only; pin remaining :latest images to currently-running tags	RUNTIME-AFFECTING — limits set too low can OOMKill; needs kubectl top baselines first (gather live before proposing numbers); roll out one service at a time with rollback ready	Revert manifest, resync
8. Documentation & DR runbook	All doc-drift items	Fix host tables, role tables, playbook lists, ingress/storage claims; add an explicit cold-rebuild runbook reflecting the now-closed GitOps gaps	None	Revert
9. CI & Claude Code guidance	New .gitea/workflows/validate.yaml (yamllint/ansible-lint/kubeconform on push); CLAUDE.md validation-commands section; widen secrets-leak-scanner.md's stated scope to include config/**; consider extending the PreToolUse hook to block Claude Code from reading .env/router/backup-base.tar.gz directly	New automation surface	None to the cluster; new CI pipeline needs approval since it's a new automation surface	Delete workflow file

Every stage gets validated with whatever's actually available (ansible-playbook --syntax-check, yamllint, ansible-lint, kubeconform/kubectl apply --dry-run=client where a live context isn't required) before being reported done — with the concrete diff and validation output shown, not just a claim of success.

---

Claude Code Project Support

• CLAUDE.md: keep it, add a short "Validation commands" section once Stage 2 lands (exact yamllint/ansible-lint/ansible-playbook --syntax-check invocations).
• Guard against accidental .env/key/backup access: partially covered already (.gitignore, sealed-secret PreToolUse hook). Extend that hook to also block Claude Code Read of .env and router/backup-base.tar.gz — small addition, Stage 9.
• One validation skill/command: rather than a new skill, widen secrets-leak-scanner.md's stated file scope to explicitly include config/** (the actual gap that let the Dashy API key through) — cheaper and more targeted than adding new tooling.