This reverts commit fd13b8a. The registry-host migration broke image
pulls for home-services/portfolio (internal CA not trusted by
containerd) and has left them Degraded for 11 days; reverting back to
gitea.nik4nao.com pending a decision on how to proceed. Not pushed yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The public domain is unreachable while moving, and the cluster had no
Traefik route to Gitea at all (public or internal), leaving every Argo
CD Application stuck in Unknown sync. Add a gitea.home.arpa
Certificate/IngressRoute, repoint Argo CD's repoURL, Gitea's own
DOMAIN/ROOT_URL/SSH_DOMAIN, the container registry references, the
Gitea Actions runner, and the watch-party clone URL at the internal
hostname.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Callers on the LAN (e.g. nik-gpu) don't trust this cluster's internal CA,
and installing it everywhere was judged not worth it for a LAN-only,
bearer-token-authenticated endpoint. Drops the internal-CA Certificate and
switches the IngressRoute from websecure to the web entrypoint.
TODO: revert once done - restores plaintext-only gRPC (no client cert
required) so synthesis can be exercised via port-forward + grpcurl
-plaintext from outside the cluster, without extracting cert material.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ansible.cfg sets inject_facts_as_vars: False repo-wide (the ollama role
already follows this via ansible_facts['system']), but these two tasks
referenced the bare ansible_hostname magic var, which doesn't exist under
that setting. Surfaced today as a hard failure re-running setup-gpu-node -
previously latent since these nodes' labels/taints were already correct
from whenever they last applied cleanly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
host_vars/gpu-node.yaml's ufw_allowed_ports has overridden (not extended)
the common role's default list since the node was added, silently dropping
the Flannel VXLAN (8472/udp), K3s API (6443/tcp), and Kubelet (10250/tcp)
rules every other node gets. Went unnoticed because kubectl logs/exec/stats
tunnel through the agent's outbound connection to the k3s server rather
than needing a direct inbound path - but real pod dataplane traffic (e.g.
tts-gateway on nik-gpu resolving DNS against CoreDNS on nik-debian) needs
actual VXLAN connectivity and was blackholing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Deploys the Go gRPC gateway and Python/libtorch inference sidecar as one
pod on nik-gpu (nodeSelector/toleration/runtimeClassName: nvidia, sidecar
requesting nvidia.com/gpu: 1), matching the existing GPU device plugin.
Model checkpoint/config are bind-mounted from /data/tts-gateway, added to
gpu-node's Ansible data_dirs for consistency with the other hosts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This points at nik-debian's raw IP (192.168.7.183) for a host-level SMB/Samba
share, not a Kubernetes Service. Stage 3 removed it based on "no Certificate/
Ingress/IngressRoute references it" -- the wrong test for a non-HTTP,
non-Kubernetes service that was never going to have one. Broke real SMB
access. Restored to both Pi-hole values files in their original position.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not used enough to justify chasing the stale-NFS-handle issue on its
photos-art mount. Removes manifests/media/immich.yaml (Namespace, PVCs,
Deployments, Service, Certificate, IngressRoute), its secret script and
sealed secret, immich.home.arpa from both Pi-hole values files, its Dashy
dashboard tile, and all other repo references (READMEs, secrets-leak-scanner
scope, regen-sealed-secret example, .env.example).
Per explicit confirmation: this also means the live namespace's PVCs
(immich-library 50Gi, immich-postgres-data 20Gi, immich-ml-cache 10Gi, all
local-path with Delete reclaim policy) are intended to be deleted along with
it -- any photos actually uploaded to Immich's own library are gone once the
namespace is deleted, separately from the read-only NFS folders it browsed
(untouched either way, those live independently on nik-debian).
Git removal alone does not delete the live cluster resources -- the media
Application has prune: false. Manual `kubectl delete namespace immich` still
needed to actually free the namespace/PVCs/data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root-causes today's Gitea outage. The official Gitea Helm chart
(dl.gitea.com/charts, gitea/helm-gitea) always builds Gitea's own [database]
config directly from the plaintext postgresql.global.postgresql.auth.password
value, confirmed from templates/gitea/config.yaml:
{{- $_ := set .Values.gitea.config.database "PASSWD" .Values.postgresql.global.postgresql.auth.password -}}
`existingSecret` only affects the bundled postgresql subchart's own credential
provisioning -- it has no effect on what Gitea itself connects with. Removing
the plaintext field in the earlier Stage 1 "secret hygiene" commit made this
silently fall back to the chart's built-in default password ("gitea", 5
chars), which got re-baked into the persisted app.ini on every pod restart
regardless of what the live Postgres role's real password was -- hence the
repeating CrashLoopBackOff no amount of Postgres-side fixing could resolve.
Restores a plaintext password (the value already live on Postgres from this
incident's recovery), with a comment explaining why, so this doesn't get
"cleaned up" back into the same breakage later. Removes the now-nonfunctional
manifests/gitea/gitea-postgres-secret.sh and its .env.example entry. Getting
this fully off plaintext would need a Config Management Plugin or similar --
not attempted here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bitnami-labs.github.io/sealed-secrets returns 404 -- Bitnami migrated the
sealed-secrets Helm repo to the main bitnami org on 2026-06-15
(bitnami/sealed-secrets#1982), and GitHub Pages doesn't redirect the old
Pages URL. Pre-existing external breakage (last successful Argo CD sync was
2026-05-16, before the migration), not something introduced this session --
the sealed-secrets-controller pod itself kept running fine throughout, this
only blocked Argo CD's ability to diff/compare state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both Deployments' rolling update briefly ran an old+new pod pair on the same
node (node-role: storage), sharing the same config storage (qbittorrent's PVC,
jdownloader's hostPath) -- both apps are effectively singletons that lock
their config directory, so the new instance conflicted with the still-running
old one:
- qbittorrent: hit a known qbittorrent:5.2.0 image bug (linuxserver/
docker-qbittorrent#432) where a stale WebUI lockfile prevents the server
from ever binding its port while a second instance is present -- surfaced
as the new pod's readiness probe getting "connection refused" indefinitely.
On top of that, my livenessProbe (30s/30s) was killing the container
(exitCode 137) before qbittorrent had any chance to come up at all. Dropped
the livenessProbe (readiness alone can never kill a container, only mark it
not-ready) and loosened the readinessProbe timing.
- jdownloader: the new pod's app process detected the old instance's lock in
the shared /data/jdownloader hostPath and exited cleanly (exitCode 0) rather
than run a duplicate copy -- nothing to do with probe timing.
Both Deployments now use `strategy: Recreate` instead of the RollingUpdate
default, so any future rollout fully stops the old pod before starting the
new one -- this is the actual fix (matches the pattern immich.yaml already
used). Applying this will briefly restart both currently-running pods to
verify it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The placeholder targetRevision reached origin/main and broke the Application's
comparison (ComparisonError: "improper constraint: REPLACE_ME"), showing as a
broken card in the Argo CD UI. The actual Authentik pods were never affected
(sync policy has no `automated` block, so nothing was ever applied against the
live release) -- confirmed all authentik namespace pods stayed Running 1/1
throughout.
Fixed using the real deployed chart version read off the live authentik-server
pod's helm.sh/chart=authentik-2026.2.1 label, not a guess.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Everything is committed locally on main; nothing has been pushed. See the
session summary for the specific items that still need your input or action
before any of this goes live (secret rotation values, Authentik chart
version, push go-ahead, live storage/Grafana decisions, router archive review).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 9 of REFACTOR_PLAN.md.
- New .gitea/workflows/validate.yaml: runs yamllint/ansible-lint/
ansible-playbook --syntax-check/kubeconform on push and PR via the
self-hosted act_runner. Read-only lint pass, no cluster access, no
apply/deploy step. This is a new automation surface -- review before
relying on it; it only takes effect once pushed and the runner picks it up.
- Widen secrets-leak-scanner.md's stated scope to explicitly cover config/**
-- the Dashy API key (finding #4) slipped through previously because the
scanner's description read as manifests/Ansible/Helm-values-centric and
didn't call out app config directories.
- Add a "Validation commands" section to CLAUDE.md with the exact
yamllint/ansible-lint/kubeconform invocations, so future sessions run them
before claiming a change is done instead of just asserting success.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 8 of REFACTOR_PLAN.md. The documented bootstrap sequence never
mentioned installing Authentik or the cert-manager ClusterIssuers -- both
required for the rest of the stack to work, per the original audit's Critical
findings #1/#2. Now that cert-manager-config and authentik-config are real
Argo CD Applications (previous commit), the runbook reflects what's actually
automatic vs. still-manual (Authentik's Helm chart install remains a one-time
manual step pending a chart-version decision -- see argocd/apps/authentik.yaml).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 7 of REFACTOR_PLAN.md (findings #16). Covers dashy, glances,
ca-installer, authentik-proxy-outpost, jellyfin, qbittorrent/jdownloader main
containers (their gluetun sidecars already had probes), and all 4 Immich
Deployments -- previously none of these had any protection against one
workload starving another on this fixed-capacity cluster, nor automatic
restart on hang.
Values are sized from live `kubectl top pod` baselines gathered this session
(not guessed): e.g. Jellyfin/Immich-server were observed at ~3.1-3.3Gi
resident, so their limits give headroom above that (4Gi) rather than an
arbitrary round number. Used tcpSocket probes instead of httpGet wherever I
wasn't certain of an app's exact health-check path (Immich, Postgres/Redis),
to avoid a wrong path causing false probe failures on a live service.
This is Kubernetes-native and takes effect on next pod restart, but should
still be rolled out watching `kubectl top`/restart counts rather than pushed
and forgotten -- limits set too low can OOMKill under real load.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 5 + part of Stage 6/7 of REFACTOR_PLAN.md. This is the highest-risk
stage per the plan -- these Applications are NOT to be pushed/synced blindly.
Each needs `kubectl diff` against live state one at a time before enabling.
New Applications (previously-live resources with zero GitOps coverage):
- cert-manager-config.yaml (manifests/cert-manager: both ClusterIssuers + the
internal CA Certificate -- every TLS cert in the cluster depends on these,
and nothing currently restores them on a cold rebuild).
- authentik-config.yaml (manifests/authentik: ingress, proxy outpost,
middleware -- raw manifests only, low risk).
- authentik.yaml (the Authentik Helm chart itself): sync is deliberately left
MANUAL and targetRevision is a REPLACE_ME placeholder -- I don't have a safe
way to read the live chart version (`helm list -n authentik`), and guessing
wrong risks an unwanted upgrade/downgrade of the SSO IdP gating Argo CD/
Grafana/Gitea logins. Needs your input before this one goes anywhere.
- network.yaml: widens coverage to the 4 non-sealed files in manifests/network
(ddns-cronjob, glances-debian-ingress, traefik-dashboard-ingress,
watch-party-ingress) that were previously invisible to Argo CD; keeps
network-secrets.yaml scoped to *-sealed.yaml only.
Fixes:
- homeassistant.yaml: destination.namespace was "homeassistant" (empty,
unused) while the actual resources are hardcoded to "default" -- corrected,
dropped CreateNamespace=true. The old empty namespace isn't auto-deleted
(prune: false); safe to remove by hand if desired.
- gitea-backup.yaml: added the missing Namespace object (nothing created
"gitea-backup" before); replaced a cluster-wide ClusterRole/ClusterRoleBinding
granting pods/exec everywhere with a Role/RoleBinding scoped to the `gitea`
namespace, matching what the backup script actually execs into. NOTE: this
is already under active sync via gitea-secrets.yaml (selfHeal: true,
prune: false) -- once pushed, the old ClusterRole/ClusterRoleBinding will
need manual `kubectl delete` since Argo CD won't prune them.
- Added sync-wave "-2" to cert-manager/sealed-secrets Applications so their
CRDs land before consumers (matches the existing -1/0 wave pattern).
- Normalized targetRevision HEAD -> main on home-services/otel-collector/tempo.
- Normalized sync policy per your decision: home-services/otel-collector/tempo
prune true -> false; pihole/pihole-debian selfHeal false -> true (repo-wide
consistency, per your call on finding #18).
Verified: kubeconform valid across all manifests + Argo CD Application objects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Marks Stages 1-4 and part of Stage 6 done in the tracker. Records three things
discovered via read-only kubectl checks during this session that weren't in
the original audit: Grafana/Loki's static PVs are orphaned (dynamically
provisioned via local-path instead), Finding #24 (GPU passthrough) is actually
working and doesn't need fixing, and live node hostnames differ cosmetically
from ansible/inventory.yaml's aliases.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 4 of REFACTOR_PLAN.md.
- Merge ansible/ansible.cfg into root ansible.cfg (single source of truth);
the dual-config setup silently broke documented commands and lost
host_key_checking=False when run from the ansible/ directory.
- Add ansible/requirements.yml pinning community.general, ansible.posix,
community.docker -- previously undocumented deps of the glances/watch-party
roles that would fail a fresh bootstrap.
- Align K3s version to v1.32.4+k3s1 across roles/k3s-server, roles/k3s-agent,
and host_vars/gpu-node.yaml defaults (was skewed: .2 vs .4). This only
changes what a *future* provision installs -- minisforum/debian are still
live on v1.32.2+k3s1 until separately upgraded.
- Fix kubeconfig fetch/replace path mismatch in k3s-server role: the `fetch`
task (flat: true) writes to ~/.kube/config, but `replace` was targeting a
/tmp/k3s-minisforum.yaml nothing creates -- would break a fresh rebuild.
- gitea-runner: only remove /run/docker.sock when it's actually a directory
(task name implied a check that wasn't there); tighten registration-token
systemd unit from 0644 to 0600.
- nvidia: stop unconditionally reporting `changed` (and restarting Docker)
on every run for an idempotent runtime-configure command.
- Gate the K3s join-token debug print and WireGuard client-config/QR display
behind opt-in vars (k3s_show_token, wireguard_show_client_configs), default
off -- both were printing real secrets to console on every run.
- Parameterize the docker role for Debian and Ubuntu; homeassistant now
depends on it (meta/main.yaml) instead of duplicating a Debian-only Docker
install inline.
- FQCN cleanup across wireguard, homeassistant, and ollama roles/handlers
(bare module names -> ansible.builtin.*/community.general.*/ansible.posix.*),
plus a few ansible-lint name-casing/idiom fixes. Handler renames verified
against their `notify:` call sites so notifications still fire.
- Update ansible/README.md and root README.md: add gpu-node/gpu_workstation
(4th host, previously undocumented), docker/nvidia roles, setup-gpu-node.yaml,
homeassistant.yaml, requirements.yml install step; correct the "Legacy"
homeassistant label (it's the only thing serving ha.home.arpa); correct the
Gitea ingress/backup-storage doc-drift; flag the Grafana/Loki static-PV
binding drift discovered via live cluster check.
Verified: all playbooks pass `ansible-playbook --syntax-check`, yamllint clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 3 + 6 of REFACTOR_PLAN.md.
- Drop nik4nao.home.arpa from both Pi-hole values files: no Certificate,
Ingress, or IngressRoute anywhere references it (confirmed via grep).
- Add the immich.home.arpa and gluetun.home.arpa entries that existed in
values/pihole.yaml but were missing from values/pihole-debian.yaml --
Pi-hole has no wildcard DNS here, so every hostname must be listed in both.
Both files' hostname lists now match exactly (aside from the differing
target IPs, which is expected for primary vs. secondary).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 2 of REFACTOR_PLAN.md. Repo had no CI or local validation tooling at
all. Tuned rules to match this repo's existing conventions (no trailing
newline on hand-written manifests, aligned comment blocks) rather than
rewriting hundreds of pre-existing files to satisfy defaults.
Baseline is clean: yamllint reports 0 issues repo-wide, kubeconform validates
117 manifests + 20 Argo CD Applications, and all Ansible playbooks pass
--syntax-check.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 1 of REFACTOR_PLAN.md. values/gitea.yaml and config/dashy/conf.yaml now
reference secrets injected at apply-time (gitea-postgres-secret.sh, .env) instead
of hardcoding a live DB password and weather API key in git. Both values must be
treated as compromised and rotated by the operator (see .env.example).
Also fixes authentik-ingress.yaml and traefik-dashboard-ingress.yaml, which
pointed at the internal-ca root ClusterIssuer instead of internal-ca-issuer,
the chained issuer every other internal Certificate uses -- causing untrusted-cert
warnings on the SSO login and Traefik dashboard.
Extends .gitignore for *.retry, .vault_pass*, kubeconfig patterns, and editor
swap files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
