The previous commit (89524d4) only captured the file rename - a git
add with one stale/invalid pathspec aborted before staging the actual
content, and the resulting empty diff wasn't caught before committing.
This commit has the real changes described in that one's message:
EXPECTED_VLAN50_IP/NET_RAW on the guard init container, REQUIRE_AUTH
dropped to false with ALLOWED_IPS removed entirely (broken as drafted
- see the file's own header for why), sync-wave 3 annotation, and the
README updates dropping the now-unneeded secret-script references.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move manifests/multus/reserved/browser-vpn-proxy.yaml into the active
multus tree (sync-wave 3) now that qBittorrent and JDownloader have
both independently passed their canary, validation, and soak gates -
the precondition this file's own header already called for.
Add EXPECTED_VLAN50_IP/NET_RAW to its guard init container, matching
the fix qBittorrent/JDownloader/canary all needed for the arping-based
checks to run.
Deliberately no auth (REQUIRE_AUTH=false) and no ALLOWED_IPS, after
discussion: access restriction relies on the home network's own
firewall (Trusted/Lab -> Lab NodePort ALLOW, other VLANs DENY except
Technitium DNS), not an app-level control. Along the way, found and
avoided shipping a real bug in the original draft - go-socks5-proxy's
ALLOWED_IPS parses values with net.ParseIP (not CIDR-aware) and
matches with exact net.IP.Equal(), so the drafted "10.10.40.0/24"
value would have produced a nil whitelist entry that matches no real
client, rejecting every connection. This library has no way to express
a subnet allowlist at all - removed rather than left silently broken.
Also drops the now-unneeded browser-vpn-proxy-credentials Secret and
its companion script/README references.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
qbittorrent.yaml and jdownloader.yaml's own header comments still
pointed at manifests/media/rollback/*-gluetun.yaml and described the
migration as pending approval/preconditions - both are live and
validated now, and that directory is being removed (no kept on-disk
Gluetun copy). Rewrite both headers as a status note instead, pointing
rollback at git history (with the exact pre-migration commit) to match
the root README's rollback table, which gets the same exact-commit
treatment here instead of the vaguer reference from the previous
commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both the root README and manifests/README were still written as if the
whole VLAN 50 migration was drafted-but-unapplied; it's now fully live
and independently validated end to end (canary matrix, both media
workloads' own fail-closed kill-switch test). Update the status line
and PIA region accordingly.
Also drop references to manifests/media/rollback/*-gluetun.yaml, which
is being removed now that qBittorrent/JDownloader are confirmed
working - rollback guidance now points at git history instead of a
kept-file copy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous ip6tables->sysctl fix (26894d1) was itself wrong: writing
/proc/sys/net/ipv6/conf/all/disable_ipv6 failed live with "Read-only
file system" despite NET_ADMIN and a passing [ -w ] check - the
container runtime mounts /proc/sys read-only by default regardless of
capabilities, independent of file permission bits. Making it writable
needs either kubelet's securityContext.sysctls (and
net.ipv6.conf.*.disable_ipv6 isn't on its default safe-sysctls
allowlist, so that means a node-level --allowed-unsafe-sysctls flag)
or securityContext.procMount: Unmasked (which needs pod-level user
namespaces) - too much blast radius for one pod's IPv6 kill switch.
nft (nftables) is genuinely present in the same image (the "nftables"
apk package, installed alongside "iptables" but not "ip6tables") and
handles the ip6 address family without a separate binary, so it needs
no image or capability change. Verified by actually running the exact
commands against the real pinned nicolaka/netshoot:v0.11 image
(digest-matched to what's on nik-debian), not just against docs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nicolaka/netshoot:v0.11 (the pinned tag actually in use) is built on
Alpine 3.18.0, where ip6tables is packaged separately from iptables;
this image's Dockerfile only installs the latter, so ip6tables does
not exist in the container at all (confirmed live: "ip6tables: not
found", and against Alpine's own v3.18 package index, not master's).
Write 1 to /proc/sys/net/ipv6/conf/all/disable_ipv6 instead - per the
kernel's own ip-sysctl docs this is equivalent to also setting
conf/default/disable_ipv6 and every existing per-interface
disable_ipv6 (lo/eth0/net1) at once, and is strictly stronger than an
iptables DROP policy would have been: no IPv6 activity happens on any
interface at all, not just filtered output. Needs no capability beyond
the NET_ADMIN the guard init container already has.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Diagnosed the vlan50-canary FailedCreatePodSandBox->net1-timeout failure
as a stale Pod artifact of the Multus DaemonSet rollout race (sandbox
created via a transient daemon/shim state mid-rollout; the current,
fully-settled daemon's own logs show no ADD for that UID, only a DEL).
No defect found in 02-daemonset.yaml or 10-nad-vlan50.yaml; both are
unchanged.
Independent of that diagnosis, harden the shared guard script per
review: print safe interface/address/route diagnostics before every
FATAL exit; validate net1 actually carries the workload's expected
static /24 address rather than just existing; add arping-based
duplicate-address and gateway-reachability checks before installing
the net1 default route (exit-code semantics verified against arping's
own source). Requires EXPECTED_VLAN50_IP and NET_RAW (for arping's raw
ARP sockets) on every consumer - wired into 20-canary.yaml now,
qbittorrent.yaml/jdownloader.yaml need the same when they're migrated.
Document the canary's Pod lifecycle: restartPolicy: Never means a
Failed canary never reruns on its own, and Argo "Synced" only reflects
manifest match, not runtime success - recreate it (new UID) after any
Multus/CNI change before trusting its result. Recommend keeping it a
manually recreated, controller-less Pod rather than a Job/Deployment,
since unattended auto-retry risks a duplicate-address race on its
static .100 IP - the exact class of bug this diagnosis just walked
through.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
k3s deliberately ships only the plugins its own flannel networking
needs (bandwidth, bridge, firewall, flannel, host-local, loopback,
portmap — confirmed against k3s's own build script; macvlan's absence
is an intentional upstream scope decision, k3s-io/k3s#9224). Multus's
macvlan delegate for the vlan50 NAD therefore had nothing to exec even
after the binDir/mount fixes.
Add a cni-plugins init container mirroring Rancher's own official
rke2-multus Helm chart (the fix k3s's own docs recommend for this
exact gap): rancher/hardened-cni-plugins, digest-pinned, writing into
our existing cnibin mount instead of the image's hardcoded
/opt/cni/bin, with SKIP_CNI_BINARIES=flannel so it doesn't overwrite
k3s's already-working flannel plugin.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
binDir alone (previous commit) wasn't sufficient: k3s's plugin bin dir
is a directory of symlinks into a content-hash-named sibling directory,
and the narrow cnibin mount didn't cover that sibling. FindInPath's
os.Stat follows symlinks, so it failed inside the container even though
the same path resolves fine on the real host. Mount the parent
(k3s-cni-data) into the main container instead so the symlinks resolve;
the init container keeps the narrower cnibin mount since it only writes
files and never follows them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause: Multus's ChrootExec resolves a delegate plugin's absolute
path from the daemon container's own filesystem view, then execs that
exact path string after chrooting into the real host root. binDir
defaulted to /opt/cni/bin, which is valid inside the container (bind-
mounted to k3s's real bin dir) but doesn't exist on the actual host, so
every pod attaching to the VLAN 50 NAD failed FailedCreatePodSandBox
trying to delegate to flannel. Confirmed against multus-cni's source at
the exact pinned v4.3.0 tag, not assumed. Sets binDir explicitly to
k3s's real path and matches the cnibin mount path to it on both sides of
the chroot boundary (main container and the install_multus init
container).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CRD, RBAC, thick-plugin DaemonSet (nik-debian only, k3s CNI paths,
digest-pinned), the VLAN 50 NAD (macvlan bridge on enp1s0.50), the
shared pod egress-guard script, a temporary canary pod, and the
not-yet-deployed browser-vpn-proxy workload (kept in reserved/, outside
this Application's non-recursive source path).
Referenced by argocd/apps/multus.yaml (already pushed) but not deployed
by it — that Application has no syncPolicy.automated, so this still
needs an explicit selective sync per resource.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
minisforum, debian, mac-mini, and gpu-node all moved from
192.168.7.0/24 to 10.10.40.0/24. Updates K3s server/agent config and
node IPs (including gpu-node's host_vars override), NFS export
allow-list and exports template, Pi-hole DNS records and
kube-vip/loadBalancerIP pins, WireGuard's pushed DNS/AllowedIPs, and
the NFS server IP baked into Jellyfin/Kavita/gitea-backup PVs and the
Ollama URL used by ai-gateway/Dashy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This reverts commit fd13b8a. The registry-host migration broke image
pulls for home-services/portfolio (internal CA not trusted by
containerd) and has left them Degraded for 11 days; reverting back to
gitea.nik4nao.com pending a decision on how to proceed. Not pushed yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The public domain is unreachable while moving, and the cluster had no
Traefik route to Gitea at all (public or internal), leaving every Argo
CD Application stuck in Unknown sync. Add a gitea.home.arpa
Certificate/IngressRoute, repoint Argo CD's repoURL, Gitea's own
DOMAIN/ROOT_URL/SSH_DOMAIN, the container registry references, the
Gitea Actions runner, and the watch-party clone URL at the internal
hostname.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Callers on the LAN (e.g. nik-gpu) don't trust this cluster's internal CA,
and installing it everywhere was judged not worth it for a LAN-only,
bearer-token-authenticated endpoint. Drops the internal-CA Certificate and
switches the IngressRoute from websecure to the web entrypoint.
TODO: revert once done - restores plaintext-only gRPC (no client cert
required) so synthesis can be exercised via port-forward + grpcurl
-plaintext from outside the cluster, without extracting cert material.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Deploys the Go gRPC gateway and Python/libtorch inference sidecar as one
pod on nik-gpu (nodeSelector/toleration/runtimeClassName: nvidia, sidecar
requesting nvidia.com/gpu: 1), matching the existing GPU device plugin.
Model checkpoint/config are bind-mounted from /data/tts-gateway, added to
gpu-node's Ansible data_dirs for consistency with the other hosts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Not used enough to justify chasing the stale-NFS-handle issue on its
photos-art mount. Removes manifests/media/immich.yaml (Namespace, PVCs,
Deployments, Service, Certificate, IngressRoute), its secret script and
sealed secret, immich.home.arpa from both Pi-hole values files, its Dashy
dashboard tile, and all other repo references (READMEs, secrets-leak-scanner
scope, regen-sealed-secret example, .env.example).
Per explicit confirmation: this also means the live namespace's PVCs
(immich-library 50Gi, immich-postgres-data 20Gi, immich-ml-cache 10Gi, all
local-path with Delete reclaim policy) are intended to be deleted along with
it -- any photos actually uploaded to Immich's own library are gone once the
namespace is deleted, separately from the read-only NFS folders it browsed
(untouched either way, those live independently on nik-debian).
Git removal alone does not delete the live cluster resources -- the media
Application has prune: false. Manual `kubectl delete namespace immich` still
needed to actually free the namespace/PVCs/data.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root-causes today's Gitea outage. The official Gitea Helm chart
(dl.gitea.com/charts, gitea/helm-gitea) always builds Gitea's own [database]
config directly from the plaintext postgresql.global.postgresql.auth.password
value, confirmed from templates/gitea/config.yaml:
{{- $_ := set .Values.gitea.config.database "PASSWD" .Values.postgresql.global.postgresql.auth.password -}}
`existingSecret` only affects the bundled postgresql subchart's own credential
provisioning -- it has no effect on what Gitea itself connects with. Removing
the plaintext field in the earlier Stage 1 "secret hygiene" commit made this
silently fall back to the chart's built-in default password ("gitea", 5
chars), which got re-baked into the persisted app.ini on every pod restart
regardless of what the live Postgres role's real password was -- hence the
repeating CrashLoopBackOff no amount of Postgres-side fixing could resolve.
Restores a plaintext password (the value already live on Postgres from this
incident's recovery), with a comment explaining why, so this doesn't get
"cleaned up" back into the same breakage later. Removes the now-nonfunctional
manifests/gitea/gitea-postgres-secret.sh and its .env.example entry. Getting
this fully off plaintext would need a Config Management Plugin or similar --
not attempted here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both Deployments' rolling update briefly ran an old+new pod pair on the same
node (node-role: storage), sharing the same config storage (qbittorrent's PVC,
jdownloader's hostPath) -- both apps are effectively singletons that lock
their config directory, so the new instance conflicted with the still-running
old one:
- qbittorrent: hit a known qbittorrent:5.2.0 image bug (linuxserver/
docker-qbittorrent#432) where a stale WebUI lockfile prevents the server
from ever binding its port while a second instance is present -- surfaced
as the new pod's readiness probe getting "connection refused" indefinitely.
On top of that, my livenessProbe (30s/30s) was killing the container
(exitCode 137) before qbittorrent had any chance to come up at all. Dropped
the livenessProbe (readiness alone can never kill a container, only mark it
not-ready) and loosened the readinessProbe timing.
- jdownloader: the new pod's app process detected the old instance's lock in
the shared /data/jdownloader hostPath and exited cleanly (exitCode 0) rather
than run a duplicate copy -- nothing to do with probe timing.
Both Deployments now use `strategy: Recreate` instead of the RollingUpdate
default, so any future rollout fully stops the old pod before starting the
new one -- this is the actual fix (matches the pattern immich.yaml already
used). Applying this will briefly restart both currently-running pods to
verify it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 7 of REFACTOR_PLAN.md (findings #16). Covers dashy, glances,
ca-installer, authentik-proxy-outpost, jellyfin, qbittorrent/jdownloader main
containers (their gluetun sidecars already had probes), and all 4 Immich
Deployments -- previously none of these had any protection against one
workload starving another on this fixed-capacity cluster, nor automatic
restart on hang.
Values are sized from live `kubectl top pod` baselines gathered this session
(not guessed): e.g. Jellyfin/Immich-server were observed at ~3.1-3.3Gi
resident, so their limits give headroom above that (4Gi) rather than an
arbitrary round number. Used tcpSocket probes instead of httpGet wherever I
wasn't certain of an app's exact health-check path (Immich, Postgres/Redis),
to avoid a wrong path causing false probe failures on a live service.
This is Kubernetes-native and takes effect on next pod restart, but should
still be rolled out watching `kubectl top`/restart counts rather than pushed
and forgotten -- limits set too low can OOMKill under real load.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 5 + part of Stage 6/7 of REFACTOR_PLAN.md. This is the highest-risk
stage per the plan -- these Applications are NOT to be pushed/synced blindly.
Each needs `kubectl diff` against live state one at a time before enabling.
New Applications (previously-live resources with zero GitOps coverage):
- cert-manager-config.yaml (manifests/cert-manager: both ClusterIssuers + the
internal CA Certificate -- every TLS cert in the cluster depends on these,
and nothing currently restores them on a cold rebuild).
- authentik-config.yaml (manifests/authentik: ingress, proxy outpost,
middleware -- raw manifests only, low risk).
- authentik.yaml (the Authentik Helm chart itself): sync is deliberately left
MANUAL and targetRevision is a REPLACE_ME placeholder -- I don't have a safe
way to read the live chart version (`helm list -n authentik`), and guessing
wrong risks an unwanted upgrade/downgrade of the SSO IdP gating Argo CD/
Grafana/Gitea logins. Needs your input before this one goes anywhere.
- network.yaml: widens coverage to the 4 non-sealed files in manifests/network
(ddns-cronjob, glances-debian-ingress, traefik-dashboard-ingress,
watch-party-ingress) that were previously invisible to Argo CD; keeps
network-secrets.yaml scoped to *-sealed.yaml only.
Fixes:
- homeassistant.yaml: destination.namespace was "homeassistant" (empty,
unused) while the actual resources are hardcoded to "default" -- corrected,
dropped CreateNamespace=true. The old empty namespace isn't auto-deleted
(prune: false); safe to remove by hand if desired.
- gitea-backup.yaml: added the missing Namespace object (nothing created
"gitea-backup" before); replaced a cluster-wide ClusterRole/ClusterRoleBinding
granting pods/exec everywhere with a Role/RoleBinding scoped to the `gitea`
namespace, matching what the backup script actually execs into. NOTE: this
is already under active sync via gitea-secrets.yaml (selfHeal: true,
prune: false) -- once pushed, the old ClusterRole/ClusterRoleBinding will
need manual `kubectl delete` since Argo CD won't prune them.
- Added sync-wave "-2" to cert-manager/sealed-secrets Applications so their
CRDs land before consumers (matches the existing -1/0 wave pattern).
- Normalized targetRevision HEAD -> main on home-services/otel-collector/tempo.
- Normalized sync policy per your decision: home-services/otel-collector/tempo
prune true -> false; pihole/pihole-debian selfHeal false -> true (repo-wide
consistency, per your call on finding #18).
Verified: kubeconform valid across all manifests + Argo CD Application objects.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Stage 1 of REFACTOR_PLAN.md. values/gitea.yaml and config/dashy/conf.yaml now
reference secrets injected at apply-time (gitea-postgres-secret.sh, .env) instead
of hardcoding a live DB password and weather API key in git. Both values must be
treated as compromised and rotated by the operator (see .env.example).
Also fixes authentik-ingress.yaml and traefik-dashboard-ingress.yaml, which
pointed at the internal-ca root ClusterIssuer instead of internal-ca-issuer,
the chained issuer every other internal Certificate uses -- causing untrusted-cert
warnings on the SSO login and Traefik dashboard.
Extends .gitignore for *.retry, .vault_pass*, kubeconfig patterns, and editor
swap files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
