From 791c0a5fa5342058fedcbe184d51ddbea198ffe7 Mon Sep 17 00:00:00 2001 From: Nik Afiq Date: Thu, 23 Jul 2026 11:25:20 +0900 Subject: [PATCH] feat: add infrastructure change reviewer and secrets leak scanner, update CLAUDE.md with operational guidance --- .claude/agents/infra-change-reviewer.md | 72 ++++++++++++ .claude/agents/secrets-leak-scanner.md | 58 ++++++++++ .claude/settings.json | 26 +++++ .claude/skills/add-service/SKILL.md | 84 ++++++++++++++ .claude/skills/regen-sealed-secret/SKILL.md | 49 ++++++++ CLAUDE.md | 120 ++++++++++++++++++++ 6 files changed, 409 insertions(+) create mode 100644 .claude/agents/infra-change-reviewer.md create mode 100644 .claude/agents/secrets-leak-scanner.md create mode 100644 .claude/settings.json create mode 100644 .claude/skills/add-service/SKILL.md create mode 100644 .claude/skills/regen-sealed-secret/SKILL.md create mode 100644 CLAUDE.md diff --git a/.claude/agents/infra-change-reviewer.md b/.claude/agents/infra-change-reviewer.md new file mode 100644 index 0000000..f30e85b --- /dev/null +++ b/.claude/agents/infra-change-reviewer.md @@ -0,0 +1,72 @@ +--- +name: infra-change-reviewer +description: Reviews changes to Ansible roles/playbooks, Kubernetes manifests, Helm values, and Argo CD Applications in this homelab repo against its documented operational invariants before they're applied. Use before running ansible-playbook, kubectl apply, or committing infra changes. +tools: Read, Grep, Glob, Bash +model: sonnet +--- + +You review infrastructure changes in this homelab repo (K3s + Argo CD + Ansible) for correctness against invariants that no linter checks. You have read-only Bash access — you may run `git diff`, `git log`, `kubectl get/describe` (read-only), and similar inspection commands, but never `apply`, `delete`, `patch`, or `ansible-playbook`. + +Check the diff (`git diff` / `git diff --staged`, or the range given to you) against this list. Only report items that actually apply to the changed files — don't pad the review with restated boilerplate. + +## Argo CD +- Applications use `prune: false` almost everywhere. If a change removes a manifest/resource that was previously deployed, flag that the live cluster resource will NOT be auto-deleted and may need manual `kubectl delete`. +- New Application should set `targetRevision: main` unless there's a stated reason to track `HEAD`. +- `syncOptions: [CreateNamespace=true]` should be present if the manifest set creates a new namespace. + +## DNS +- Any new or changed `*.home.arpa` hostname must appear in **both** + `values/pihole.yaml` and `values/pihole-debian.yaml` (`dnsmasq.customDnsEntries`). + If only one file changed, flag it. +- `home.arpa` has no wildcard DNS — every hostname is explicit. + +## Certificates +- Internal (`home.arpa`) resources should use `issuerRef.name: internal-ca-issuer`. +- Public (`nik4nao.com`) resources should use `issuerRef.name: letsencrypt-prod`. +- Flag a `home.arpa` host using the Let's Encrypt issuer or vice versa. + +## Secrets +- No plaintext secret values (tokens, passwords, keys) in tracked YAML. Values + should come from `.env` via a `*-secret.sh` script (runtime, not committed) + or be sealed via `kubeseal` into a `*-sealed.yaml` (ciphertext under + `spec.encryptedData`, safe to commit). +- A hand-edited `*-sealed.yaml` (anything other than regenerating it via its + matching script) is almost always wrong — sealed output should never be + hand-authored. +- New secret-consuming manifests should reference an existing Secret name + rather than inlining `stringData`. + +## Gitea +- Gitea's public ingress is a manual `IngressRoute`; the Helm chart's own + ingress is disabled in `values/gitea.yaml` — don't re-enable it as a side + effect of a values change. +- A `ROOT_URL` change may require deleting the generated inline config secret + before Argo CD reconciles cleanly — flag this if `values/gitea.yaml` touches + `ROOT_URL`/server config. + +## GPU node / node scheduling +- `gpu-node` is tainted `spot=true:NoSchedule` (see `ansible/host_vars/gpu-node.yaml`). + Any workload meant to run there needs a matching toleration, or it won't + schedule. Flag new workloads targeting `node-role: gpu` / `nik4nao.com/gpu` + without one. + +## Ansible +- The `ollama` role branches on `ansible_facts['system']` to support both + macOS (Homebrew + launchd) and Linux (install script + systemd). A change + to one branch that isn't mirrored in the other is a likely bug — flag it. +- There are two `ansible.cfg` files (repo root and `ansible/`) with different + `inventory`/`roles_path` settings. A playbook or role-path change should be + checked against whichever `ansible.cfg` is actually active for the working + directory the command runs from. +- Playbooks are expected to be idempotent/rerunnable — flag tasks without + `creates`/`state` guards that would fail or duplicate work on a rerun. + +## Pi-hole HA +- `values/pihole-debian.yaml`'s external IPs are known to get dropped on + chart upgrades, requiring `manifests/network/pihole-debian-patch.sh` to be + rerun. If this change touches the Pi-hole chart version or its Service + spec, mention that the patch script may need to run again post-sync. + +Report findings as a short list: what's wrong, which file/line, and why it +matters (cite the invariant above). If nothing applies, say so briefly — +don't invent issues. diff --git a/.claude/agents/secrets-leak-scanner.md b/.claude/agents/secrets-leak-scanner.md new file mode 100644 index 0000000..976fe03 --- /dev/null +++ b/.claude/agents/secrets-leak-scanner.md @@ -0,0 +1,58 @@ +--- +name: secrets-leak-scanner +description: Scans staged/diffed files in this homelab repo for plaintext secrets that should instead come from .env or be sealed via kubeseal. Use before committing changes to manifests, Ansible vars, or Helm values. +tools: Read, Grep, Glob, Bash +model: sonnet +--- + +You scan changes in this repo for secrets that are about to be committed in +plaintext. You have read-only Bash access (`git diff`, `git status`, `grep`) +— never modify or stage files yourself. + +## What "should never be plaintext in git" looks like here + +Cross-reference `.env.example` for the full list of secret-shaped variable +names this repo uses: `PORKBUN_API_KEY`, `PORKBUN_SECRET_KEY`, +`K3S_NODE_TOKEN`, `GITEA_RUNNER_TOKEN`, `GRAFANA_ADMIN_PASSWORD`, +`AUTHENTIK_PROXY_TOKEN`, `AUTHENTIK_*_CLIENT_ID`/`_CLIENT_SECRET`, +`REGISTRY_PASSWORD`, `HA_TOKEN`, `DISCORD_TOKEN`, `GUILD_ID`, +`IMMICH_POSTGRES_PASSWORD`, `PIA_USER`, `PIA_PASSWORD`, and Ansible's +`vault_k3s_node_token` (`ansible/group_vars/all/vault.yaml`). + +A finding is real if a tracked (non-`.env`) file contains what looks like an +actual value for one of these — not a template placeholder +(`your_x_here`, `pk1_your_key_here`), not a Jinja reference (`{{ vault_x }}` +or `{{ item }}`), and not a shell variable expansion (`"${X}"`). + +## Where legitimate secrets are allowed to live + +- `.env` itself (gitignored — flag if it's ever staged: `git status` showing + `.env` as staged/tracked is itself a finding). +- `*-sealed.yaml` files, but only as ciphertext under `spec.encryptedData` — + if one of these contains a plausible plaintext value instead of encrypted + blob data, that's a finding (it means kubeseal wasn't actually used, or the + file was hand-edited). +- `ansible/group_vars/all/vault.yaml`, but only if it's actually + Ansible-Vault-encrypted (`$ANSIBLE_VAULT;...` header) — an unencrypted + value there is a finding. + +## Also check for + +- High-entropy strings assigned to obviously credential-shaped keys + (`password:`, `token:`, `secret:`, `apiKey:`, `-----BEGIN ... PRIVATE + KEY-----`) in any manifest, values file, or playbook. +- A `kubectl create secret ... --from-literal=X=` committed + directly instead of piped through `kubeseal` or run at apply-time from + `.env`. +- Real IPs/hostnames are not secrets and should not be flagged — this repo's + README documents its network topology openly; don't waste findings on + `192.168.7.x` addresses or `*.home.arpa`/`*.nik4nao.com` hostnames. + +## Output + +List only real findings: file, line, the variable/value in question (redact +the actual secret value in your report — show the key name and enough +context to locate it, not the secret itself), and which of the two +"never plaintext" mechanisms it should be using instead (`.env` + +`*-secret.sh`, or `kubeseal` → `*-sealed.yaml`). If nothing is found, say so +briefly. diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..d751972 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,26 @@ +{ + "hooks": { + "PreToolUse": [ + { + "matcher": "Edit|Write", + "hooks": [ + { + "type": "command", + "command": "file=$(jq -r '.tool_input.file_path // empty'); case \"$file\" in *-sealed.yaml) echo \"Blocked: $file is generated by kubeseal. Don't hand-edit sealed output -- update the source (.env / the raw secret) and rerun the matching manifests/**/*-secret.sh script instead.\" >&2; exit 2 ;; esac; exit 0" + } + ] + } + ], + "PostToolUse": [ + { + "matcher": "Edit|Write", + "hooks": [ + { + "type": "command", + "command": "file=$(jq -r '.tool_input.file_path // empty'); case \"$file\" in */values/pihole.yaml) echo \"Reminder: home.arpa has no wildcard DNS -- mirror this hostname change in values/pihole-debian.yaml too.\" >&2; exit 2 ;; */values/pihole-debian.yaml) echo \"Reminder: home.arpa has no wildcard DNS -- mirror this hostname change in values/pihole.yaml too.\" >&2; exit 2 ;; esac; exit 0" + } + ] + } + ] + } +} diff --git a/.claude/skills/add-service/SKILL.md b/.claude/skills/add-service/SKILL.md new file mode 100644 index 0000000..1a30043 --- /dev/null +++ b/.claude/skills/add-service/SKILL.md @@ -0,0 +1,84 @@ +--- +name: add-service +description: Add a new service to the homelab cluster end-to-end (manifests, Argo CD Application, DNS, certs/secrets). Use when the user wants to deploy a new app or service to the K3s cluster. +--- + +Add a new service to the cluster: $ARGUMENTS + +Follow the checklist from `argocd/README.md`, in order: + +## 1. Manifests or Helm values + +- Raw resources: create `manifests//.yaml`. Follow the style of + existing files (see `manifests/portfolio/portfolio.yaml` for a full example: + `Namespace` → `Deployment` → `Service` → `Certificate` → `IngressRoute`, all + in one file separated by `---`). Start the file with an `# Apply:` / + `# Delete:` / `# Description:` comment block matching sibling files. +- Helm chart instead: add `values/.yaml` with the chart's values. + +## 2. Argo CD Application + +Add `argocd/apps/.yaml`. For a raw manifest directory: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: + namespace: argocd +spec: + project: default + source: + repoURL: https://gitea.nik4nao.com/nik/homelab.git + targetRevision: main + path: manifests/ + directory: + recurse: true + include: '*.yaml' + destination: + server: https://kubernetes.default.svc + namespace: + syncPolicy: + automated: + prune: false + selfHeal: true + syncOptions: + - CreateNamespace=true +``` + +For a Helm chart, mirror an existing chart-backed Application (e.g. +`argocd/apps/traefik.yaml`) and point it at `values/.yaml`. + +`prune: false` is deliberate — deleting a manifest from Git will NOT remove +the resource from the cluster automatically; say so if the user is removing +something, not just adding. + +## 3. DNS (internal services only) + +`home.arpa` has no wildcard DNS. If this service gets a `*.home.arpa` +hostname, add it to **both**: +- `values/pihole.yaml` (`dnsmasq.customDnsEntries`, format + `address=/.home.arpa/`, plus an `ingress.hosts` entry if the + service itself is exposed through Pi-hole's own ingress block — most + services don't need that part, just the `customDnsEntries` line) +- `values/pihole-debian.yaml` (same entry, kept in sync with the primary) + +Public services use `nik4nao.com` and don't need Pi-hole entries. + +## 4. Certificates and secrets + +- Internal (`home.arpa`): `Certificate` with `issuerRef.name: internal-ca-issuer`. +- Public (`nik4nao.com`): `Certificate` with `issuerRef.name: letsencrypt-prod`. +- If the service needs credentials, decide runtime-script vs sealed-secret — + see the `regen-sealed-secret` skill / `manifests/README.md` for the + distinction. Don't hardcode secret values into the manifest. +- If the image is pulled from the private Gitea registry, add an + `imagePullSecrets` reference and reuse/create the matching + `registry-secret.sh` pattern (see `manifests/portfolio/registry-secret.sh`). + +## 5. Commit + +Commit manifests, the Application file, and any DNS/values changes together. +Do not commit `.env` or any plaintext secret values. Let the app-of-apps +reconcile — don't `kubectl apply` the new Application by hand unless the user +asks for an immediate manual sync. diff --git a/.claude/skills/regen-sealed-secret/SKILL.md b/.claude/skills/regen-sealed-secret/SKILL.md new file mode 100644 index 0000000..25d3cba --- /dev/null +++ b/.claude/skills/regen-sealed-secret/SKILL.md @@ -0,0 +1,49 @@ +--- +name: regen-sealed-secret +description: Regenerate a committed *-sealed.yaml SealedSecret from .env values. Use when rotating a secret or when a *-secret.sh script needs to be run to refresh sealed output. +disable-model-invocation: true +--- + +Regenerate the sealed secret for: $ARGUMENTS + +This repo keeps sealed secrets and their source script side by side (e.g. +`manifests/home-services/discord-bot-secret.sh` → +`manifests/home-services/discord-bot-sealed.yaml`, +`manifests/media/immich-postgres-secret.sh` → +`manifests/media/immich-postgres-sealed.yaml`). The script is the source of +truth; the `*-sealed.yaml` file is generated output. + +## Steps + +1. Find the matching `*-secret.sh` script for the target secret (search + `manifests/**/*-secret.sh`). If none exists yet, model the new one on + `manifests/home-services/discord-bot-secret.sh`: + ```bash + source "$(dirname "$0")/../../.env" + kubectl create secret generic \ + --namespace= \ + --from-literal=="${ENV_VAR}" \ + --dry-run=client -o yaml \ + | kubeseal --controller-namespace=kube-system \ + --controller-name=sealed-secrets-controller \ + --format=yaml \ + > "$(dirname "$0")/-sealed.yaml" + ``` +2. Confirm the source value is up to date in `.env` (copy from + `.env.example` if the key is missing, then ask the user to fill it in — + never invent a secret value). +3. Confirm `kubeseal` can reach the cluster's controller: + `kubeseal --controller-namespace=kube-system --controller-name=sealed-secrets-controller --fetch-cert` + should succeed against the live cluster (`minisforum`, context `default`). +4. Run the script: `bash manifests//-secret.sh`. +5. Diff the resulting `*-sealed.yaml` — it should be the only file that + changed, and it should NOT contain plaintext (sealed secrets are + ciphertext under `spec.encryptedData`, unlike the raw + `kubectl create secret ... -o yaml` output piped into it). +6. Commit only the regenerated `*-sealed.yaml`. Never commit the raw + `kubectl create secret --dry-run=client -o yaml` output or `.env` itself. + +If the secret should instead be created directly in-cluster without being +committed (the runtime-script pattern, e.g. `authentik-secret.sh`), skip +`kubeseal` entirely and pipe straight to `kubectl apply -f -` — don't +sealed-secret something that was never meant to be committed. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..86de43e --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,120 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What this is + +Infrastructure-as-code for a small K3s homelab. There is no application source +code to build or test here — this repo *is* the infrastructure: Ansible for +host bootstrap, Argo CD for GitOps reconciliation of the cluster, and raw +Kubernetes manifests / Helm values for service configuration. "Testing" a +change means applying it (via `ansible-playbook`, `kubectl`, or letting Argo +CD sync) and checking real cluster/service state — there are no unit tests. + +Full operational detail (bootstrap sequence, secrets workflow, storage +layout, URL list, gotchas) lives in the root `README.md` — read it before +making non-trivial changes. Scoped `README.md` files in `ansible/`, +`argocd/`, and `manifests/` cover directory-specific workflows. + +## Operating rules for Claude Code + +- **There is no staging environment.** Argo CD watches `main` on the Gitea + remote directly, and most `Application`s run with `selfHeal: true` (see + Architecture below) — so `git push` to `main`, especially anything under + `manifests/`, `values/`, or `argocd/`, is a production deploy, not just a + code change, and Argo CD will keep re-asserting it even against manual + cluster edits. Never push to `main` without the user's explicit go-ahead + for that specific push — an earlier approval to commit is not approval to + push. +- **Never run a command that needs the user's own credentials** (API/personal + access tokens, passwords, anything destined for `.env` or an MCP `-e` + flag) directly, and don't hand it back as an inline `! command` either. + Write the exact command to a file (e.g. under the session scratch/tmp + directory — never committed) and ask the user to open it, review it, and + run it themselves. This keeps secrets out of the conversation transcript. + +## Architecture + +Three-layer flow: **Ansible** brings up hosts and anything that must run +outside Kubernetes → **Argo CD** (`manifests/argocd/app-of-apps.yaml` → +`argocd/apps/*.yaml`) reconciles everything else from Git → each +`Application` points at either a raw manifest directory under `manifests/` +or a Helm chart configured by `values/*.yaml`. + +Hosts (see `ansible/inventory.yaml`): + +| Host | IP | Role | +| --- | --- | --- | +| `minisforum` | `192.168.7.77` | K3s server, Traefik entrypoint, primary app node | +| `debian` | `192.168.7.183` | K3s agent, NFS storage (`/mnt/storage`), secondary Pi-hole | +| `mac-mini` | `192.168.7.96` | Standalone Docker host — Watch Party, Ollama (not in the K3s cluster) | +| `gpu-node` | `192.168.7.98` | K3s agent with NVIDIA GPU, tainted `spot=true:NoSchedule`, labeled `node-role: gpu`; runs Ollama directly on the host, not as a pod | + +K3s's bundled Traefik and ingress controller are disabled — Traefik is +installed and managed through Argo CD instead. Internal services live under +`home.arpa` (internal CA via cert-manager, no wildcard DNS — every hostname +must be added explicitly to **both** `values/pihole.yaml` and +`values/pihole-debian.yaml`). Public services live under `nik4nao.com` +(Let's Encrypt). + +### Repo layout + +| Path | Purpose | +| --- | --- | +| `ansible/` | Host bootstrap and non-Kubernetes services (inventory, roles, playbooks, `group_vars/`, `host_vars/`) | +| `argocd/apps/` | One file per Argo CD `Application` | +| `argocd/values/` | Helm values for installing/reconciling Argo CD itself | +| `manifests/` | Raw Kubernetes resources grouped by service area (see `manifests/README.md`) | +| `values/` | Helm values consumed by Argo CD `Application`s for third-party charts | +| `config/dashy/` | Dashy dashboard config, injected via a script (not the raw manifest) | +| `router/` | OpenWRT (UCI) config backup for the network router — not part of the Ansible/K3s flow | + +### Ansible + +Three inventory groups (`k3s_server`, `k3s_agents`, `mac_mini`) plus +`gpu_workstation` (see `ansible/inventory.yaml`). Per-host variables live in +`ansible/host_vars/.yaml`; shared/secret vars in +`ansible/group_vars/all/vault.yaml`. Roles are composed per playbook, e.g. +`ansible/playbooks/setup-gpu-node.yaml` runs `common → docker → nvidia → +k3s-agent → ollama → glances` against `gpu_workstation`. Note there are two +`ansible.cfg` files (repo root and `ansible/`) — check which one is active +for the working directory a command is run from before assuming inventory +defaults. + +The `ollama` role branches on `ansible_facts['system']` to support both the +Mac Mini (Homebrew + launchd) and Linux GPU nodes (install script + +systemd), so changes to it must keep both code paths working. + +### Argo CD + +Most `Application`s use automated sync with `selfHeal: true` and +`prune: false` — Argo CD corrects drift but does **not** delete cluster +resources removed from Git; matching manual cleanup is often required. +`targetRevision: main` is the default for repo-managed sources. + +## Secrets + +Never commit plaintext secrets. Two patterns coexist, both listed in +`manifests/README.md`: + +- **Runtime scripts** (`manifests/**/*-secret.sh`) read `.env` (copy from + `.env.example`) and create Kubernetes `Secret`s directly against the live + cluster — nothing plaintext is committed. +- **Sealed Secrets** (`*-sealed.yaml`) are safe to commit; regenerate them + with the matching script and commit the resulting YAML. `kubeseal` must + point at the in-cluster `sealed-secrets-controller` (`kube-system`). + +`ansible/group_vars/all/vault.yaml` holds Ansible-side secrets (e.g. +`vault_k3s_node_token`) referenced from `host_vars`. + +## Key gotchas (see README.md "Gotchas" for the full list) + +- Pi-hole has no wildcard DNS — add every new `home.arpa` hostname to both + Pi-hole values files. +- Gitea uses a manual public `IngressRoute`; its Helm-chart ingress is + disabled in `values/gitea.yaml`. Changing `ROOT_URL` may require deleting + the generated inline config secret before Argo CD reconciles cleanly. +- Secondary Pi-hole's external IPs can be lost on chart upgrades — rerun + `manifests/network/pihole-debian-patch.sh`. +- Dashy's manifest ships an empty ConfigMap shell; real config only lands + after running `manifests/core/apply-dashy-config.sh`.