From 2c26c227ee9d2560b5a7faf5691e51388cb18276 Mon Sep 17 00:00:00 2001 From: Nik Afiq Date: Wed, 16 Sep 2026 23:30:38 +0900 Subject: [PATCH] docs: update CLAUDE.md and README.md to include me.nik4nao.com deployment details --- CLAUDE.md | 5 ++++ README.md | 62 +++++++++++++++++++++++++++++++++++++++++++-- manifests/README.md | 1 + 3 files changed, 66 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index f3304b9..6cfdc68 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -140,3 +140,8 @@ otherwise it reports spurious `unknown-module` errors for `community.general`/ `manifests/network/pihole-debian-patch.sh`. - Dashy's manifest ships an empty ConfigMap shell; real config only lands after running `manifests/core/apply-dashy-config.sh`. +- A newly pushed Argo CD `Application` doesn't create its namespace + instantly — `app-of-apps` only picks it up on its next Git poll. A + namespace-scoped secret script run immediately after pushing can fail + with `namespaces "" not found` until then (or until a forced + refresh). diff --git a/README.md b/README.md index e5dded1..e428a00 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ Public services under `nik4nao.com` use Let's Encrypt. | Git and CI | Gitea, Gitea Actions runner, registry pull secrets, Gitea backup CronJob | | Media | Jellyfin, qBittorrent, JDownloader | | Home services | Home Assistant ingress, HA gateway, AI gateway, Discord bot, Alexa bridge, Alert bridge | -| Public apps | Portfolio, Watch Party ingress to the Mac Mini | +| Public apps | Portfolio, me.nik4nao.com, Watch Party ingress to the Mac Mini | | Dashboard | Dashy | ## Important URLs @@ -64,6 +64,7 @@ Public services under `nik4nao.com` use Let's Encrypt. | `http://alert-bridge.home.arpa` | Alert bridge (Discord webhook relay) | Plain HTTP | | `https://watch-party.nik4nao.com` | Watch Party on Mac Mini | Let's Encrypt | | `https://nik4nao.com` | Portfolio | Let's Encrypt | +| `https://me.nik4nao.com` | me.nik4nao.com (terminal-style personal site) | Let's Encrypt | | `http://ca.home.arpa` | Internal CA installer | Plain HTTP | `home.arpa` names are defined explicitly in `values/pihole.yaml` and @@ -190,6 +191,7 @@ bash manifests/home-services/registry-secret.sh bash manifests/monitoring/grafana-secret.sh bash manifests/network/ddns-secret.sh bash manifests/portfolio/registry-secret.sh +bash manifests/me-nik4nao-com/registry-secret.sh ``` Sealed Secret regeneration: @@ -223,7 +225,8 @@ The Debian NFS server exports both `/mnt/storage` and `/home/nik/backups` to ## TLS and Trust -`*.home.arpa` certificates use `internal-ca-issuer` from `manifests/cert-manager`. +`*.home.arpa` certificates use `internal-ca-issuer` from `manifests/cert-manager`, +which signs off the self-signed root `internal-ca-cert` (`cert-manager` namespace). The CA installer at `http://ca.home.arpa` serves `ca.crt` and an iOS/macOS mobileconfig profile. The `ca-sync` CronJob updates those files from the `cert-manager/internal-ca-cert` secret when the CA changes. @@ -231,6 +234,49 @@ mobileconfig profile. The `ca-sync` CronJob updates those files from the `*.nik4nao.com` certificates use the Let's Encrypt issuers in `manifests/cert-manager/cluster-issuer-letsencrypt.yaml`. +### Renewing the internal CA + +`internal-ca-cert`'s default `privateKey.rotationPolicy` is `Never`, so a +normal renewal reuses the existing private key — every already-issued leaf +cert and every device that already trusts the CA keep working unchanged. +**Deleting the `internal-ca-cert` secret instead of renewing it in place +generates a brand-new key pair**, which invalidates every leaf cert and every +device's trust simultaneously (this happened 2026-09-02: leaf certs issued +before the deletion failed `openssl verify` against the recreated CA +entirely, and Jellyfin/Kavita/etc. all showed as insecure on client devices). + +To renew correctly: + +```bash +cmctl renew certificate internal-ca-cert -n cert-manager +``` + +This reissues the CA certificate (new serial/validity) while keeping the same +key, so it should not require any downstream action. If the key ever *does* +change (intentional rotation, or the secret was lost and had to be +recreated), do both of the following — the CA rotating does **not** +automatically cascade to either: + +1. **Force-reissue every leaf cert**, since each `Certificate` renews on its + own independent schedule and won't notice the issuing CA changed: + ```bash + for c in $(kubectl get certificate -A -o json | \ + jq -r '.items[] | select(.spec.issuerRef.name=="internal-ca-issuer") | "\(.metadata.namespace)/\(.metadata.name)"'); do + ns="${c%%/*}"; name="${c##*/}" + secret=$(kubectl get certificate "$name" -n "$ns" -o jsonpath='{.spec.secretName}') + kubectl delete secret "$secret" -n "$ns" + done + ``` + Verify afterward that every leaf chains to the current CA: + ```bash + kubectl get secret internal-ca-cert -n cert-manager -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/current-ca.crt + openssl verify -CAfile /tmp/current-ca.crt <(kubectl get secret -n -o jsonpath='{.data.tls\.crt}' | base64 -d) + ``` +2. **Re-trust the CA on every client device** — visit `http://ca.home.arpa`, + reinstall the mobileconfig, and re-enable full trust under Settings → + General → About → Certificate Trust Settings (iOS/macOS). The old trust + profile does not update itself; nothing pushes the new CA to devices. + ## VPN VLAN 50 (PIA gateway) — status and runbook `minisforum` is a PIA WireGuard egress gateway for a dedicated VPN VLAN @@ -329,8 +375,20 @@ plane. - Authentik forward-auth depends on the `Cookie` header in `authRequestHeaders`; removing it causes redirect loops. - CoreDNS custom zone snippets must use the `.server` key suffix. +- Renewing `internal-ca-cert` by deleting its secret (instead of `cmctl + renew`) generates a new key pair, silently breaking every leaf cert and + every device's trust at once — see "Renewing the internal CA" under [TLS + and Trust](#tls-and-trust) for the recovery/prevention procedure. - The Dashy manifest contains an empty ConfigMap shell; use `manifests/core/apply-dashy-config.sh` to load the real config. +- A newly pushed Argo CD `Application` (a new file under `argocd/apps/`) + doesn't create its namespace/resources the instant you push — `app-of-apps` + only picks it up on its next Git poll. Running a namespace-scoped + `registry-secret.sh` (or anything else assuming the namespace already + exists) right after pushing can fail with `namespaces "" not found`; + either wait for the next poll or force one: `kubectl patch application + app-of-apps -n argocd --type merge -p + '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'`. See the scoped READMEs in `ansible/`, `argocd/`, and `manifests/` for workflows specific to those directories. diff --git a/manifests/README.md b/manifests/README.md index ed6e7f6..210fb9b 100644 --- a/manifests/README.md +++ b/manifests/README.md @@ -14,6 +14,7 @@ subdirectories are consumed by Argo CD Applications in `argocd/apps`. | `gitea/` | Gitea storage, backup, public ingress, runner and OIDC/admin secrets | | `home-services/` | HA gateway, AI gateway, Discord bot, service TLS, registry secret | | `homeassistant/` | Home Assistant external service and ingress | +| `me-nik4nao-com/` | me.nik4nao.com deployment, ingress, registry pull secret | | `media/` | Jellyfin, qBittorrent, JDownloader ([sending links](media/jdownloader.md)) | | `monitoring/` | Monitoring PVs, Grafana datasource, Grafana/Auth OIDC secrets | | `multus/` | Multus CNI, VLAN 50 NetworkAttachmentDefinition, canary, browser-vpn-proxy SOCKS5 proxy ([details](multus/README.md)) — manual-sync Application, see `argocd/apps/multus.yaml` |