homelab/manifests/multus/README.md

# Multus + VLAN 50 NAD + canary

Implements plan.md Phase 4/5. Reconciled by `argocd/apps/multus.yaml`,
which is deliberately **manual-sync only** — see that file's header
comment. Nothing here applies to the cluster just because it's merged to
`main`.

## Files (sync-wave order)

| File | Wave | What |
| --- | --- | --- |
| `00-crd.yaml` | -1 | `NetworkAttachmentDefinition` CRD |
| `01-rbac.yaml` | 0 | Multus ServiceAccount/ClusterRole/ClusterRoleBinding |
| `02-daemonset.yaml` | 0 | Multus thick-plugin DaemonSet — **nodeSelector restricted to `nik-debian` only**, k3s-specific CNI paths |
| `10-nad-vlan50.yaml` | 1 | VLAN 50 `NetworkAttachmentDefinition`, macvlan bridge on `enp1s0.50`, static IPAM |
| `vlan50-egress-guard-script.yaml` | (no wave annotation — apply alongside 10) | Shared init-container script consumed by the canary and, later, qBittorrent/JDownloader |
| `20-canary.yaml` | 2 | Temporary canary pod — **do not sync without separate explicit approval**, see its own header |
| `reserved/browser-vpn-proxy.yaml` | n/a | The `.12` third workload, prepared not deployed — see its own header for why it lives in a subdirectory `argocd/apps/multus.yaml` never scans at all |

## Before applying anything here

1. Ansible Phases 2/3 (`ansible/roles/pia-gateway`, `ansible/roles/
   vlan50-parent`) must already be live and verified — the NAD's `master:
   enp1s0.50` and the whole VLAN 50 return path depend on both.
2. **Verify the Multus DaemonSet manifest** (`02-daemonset.yaml`) against
   the current `k8snetworkplumbingwg/multus-cni` `deployments/multus-
   daemonset-thick.yml` — see that file's own header comment for exactly
   what's unverified (image tag, daemon flags) versus live-confirmed (the
   k3s CNI paths).
3. Sync order matters even within this one manual-sync Application: 00 →
   01 → 02, confirm the Multus pod is actually Running on `nik-debian`
   and `/var/lib/rancher/k3s/agent/etc/cni/net.d` now has a generated
   `00-multus.conf`, *then* 10, *then* — only with separate approval — 20.

## Canary lifecycle — re-running it after a Multus/CNI change

**A canary that already reached `Failed` will never rerun on its own, and
an Argo "Synced" status does not mean it passed.** `20-canary.yaml` sets
`restartPolicy: Never` and has a fixed name (`vlan50-canary`, no owning
Deployment/Job/controller) — deliberately, so nothing ever recreates it
automatically (see "Should this become a Deployment/Job instead?" below
for why that's load-bearing, not incidental). Two consequences that have
each caused real confusion live (2026-08-24), diagnosed from cluster
evidence rather than assumed:

- Once the Pod object reaches `phase: Failed`, it stays exactly as it is
  forever — Kubernetes does not retry a `restartPolicy: Never` Pod, full
  stop. Fixing the underlying Multus/CNI DaemonSet does **not** make this
  Pod try again; it is inert. The only way to get a fresh attempt is to
  delete and recreate the Pod object itself (new UID, new sandbox, new
  CNI ADD from scratch).
- Argo reporting `multus` as `Synced` only means the *live Pod manifest
  matches what's in Git* — for a bare Pod (not a Deployment), that
  comparison is satisfied by the existing (possibly long-Failed) Pod
  object as-is. Argo has no reason to touch it, so "Synced" here proves
  nothing about whether the canary actually ran successfully, or even
  about when it last ran. Don't read Synced as "passing."

A real example of what this produces if missed: after a DaemonSet fix
lands and rolls out, the *node* briefly passes through an inconsistent
state (old Multus daemon torn down, new one's init containers still
installing plugins, new daemon not started yet — a few seconds to low
tens of seconds depending on image pull/init time). A `FailedCreatePodSandBox`
retry loop that happens to land its next attempt inside that exact
window can produce a Pod that *looks* alive (sandbox created, init
container starts, primary `eth0` interface up) but whose secondary
`net1` attach never actually completed — no `AddedInterface` event for
it, no `k8s.v1.cni.cncf.io/network-status` annotation, the guard script's
own `net1` wait times out. That Pod is now permanently `Failed` and
permanently stale evidence about the *current* DaemonSet state — its logs
and events describe a multus-daemon instance that may no longer exist.
Treating that as proof of a live code defect (rather than recreating the
Pod first) risks chasing a bug that was already fixed.

**After any change to `02-daemonset.yaml`, `10-nad-vlan50.yaml`, or the
node's CNI binaries**, the only trustworthy re-test is:

1. Wait for the DaemonSet rollout to actually finish and be Ready —
   `kubectl -n kube-system rollout status daemonset/kube-multus-ds`.
2. Delete the old (Failed) canary Pod — `kubectl -n downloads delete pod
   vlan50-canary` (or via the Argo UI).
3. Recreate it fresh — `kubectl apply -f manifests/multus/20-canary.yaml`
   (or a selective Argo sync of just that resource).
4. Record the **new** Pod UID (`kubectl -n downloads get pod vlan50-canary
   -o jsonpath='{.metadata.uid}'`) and evaluate only logs/events tied to
   that UID going forward — old aggregated events for the previous UID
   are not evidence about this attempt.

### Should this become a Deployment/Job instead, so it retries itself?

Deliberately not, at least not while `ipam.type: static` has no
allocator behind it (see above) and this canary is hardcoded to
`10.10.50.100`. A controller that recreates the Pod automatically (a
`Job`, a `Deployment` with `replicas: 1`) would retry through exactly the
kind of transient window described above *without* a human confirming
the previous attempt's netns was actually torn down cleanly first — two
overlapping attempts (a slow-terminating old Pod and a freshly-started
new one) both claiming `.100` on the same L2 segment is precisely the
duplicate-address scenario the guard script's `arping -D` check now
exists to catch, not something to engineer back in via unattended
retries. Keeping this a manually recreated, controller-less Pod means
recreation only ever happens when someone is actually watching the
outcome — that property is worth more here than convenience. If this
changes (e.g. a real per-IP allocator is introduced later), revisit.

The managed switch accepts multiple source MAC addresses on nik-debian's
port (confirmed in the session that authored this plan — see the
coordination handoff in the final report of that session). macvlan bridge
is plan.md's stated preference whenever that's true, and is simpler than
ipvlan L2 (each attached pod gets its own real MAC, no L2 address-sharing
edge cases).

## Why `ipam.type: static` and no shared allocator (Whereabouts etc.)

Only ever a handful of fixed IPs on this network — qBittorrent (`.10`),
JDownloader (`.11`), one reserved-not-deployed (`.12`), and a single
canary (`.100`). A dynamic IPAM controller (CRD, webhook, its own failure
modes) is unjustified complexity for that; each pod's own
`k8s.v1.cni.cncf.io/networks` annotation just states its IP directly.

## Rollback

**Deleting the Kubernetes-side resources alone is not safe and can break
all new pod scheduling on nik-debian** — not just Multus-attached pods.
An earlier version of this doc claimed leaving the generated
`00-multus.conf` in place was harmless once Multus was gone; that's
wrong. Multus's "auto" config mode does **not** replace or rename the
original Flannel CNI conf file — it adds `00-multus.conf` alongside it,
which sorts first and wins by kubelet's own file-ordering convention.
`kubectl delete`ing the DaemonSet removes the running daemon and its pod,
but does **not** remove that generated conf file or the multus shim
binary the init container copied onto the host — both are host
filesystem side effects `kubectl delete` has no reach into. If
`00-multus.conf` is left behind after the daemon it depends on is gone,
containerd/kubelet will try to invoke a CNI plugin that's no longer
there for **every** new pod sandbox on that node, Multus-attached or not
— the node effectively stops being able to start any new pod until that
file is dealt with.

Correct order — host-side cleanup on nik-debian *before* the Kubernetes
side:

```bash
# 1. On nik-debian (needs root — not reachable via kubectl):
ssh -p 430 nik@10.10.40.20
sudo ls /var/lib/rancher/k3s/agent/etc/cni/net.d/
#   Confirm both files are present: 00-multus.conf (generated) and the
#   original Flannel conf (something like 10-flannel.conflist — the
#   exact name wasn't captured during discovery; read whatever's there).
sudo cp /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conf \
        /root/00-multus.conf.removed-$(date +%Y%m%d%H%M%S)   # capture for diagnosis first
sudo rm /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conf
#   Optional cleanup, not required for correctness once the conf file
#   above is gone — the multus shim binary itself, if you want it off
#   the host too:
sudo rm -f /var/lib/rancher/k3s/data/cni/multus-shim  # confirm the actual filename first; do not guess-delete

# 2. Prove an ordinary pod can still be newly scheduled on nik-debian
#    before touching anything else (from your workstation):
kubectl run vlan50-rollback-probe --image=busybox:1.36 --restart=Never \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"nik-debian"}}}' \
  -- sleep 60
kubectl wait --for=condition=Ready pod/vlan50-rollback-probe --timeout=60s
kubectl delete pod vlan50-rollback-probe

# 3. Only now, the Kubernetes-side resources, in this order:
kubectl delete -f manifests/multus/20-canary.yaml     # if it was ever applied
kubectl delete -f manifests/multus/10-nad-vlan50.yaml  # only after no pod still references it
kubectl delete -f manifests/multus/vlan50-egress-guard-script.yaml
kubectl delete -f manifests/multus/02-daemonset.yaml
kubectl delete -f manifests/multus/01-rbac.yaml
kubectl delete -f manifests/multus/00-crd.yaml         # only after no NAD remains
```

None of this has been exercised against the live cluster — step 2's
probe pod is exactly how to prove it actually worked before considering
the rollback complete, not an optional extra.