homelab/manifests/multus/README.md

Multus + VLAN 50 NAD + canary

Implements plan.md Phase 4/5. Reconciled by argocd/apps/multus.yaml, which is deliberately manual-sync only — see that file's header comment. Nothing here applies to the cluster just because it's merged to main.

Files (sync-wave order)

File	Wave	What
00-crd.yaml	-1	NetworkAttachmentDefinition CRD
01-rbac.yaml	0	Multus ServiceAccount/ClusterRole/ClusterRoleBinding
02-daemonset.yaml	0	Multus thick-plugin DaemonSet — nodeSelector restricted to nik-debian only, k3s-specific CNI paths
10-nad-vlan50.yaml	1	VLAN 50 NetworkAttachmentDefinition, macvlan bridge on enp1s0.50, static IPAM
vlan50-egress-guard-script.yaml	(no wave annotation — apply alongside 10)	Shared init-container script consumed by the canary, qBittorrent, JDownloader, and browser-vpn-proxy
20-canary.yaml	2	Temporary canary pod — do not sync without separate explicit approval, see its own header
30-browser-vpn-proxy.yaml	3	.12 — SOCKS5 proxy for browser traffic via PIA, replaces manifests/media/gluetun-proxy.yaml. No auth, no app-level IP allowlist (see its own header for why and what actually restricts access instead) — do not sync without separate explicit approval

Before applying anything here

1. Ansible Phases 2/3 (ansible/roles/pia-gateway, ansible/roles/ vlan50-parent) must already be live and verified — the NAD's master: enp1s0.50 and the whole VLAN 50 return path depend on both.
2. Verify the Multus DaemonSet manifest (02-daemonset.yaml) against the current k8snetworkplumbingwg/multus-cni deployments/multus- daemonset-thick.yml — see that file's own header comment for exactly what's unverified (image tag, daemon flags) versus live-confirmed (the k3s CNI paths).
3. Sync order matters even within this one manual-sync Application: 00 → 01 → 02, confirm the Multus pod is actually Running on nik-debian and /var/lib/rancher/k3s/agent/etc/cni/net.d now has a generated 00-multus.conf, then 10, then — only with separate approval — 20. 30-browser-vpn-proxy.yaml follows the same "separate approval" gate as 20 — no Secret precondition, it needs no credentials at all (see its own header for why).

Canary lifecycle — re-running it after a Multus/CNI change

A canary that already reached Failed will never rerun on its own, and an Argo "Synced" status does not mean it passed. 20-canary.yaml sets restartPolicy: Never and has a fixed name (vlan50-canary, no owning Deployment/Job/controller) — deliberately, so nothing ever recreates it automatically (see "Should this become a Deployment/Job instead?" below for why that's load-bearing, not incidental). Two consequences that have each caused real confusion live (2026-08-24), diagnosed from cluster evidence rather than assumed:

• Once the Pod object reaches phase: Failed, it stays exactly as it is forever — Kubernetes does not retry a restartPolicy: Never Pod, full stop. Fixing the underlying Multus/CNI DaemonSet does not make this Pod try again; it is inert. The only way to get a fresh attempt is to delete and recreate the Pod object itself (new UID, new sandbox, new CNI ADD from scratch).
• Argo reporting multus as Synced only means the live Pod manifest matches what's in Git — for a bare Pod (not a Deployment), that comparison is satisfied by the existing (possibly long-Failed) Pod object as-is. Argo has no reason to touch it, so "Synced" here proves nothing about whether the canary actually ran successfully, or even about when it last ran. Don't read Synced as "passing."

A real example of what this produces if missed: after a DaemonSet fix lands and rolls out, the node briefly passes through an inconsistent state (old Multus daemon torn down, new one's init containers still installing plugins, new daemon not started yet — a few seconds to low tens of seconds depending on image pull/init time). A FailedCreatePodSandBox retry loop that happens to land its next attempt inside that exact window can produce a Pod that looks alive (sandbox created, init container starts, primary eth0 interface up) but whose secondary net1 attach never actually completed — no AddedInterface event for it, no k8s.v1.cni.cncf.io/network-status annotation, the guard script's own net1 wait times out. That Pod is now permanently Failed and permanently stale evidence about the current DaemonSet state — its logs and events describe a multus-daemon instance that may no longer exist. Treating that as proof of a live code defect (rather than recreating the Pod first) risks chasing a bug that was already fixed.

After any change to 02-daemonset.yaml, 10-nad-vlan50.yaml, or the node's CNI binaries, the only trustworthy re-test is:

1. Wait for the DaemonSet rollout to actually finish and be Ready — kubectl -n kube-system rollout status daemonset/kube-multus-ds.
2. Delete the old (Failed) canary Pod — kubectl -n downloads delete pod vlan50-canary (or via the Argo UI).
3. Recreate it fresh — kubectl apply -f manifests/multus/20-canary.yaml (or a selective Argo sync of just that resource).
4. Record the new Pod UID (kubectl -n downloads get pod vlan50-canary -o jsonpath='{.metadata.uid}') and evaluate only logs/events tied to that UID going forward — old aggregated events for the previous UID are not evidence about this attempt.

Should this become a Deployment/Job instead, so it retries itself?

Deliberately not, at least not while ipam.type: static has no allocator behind it (see above) and this canary is hardcoded to 10.10.50.100. A controller that recreates the Pod automatically (a Job, a Deployment with replicas: 1) would retry through exactly the kind of transient window described above without a human confirming the previous attempt's netns was actually torn down cleanly first — two overlapping attempts (a slow-terminating old Pod and a freshly-started new one) both claiming .100 on the same L2 segment is precisely the duplicate-address scenario the guard script's arping -D check now exists to catch, not something to engineer back in via unattended retries. Keeping this a manually recreated, controller-less Pod means recreation only ever happens when someone is actually watching the outcome — that property is worth more here than convenience. If this changes (e.g. a real per-IP allocator is introduced later), revisit.

The managed switch accepts multiple source MAC addresses on nik-debian's port (confirmed in the session that authored this plan — see the coordination handoff in the final report of that session). macvlan bridge is plan.md's stated preference whenever that's true, and is simpler than ipvlan L2 (each attached pod gets its own real MAC, no L2 address-sharing edge cases).

Why ipam.type: static and no shared allocator (Whereabouts etc.)

Only ever a handful of fixed IPs on this network — qBittorrent (.10), JDownloader (.11), browser-vpn-proxy (.12), and a single canary (.100). A dynamic IPAM controller (CRD, webhook, its own failure modes) is unjustified complexity for that; each pod's own k8s.v1.cni.cncf.io/networks annotation just states its IP directly.

Rollback

Deleting the Kubernetes-side resources alone is not safe and can break all new pod scheduling on nik-debian — not just Multus-attached pods. An earlier version of this doc claimed leaving the generated 00-multus.conf in place was harmless once Multus was gone; that's wrong. Multus's "auto" config mode does not replace or rename the original Flannel CNI conf file — it adds 00-multus.conf alongside it, which sorts first and wins by kubelet's own file-ordering convention. kubectl deleteing the DaemonSet removes the running daemon and its pod, but does not remove that generated conf file or the multus shim binary the init container copied onto the host — both are host filesystem side effects kubectl delete has no reach into. If 00-multus.conf is left behind after the daemon it depends on is gone, containerd/kubelet will try to invoke a CNI plugin that's no longer there for every new pod sandbox on that node, Multus-attached or not — the node effectively stops being able to start any new pod until that file is dealt with.

Correct order — host-side cleanup on nik-debian before the Kubernetes side:

# 1. On nik-debian (needs root — not reachable via kubectl):
ssh -p 430 nik@10.10.40.20
sudo ls /var/lib/rancher/k3s/agent/etc/cni/net.d/
#   Confirm both files are present: 00-multus.conf (generated) and the
#   original Flannel conf (something like 10-flannel.conflist — the
#   exact name wasn't captured during discovery; read whatever's there).
sudo cp /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conf \
        /root/00-multus.conf.removed-$(date +%Y%m%d%H%M%S)   # capture for diagnosis first
sudo rm /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conf
#   Optional cleanup, not required for correctness once the conf file
#   above is gone — the multus shim binary itself, if you want it off
#   the host too:
sudo rm -f /var/lib/rancher/k3s/data/cni/multus-shim  # confirm the actual filename first; do not guess-delete

# 2. Prove an ordinary pod can still be newly scheduled on nik-debian
#    before touching anything else (from your workstation):
kubectl run vlan50-rollback-probe --image=busybox:1.36 --restart=Never \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"nik-debian"}}}' \
  -- sleep 60
kubectl wait --for=condition=Ready pod/vlan50-rollback-probe --timeout=60s
kubectl delete pod vlan50-rollback-probe

# 3. Only now, the Kubernetes-side resources, in this order:
kubectl delete -f manifests/multus/30-browser-vpn-proxy.yaml  # if it was ever applied
kubectl delete -f manifests/multus/20-canary.yaml     # if it was ever applied
kubectl delete -f manifests/multus/10-nad-vlan50.yaml  # only after no pod still references it
kubectl delete -f manifests/multus/vlan50-egress-guard-script.yaml
kubectl delete -f manifests/multus/02-daemonset.yaml
kubectl delete -f manifests/multus/01-rbac.yaml
kubectl delete -f manifests/multus/00-crd.yaml         # only after no NAD remains

None of this has been exercised against the live cluster — step 2's probe pod is exactly how to prove it actually worked before considering the rollback complete, not an optional extra.