homelab/manifests/multus

Multus + VLAN 50 NAD + canary

Implements plan.md Phase 4/5. Reconciled by argocd/apps/multus.yaml, which is deliberately manual-sync only — see that file's header comment. Nothing here applies to the cluster just because it's merged to main.

Files (sync-wave order)

File	Wave	What
00-crd.yaml	-1	NetworkAttachmentDefinition CRD
01-rbac.yaml	0	Multus ServiceAccount/ClusterRole/ClusterRoleBinding
02-daemonset.yaml	0	Multus thick-plugin DaemonSet — nodeSelector restricted to nik-debian only, k3s-specific CNI paths
10-nad-vlan50.yaml	1	VLAN 50 NetworkAttachmentDefinition, macvlan bridge on enp1s0.50, static IPAM
vlan50-egress-guard-script.yaml	(no wave annotation — apply alongside 10)	Shared init-container script consumed by the canary and, later, qBittorrent/JDownloader
20-canary.yaml	2	Temporary canary pod — do not sync without separate explicit approval, see its own header
reserved/browser-vpn-proxy.yaml	n/a	The .12 third workload, prepared not deployed — see its own header for why it lives in a subdirectory argocd/apps/multus.yaml never scans at all

Before applying anything here

1. Ansible Phases 2/3 (ansible/roles/pia-gateway, ansible/roles/ vlan50-parent) must already be live and verified — the NAD's master: enp1s0.50 and the whole VLAN 50 return path depend on both.
2. Verify the Multus DaemonSet manifest (02-daemonset.yaml) against the current k8snetworkplumbingwg/multus-cni deployments/multus- daemonset-thick.yml — see that file's own header comment for exactly what's unverified (image tag, daemon flags) versus live-confirmed (the k3s CNI paths).
3. Sync order matters even within this one manual-sync Application: 00 → 01 → 02, confirm the Multus pod is actually Running on nik-debian and /var/lib/rancher/k3s/agent/etc/cni/net.d now has a generated 00-multus.conf, then 10, then — only with separate approval — 20.

Why macvlan bridge, not ipvlan

The managed switch accepts multiple source MAC addresses on nik-debian's port (confirmed in the session that authored this plan — see the coordination handoff in the final report of that session). macvlan bridge is plan.md's stated preference whenever that's true, and is simpler than ipvlan L2 (each attached pod gets its own real MAC, no L2 address-sharing edge cases).

Why ipam.type: static and no shared allocator (Whereabouts etc.)

Only ever a handful of fixed IPs on this network — qBittorrent (.10), JDownloader (.11), one reserved-not-deployed (.12), and a single canary (.100). A dynamic IPAM controller (CRD, webhook, its own failure modes) is unjustified complexity for that; each pod's own k8s.v1.cni.cncf.io/networks annotation just states its IP directly.

Rollback

Deleting the Kubernetes-side resources alone is not safe and can break all new pod scheduling on nik-debian — not just Multus-attached pods. An earlier version of this doc claimed leaving the generated 00-multus.conf in place was harmless once Multus was gone; that's wrong. Multus's "auto" config mode does not replace or rename the original Flannel CNI conf file — it adds 00-multus.conf alongside it, which sorts first and wins by kubelet's own file-ordering convention. kubectl deleteing the DaemonSet removes the running daemon and its pod, but does not remove that generated conf file or the multus shim binary the init container copied onto the host — both are host filesystem side effects kubectl delete has no reach into. If 00-multus.conf is left behind after the daemon it depends on is gone, containerd/kubelet will try to invoke a CNI plugin that's no longer there for every new pod sandbox on that node, Multus-attached or not — the node effectively stops being able to start any new pod until that file is dealt with.

Correct order — host-side cleanup on nik-debian before the Kubernetes side:

# 1. On nik-debian (needs root — not reachable via kubectl):
ssh -p 430 nik@10.10.40.20
sudo ls /var/lib/rancher/k3s/agent/etc/cni/net.d/
#   Confirm both files are present: 00-multus.conf (generated) and the
#   original Flannel conf (something like 10-flannel.conflist — the
#   exact name wasn't captured during discovery; read whatever's there).
sudo cp /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conf \
        /root/00-multus.conf.removed-$(date +%Y%m%d%H%M%S)   # capture for diagnosis first
sudo rm /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conf
#   Optional cleanup, not required for correctness once the conf file
#   above is gone — the multus shim binary itself, if you want it off
#   the host too:
sudo rm -f /var/lib/rancher/k3s/data/cni/multus-shim  # confirm the actual filename first; do not guess-delete

# 2. Prove an ordinary pod can still be newly scheduled on nik-debian
#    before touching anything else (from your workstation):
kubectl run vlan50-rollback-probe --image=busybox:1.36 --restart=Never \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"nik-debian"}}}' \
  -- sleep 60
kubectl wait --for=condition=Ready pod/vlan50-rollback-probe --timeout=60s
kubectl delete pod vlan50-rollback-probe

# 3. Only now, the Kubernetes-side resources, in this order:
kubectl delete -f manifests/multus/20-canary.yaml     # if it was ever applied
kubectl delete -f manifests/multus/10-nad-vlan50.yaml  # only after no pod still references it
kubectl delete -f manifests/multus/vlan50-egress-guard-script.yaml
kubectl delete -f manifests/multus/02-daemonset.yaml
kubectl delete -f manifests/multus/01-rbac.yaml
kubectl delete -f manifests/multus/00-crd.yaml         # only after no NAD remains

None of this has been exercised against the live cluster — step 2's probe pod is exactly how to prove it actually worked before considering the rollback complete, not an optional extra.