homelab/manifests/multus/README.md

# Multus + VLAN 50 NAD + canary

Implements plan.md Phase 4/5. Reconciled by `argocd/apps/multus.yaml`,
which is deliberately **manual-sync only** — see that file's header
comment. Nothing here applies to the cluster just because it's merged to
`main`.

## Files (sync-wave order)

| File | Wave | What |
| --- | --- | --- |
| `00-crd.yaml` | -1 | `NetworkAttachmentDefinition` CRD |
| `01-rbac.yaml` | 0 | Multus ServiceAccount/ClusterRole/ClusterRoleBinding |
| `02-daemonset.yaml` | 0 | Multus thick-plugin DaemonSet — **nodeSelector restricted to `nik-debian` only**, k3s-specific CNI paths |
| `10-nad-vlan50.yaml` | 1 | VLAN 50 `NetworkAttachmentDefinition`, macvlan bridge on `enp1s0.50`, static IPAM |
| `vlan50-egress-guard-script.yaml` | (no wave annotation — apply alongside 10) | Shared init-container script consumed by the canary and, later, qBittorrent/JDownloader |
| `20-canary.yaml` | 2 | Temporary canary pod — **do not sync without separate explicit approval**, see its own header |
| `reserved/browser-vpn-proxy.yaml` | n/a | The `.12` third workload, prepared not deployed — see its own header for why it lives in a subdirectory `argocd/apps/multus.yaml` never scans at all |

## Before applying anything here

1. Ansible Phases 2/3 (`ansible/roles/pia-gateway`, `ansible/roles/
   vlan50-parent`) must already be live and verified — the NAD's `master:
   enp1s0.50` and the whole VLAN 50 return path depend on both.
2. **Verify the Multus DaemonSet manifest** (`02-daemonset.yaml`) against
   the current `k8snetworkplumbingwg/multus-cni` `deployments/multus-
   daemonset-thick.yml` — see that file's own header comment for exactly
   what's unverified (image tag, daemon flags) versus live-confirmed (the
   k3s CNI paths).
3. Sync order matters even within this one manual-sync Application: 00 →
   01 → 02, confirm the Multus pod is actually Running on `nik-debian`
   and `/var/lib/rancher/k3s/agent/etc/cni/net.d` now has a generated
   `00-multus.conf`, *then* 10, *then* — only with separate approval — 20.

## Why macvlan bridge, not ipvlan

The managed switch accepts multiple source MAC addresses on nik-debian's
port (confirmed in the session that authored this plan — see the
coordination handoff in the final report of that session). macvlan bridge
is plan.md's stated preference whenever that's true, and is simpler than
ipvlan L2 (each attached pod gets its own real MAC, no L2 address-sharing
edge cases).

## Why `ipam.type: static` and no shared allocator (Whereabouts etc.)

Only ever a handful of fixed IPs on this network — qBittorrent (`.10`),
JDownloader (`.11`), one reserved-not-deployed (`.12`), and a single
canary (`.100`). A dynamic IPAM controller (CRD, webhook, its own failure
modes) is unjustified complexity for that; each pod's own
`k8s.v1.cni.cncf.io/networks` annotation just states its IP directly.

## Rollback

**Deleting the Kubernetes-side resources alone is not safe and can break
all new pod scheduling on nik-debian** — not just Multus-attached pods.
An earlier version of this doc claimed leaving the generated
`00-multus.conf` in place was harmless once Multus was gone; that's
wrong. Multus's "auto" config mode does **not** replace or rename the
original Flannel CNI conf file — it adds `00-multus.conf` alongside it,
which sorts first and wins by kubelet's own file-ordering convention.
`kubectl delete`ing the DaemonSet removes the running daemon and its pod,
but does **not** remove that generated conf file or the multus shim
binary the init container copied onto the host — both are host
filesystem side effects `kubectl delete` has no reach into. If
`00-multus.conf` is left behind after the daemon it depends on is gone,
containerd/kubelet will try to invoke a CNI plugin that's no longer
there for **every** new pod sandbox on that node, Multus-attached or not
— the node effectively stops being able to start any new pod until that
file is dealt with.

Correct order — host-side cleanup on nik-debian *before* the Kubernetes
side:

```bash
# 1. On nik-debian (needs root — not reachable via kubectl):
ssh -p 430 nik@10.10.40.20
sudo ls /var/lib/rancher/k3s/agent/etc/cni/net.d/
#   Confirm both files are present: 00-multus.conf (generated) and the
#   original Flannel conf (something like 10-flannel.conflist — the
#   exact name wasn't captured during discovery; read whatever's there).
sudo cp /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conf \
        /root/00-multus.conf.removed-$(date +%Y%m%d%H%M%S)   # capture for diagnosis first
sudo rm /var/lib/rancher/k3s/agent/etc/cni/net.d/00-multus.conf
#   Optional cleanup, not required for correctness once the conf file
#   above is gone — the multus shim binary itself, if you want it off
#   the host too:
sudo rm -f /var/lib/rancher/k3s/data/cni/multus-shim  # confirm the actual filename first; do not guess-delete

# 2. Prove an ordinary pod can still be newly scheduled on nik-debian
#    before touching anything else (from your workstation):
kubectl run vlan50-rollback-probe --image=busybox:1.36 --restart=Never \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"nik-debian"}}}' \
  -- sleep 60
kubectl wait --for=condition=Ready pod/vlan50-rollback-probe --timeout=60s
kubectl delete pod vlan50-rollback-probe

# 3. Only now, the Kubernetes-side resources, in this order:
kubectl delete -f manifests/multus/20-canary.yaml     # if it was ever applied
kubectl delete -f manifests/multus/10-nad-vlan50.yaml  # only after no pod still references it
kubectl delete -f manifests/multus/vlan50-egress-guard-script.yaml
kubectl delete -f manifests/multus/02-daemonset.yaml
kubectl delete -f manifests/multus/01-rbac.yaml
kubectl delete -f manifests/multus/00-crd.yaml         # only after no NAD remains
```

None of this has been exercised against the live cluster — step 2's
probe pod is exactly how to prove it actually worked before considering
the rollback complete, not an optional extra.