fix: install missing CNI reference plugins (macvlan) for Multus on k3s

k3s deliberately ships only the plugins its own flannel networking
needs (bandwidth, bridge, firewall, flannel, host-local, loopback,
portmap — confirmed against k3s's own build script; macvlan's absence
is an intentional upstream scope decision, k3s-io/k3s#9224). Multus's
macvlan delegate for the vlan50 NAD therefore had nothing to exec even
after the binDir/mount fixes.

Add a cni-plugins init container mirroring Rancher's own official
rke2-multus Helm chart (the fix k3s's own docs recommend for this
exact gap): rancher/hardened-cni-plugins, digest-pinned, writing into
our existing cnibin mount instead of the image's hardcoded
/opt/cni/bin, with SKIP_CNI_BINARIES=flannel so it doesn't overwrite
k3s's already-working flannel plugin.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

manifests/multus/02-daemonset.yaml

@@ -93,15 +93,38 @@
 #      (k3s-cni-data, at the same path in-container as on the host) lets
 #      the symlink target resolve inside the container too, without
 #      changing binDir or the resolved host path at all. cnibin itself is
-#      kept, narrowed to only the install-multus-binary init container's
+#      kept, narrowed to the two init containers' own plain file writes
-#      use (a plain file write into that directory, which never follows
+#      (which never follow the symlinks and so never needed the wider
-#      the symlinks and so never needed the wider mount).
+#      mount) — see point 7 for the second of those two.
+#   7. A cni-plugins init container, absent from upstream's own
+#      daemonset-thick.yml entirely, installs the standard CNI reference
+#      plugins (bridge, macvlan, ipvlan, vlan, host-local, loopback,
+#      portmap, bandwidth, firewall, dhcp, static, tuning, sbr, vrf,
+#      host-device, ptp, dummy) that k3s does not bundle. Root-caused live
+#      (2026-08-24), same incident as points 5/6 continued: with binDir
+#      and the mount both fixed, sandbox creation failed a third time,
+#      now with `failed to find plugin "macvlan" in path
+#      [/var/lib/rancher/k3s/data/cni]` — a genuinely absent binary this
+#      time, not another path/mount problem. Confirmed against k3s's own
+#      real build script (k3s-io/k3s's scripts/package-cli at this exact
+#      tag, read in full): its cni_binaries array is hardcoded to exactly
+#      bandwidth, bridge, firewall, flannel, host-local, loopback,
+#      portmap — macvlan is deliberately excluded by k3s's own project
+#      scope decision (k3s-io/k3s issue #9224 confirms this is
+#      intentional, not an oversight). k3s's own documentation
+#      (https://docs.k3s.io/networking/multus-ipams) names the fix for
+#      exactly this gap: install the missing plugins via the same
+#      mechanism Rancher's own rke2-multus Helm chart uses — see this
+#      container's own inline comment for exactly which real chart/image
+#      that is and how it was verified (image digest confirmed live
+#      against the real registry, not assumed from the tag).
 #
 # Do not hand-edit the daemon-config.json keys, the container
 # command/args, or the mount set below without re-diffing against the
 # URL above — the parts still matching upstream verbatim were not
-# invented, they're copied; the binDir addition and the cnibin mountPath
+# invented, they're copied; the binDir addition, the cnibin/k3s-cni-data
-# change are the two deliberate, root-caused departures.
+# mount split, and the added cni-plugins init container are the three
+# deliberate, root-caused departures.
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -227,6 +250,46 @@ spec:
                 fieldRef:
                   fieldPath: spec.nodeName
       initContainers:
+        - name: cni-plugins
+          # NOT part of upstream k8snetworkplumbingwg/multus-cni's own
+          # daemonset-thick.yml (that manifest assumes the standard CNI
+          # reference plugins already exist on the host, which is true on
+          # most distributions but NOT k3s — see the header comment's
+          # point 7). This container and image are copied from Rancher's
+          # own official rke2-multus Helm chart (the one k3s's own docs at
+          # https://docs.k3s.io/networking/multus-ipams recommend for
+          # exactly this situation), fetched and read in full 2026-08-24
+          # from the real packaged chart at https://rke2-charts.rancher.io
+          # /assets/rke2-multus/rke2-multus-v4.3.017.tgz — not fabricated.
+          image: "docker.io/rancher/hardened-cni-plugins@sha256:935b2e954695ecc153d4a671f51717dc93b613e08b5f4230ba0eb8dbf273495a" # v1.9.1-build20260819
+          securityContext:
+            privileged: true
+          env:
+            # Skip flannel specifically: k3s's own binDir already has a
+            # working flannel plugin (from k3s's own build, version-locked
+            # to this k3s release) — see header comment's points 5 and 6.
+            # Letting this image overwrite it with its own separately
+            # versioned flannel would be a needless, unverified swap of a
+            # plugin that already works, for zero benefit (this image is
+            # only needed here for the plugins k3s doesn't ship at all,
+            # e.g. macvlan). Matches Rancher's own chart's default
+            # cniplugins.skipcnis value for this exact reason.
+            - name: SKIP_CNI_BINARIES
+              value: flannel
+          volumeMounts:
+            # install-cnis.sh (this image's entrypoint, read in full) has
+            # a hardcoded destination of /host/opt/cni/bin — not
+            # configurable via any env var or arg. Mounting cnibin (this
+            # role's real k3s CNI bin dir, not the standard /opt/cni/bin)
+            # at that container path redirects the writes to the right
+            # place without needing a different image or a patched
+            # script. It copies plain regular files directly into that
+            # directory (not through the flannel-style symlink-to-hash-dir
+            # indirection those specific k3s-provided binaries use), so
+            # this narrower cnibin mount is sufficient here — no need for
+            # the wider k3s-cni-data mount the main container uses.
+            - name: cnibin
+              mountPath: /host/opt/cni/bin
         - name: install-multus-binary
           # Same image/digest as the main container — see header comment.
           image: "ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896d56ae51c9" # v4.3.0-thick