From f1bdf11f1ef30ee6e049c85211d5d3bf70498b68 Mon Sep 17 00:00:00 2001 From: Nik Afiq Date: Mon, 24 Aug 2026 21:39:23 +0900 Subject: [PATCH] fix: install missing CNI reference plugins (macvlan) for Multus on k3s MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit k3s deliberately ships only the plugins its own flannel networking needs (bandwidth, bridge, firewall, flannel, host-local, loopback, portmap — confirmed against k3s's own build script; macvlan's absence is an intentional upstream scope decision, k3s-io/k3s#9224). Multus's macvlan delegate for the vlan50 NAD therefore had nothing to exec even after the binDir/mount fixes. Add a cni-plugins init container mirroring Rancher's own official rke2-multus Helm chart (the fix k3s's own docs recommend for this exact gap): rancher/hardened-cni-plugins, digest-pinned, writing into our existing cnibin mount instead of the image's hardcoded /opt/cni/bin, with SKIP_CNI_BINARIES=flannel so it doesn't overwrite k3s's already-working flannel plugin. Co-Authored-By: Claude Sonnet 5 --- manifests/multus/02-daemonset.yaml | 73 ++++++++++++++++++++++++++++-- 1 file changed, 68 insertions(+), 5 deletions(-) diff --git a/manifests/multus/02-daemonset.yaml b/manifests/multus/02-daemonset.yaml index bd17e2d..3f0f80b 100644 --- a/manifests/multus/02-daemonset.yaml +++ b/manifests/multus/02-daemonset.yaml @@ -93,15 +93,38 @@ # (k3s-cni-data, at the same path in-container as on the host) lets # the symlink target resolve inside the container too, without # changing binDir or the resolved host path at all. cnibin itself is -# kept, narrowed to only the install-multus-binary init container's -# use (a plain file write into that directory, which never follows -# the symlinks and so never needed the wider mount). +# kept, narrowed to the two init containers' own plain file writes +# (which never follow the symlinks and so never needed the wider +# mount) — see point 7 for the second of those two. +# 7. A cni-plugins init container, absent from upstream's own +# daemonset-thick.yml entirely, installs the standard CNI reference +# plugins (bridge, macvlan, ipvlan, vlan, host-local, loopback, +# portmap, bandwidth, firewall, dhcp, static, tuning, sbr, vrf, +# host-device, ptp, dummy) that k3s does not bundle. Root-caused live +# (2026-08-24), same incident as points 5/6 continued: with binDir +# and the mount both fixed, sandbox creation failed a third time, +# now with `failed to find plugin "macvlan" in path +# [/var/lib/rancher/k3s/data/cni]` — a genuinely absent binary this +# time, not another path/mount problem. Confirmed against k3s's own +# real build script (k3s-io/k3s's scripts/package-cli at this exact +# tag, read in full): its cni_binaries array is hardcoded to exactly +# bandwidth, bridge, firewall, flannel, host-local, loopback, +# portmap — macvlan is deliberately excluded by k3s's own project +# scope decision (k3s-io/k3s issue #9224 confirms this is +# intentional, not an oversight). k3s's own documentation +# (https://docs.k3s.io/networking/multus-ipams) names the fix for +# exactly this gap: install the missing plugins via the same +# mechanism Rancher's own rke2-multus Helm chart uses — see this +# container's own inline comment for exactly which real chart/image +# that is and how it was verified (image digest confirmed live +# against the real registry, not assumed from the tag). # # Do not hand-edit the daemon-config.json keys, the container # command/args, or the mount set below without re-diffing against the # URL above — the parts still matching upstream verbatim were not -# invented, they're copied; the binDir addition and the cnibin mountPath -# change are the two deliberate, root-caused departures. +# invented, they're copied; the binDir addition, the cnibin/k3s-cni-data +# mount split, and the added cni-plugins init container are the three +# deliberate, root-caused departures. apiVersion: v1 kind: ConfigMap metadata: @@ -227,6 +250,46 @@ spec: fieldRef: fieldPath: spec.nodeName initContainers: + - name: cni-plugins + # NOT part of upstream k8snetworkplumbingwg/multus-cni's own + # daemonset-thick.yml (that manifest assumes the standard CNI + # reference plugins already exist on the host, which is true on + # most distributions but NOT k3s — see the header comment's + # point 7). This container and image are copied from Rancher's + # own official rke2-multus Helm chart (the one k3s's own docs at + # https://docs.k3s.io/networking/multus-ipams recommend for + # exactly this situation), fetched and read in full 2026-08-24 + # from the real packaged chart at https://rke2-charts.rancher.io + # /assets/rke2-multus/rke2-multus-v4.3.017.tgz — not fabricated. + image: "docker.io/rancher/hardened-cni-plugins@sha256:935b2e954695ecc153d4a671f51717dc93b613e08b5f4230ba0eb8dbf273495a" # v1.9.1-build20260819 + securityContext: + privileged: true + env: + # Skip flannel specifically: k3s's own binDir already has a + # working flannel plugin (from k3s's own build, version-locked + # to this k3s release) — see header comment's points 5 and 6. + # Letting this image overwrite it with its own separately + # versioned flannel would be a needless, unverified swap of a + # plugin that already works, for zero benefit (this image is + # only needed here for the plugins k3s doesn't ship at all, + # e.g. macvlan). Matches Rancher's own chart's default + # cniplugins.skipcnis value for this exact reason. + - name: SKIP_CNI_BINARIES + value: flannel + volumeMounts: + # install-cnis.sh (this image's entrypoint, read in full) has + # a hardcoded destination of /host/opt/cni/bin — not + # configurable via any env var or arg. Mounting cnibin (this + # role's real k3s CNI bin dir, not the standard /opt/cni/bin) + # at that container path redirects the writes to the right + # place without needing a different image or a patched + # script. It copies plain regular files directly into that + # directory (not through the flannel-style symlink-to-hash-dir + # indirection those specific k3s-provided binaries use), so + # this narrower cnibin mount is sufficient here — no need for + # the wider k3s-cni-data mount the main container uses. + - name: cnibin + mountPath: /host/opt/cni/bin - name: install-multus-binary # Same image/digest as the main container — see header comment. image: "ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896d56ae51c9" # v4.3.0-thick