fix: correct Multus binDir/cnibin path for k3s delegate plugin exec
Some checks failed
validate / lint (push) Failing after 0s

Root cause: Multus's ChrootExec resolves a delegate plugin's absolute
path from the daemon container's own filesystem view, then execs that
exact path string after chrooting into the real host root. binDir
defaulted to /opt/cni/bin, which is valid inside the container (bind-
mounted to k3s's real bin dir) but doesn't exist on the actual host, so
every pod attaching to the VLAN 50 NAD failed FailedCreatePodSandBox
trying to delegate to flannel. Confirmed against multus-cni's source at
the exact pinned v4.3.0 tag, not assumed. Sets binDir explicitly to
k3s's real path and matches the cnibin mount path to it on both sides of
the chroot boundary (main container and the install_multus init
container).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Nik Afiq 2026-08-24 20:06:04 +09:00
parent e980fcb755
commit d410c05e6d

View File

@ -44,10 +44,38 @@
# sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896 # sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896
# d56ae51c9, i.e. v4.3.0 is genuinely the current stable release, # d56ae51c9, i.e. v4.3.0 is genuinely the current stable release,
# not just a same-named coincidence. # not just a same-named coincidence.
# 5. daemon-config.json has one key added beyond upstream's own
# example: "binDir". Root-caused live (2026-08-24) after Multus
# itself came up correctly but every pod on nik-debian using the
# NAD failed FailedCreatePodSandBox with `plugin type="flannel"
# failed (add): failed to find plugin "flannel" in path
# [/opt/cni/bin]`. Read multus-cni's actual source (pinned to the
# v4.3.0 tag, not master) to confirm the mechanism rather than
# guess: pkg/server/exec_chroot.go's ChrootExec resolves a delegate
# plugin's absolute path using the DAEMON CONTAINER's own
# filesystem view (FindInPath), then execs that exact path string
# *after* chrooting the child process into "hostroot" (the real
# host root, bind-mounted). binDir defaults to "/opt/cni/bin"
# (pkg/types/conf.go's defaultBinDir) when unset — a path that is
# valid inside this container (bind-mounted to k3s's real bin dir
# below) but does not exist on the actual host filesystem at all,
# so the post-chroot exec fails even though the daemon "found" the
# plugin from its own point of view first. Confirmed the real host
# path live via SSH (flannel/bridge/host-local genuinely present at
# /var/lib/rancher/k3s/data/cni), confirmed binDir's exact JSON tag
# against pkg/server/config/generator.go's MultusConf struct at the
# v4.3.0 tag specifically (not assumed from master, in case the
# schema had drifted between versions — it hadn't, but that was
# checked, not assumed). Setting binDir alone isn't sufficient by
# itself: it also has to resolve correctly in the *pre-chroot*
# container view (FindInPath's side), which is exactly why the
# cnibin mountPath below changed too — see that comment.
# #
# Do not hand-edit the daemon-config.json keys, the container # Do not hand-edit the daemon-config.json keys, the container
# command/args, or the mount set below without re-diffing against the # command/args, or the mount set below without re-diffing against the
# URL above — none of that was invented this time, it's copied. # URL above — the parts still matching upstream verbatim were not
# invented, they're copied; the binDir addition and the cnibin mountPath
# change are the two deliberate, root-caused departures.
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
@ -68,7 +96,8 @@ data:
"cniConfigDir": "/host/etc/cni/net.d", "cniConfigDir": "/host/etc/cni/net.d",
"multusAutoconfigDir": "/host/etc/cni/net.d", "multusAutoconfigDir": "/host/etc/cni/net.d",
"multusConfigFile": "auto", "multusConfigFile": "auto",
"socketDir": "/host/run/multus/" "socketDir": "/host/run/multus/",
"binDir": "/var/lib/rancher/k3s/data/cni"
} }
--- ---
apiVersion: apps/v1 apiVersion: apps/v1
@ -129,8 +158,17 @@ spec:
# multus-daemon expects that cnibin path must be identical between pod and container host. # multus-daemon expects that cnibin path must be identical between pod and container host.
# e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon, # e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon,
# not to any other directory, like '/opt/bin' or '/usr/bin'. # not to any other directory, like '/opt/bin' or '/usr/bin'.
#
# On THIS host that means it must be
# /var/lib/rancher/k3s/data/cni, not the standard /opt/cni/bin
# this comment's own example uses — k3s's real bin dir is not
# the standard one (see the volume definition below), and this
# mountPath has to be identical to it, not to the standard
# path, for delegate plugin exec-after-chroot to find anything
# here at all. See the header comment's point 5 for the full
# mechanism (this was live-broken before that value matched).
- name: cnibin - name: cnibin
mountPath: /opt/cni/bin mountPath: /var/lib/rancher/k3s/data/cni
- name: host-run - name: host-run
mountPath: /host/run mountPath: /host/run
- name: host-var-lib-cni-multus - name: host-var-lib-cni-multus
@ -163,7 +201,7 @@ spec:
command: command:
- "/usr/src/multus-cni/bin/install_multus" - "/usr/src/multus-cni/bin/install_multus"
- "-d" - "-d"
- "/host/opt/cni/bin" - "/host/var/lib/rancher/k3s/data/cni"
- "-t" - "-t"
- "thick" - "thick"
resources: resources:
@ -175,7 +213,7 @@ spec:
terminationMessagePolicy: FallbackToLogsOnError terminationMessagePolicy: FallbackToLogsOnError
volumeMounts: volumeMounts:
- name: cnibin - name: cnibin
mountPath: /host/opt/cni/bin mountPath: /host/var/lib/rancher/k3s/data/cni
mountPropagation: Bidirectional mountPropagation: Bidirectional
terminationGracePeriodSeconds: 30 terminationGracePeriodSeconds: 30
volumes: volumes: