diff --git a/manifests/multus/02-daemonset.yaml b/manifests/multus/02-daemonset.yaml index 77c6d34..bd17e2d 100644 --- a/manifests/multus/02-daemonset.yaml +++ b/manifests/multus/02-daemonset.yaml @@ -70,6 +70,32 @@ # itself: it also has to resolve correctly in the *pre-chroot* # container view (FindInPath's side), which is exactly why the # cnibin mountPath below changed too — see that comment. +# 6. The main kube-multus container mounts a second hostPath, +# k3s-cni-data, at /var/lib/rancher/k3s/data (cnibin's parent), in +# addition to cnibin itself. Root-caused live (2026-08-24), same +# incident as point 5 continued: after binDir was corrected to +# /var/lib/rancher/k3s/data/cni, pod sandbox creation still failed +# with the identical error, `failed to find plugin "flannel" in +# path [/var/lib/rancher/k3s/data/cni]` — this time from inside the +# corrected path itself. Read containernetworking/cni's actual +# source (pkg/invoke/find.go's FindInPath) to confirm the mechanism: +# it calls os.Stat on each candidate path, which follows symlinks, +# and requires the resolved target to be a regular file. A live SSH +# check on nik-debian (already done for point 5) had shown +# /var/lib/rancher/k3s/data/cni is itself a directory of SYMLINKS — +# e.g. .../data/cni/flannel -> .../data//bin/cni — not +# the real binaries. The narrow cnibin mount only bind-mounts the +# cni subdirectory, so inside the container those symlinks point at +# a hash-named sibling directory that was never mounted at all, and +# os.Stat fails on the daemon container's own (pre-chroot) side — +# exactly the FindInPath call point 5 identified, just one symlink +# hop further than binDir alone could fix. Mounting the parent +# (k3s-cni-data, at the same path in-container as on the host) lets +# the symlink target resolve inside the container too, without +# changing binDir or the resolved host path at all. cnibin itself is +# kept, narrowed to only the install-multus-binary init container's +# use (a plain file write into that directory, which never follows +# the symlinks and so never needed the wider mount). # # Do not hand-edit the daemon-config.json keys, the container # command/args, or the mount set below without re-diffing against the @@ -159,16 +185,22 @@ spec: # e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon, # not to any other directory, like '/opt/bin' or '/usr/bin'. # - # On THIS host that means it must be - # /var/lib/rancher/k3s/data/cni, not the standard /opt/cni/bin - # this comment's own example uses — k3s's real bin dir is not - # the standard one (see the volume definition below), and this - # mountPath has to be identical to it, not to the standard - # path, for delegate plugin exec-after-chroot to find anything - # here at all. See the header comment's point 5 for the full - # mechanism (this was live-broken before that value matched). - - name: cnibin - mountPath: /var/lib/rancher/k3s/data/cni + # On THIS host that mount is k3s-cni-data (below), not cnibin + # — see the header comment's point 6. k3s's actual plugin + # binaries live in a content-hash-named sibling directory + # (/var/lib/rancher/k3s/data//bin/cni) and + # /var/lib/rancher/k3s/data/cni/flannel etc. are only + # *symlinks* to that; mounting just the cni/ subdirectory + # (cnibin, still used by the init container below for its own + # narrower purpose) left those symlinks dangling from this + # container's own point of view, which is exactly where + # FindInPath (invoke.FindInPath, os.Stat-based, follows + # symlinks) checks each candidate before the chroot-exec ever + # happens. Mounting the whole parent directory at an identical + # path is what makes both the symlink and its target resolve + # the same way here as they do on the real host. + - name: k3s-cni-data + mountPath: /var/lib/rancher/k3s/data - name: host-run mountPath: /host/run - name: host-var-lib-cni-multus @@ -223,8 +255,19 @@ spec: path: /var/lib/rancher/k3s/agent/etc/cni/net.d - name: cnibin hostPath: - # k3s CNI bin dir — NOT the standard /opt/cni/bin. + # k3s CNI bin dir — NOT the standard /opt/cni/bin. Used only + # by the install-multus-binary init container (a plain file + # write, doesn't need the wider parent — see k3s-cni-data + # below for why the main container needs more than this). path: /var/lib/rancher/k3s/data/cni + - name: k3s-cni-data + hostPath: + # The parent of cnibin above, mounted identically in the main + # container so that k3s's plugin symlinks (e.g. + # .../data/cni/flannel -> .../data//bin/cni) resolve + # correctly from inside the container too, not just on the + # real host — see the volumeMounts comment above for why. + path: /var/lib/rancher/k3s/data - name: hostroot hostPath: path: /