From c7cdcc967dd793a471fc95cb2aaa9da1ee6c2f51 Mon Sep 17 00:00:00 2001 From: Nik Afiq Date: Mon, 24 Aug 2026 21:25:12 +0900 Subject: [PATCH] fix: mount k3s CNI data parent dir into multus for symlink resolution binDir alone (previous commit) wasn't sufficient: k3s's plugin bin dir is a directory of symlinks into a content-hash-named sibling directory, and the narrow cnibin mount didn't cover that sibling. FindInPath's os.Stat follows symlinks, so it failed inside the container even though the same path resolves fine on the real host. Mount the parent (k3s-cni-data) into the main container instead so the symlinks resolve; the init container keeps the narrower cnibin mount since it only writes files and never follows them. Co-Authored-By: Claude Sonnet 5 --- manifests/multus/02-daemonset.yaml | 65 +++++++++++++++++++++++++----- 1 file changed, 54 insertions(+), 11 deletions(-) diff --git a/manifests/multus/02-daemonset.yaml b/manifests/multus/02-daemonset.yaml index 77c6d34..bd17e2d 100644 --- a/manifests/multus/02-daemonset.yaml +++ b/manifests/multus/02-daemonset.yaml @@ -70,6 +70,32 @@ # itself: it also has to resolve correctly in the *pre-chroot* # container view (FindInPath's side), which is exactly why the # cnibin mountPath below changed too — see that comment. +# 6. The main kube-multus container mounts a second hostPath, +# k3s-cni-data, at /var/lib/rancher/k3s/data (cnibin's parent), in +# addition to cnibin itself. Root-caused live (2026-08-24), same +# incident as point 5 continued: after binDir was corrected to +# /var/lib/rancher/k3s/data/cni, pod sandbox creation still failed +# with the identical error, `failed to find plugin "flannel" in +# path [/var/lib/rancher/k3s/data/cni]` — this time from inside the +# corrected path itself. Read containernetworking/cni's actual +# source (pkg/invoke/find.go's FindInPath) to confirm the mechanism: +# it calls os.Stat on each candidate path, which follows symlinks, +# and requires the resolved target to be a regular file. A live SSH +# check on nik-debian (already done for point 5) had shown +# /var/lib/rancher/k3s/data/cni is itself a directory of SYMLINKS — +# e.g. .../data/cni/flannel -> .../data//bin/cni — not +# the real binaries. The narrow cnibin mount only bind-mounts the +# cni subdirectory, so inside the container those symlinks point at +# a hash-named sibling directory that was never mounted at all, and +# os.Stat fails on the daemon container's own (pre-chroot) side — +# exactly the FindInPath call point 5 identified, just one symlink +# hop further than binDir alone could fix. Mounting the parent +# (k3s-cni-data, at the same path in-container as on the host) lets +# the symlink target resolve inside the container too, without +# changing binDir or the resolved host path at all. cnibin itself is +# kept, narrowed to only the install-multus-binary init container's +# use (a plain file write into that directory, which never follows +# the symlinks and so never needed the wider mount). # # Do not hand-edit the daemon-config.json keys, the container # command/args, or the mount set below without re-diffing against the @@ -159,16 +185,22 @@ spec: # e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon, # not to any other directory, like '/opt/bin' or '/usr/bin'. # - # On THIS host that means it must be - # /var/lib/rancher/k3s/data/cni, not the standard /opt/cni/bin - # this comment's own example uses — k3s's real bin dir is not - # the standard one (see the volume definition below), and this - # mountPath has to be identical to it, not to the standard - # path, for delegate plugin exec-after-chroot to find anything - # here at all. See the header comment's point 5 for the full - # mechanism (this was live-broken before that value matched). - - name: cnibin - mountPath: /var/lib/rancher/k3s/data/cni + # On THIS host that mount is k3s-cni-data (below), not cnibin + # — see the header comment's point 6. k3s's actual plugin + # binaries live in a content-hash-named sibling directory + # (/var/lib/rancher/k3s/data//bin/cni) and + # /var/lib/rancher/k3s/data/cni/flannel etc. are only + # *symlinks* to that; mounting just the cni/ subdirectory + # (cnibin, still used by the init container below for its own + # narrower purpose) left those symlinks dangling from this + # container's own point of view, which is exactly where + # FindInPath (invoke.FindInPath, os.Stat-based, follows + # symlinks) checks each candidate before the chroot-exec ever + # happens. Mounting the whole parent directory at an identical + # path is what makes both the symlink and its target resolve + # the same way here as they do on the real host. + - name: k3s-cni-data + mountPath: /var/lib/rancher/k3s/data - name: host-run mountPath: /host/run - name: host-var-lib-cni-multus @@ -223,8 +255,19 @@ spec: path: /var/lib/rancher/k3s/agent/etc/cni/net.d - name: cnibin hostPath: - # k3s CNI bin dir — NOT the standard /opt/cni/bin. + # k3s CNI bin dir — NOT the standard /opt/cni/bin. Used only + # by the install-multus-binary init container (a plain file + # write, doesn't need the wider parent — see k3s-cni-data + # below for why the main container needs more than this). path: /var/lib/rancher/k3s/data/cni + - name: k3s-cni-data + hostPath: + # The parent of cnibin above, mounted identically in the main + # container so that k3s's plugin symlinks (e.g. + # .../data/cni/flannel -> .../data//bin/cni) resolve + # correctly from inside the container too, not just on the + # real host — see the volumeMounts comment above for why. + path: /var/lib/rancher/k3s/data - name: hostroot hostPath: path: /