fix: mount k3s CNI data parent dir into multus for symlink resolution
Some checks failed
validate / lint (push) Failing after 0s

binDir alone (previous commit) wasn't sufficient: k3s's plugin bin dir
is a directory of symlinks into a content-hash-named sibling directory,
and the narrow cnibin mount didn't cover that sibling. FindInPath's
os.Stat follows symlinks, so it failed inside the container even though
the same path resolves fine on the real host. Mount the parent
(k3s-cni-data) into the main container instead so the symlinks resolve;
the init container keeps the narrower cnibin mount since it only writes
files and never follows them.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Nik Afiq 2026-08-24 21:25:12 +09:00
parent d410c05e6d
commit c7cdcc967d

View File

@ -70,6 +70,32 @@
# itself: it also has to resolve correctly in the *pre-chroot*
# container view (FindInPath's side), which is exactly why the
# cnibin mountPath below changed too — see that comment.
# 6. The main kube-multus container mounts a second hostPath,
# k3s-cni-data, at /var/lib/rancher/k3s/data (cnibin's parent), in
# addition to cnibin itself. Root-caused live (2026-08-24), same
# incident as point 5 continued: after binDir was corrected to
# /var/lib/rancher/k3s/data/cni, pod sandbox creation still failed
# with the identical error, `failed to find plugin "flannel" in
# path [/var/lib/rancher/k3s/data/cni]` — this time from inside the
# corrected path itself. Read containernetworking/cni's actual
# source (pkg/invoke/find.go's FindInPath) to confirm the mechanism:
# it calls os.Stat on each candidate path, which follows symlinks,
# and requires the resolved target to be a regular file. A live SSH
# check on nik-debian (already done for point 5) had shown
# /var/lib/rancher/k3s/data/cni is itself a directory of SYMLINKS —
# e.g. .../data/cni/flannel -> .../data/<content-hash>/bin/cni — not
# the real binaries. The narrow cnibin mount only bind-mounts the
# cni subdirectory, so inside the container those symlinks point at
# a hash-named sibling directory that was never mounted at all, and
# os.Stat fails on the daemon container's own (pre-chroot) side —
# exactly the FindInPath call point 5 identified, just one symlink
# hop further than binDir alone could fix. Mounting the parent
# (k3s-cni-data, at the same path in-container as on the host) lets
# the symlink target resolve inside the container too, without
# changing binDir or the resolved host path at all. cnibin itself is
# kept, narrowed to only the install-multus-binary init container's
# use (a plain file write into that directory, which never follows
# the symlinks and so never needed the wider mount).
#
# Do not hand-edit the daemon-config.json keys, the container
# command/args, or the mount set below without re-diffing against the
@ -159,16 +185,22 @@ spec:
# e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon,
# not to any other directory, like '/opt/bin' or '/usr/bin'.
#
# On THIS host that means it must be
# /var/lib/rancher/k3s/data/cni, not the standard /opt/cni/bin
# this comment's own example uses — k3s's real bin dir is not
# the standard one (see the volume definition below), and this
# mountPath has to be identical to it, not to the standard
# path, for delegate plugin exec-after-chroot to find anything
# here at all. See the header comment's point 5 for the full
# mechanism (this was live-broken before that value matched).
- name: cnibin
mountPath: /var/lib/rancher/k3s/data/cni
# On THIS host that mount is k3s-cni-data (below), not cnibin
# — see the header comment's point 6. k3s's actual plugin
# binaries live in a content-hash-named sibling directory
# (/var/lib/rancher/k3s/data/<hash>/bin/cni) and
# /var/lib/rancher/k3s/data/cni/flannel etc. are only
# *symlinks* to that; mounting just the cni/ subdirectory
# (cnibin, still used by the init container below for its own
# narrower purpose) left those symlinks dangling from this
# container's own point of view, which is exactly where
# FindInPath (invoke.FindInPath, os.Stat-based, follows
# symlinks) checks each candidate before the chroot-exec ever
# happens. Mounting the whole parent directory at an identical
# path is what makes both the symlink and its target resolve
# the same way here as they do on the real host.
- name: k3s-cni-data
mountPath: /var/lib/rancher/k3s/data
- name: host-run
mountPath: /host/run
- name: host-var-lib-cni-multus
@ -223,8 +255,19 @@ spec:
path: /var/lib/rancher/k3s/agent/etc/cni/net.d
- name: cnibin
hostPath:
# k3s CNI bin dir — NOT the standard /opt/cni/bin.
# k3s CNI bin dir — NOT the standard /opt/cni/bin. Used only
# by the install-multus-binary init container (a plain file
# write, doesn't need the wider parent — see k3s-cni-data
# below for why the main container needs more than this).
path: /var/lib/rancher/k3s/data/cni
- name: k3s-cni-data
hostPath:
# The parent of cnibin above, mounted identically in the main
# container so that k3s's plugin symlinks (e.g.
# .../data/cni/flannel -> .../data/<hash>/bin/cni) resolve
# correctly from inside the container too, not just on the
# real host — see the volumeMounts comment above for why.
path: /var/lib/rancher/k3s/data
- name: hostroot
hostPath:
path: /