nik/homelab
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
homelab/manifests/multus/02-daemonset.yaml
Nik Afiq d410c05e6d
Some checks failed
validate / lint (push) Failing after 0s
Details
fix: correct Multus binDir/cnibin path for k3s delegate plugin exec 
Root cause: Multus's ChrootExec resolves a delegate plugin's absolute
path from the daemon container's own filesystem view, then execs that
exact path string after chrooting into the real host root. binDir
defaulted to /opt/cni/bin, which is valid inside the container (bind-
mounted to k3s's real bin dir) but doesn't exist on the actual host, so
every pod attaching to the VLAN 50 NAD failed FailedCreatePodSandBox
trying to delegate to flannel. Confirmed against multus-cni's source at
the exact pinned v4.3.0 tag, not assumed. Sets binDir explicitly to
k3s's real path and matches the cnibin mount path to it on both sides of
the chroot boundary (main container and the install_multus init
container).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-24 20:06:04 +09:00

255 lines
11 KiB
YAML
Raw Blame History

# Apply: kubectl apply -f manifests/multus/02-daemonset.yaml
# Description: Multus thick-plugin DaemonSet, restricted by nodeSelector
# to nik-debian only (the sole node with a VLAN 50 trunk — deliberately
# not cluster-wide, so minisforum and nik-gpu's CNI chain is never
# touched by this at all). sync-wave 0 — after RBAC.
#
# Rebuilt from k8snetworkplumbingwg/multus-cni's official
# deployments/multus-daemonset-thick.yml, commit
# 95a0932350f0060cd232096876e24672a1de83fa (master, fetched and read in
# full 2026-08-24 — https://raw.githubusercontent.com/
# k8snetworkplumbingwg/multus-cni/master/deployments/
# multus-daemonset-thick.yml). An earlier version of this file was NOT
# a faithful copy of that manifest — it used an invented entrypoint
# (/entrypoint/cni-installer.sh, which does not exist in the real image)
# and an incomplete set of mounts/config keys, reconstructed from memory
# rather than the real source. This version changes only what k3s and
# this topology actually require, everything else (entrypoints, daemon
# config keys, volume set, RBAC-adjacent mounts) is verbatim:
#
# 1. cni/cnibin volumes' hostPath: k3s's real CNI conf/bin dirs
# (/var/lib/rancher/k3s/agent/etc/cni/net.d,
# /var/lib/rancher/k3s/data/cni — confirmed LIVE on nik-debian
# 2026-08-23; there is no /opt/cni/bin or /etc/cni/net.d on this
# host, upstream's own defaults would silently no-op on k3s
# without this). The container-side mountPaths upstream defines
# (/host/etc/cni/net.d, /host/opt/cni/bin) are unchanged — only
# which host directory backs them moves.
# 2. All OTHER hostPath volumes (host-run, host-var-lib-cni-multus,
# host-var-lib-kubelet, host-run-k8s-cni-cncf-io, host-run-netns,
# hostroot, multus-conf-dir) are left at upstream's standard paths
# — k3s does not relocate /run, /var/lib/kubelet, /run/netns, or /
# itself, only the CNI-specific directories. This assumption (that
# kubelet's own root-dir is standard on k3s) is NOT independently
# verified against a live k3s node the way the CNI paths are —
# confirm before applying if this cluster ever changes
# --root-dir on the kubelet.
# 3. nodeSelector restricting it to nik-debian (upstream is
# cluster-wide by default).
# 4. Image pinned to the v4.3.0-thick digest (upstream's own quickstart
# file uses the "snapshot-thick" moving tag, which is explicitly a
# dev/nightly build, not something to run unpinned in production).
# Digest confirmed live via the GHCR OCI Distribution API
# 2026-08-24: v4.3.0-thick and stable-thick both resolve to
# sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896
# d56ae51c9, i.e. v4.3.0 is genuinely the current stable release,
# not just a same-named coincidence.
# 5. daemon-config.json has one key added beyond upstream's own
# example: "binDir". Root-caused live (2026-08-24) after Multus
# itself came up correctly but every pod on nik-debian using the
# NAD failed FailedCreatePodSandBox with `plugin type="flannel"
# failed (add): failed to find plugin "flannel" in path
# [/opt/cni/bin]`. Read multus-cni's actual source (pinned to the
# v4.3.0 tag, not master) to confirm the mechanism rather than
# guess: pkg/server/exec_chroot.go's ChrootExec resolves a delegate
# plugin's absolute path using the DAEMON CONTAINER's own
# filesystem view (FindInPath), then execs that exact path string
# *after* chrooting the child process into "hostroot" (the real
# host root, bind-mounted). binDir defaults to "/opt/cni/bin"
# (pkg/types/conf.go's defaultBinDir) when unset — a path that is
# valid inside this container (bind-mounted to k3s's real bin dir
# below) but does not exist on the actual host filesystem at all,
# so the post-chroot exec fails even though the daemon "found" the
# plugin from its own point of view first. Confirmed the real host
# path live via SSH (flannel/bridge/host-local genuinely present at
# /var/lib/rancher/k3s/data/cni), confirmed binDir's exact JSON tag
# against pkg/server/config/generator.go's MultusConf struct at the
# v4.3.0 tag specifically (not assumed from master, in case the
# schema had drifted between versions — it hadn't, but that was
# checked, not assumed). Setting binDir alone isn't sufficient by
# itself: it also has to resolve correctly in the *pre-chroot*
# container view (FindInPath's side), which is exactly why the
# cnibin mountPath below changed too — see that comment.
#
# Do not hand-edit the daemon-config.json keys, the container
# command/args, or the mount set below without re-diffing against the
# URL above — the parts still matching upstream verbatim were not
# invented, they're copied; the binDir addition and the cnibin mountPath
# change are the two deliberate, root-caused departures.
apiVersion: v1
kind: ConfigMap
metadata:
name: multus-daemon-config
namespace: kube-system
labels:
tier: node
app: multus
annotations:
argocd.argoproj.io/sync-wave: "0"
data:
daemon-config.json: |
{
"chrootDir": "/hostroot",
"cniVersion": "0.3.1",
"logLevel": "verbose",
"logToStderr": true,
"cniConfigDir": "/host/etc/cni/net.d",
"multusAutoconfigDir": "/host/etc/cni/net.d",
"multusConfigFile": "auto",
"socketDir": "/host/run/multus/",
"binDir": "/var/lib/rancher/k3s/data/cni"
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-multus-ds
namespace: kube-system
labels:
tier: node
app: multus
name: multus
annotations:
argocd.argoproj.io/sync-wave: "0"
spec:
selector:
matchLabels:
name: multus
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
tier: node
app: multus
name: multus
spec:
# Not upstream — restricts this DaemonSet to the one node with a
# VLAN 50 trunk. minisforum and nik-gpu never run Multus at all.
nodeSelector:
kubernetes.io/hostname: nik-debian
hostNetwork: true
hostPID: true
tolerations:
- operator: Exists
effect: NoSchedule
- operator: Exists
effect: NoExecute
serviceAccountName: multus
containers:
- name: kube-multus
# VERIFY the digest still resolves before applying — see
# header comment for how it was obtained and when.
image: "ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896d56ae51c9" # v4.3.0-thick
command: ["/usr/src/multus-cni/bin/multus-daemon"]
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: true
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- name: cni
mountPath: /host/etc/cni/net.d
# multus-daemon expects that cnibin path must be identical between pod and container host.
# e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon,
# not to any other directory, like '/opt/bin' or '/usr/bin'.
#
# On THIS host that means it must be
# /var/lib/rancher/k3s/data/cni, not the standard /opt/cni/bin
# this comment's own example uses — k3s's real bin dir is not
# the standard one (see the volume definition below), and this
# mountPath has to be identical to it, not to the standard
# path, for delegate plugin exec-after-chroot to find anything
# here at all. See the header comment's point 5 for the full
# mechanism (this was live-broken before that value matched).
- name: cnibin
mountPath: /var/lib/rancher/k3s/data/cni
- name: host-run
mountPath: /host/run
- name: host-var-lib-cni-multus
mountPath: /var/lib/cni/multus
- name: host-var-lib-kubelet
mountPath: /var/lib/kubelet
mountPropagation: HostToContainer
- name: host-run-k8s-cni-cncf-io
mountPath: /run/k8s.cni.cncf.io
- name: host-run-netns
mountPath: /run/netns
mountPropagation: HostToContainer
- name: multus-daemon-config
mountPath: /etc/cni/net.d/multus.d
readOnly: true
- name: hostroot
mountPath: /hostroot
mountPropagation: HostToContainer
- mountPath: /etc/cni/multus/net.d
name: multus-conf-dir
env:
- name: MULTUS_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
initContainers:
- name: install-multus-binary
# Same image/digest as the main container — see header comment.
image: "ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896d56ae51c9" # v4.3.0-thick
command:
- "/usr/src/multus-cni/bin/install_multus"
- "-d"
- "/host/var/lib/rancher/k3s/data/cni"
- "-t"
- "thick"
resources:
requests:
cpu: "10m"
memory: "15Mi"
securityContext:
privileged: true
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- name: cnibin
mountPath: /host/var/lib/rancher/k3s/data/cni
mountPropagation: Bidirectional
terminationGracePeriodSeconds: 30
volumes:
- name: cni
hostPath:
# k3s CNI conf dir — NOT the standard /etc/cni/net.d.
path: /var/lib/rancher/k3s/agent/etc/cni/net.d
- name: cnibin
hostPath:
# k3s CNI bin dir — NOT the standard /opt/cni/bin.
path: /var/lib/rancher/k3s/data/cni
- name: hostroot
hostPath:
path: /
- name: multus-daemon-config
configMap:
name: multus-daemon-config
items:
- key: daemon-config.json
path: daemon-config.json
- name: host-run
hostPath:
path: /run
- name: host-var-lib-cni-multus
hostPath:
path: /var/lib/cni/multus
- name: host-var-lib-kubelet
hostPath:
path: /var/lib/kubelet
- name: host-run-k8s-cni-cncf-io
hostPath:
path: /run/k8s.cni.cncf.io
- name: host-run-netns
hostPath:
path: /run/netns/
- name: multus-conf-dir
hostPath:
path: /etc/cni/multus/net.d
Reference in New Issue View Git Blame Copy Permalink