Nik Afiq
f1bdf11f1e
Some checks failed
validate / lint (push) Failing after 1s
k3s deliberately ships only the plugins its own flannel networking needs (bandwidth, bridge, firewall, flannel, host-local, loopback, portmap — confirmed against k3s's own build script; macvlan's absence is an intentional upstream scope decision, k3s-io/k3s#9224). Multus's macvlan delegate for the vlan50 NAD therefore had nothing to exec even after the binDir/mount fixes. Add a cni-plugins init container mirroring Rancher's own official rke2-multus Helm chart (the fix k3s's own docs recommend for this exact gap): rancher/hardened-cni-plugins, digest-pinned, writing into our existing cnibin mount instead of the image's hardcoded /opt/cni/bin, with SKIP_CNI_BINARIES=flannel so it doesn't overwrite k3s's already-working flannel plugin. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
361 lines
18 KiB
YAML
361 lines
18 KiB
YAML
# Apply: kubectl apply -f manifests/multus/02-daemonset.yaml
|
|
# Description: Multus thick-plugin DaemonSet, restricted by nodeSelector
|
|
# to nik-debian only (the sole node with a VLAN 50 trunk — deliberately
|
|
# not cluster-wide, so minisforum and nik-gpu's CNI chain is never
|
|
# touched by this at all). sync-wave 0 — after RBAC.
|
|
#
|
|
# Rebuilt from k8snetworkplumbingwg/multus-cni's official
|
|
# deployments/multus-daemonset-thick.yml, commit
|
|
# 95a0932350f0060cd232096876e24672a1de83fa (master, fetched and read in
|
|
# full 2026-08-24 — https://raw.githubusercontent.com/
|
|
# k8snetworkplumbingwg/multus-cni/master/deployments/
|
|
# multus-daemonset-thick.yml). An earlier version of this file was NOT
|
|
# a faithful copy of that manifest — it used an invented entrypoint
|
|
# (/entrypoint/cni-installer.sh, which does not exist in the real image)
|
|
# and an incomplete set of mounts/config keys, reconstructed from memory
|
|
# rather than the real source. This version changes only what k3s and
|
|
# this topology actually require, everything else (entrypoints, daemon
|
|
# config keys, volume set, RBAC-adjacent mounts) is verbatim:
|
|
#
|
|
# 1. cni/cnibin volumes' hostPath: k3s's real CNI conf/bin dirs
|
|
# (/var/lib/rancher/k3s/agent/etc/cni/net.d,
|
|
# /var/lib/rancher/k3s/data/cni — confirmed LIVE on nik-debian
|
|
# 2026-08-23; there is no /opt/cni/bin or /etc/cni/net.d on this
|
|
# host, upstream's own defaults would silently no-op on k3s
|
|
# without this). The container-side mountPaths upstream defines
|
|
# (/host/etc/cni/net.d, /host/opt/cni/bin) are unchanged — only
|
|
# which host directory backs them moves.
|
|
# 2. All OTHER hostPath volumes (host-run, host-var-lib-cni-multus,
|
|
# host-var-lib-kubelet, host-run-k8s-cni-cncf-io, host-run-netns,
|
|
# hostroot, multus-conf-dir) are left at upstream's standard paths
|
|
# — k3s does not relocate /run, /var/lib/kubelet, /run/netns, or /
|
|
# itself, only the CNI-specific directories. This assumption (that
|
|
# kubelet's own root-dir is standard on k3s) is NOT independently
|
|
# verified against a live k3s node the way the CNI paths are —
|
|
# confirm before applying if this cluster ever changes
|
|
# --root-dir on the kubelet.
|
|
# 3. nodeSelector restricting it to nik-debian (upstream is
|
|
# cluster-wide by default).
|
|
# 4. Image pinned to the v4.3.0-thick digest (upstream's own quickstart
|
|
# file uses the "snapshot-thick" moving tag, which is explicitly a
|
|
# dev/nightly build, not something to run unpinned in production).
|
|
# Digest confirmed live via the GHCR OCI Distribution API
|
|
# 2026-08-24: v4.3.0-thick and stable-thick both resolve to
|
|
# sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896
|
|
# d56ae51c9, i.e. v4.3.0 is genuinely the current stable release,
|
|
# not just a same-named coincidence.
|
|
# 5. daemon-config.json has one key added beyond upstream's own
|
|
# example: "binDir". Root-caused live (2026-08-24) after Multus
|
|
# itself came up correctly but every pod on nik-debian using the
|
|
# NAD failed FailedCreatePodSandBox with `plugin type="flannel"
|
|
# failed (add): failed to find plugin "flannel" in path
|
|
# [/opt/cni/bin]`. Read multus-cni's actual source (pinned to the
|
|
# v4.3.0 tag, not master) to confirm the mechanism rather than
|
|
# guess: pkg/server/exec_chroot.go's ChrootExec resolves a delegate
|
|
# plugin's absolute path using the DAEMON CONTAINER's own
|
|
# filesystem view (FindInPath), then execs that exact path string
|
|
# *after* chrooting the child process into "hostroot" (the real
|
|
# host root, bind-mounted). binDir defaults to "/opt/cni/bin"
|
|
# (pkg/types/conf.go's defaultBinDir) when unset — a path that is
|
|
# valid inside this container (bind-mounted to k3s's real bin dir
|
|
# below) but does not exist on the actual host filesystem at all,
|
|
# so the post-chroot exec fails even though the daemon "found" the
|
|
# plugin from its own point of view first. Confirmed the real host
|
|
# path live via SSH (flannel/bridge/host-local genuinely present at
|
|
# /var/lib/rancher/k3s/data/cni), confirmed binDir's exact JSON tag
|
|
# against pkg/server/config/generator.go's MultusConf struct at the
|
|
# v4.3.0 tag specifically (not assumed from master, in case the
|
|
# schema had drifted between versions — it hadn't, but that was
|
|
# checked, not assumed). Setting binDir alone isn't sufficient by
|
|
# itself: it also has to resolve correctly in the *pre-chroot*
|
|
# container view (FindInPath's side), which is exactly why the
|
|
# cnibin mountPath below changed too — see that comment.
|
|
# 6. The main kube-multus container mounts a second hostPath,
|
|
# k3s-cni-data, at /var/lib/rancher/k3s/data (cnibin's parent), in
|
|
# addition to cnibin itself. Root-caused live (2026-08-24), same
|
|
# incident as point 5 continued: after binDir was corrected to
|
|
# /var/lib/rancher/k3s/data/cni, pod sandbox creation still failed
|
|
# with the identical error, `failed to find plugin "flannel" in
|
|
# path [/var/lib/rancher/k3s/data/cni]` — this time from inside the
|
|
# corrected path itself. Read containernetworking/cni's actual
|
|
# source (pkg/invoke/find.go's FindInPath) to confirm the mechanism:
|
|
# it calls os.Stat on each candidate path, which follows symlinks,
|
|
# and requires the resolved target to be a regular file. A live SSH
|
|
# check on nik-debian (already done for point 5) had shown
|
|
# /var/lib/rancher/k3s/data/cni is itself a directory of SYMLINKS —
|
|
# e.g. .../data/cni/flannel -> .../data/<content-hash>/bin/cni — not
|
|
# the real binaries. The narrow cnibin mount only bind-mounts the
|
|
# cni subdirectory, so inside the container those symlinks point at
|
|
# a hash-named sibling directory that was never mounted at all, and
|
|
# os.Stat fails on the daemon container's own (pre-chroot) side —
|
|
# exactly the FindInPath call point 5 identified, just one symlink
|
|
# hop further than binDir alone could fix. Mounting the parent
|
|
# (k3s-cni-data, at the same path in-container as on the host) lets
|
|
# the symlink target resolve inside the container too, without
|
|
# changing binDir or the resolved host path at all. cnibin itself is
|
|
# kept, narrowed to the two init containers' own plain file writes
|
|
# (which never follow the symlinks and so never needed the wider
|
|
# mount) — see point 7 for the second of those two.
|
|
# 7. A cni-plugins init container, absent from upstream's own
|
|
# daemonset-thick.yml entirely, installs the standard CNI reference
|
|
# plugins (bridge, macvlan, ipvlan, vlan, host-local, loopback,
|
|
# portmap, bandwidth, firewall, dhcp, static, tuning, sbr, vrf,
|
|
# host-device, ptp, dummy) that k3s does not bundle. Root-caused live
|
|
# (2026-08-24), same incident as points 5/6 continued: with binDir
|
|
# and the mount both fixed, sandbox creation failed a third time,
|
|
# now with `failed to find plugin "macvlan" in path
|
|
# [/var/lib/rancher/k3s/data/cni]` — a genuinely absent binary this
|
|
# time, not another path/mount problem. Confirmed against k3s's own
|
|
# real build script (k3s-io/k3s's scripts/package-cli at this exact
|
|
# tag, read in full): its cni_binaries array is hardcoded to exactly
|
|
# bandwidth, bridge, firewall, flannel, host-local, loopback,
|
|
# portmap — macvlan is deliberately excluded by k3s's own project
|
|
# scope decision (k3s-io/k3s issue #9224 confirms this is
|
|
# intentional, not an oversight). k3s's own documentation
|
|
# (https://docs.k3s.io/networking/multus-ipams) names the fix for
|
|
# exactly this gap: install the missing plugins via the same
|
|
# mechanism Rancher's own rke2-multus Helm chart uses — see this
|
|
# container's own inline comment for exactly which real chart/image
|
|
# that is and how it was verified (image digest confirmed live
|
|
# against the real registry, not assumed from the tag).
|
|
#
|
|
# Do not hand-edit the daemon-config.json keys, the container
|
|
# command/args, or the mount set below without re-diffing against the
|
|
# URL above — the parts still matching upstream verbatim were not
|
|
# invented, they're copied; the binDir addition, the cnibin/k3s-cni-data
|
|
# mount split, and the added cni-plugins init container are the three
|
|
# deliberate, root-caused departures.
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: multus-daemon-config
|
|
namespace: kube-system
|
|
labels:
|
|
tier: node
|
|
app: multus
|
|
annotations:
|
|
argocd.argoproj.io/sync-wave: "0"
|
|
data:
|
|
daemon-config.json: |
|
|
{
|
|
"chrootDir": "/hostroot",
|
|
"cniVersion": "0.3.1",
|
|
"logLevel": "verbose",
|
|
"logToStderr": true,
|
|
"cniConfigDir": "/host/etc/cni/net.d",
|
|
"multusAutoconfigDir": "/host/etc/cni/net.d",
|
|
"multusConfigFile": "auto",
|
|
"socketDir": "/host/run/multus/",
|
|
"binDir": "/var/lib/rancher/k3s/data/cni"
|
|
}
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: DaemonSet
|
|
metadata:
|
|
name: kube-multus-ds
|
|
namespace: kube-system
|
|
labels:
|
|
tier: node
|
|
app: multus
|
|
name: multus
|
|
annotations:
|
|
argocd.argoproj.io/sync-wave: "0"
|
|
spec:
|
|
selector:
|
|
matchLabels:
|
|
name: multus
|
|
updateStrategy:
|
|
type: RollingUpdate
|
|
template:
|
|
metadata:
|
|
labels:
|
|
tier: node
|
|
app: multus
|
|
name: multus
|
|
spec:
|
|
# Not upstream — restricts this DaemonSet to the one node with a
|
|
# VLAN 50 trunk. minisforum and nik-gpu never run Multus at all.
|
|
nodeSelector:
|
|
kubernetes.io/hostname: nik-debian
|
|
hostNetwork: true
|
|
hostPID: true
|
|
tolerations:
|
|
- operator: Exists
|
|
effect: NoSchedule
|
|
- operator: Exists
|
|
effect: NoExecute
|
|
serviceAccountName: multus
|
|
containers:
|
|
- name: kube-multus
|
|
# VERIFY the digest still resolves before applying — see
|
|
# header comment for how it was obtained and when.
|
|
image: "ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896d56ae51c9" # v4.3.0-thick
|
|
command: ["/usr/src/multus-cni/bin/multus-daemon"]
|
|
resources:
|
|
requests:
|
|
cpu: "100m"
|
|
memory: "50Mi"
|
|
limits:
|
|
cpu: "100m"
|
|
memory: "50Mi"
|
|
securityContext:
|
|
privileged: true
|
|
terminationMessagePolicy: FallbackToLogsOnError
|
|
volumeMounts:
|
|
- name: cni
|
|
mountPath: /host/etc/cni/net.d
|
|
# multus-daemon expects that cnibin path must be identical between pod and container host.
|
|
# e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon,
|
|
# not to any other directory, like '/opt/bin' or '/usr/bin'.
|
|
#
|
|
# On THIS host that mount is k3s-cni-data (below), not cnibin
|
|
# — see the header comment's point 6. k3s's actual plugin
|
|
# binaries live in a content-hash-named sibling directory
|
|
# (/var/lib/rancher/k3s/data/<hash>/bin/cni) and
|
|
# /var/lib/rancher/k3s/data/cni/flannel etc. are only
|
|
# *symlinks* to that; mounting just the cni/ subdirectory
|
|
# (cnibin, still used by the init container below for its own
|
|
# narrower purpose) left those symlinks dangling from this
|
|
# container's own point of view, which is exactly where
|
|
# FindInPath (invoke.FindInPath, os.Stat-based, follows
|
|
# symlinks) checks each candidate before the chroot-exec ever
|
|
# happens. Mounting the whole parent directory at an identical
|
|
# path is what makes both the symlink and its target resolve
|
|
# the same way here as they do on the real host.
|
|
- name: k3s-cni-data
|
|
mountPath: /var/lib/rancher/k3s/data
|
|
- name: host-run
|
|
mountPath: /host/run
|
|
- name: host-var-lib-cni-multus
|
|
mountPath: /var/lib/cni/multus
|
|
- name: host-var-lib-kubelet
|
|
mountPath: /var/lib/kubelet
|
|
mountPropagation: HostToContainer
|
|
- name: host-run-k8s-cni-cncf-io
|
|
mountPath: /run/k8s.cni.cncf.io
|
|
- name: host-run-netns
|
|
mountPath: /run/netns
|
|
mountPropagation: HostToContainer
|
|
- name: multus-daemon-config
|
|
mountPath: /etc/cni/net.d/multus.d
|
|
readOnly: true
|
|
- name: hostroot
|
|
mountPath: /hostroot
|
|
mountPropagation: HostToContainer
|
|
- mountPath: /etc/cni/multus/net.d
|
|
name: multus-conf-dir
|
|
env:
|
|
- name: MULTUS_NODE_NAME
|
|
valueFrom:
|
|
fieldRef:
|
|
fieldPath: spec.nodeName
|
|
initContainers:
|
|
- name: cni-plugins
|
|
# NOT part of upstream k8snetworkplumbingwg/multus-cni's own
|
|
# daemonset-thick.yml (that manifest assumes the standard CNI
|
|
# reference plugins already exist on the host, which is true on
|
|
# most distributions but NOT k3s — see the header comment's
|
|
# point 7). This container and image are copied from Rancher's
|
|
# own official rke2-multus Helm chart (the one k3s's own docs at
|
|
# https://docs.k3s.io/networking/multus-ipams recommend for
|
|
# exactly this situation), fetched and read in full 2026-08-24
|
|
# from the real packaged chart at https://rke2-charts.rancher.io
|
|
# /assets/rke2-multus/rke2-multus-v4.3.017.tgz — not fabricated.
|
|
image: "docker.io/rancher/hardened-cni-plugins@sha256:935b2e954695ecc153d4a671f51717dc93b613e08b5f4230ba0eb8dbf273495a" # v1.9.1-build20260819
|
|
securityContext:
|
|
privileged: true
|
|
env:
|
|
# Skip flannel specifically: k3s's own binDir already has a
|
|
# working flannel plugin (from k3s's own build, version-locked
|
|
# to this k3s release) — see header comment's points 5 and 6.
|
|
# Letting this image overwrite it with its own separately
|
|
# versioned flannel would be a needless, unverified swap of a
|
|
# plugin that already works, for zero benefit (this image is
|
|
# only needed here for the plugins k3s doesn't ship at all,
|
|
# e.g. macvlan). Matches Rancher's own chart's default
|
|
# cniplugins.skipcnis value for this exact reason.
|
|
- name: SKIP_CNI_BINARIES
|
|
value: flannel
|
|
volumeMounts:
|
|
# install-cnis.sh (this image's entrypoint, read in full) has
|
|
# a hardcoded destination of /host/opt/cni/bin — not
|
|
# configurable via any env var or arg. Mounting cnibin (this
|
|
# role's real k3s CNI bin dir, not the standard /opt/cni/bin)
|
|
# at that container path redirects the writes to the right
|
|
# place without needing a different image or a patched
|
|
# script. It copies plain regular files directly into that
|
|
# directory (not through the flannel-style symlink-to-hash-dir
|
|
# indirection those specific k3s-provided binaries use), so
|
|
# this narrower cnibin mount is sufficient here — no need for
|
|
# the wider k3s-cni-data mount the main container uses.
|
|
- name: cnibin
|
|
mountPath: /host/opt/cni/bin
|
|
- name: install-multus-binary
|
|
# Same image/digest as the main container — see header comment.
|
|
image: "ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896d56ae51c9" # v4.3.0-thick
|
|
command:
|
|
- "/usr/src/multus-cni/bin/install_multus"
|
|
- "-d"
|
|
- "/host/var/lib/rancher/k3s/data/cni"
|
|
- "-t"
|
|
- "thick"
|
|
resources:
|
|
requests:
|
|
cpu: "10m"
|
|
memory: "15Mi"
|
|
securityContext:
|
|
privileged: true
|
|
terminationMessagePolicy: FallbackToLogsOnError
|
|
volumeMounts:
|
|
- name: cnibin
|
|
mountPath: /host/var/lib/rancher/k3s/data/cni
|
|
mountPropagation: Bidirectional
|
|
terminationGracePeriodSeconds: 30
|
|
volumes:
|
|
- name: cni
|
|
hostPath:
|
|
# k3s CNI conf dir — NOT the standard /etc/cni/net.d.
|
|
path: /var/lib/rancher/k3s/agent/etc/cni/net.d
|
|
- name: cnibin
|
|
hostPath:
|
|
# k3s CNI bin dir — NOT the standard /opt/cni/bin. Used only
|
|
# by the install-multus-binary init container (a plain file
|
|
# write, doesn't need the wider parent — see k3s-cni-data
|
|
# below for why the main container needs more than this).
|
|
path: /var/lib/rancher/k3s/data/cni
|
|
- name: k3s-cni-data
|
|
hostPath:
|
|
# The parent of cnibin above, mounted identically in the main
|
|
# container so that k3s's plugin symlinks (e.g.
|
|
# .../data/cni/flannel -> .../data/<hash>/bin/cni) resolve
|
|
# correctly from inside the container too, not just on the
|
|
# real host — see the volumeMounts comment above for why.
|
|
path: /var/lib/rancher/k3s/data
|
|
- name: hostroot
|
|
hostPath:
|
|
path: /
|
|
- name: multus-daemon-config
|
|
configMap:
|
|
name: multus-daemon-config
|
|
items:
|
|
- key: daemon-config.json
|
|
path: daemon-config.json
|
|
- name: host-run
|
|
hostPath:
|
|
path: /run
|
|
- name: host-var-lib-cni-multus
|
|
hostPath:
|
|
path: /var/lib/cni/multus
|
|
- name: host-var-lib-kubelet
|
|
hostPath:
|
|
path: /var/lib/kubelet
|
|
- name: host-run-k8s-cni-cncf-io
|
|
hostPath:
|
|
path: /run/k8s.cni.cncf.io
|
|
- name: host-run-netns
|
|
hostPath:
|
|
path: /run/netns/
|
|
- name: multus-conf-dir
|
|
hostPath:
|
|
path: /etc/cni/multus/net.d
|