# Apply: kubectl apply -f manifests/multus/02-daemonset.yaml
# Description: Multus thick-plugin DaemonSet, restricted by nodeSelector
#   to nik-debian only (the sole node with a VLAN 50 trunk — deliberately
#   not cluster-wide, so minisforum and nik-gpu's CNI chain is never
#   touched by this at all). sync-wave 0 — after RBAC.
#
# Rebuilt from k8snetworkplumbingwg/multus-cni's official
# deployments/multus-daemonset-thick.yml, commit
# 95a0932350f0060cd232096876e24672a1de83fa (master, fetched and read in
# full 2026-08-24 — https://raw.githubusercontent.com/
# k8snetworkplumbingwg/multus-cni/master/deployments/
# multus-daemonset-thick.yml). An earlier version of this file was NOT
# a faithful copy of that manifest — it used an invented entrypoint
# (/entrypoint/cni-installer.sh, which does not exist in the real image)
# and an incomplete set of mounts/config keys, reconstructed from memory
# rather than the real source. This version changes only what k3s and
# this topology actually require, everything else (entrypoints, daemon
# config keys, volume set, RBAC-adjacent mounts) is verbatim:
#
#   1. cni/cnibin volumes' hostPath: k3s's real CNI conf/bin dirs
#      (/var/lib/rancher/k3s/agent/etc/cni/net.d,
#      /var/lib/rancher/k3s/data/cni — confirmed LIVE on nik-debian
#      2026-08-23; there is no /opt/cni/bin or /etc/cni/net.d on this
#      host, upstream's own defaults would silently no-op on k3s
#      without this). The container-side mountPaths upstream defines
#      (/host/etc/cni/net.d, /host/opt/cni/bin) are unchanged — only
#      which host directory backs them moves.
#   2. All OTHER hostPath volumes (host-run, host-var-lib-cni-multus,
#      host-var-lib-kubelet, host-run-k8s-cni-cncf-io, host-run-netns,
#      hostroot, multus-conf-dir) are left at upstream's standard paths
#      — k3s does not relocate /run, /var/lib/kubelet, /run/netns, or /
#      itself, only the CNI-specific directories. This assumption (that
#      kubelet's own root-dir is standard on k3s) is NOT independently
#      verified against a live k3s node the way the CNI paths are —
#      confirm before applying if this cluster ever changes
#      --root-dir on the kubelet.
#   3. nodeSelector restricting it to nik-debian (upstream is
#      cluster-wide by default).
#   4. Image pinned to the v4.3.0-thick digest (upstream's own quickstart
#      file uses the "snapshot-thick" moving tag, which is explicitly a
#      dev/nightly build, not something to run unpinned in production).
#      Digest confirmed live via the GHCR OCI Distribution API
#      2026-08-24: v4.3.0-thick and stable-thick both resolve to
#      sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896
#      d56ae51c9, i.e. v4.3.0 is genuinely the current stable release,
#      not just a same-named coincidence.
#   5. daemon-config.json has one key added beyond upstream's own
#      example: "binDir". Root-caused live (2026-08-24) after Multus
#      itself came up correctly but every pod on nik-debian using the
#      NAD failed FailedCreatePodSandBox with `plugin type="flannel"
#      failed (add): failed to find plugin "flannel" in path
#      [/opt/cni/bin]`. Read multus-cni's actual source (pinned to the
#      v4.3.0 tag, not master) to confirm the mechanism rather than
#      guess: pkg/server/exec_chroot.go's ChrootExec resolves a delegate
#      plugin's absolute path using the DAEMON CONTAINER's own
#      filesystem view (FindInPath), then execs that exact path string
#      *after* chrooting the child process into "hostroot" (the real
#      host root, bind-mounted). binDir defaults to "/opt/cni/bin"
#      (pkg/types/conf.go's defaultBinDir) when unset — a path that is
#      valid inside this container (bind-mounted to k3s's real bin dir
#      below) but does not exist on the actual host filesystem at all,
#      so the post-chroot exec fails even though the daemon "found" the
#      plugin from its own point of view first. Confirmed the real host
#      path live via SSH (flannel/bridge/host-local genuinely present at
#      /var/lib/rancher/k3s/data/cni), confirmed binDir's exact JSON tag
#      against pkg/server/config/generator.go's MultusConf struct at the
#      v4.3.0 tag specifically (not assumed from master, in case the
#      schema had drifted between versions — it hadn't, but that was
#      checked, not assumed). Setting binDir alone isn't sufficient by
#      itself: it also has to resolve correctly in the *pre-chroot*
#      container view (FindInPath's side), which is exactly why the
#      cnibin mountPath below changed too — see that comment.
#   6. The main kube-multus container mounts a second hostPath,
#      k3s-cni-data, at /var/lib/rancher/k3s/data (cnibin's parent), in
#      addition to cnibin itself. Root-caused live (2026-08-24), same
#      incident as point 5 continued: after binDir was corrected to
#      /var/lib/rancher/k3s/data/cni, pod sandbox creation still failed
#      with the identical error, `failed to find plugin "flannel" in
#      path [/var/lib/rancher/k3s/data/cni]` — this time from inside the
#      corrected path itself. Read containernetworking/cni's actual
#      source (pkg/invoke/find.go's FindInPath) to confirm the mechanism:
#      it calls os.Stat on each candidate path, which follows symlinks,
#      and requires the resolved target to be a regular file. A live SSH
#      check on nik-debian (already done for point 5) had shown
#      /var/lib/rancher/k3s/data/cni is itself a directory of SYMLINKS —
#      e.g. .../data/cni/flannel -> .../data/<content-hash>/bin/cni — not
#      the real binaries. The narrow cnibin mount only bind-mounts the
#      cni subdirectory, so inside the container those symlinks point at
#      a hash-named sibling directory that was never mounted at all, and
#      os.Stat fails on the daemon container's own (pre-chroot) side —
#      exactly the FindInPath call point 5 identified, just one symlink
#      hop further than binDir alone could fix. Mounting the parent
#      (k3s-cni-data, at the same path in-container as on the host) lets
#      the symlink target resolve inside the container too, without
#      changing binDir or the resolved host path at all. cnibin itself is
#      kept, narrowed to the two init containers' own plain file writes
#      (which never follow the symlinks and so never needed the wider
#      mount) — see point 7 for the second of those two.
#   7. A cni-plugins init container, absent from upstream's own
#      daemonset-thick.yml entirely, installs the standard CNI reference
#      plugins (bridge, macvlan, ipvlan, vlan, host-local, loopback,
#      portmap, bandwidth, firewall, dhcp, static, tuning, sbr, vrf,
#      host-device, ptp, dummy) that k3s does not bundle. Root-caused live
#      (2026-08-24), same incident as points 5/6 continued: with binDir
#      and the mount both fixed, sandbox creation failed a third time,
#      now with `failed to find plugin "macvlan" in path
#      [/var/lib/rancher/k3s/data/cni]` — a genuinely absent binary this
#      time, not another path/mount problem. Confirmed against k3s's own
#      real build script (k3s-io/k3s's scripts/package-cli at this exact
#      tag, read in full): its cni_binaries array is hardcoded to exactly
#      bandwidth, bridge, firewall, flannel, host-local, loopback,
#      portmap — macvlan is deliberately excluded by k3s's own project
#      scope decision (k3s-io/k3s issue #9224 confirms this is
#      intentional, not an oversight). k3s's own documentation
#      (https://docs.k3s.io/networking/multus-ipams) names the fix for
#      exactly this gap: install the missing plugins via the same
#      mechanism Rancher's own rke2-multus Helm chart uses — see this
#      container's own inline comment for exactly which real chart/image
#      that is and how it was verified (image digest confirmed live
#      against the real registry, not assumed from the tag).
#
# Do not hand-edit the daemon-config.json keys, the container
# command/args, or the mount set below without re-diffing against the
# URL above — the parts still matching upstream verbatim were not
# invented, they're copied; the binDir addition, the cnibin/k3s-cni-data
# mount split, and the added cni-plugins init container are the three
# deliberate, root-caused departures.
apiVersion: v1
kind: ConfigMap
metadata:
  name: multus-daemon-config
  namespace: kube-system
  labels:
    tier: node
    app: multus
  annotations:
    argocd.argoproj.io/sync-wave: "0"
data:
  daemon-config.json: |
    {
        "chrootDir": "/hostroot",
        "cniVersion": "0.3.1",
        "logLevel": "verbose",
        "logToStderr": true,
        "cniConfigDir": "/host/etc/cni/net.d",
        "multusAutoconfigDir": "/host/etc/cni/net.d",
        "multusConfigFile": "auto",
        "socketDir": "/host/run/multus/",
        "binDir": "/var/lib/rancher/k3s/data/cni"
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-multus-ds
  namespace: kube-system
  labels:
    tier: node
    app: multus
    name: multus
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  selector:
    matchLabels:
      name: multus
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        tier: node
        app: multus
        name: multus
    spec:
      # Not upstream — restricts this DaemonSet to the one node with a
      # VLAN 50 trunk. minisforum and nik-gpu never run Multus at all.
      nodeSelector:
        kubernetes.io/hostname: nik-debian
      hostNetwork: true
      hostPID: true
      tolerations:
        - operator: Exists
          effect: NoSchedule
        - operator: Exists
          effect: NoExecute
      serviceAccountName: multus
      containers:
        - name: kube-multus
          # VERIFY the digest still resolves before applying — see
          # header comment for how it was obtained and when.
          image: "ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896d56ae51c9" # v4.3.0-thick
          command: ["/usr/src/multus-cni/bin/multus-daemon"]
          resources:
            requests:
              cpu: "100m"
              memory: "50Mi"
            limits:
              cpu: "100m"
              memory: "50Mi"
          securityContext:
            privileged: true
          terminationMessagePolicy: FallbackToLogsOnError
          volumeMounts:
            - name: cni
              mountPath: /host/etc/cni/net.d
            # multus-daemon expects that cnibin path must be identical between pod and container host.
            # e.g. if the cni bin is in '/opt/cni/bin' on the container host side, then it should be mount to '/opt/cni/bin' in multus-daemon,
            # not to any other directory, like '/opt/bin' or '/usr/bin'.
            #
            # On THIS host that mount is k3s-cni-data (below), not cnibin
            # — see the header comment's point 6. k3s's actual plugin
            # binaries live in a content-hash-named sibling directory
            # (/var/lib/rancher/k3s/data/<hash>/bin/cni) and
            # /var/lib/rancher/k3s/data/cni/flannel etc. are only
            # *symlinks* to that; mounting just the cni/ subdirectory
            # (cnibin, still used by the init container below for its own
            # narrower purpose) left those symlinks dangling from this
            # container's own point of view, which is exactly where
            # FindInPath (invoke.FindInPath, os.Stat-based, follows
            # symlinks) checks each candidate before the chroot-exec ever
            # happens. Mounting the whole parent directory at an identical
            # path is what makes both the symlink and its target resolve
            # the same way here as they do on the real host.
            - name: k3s-cni-data
              mountPath: /var/lib/rancher/k3s/data
            - name: host-run
              mountPath: /host/run
            - name: host-var-lib-cni-multus
              mountPath: /var/lib/cni/multus
            - name: host-var-lib-kubelet
              mountPath: /var/lib/kubelet
              mountPropagation: HostToContainer
            - name: host-run-k8s-cni-cncf-io
              mountPath: /run/k8s.cni.cncf.io
            - name: host-run-netns
              mountPath: /run/netns
              mountPropagation: HostToContainer
            - name: multus-daemon-config
              mountPath: /etc/cni/net.d/multus.d
              readOnly: true
            - name: hostroot
              mountPath: /hostroot
              mountPropagation: HostToContainer
            - mountPath: /etc/cni/multus/net.d
              name: multus-conf-dir
          env:
            - name: MULTUS_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
      initContainers:
        - name: cni-plugins
          # NOT part of upstream k8snetworkplumbingwg/multus-cni's own
          # daemonset-thick.yml (that manifest assumes the standard CNI
          # reference plugins already exist on the host, which is true on
          # most distributions but NOT k3s — see the header comment's
          # point 7). This container and image are copied from Rancher's
          # own official rke2-multus Helm chart (the one k3s's own docs at
          # https://docs.k3s.io/networking/multus-ipams recommend for
          # exactly this situation), fetched and read in full 2026-08-24
          # from the real packaged chart at https://rke2-charts.rancher.io
          # /assets/rke2-multus/rke2-multus-v4.3.017.tgz — not fabricated.
          image: "docker.io/rancher/hardened-cni-plugins@sha256:935b2e954695ecc153d4a671f51717dc93b613e08b5f4230ba0eb8dbf273495a" # v1.9.1-build20260819
          securityContext:
            privileged: true
          env:
            # Skip flannel specifically: k3s's own binDir already has a
            # working flannel plugin (from k3s's own build, version-locked
            # to this k3s release) — see header comment's points 5 and 6.
            # Letting this image overwrite it with its own separately
            # versioned flannel would be a needless, unverified swap of a
            # plugin that already works, for zero benefit (this image is
            # only needed here for the plugins k3s doesn't ship at all,
            # e.g. macvlan). Matches Rancher's own chart's default
            # cniplugins.skipcnis value for this exact reason.
            - name: SKIP_CNI_BINARIES
              value: flannel
          volumeMounts:
            # install-cnis.sh (this image's entrypoint, read in full) has
            # a hardcoded destination of /host/opt/cni/bin — not
            # configurable via any env var or arg. Mounting cnibin (this
            # role's real k3s CNI bin dir, not the standard /opt/cni/bin)
            # at that container path redirects the writes to the right
            # place without needing a different image or a patched
            # script. It copies plain regular files directly into that
            # directory (not through the flannel-style symlink-to-hash-dir
            # indirection those specific k3s-provided binaries use), so
            # this narrower cnibin mount is sufficient here — no need for
            # the wider k3s-cni-data mount the main container uses.
            - name: cnibin
              mountPath: /host/opt/cni/bin
        - name: install-multus-binary
          # Same image/digest as the main container — see header comment.
          image: "ghcr.io/k8snetworkplumbingwg/multus-cni@sha256:2b9671447f3ea4e7e56730843dbf59445b9307246f393b61386b896d56ae51c9" # v4.3.0-thick
          command:
            - "/usr/src/multus-cni/bin/install_multus"
            - "-d"
            - "/host/var/lib/rancher/k3s/data/cni"
            - "-t"
            - "thick"
          resources:
            requests:
              cpu: "10m"
              memory: "15Mi"
          securityContext:
            privileged: true
          terminationMessagePolicy: FallbackToLogsOnError
          volumeMounts:
            - name: cnibin
              mountPath: /host/var/lib/rancher/k3s/data/cni
              mountPropagation: Bidirectional
      terminationGracePeriodSeconds: 30
      volumes:
        - name: cni
          hostPath:
            # k3s CNI conf dir — NOT the standard /etc/cni/net.d.
            path: /var/lib/rancher/k3s/agent/etc/cni/net.d
        - name: cnibin
          hostPath:
            # k3s CNI bin dir — NOT the standard /opt/cni/bin. Used only
            # by the install-multus-binary init container (a plain file
            # write, doesn't need the wider parent — see k3s-cni-data
            # below for why the main container needs more than this).
            path: /var/lib/rancher/k3s/data/cni
        - name: k3s-cni-data
          hostPath:
            # The parent of cnibin above, mounted identically in the main
            # container so that k3s's plugin symlinks (e.g.
            # .../data/cni/flannel -> .../data/<hash>/bin/cni) resolve
            # correctly from inside the container too, not just on the
            # real host — see the volumeMounts comment above for why.
            path: /var/lib/rancher/k3s/data
        - name: hostroot
          hostPath:
            path: /
        - name: multus-daemon-config
          configMap:
            name: multus-daemon-config
            items:
              - key: daemon-config.json
                path: daemon-config.json
        - name: host-run
          hostPath:
            path: /run
        - name: host-var-lib-cni-multus
          hostPath:
            path: /var/lib/cni/multus
        - name: host-var-lib-kubelet
          hostPath:
            path: /var/lib/kubelet
        - name: host-run-k8s-cni-cncf-io
          hostPath:
            path: /run/k8s.cni.cncf.io
        - name: host-run-netns
          hostPath:
            path: /run/netns/
        - name: multus-conf-dir
          hostPath:
            path: /etc/cni/multus/net.d