From 26894d16ce9022908dbde7cf85dc9c6bc6642e50 Mon Sep 17 00:00:00 2001 From: Nik Afiq Date: Mon, 24 Aug 2026 22:02:27 +0900 Subject: [PATCH] feat: strengthen VLAN50 canary egress guard and document its lifecycle Diagnosed the vlan50-canary FailedCreatePodSandBox->net1-timeout failure as a stale Pod artifact of the Multus DaemonSet rollout race (sandbox created via a transient daemon/shim state mid-rollout; the current, fully-settled daemon's own logs show no ADD for that UID, only a DEL). No defect found in 02-daemonset.yaml or 10-nad-vlan50.yaml; both are unchanged. Independent of that diagnosis, harden the shared guard script per review: print safe interface/address/route diagnostics before every FATAL exit; validate net1 actually carries the workload's expected static /24 address rather than just existing; add arping-based duplicate-address and gateway-reachability checks before installing the net1 default route (exit-code semantics verified against arping's own source). Requires EXPECTED_VLAN50_IP and NET_RAW (for arping's raw ARP sockets) on every consumer - wired into 20-canary.yaml now, qbittorrent.yaml/jdownloader.yaml need the same when they're migrated. Document the canary's Pod lifecycle: restartPolicy: Never means a Failed canary never reruns on its own, and Argo "Synced" only reflects manifest match, not runtime success - recreate it (new UID) after any Multus/CNI change before trusting its result. Recommend keeping it a manually recreated, controller-less Pod rather than a Job/Deployment, since unattended auto-retry risks a duplicate-address race on its static .100 IP - the exact class of bug this diagnosis just walked through. Co-Authored-By: Claude Sonnet 5 --- manifests/multus/20-canary.yaml | 8 +- manifests/multus/README.md | 71 +++++++++++++++- .../multus/vlan50-egress-guard-script.yaml | 80 ++++++++++++++++++- 3 files changed, 156 insertions(+), 3 deletions(-) diff --git a/manifests/multus/20-canary.yaml b/manifests/multus/20-canary.yaml index 1691232..e9a7a45 100644 --- a/manifests/multus/20-canary.yaml +++ b/manifests/multus/20-canary.yaml @@ -55,10 +55,16 @@ spec: value: "10.43.0.0/16" - name: NODE_IP value: "10.10.40.20" + - name: EXPECTED_VLAN50_IP + value: "10.10.50.100" securityContext: capabilities: drop: ["ALL"] - add: ["NET_ADMIN"] + # NET_RAW alongside NET_ADMIN: the guard script's arping-based + # duplicate-address and gateway-reachability checks build raw + # ARP frames, which the kernel gates on CAP_NET_RAW specifically + # — see vlan50-egress-guard-script.yaml's header comment. + add: ["NET_ADMIN", "NET_RAW"] volumeMounts: - name: guard-script mountPath: /scripts diff --git a/manifests/multus/README.md b/manifests/multus/README.md index 870420b..2a6c07f 100644 --- a/manifests/multus/README.md +++ b/manifests/multus/README.md @@ -32,7 +32,76 @@ comment. Nothing here applies to the cluster just because it's merged to and `/var/lib/rancher/k3s/agent/etc/cni/net.d` now has a generated `00-multus.conf`, *then* 10, *then* — only with separate approval — 20. -## Why macvlan bridge, not ipvlan +## Canary lifecycle — re-running it after a Multus/CNI change + +**A canary that already reached `Failed` will never rerun on its own, and +an Argo "Synced" status does not mean it passed.** `20-canary.yaml` sets +`restartPolicy: Never` and has a fixed name (`vlan50-canary`, no owning +Deployment/Job/controller) — deliberately, so nothing ever recreates it +automatically (see "Should this become a Deployment/Job instead?" below +for why that's load-bearing, not incidental). Two consequences that have +each caused real confusion live (2026-08-24), diagnosed from cluster +evidence rather than assumed: + +- Once the Pod object reaches `phase: Failed`, it stays exactly as it is + forever — Kubernetes does not retry a `restartPolicy: Never` Pod, full + stop. Fixing the underlying Multus/CNI DaemonSet does **not** make this + Pod try again; it is inert. The only way to get a fresh attempt is to + delete and recreate the Pod object itself (new UID, new sandbox, new + CNI ADD from scratch). +- Argo reporting `multus` as `Synced` only means the *live Pod manifest + matches what's in Git* — for a bare Pod (not a Deployment), that + comparison is satisfied by the existing (possibly long-Failed) Pod + object as-is. Argo has no reason to touch it, so "Synced" here proves + nothing about whether the canary actually ran successfully, or even + about when it last ran. Don't read Synced as "passing." + +A real example of what this produces if missed: after a DaemonSet fix +lands and rolls out, the *node* briefly passes through an inconsistent +state (old Multus daemon torn down, new one's init containers still +installing plugins, new daemon not started yet — a few seconds to low +tens of seconds depending on image pull/init time). A `FailedCreatePodSandBox` +retry loop that happens to land its next attempt inside that exact +window can produce a Pod that *looks* alive (sandbox created, init +container starts, primary `eth0` interface up) but whose secondary +`net1` attach never actually completed — no `AddedInterface` event for +it, no `k8s.v1.cni.cncf.io/network-status` annotation, the guard script's +own `net1` wait times out. That Pod is now permanently `Failed` and +permanently stale evidence about the *current* DaemonSet state — its logs +and events describe a multus-daemon instance that may no longer exist. +Treating that as proof of a live code defect (rather than recreating the +Pod first) risks chasing a bug that was already fixed. + +**After any change to `02-daemonset.yaml`, `10-nad-vlan50.yaml`, or the +node's CNI binaries**, the only trustworthy re-test is: + +1. Wait for the DaemonSet rollout to actually finish and be Ready — + `kubectl -n kube-system rollout status daemonset/kube-multus-ds`. +2. Delete the old (Failed) canary Pod — `kubectl -n downloads delete pod + vlan50-canary` (or via the Argo UI). +3. Recreate it fresh — `kubectl apply -f manifests/multus/20-canary.yaml` + (or a selective Argo sync of just that resource). +4. Record the **new** Pod UID (`kubectl -n downloads get pod vlan50-canary + -o jsonpath='{.metadata.uid}'`) and evaluate only logs/events tied to + that UID going forward — old aggregated events for the previous UID + are not evidence about this attempt. + +### Should this become a Deployment/Job instead, so it retries itself? + +Deliberately not, at least not while `ipam.type: static` has no +allocator behind it (see above) and this canary is hardcoded to +`10.10.50.100`. A controller that recreates the Pod automatically (a +`Job`, a `Deployment` with `replicas: 1`) would retry through exactly the +kind of transient window described above *without* a human confirming +the previous attempt's netns was actually torn down cleanly first — two +overlapping attempts (a slow-terminating old Pod and a freshly-started +new one) both claiming `.100` on the same L2 segment is precisely the +duplicate-address scenario the guard script's `arping -D` check now +exists to catch, not something to engineer back in via unattended +retries. Keeping this a manually recreated, controller-less Pod means +recreation only ever happens when someone is actually watching the +outcome — that property is worth more here than convenience. If this +changes (e.g. a real per-IP allocator is introduced later), revisit. The managed switch accepts multiple source MAC addresses on nik-debian's port (confirmed in the session that authored this plan — see the diff --git a/manifests/multus/vlan50-egress-guard-script.yaml b/manifests/multus/vlan50-egress-guard-script.yaml index 159406b..fb06a9e 100644 --- a/manifests/multus/vlan50-egress-guard-script.yaml +++ b/manifests/multus/vlan50-egress-guard-script.yaml @@ -21,12 +21,26 @@ # Required env on the init container: VLAN50_GATEWAY, TECHNITIUM_IP, # POD_CIDR, SERVICE_CIDR, NODE_IP (nik-debian's own node IP — # kubelet-originated probe traffic to the pod arrives via the primary -# eth0 gateway and its replies must go back the same way, not out net1). +# eth0 gateway and its replies must go back the same way, not out net1), +# EXPECTED_VLAN50_IP (this workload's static VLAN 50 address, without a +# prefix — .100 for the canary, .10/.11/.12 for qBittorrent/JDownloader/ +# the future browser-vpn-proxy — must match the /24 the NAD's static +# IPAM assigned via the pod's own k8s.v1.cni.cncf.io/networks annotation, +# checked below rather than trusted blindly). # Optional: EXTRA_ETH0_CIDR (space-separated, for any additional # narrowly-required cluster path — leave unset unless a specific need is # identified and reviewed; do not widen this to a blanket 10.0.0.0/8 or # similar). # +# The init container needs NET_RAW in addition to NET_ADMIN (both, drop +# ALL otherwise) — arping (used below for the duplicate-address and +# gateway-reachability checks) builds raw ARP frames over an AF_PACKET +# socket, which the kernel gates on CAP_NET_RAW specifically; NET_ADMIN +# alone is not sufficient and arping fails immediately without it. This +# is a hard runtime dependency of this script now, not optional +# hardening — a pod wiring this ConfigMap in without also granting +# NET_RAW will fail closed at the arping step every time. +# # Corrected from an earlier version after code review: that version # deleted the eth0 default route and then routed pod/service CIDR # directly `dev eth0` with no gateway — remote pod addresses (on other @@ -55,6 +69,18 @@ data: : "${POD_CIDR:?required}" : "${SERVICE_CIDR:?required}" : "${NODE_IP:?required}" + : "${EXPECTED_VLAN50_IP:?required}" + + # Safe to print in full: interface/address/route state only, never + # touches the PIA token, WireGuard keys, or any other credential — + # those live entirely on minisforum's side of the tunnel, not in this + # pod's netns at all. + print_diagnostics() { + echo "[vlan50-egress-guard] diagnostics follow:" >&2 + ip -details link show >&2 || true + ip -4 address show >&2 || true + ip -4 route show table all >&2 || true + } echo "[vlan50-egress-guard] waiting for net1" ready=0 @@ -67,13 +93,65 @@ data: done if [ "${ready}" -ne 1 ]; then echo "[vlan50-egress-guard] FATAL: net1 did not appear after 30s — refusing to start without the VLAN 50 attachment (Multus/NAD required, no eth0-only fallback)" >&2 + print_diagnostics exit 1 fi + echo "[vlan50-egress-guard] validating net1's address: expecting ${EXPECTED_VLAN50_IP}/24" + addr_ok=0 + for _ in $(seq 1 15); do + if ip -4 -o address show dev net1 | awk '{print $4}' | grep -qx "${EXPECTED_VLAN50_IP}/24"; then + addr_ok=1 + break + fi + sleep 1 + done + if [ "${addr_ok}" -ne 1 ]; then + echo "[vlan50-egress-guard] FATAL: net1 does not carry the expected ${EXPECTED_VLAN50_IP}/24 after 15s (interface existing is not enough — the static IPAM result itself has to match what this workload is supposed to have). Actual net1 address(es):" >&2 + ip -4 -o address show dev net1 >&2 || true + print_diagnostics + exit 1 + fi + echo "[vlan50-egress-guard] net1 has ${EXPECTED_VLAN50_IP}/24" + + # "Duplicate or tentative" for IPv4: the kernel's IFA_F_TENTATIVE flag + # (what `ip addr show` reports as "tentative") is an IPv6 DAD concept + # only — the static IPAM CNI plugin used here does no IPv4 DAD of its + # own, so there is no tentative *state* to check for IPv4. What + # actually matters — whether some OTHER host on VLAN 50 already holds + # this address — is checked directly with an ARP probe instead. + echo "[vlan50-egress-guard] checking ${EXPECTED_VLAN50_IP} is not already in use on VLAN 50 (ARP duplicate check)" + if ! arping -D -c 2 -w 2 -I net1 "${EXPECTED_VLAN50_IP}" >/tmp/arping-dup.log 2>&1; then + echo "[vlan50-egress-guard] FATAL: ${EXPECTED_VLAN50_IP} appears to already be in use on VLAN 50 — arping -D got a reply from another host, meaning this address is a duplicate. Refusing to proceed with a conflicting static IP. arping output:" >&2 + cat /tmp/arping-dup.log >&2 || true + print_diagnostics + exit 1 + fi + echo "[vlan50-egress-guard] no duplicate detected for ${EXPECTED_VLAN50_IP}" + + # L2 (ARP) reachability, checked before this gateway is trusted with + # the default route below — deliberately not an ICMP ping instead: + # ARP resolution is a hard prerequisite for delivering any IP packet + # over Ethernet at all, so it is a reliable, low-false-positive proof + # the gateway is actually there; an ICMP probe would additionally + # depend on the gateway choosing to answer echo requests, which many + # routers disable for unrelated security reasons while still routing + # traffic completely normally — that would risk failing this check + # closed for a gateway that works fine. + echo "[vlan50-egress-guard] checking VLAN 50 gateway ${VLAN50_GATEWAY} answers ARP on net1 before trusting it" + if ! arping -c 3 -w 3 -I net1 "${VLAN50_GATEWAY}" >/tmp/arping-gw.log 2>&1; then + echo "[vlan50-egress-guard] FATAL: VLAN 50 gateway ${VLAN50_GATEWAY} did not answer ARP on net1 — refusing to install it as the default route. arping output:" >&2 + cat /tmp/arping-gw.log >&2 || true + print_diagnostics + exit 1 + fi + echo "[vlan50-egress-guard] gateway ${VLAN50_GATEWAY} is reachable" + echo "[vlan50-egress-guard] capturing the original eth0 default gateway" ETH0_GATEWAY=$(ip route show default dev eth0 2>/dev/null | awk '/^default/ {print $3; exit}') if [ -z "${ETH0_GATEWAY}" ]; then echo "[vlan50-egress-guard] FATAL: could not identify eth0's original default gateway — refusing to proceed. Pod/service CIDR routes below need it explicitly (those destinations are not on-link on eth0), and continuing without it would either leave them unreachable or silently do nothing." >&2 + print_diagnostics exit 1 fi echo "[vlan50-egress-guard] eth0 gateway: ${ETH0_GATEWAY}"