feat: strengthen VLAN50 canary egress guard and document its lifecycle

Diagnosed the vlan50-canary FailedCreatePodSandBox->net1-timeout failure
as a stale Pod artifact of the Multus DaemonSet rollout race (sandbox
created via a transient daemon/shim state mid-rollout; the current,
fully-settled daemon's own logs show no ADD for that UID, only a DEL).
No defect found in 02-daemonset.yaml or 10-nad-vlan50.yaml; both are
unchanged.

Independent of that diagnosis, harden the shared guard script per
review: print safe interface/address/route diagnostics before every
FATAL exit; validate net1 actually carries the workload's expected
static /24 address rather than just existing; add arping-based
duplicate-address and gateway-reachability checks before installing
the net1 default route (exit-code semantics verified against arping's
own source). Requires EXPECTED_VLAN50_IP and NET_RAW (for arping's raw
ARP sockets) on every consumer - wired into 20-canary.yaml now,
qbittorrent.yaml/jdownloader.yaml need the same when they're migrated.

Document the canary's Pod lifecycle: restartPolicy: Never means a
Failed canary never reruns on its own, and Argo "Synced" only reflects
manifest match, not runtime success - recreate it (new UID) after any
Multus/CNI change before trusting its result. Recommend keeping it a
manually recreated, controller-less Pod rather than a Job/Deployment,
since unattended auto-retry risks a duplicate-address race on its
static .100 IP - the exact class of bug this diagnosis just walked
through.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

manifests/multus/20-canary.yaml

@@ -55,10 +55,16 @@ spec:
           value: "10.43.0.0/16"
         - name: NODE_IP
           value: "10.10.40.20"
+        - name: EXPECTED_VLAN50_IP
+          value: "10.10.50.100"
       securityContext:
         capabilities:
           drop: ["ALL"]
-          add: ["NET_ADMIN"]
+          # NET_RAW alongside NET_ADMIN: the guard script's arping-based
+          # duplicate-address and gateway-reachability checks build raw
+          # ARP frames, which the kernel gates on CAP_NET_RAW specifically
+          # — see vlan50-egress-guard-script.yaml's header comment.
+          add: ["NET_ADMIN", "NET_RAW"]
       volumeMounts:
         - name: guard-script
           mountPath: /scripts

manifests/multus/README.md

@@ -32,7 +32,76 @@ comment. Nothing here applies to the cluster just because it's merged to
    and `/var/lib/rancher/k3s/agent/etc/cni/net.d` now has a generated
    `00-multus.conf`, *then* 10, *then* — only with separate approval — 20.
 
-## Why macvlan bridge, not ipvlan
+## Canary lifecycle — re-running it after a Multus/CNI change
+
+**A canary that already reached `Failed` will never rerun on its own, and
+an Argo "Synced" status does not mean it passed.** `20-canary.yaml` sets
+`restartPolicy: Never` and has a fixed name (`vlan50-canary`, no owning
+Deployment/Job/controller) — deliberately, so nothing ever recreates it
+automatically (see "Should this become a Deployment/Job instead?" below
+for why that's load-bearing, not incidental). Two consequences that have
+each caused real confusion live (2026-08-24), diagnosed from cluster
+evidence rather than assumed:
+
+- Once the Pod object reaches `phase: Failed`, it stays exactly as it is
+  forever — Kubernetes does not retry a `restartPolicy: Never` Pod, full
+  stop. Fixing the underlying Multus/CNI DaemonSet does **not** make this
+  Pod try again; it is inert. The only way to get a fresh attempt is to
+  delete and recreate the Pod object itself (new UID, new sandbox, new
+  CNI ADD from scratch).
+- Argo reporting `multus` as `Synced` only means the *live Pod manifest
+  matches what's in Git* — for a bare Pod (not a Deployment), that
+  comparison is satisfied by the existing (possibly long-Failed) Pod
+  object as-is. Argo has no reason to touch it, so "Synced" here proves
+  nothing about whether the canary actually ran successfully, or even
+  about when it last ran. Don't read Synced as "passing."
+
+A real example of what this produces if missed: after a DaemonSet fix
+lands and rolls out, the *node* briefly passes through an inconsistent
+state (old Multus daemon torn down, new one's init containers still
+installing plugins, new daemon not started yet — a few seconds to low
+tens of seconds depending on image pull/init time). A `FailedCreatePodSandBox`
+retry loop that happens to land its next attempt inside that exact
+window can produce a Pod that *looks* alive (sandbox created, init
+container starts, primary `eth0` interface up) but whose secondary
+`net1` attach never actually completed — no `AddedInterface` event for
+it, no `k8s.v1.cni.cncf.io/network-status` annotation, the guard script's
+own `net1` wait times out. That Pod is now permanently `Failed` and
+permanently stale evidence about the *current* DaemonSet state — its logs
+and events describe a multus-daemon instance that may no longer exist.
+Treating that as proof of a live code defect (rather than recreating the
+Pod first) risks chasing a bug that was already fixed.
+
+**After any change to `02-daemonset.yaml`, `10-nad-vlan50.yaml`, or the
+node's CNI binaries**, the only trustworthy re-test is:
+
+1. Wait for the DaemonSet rollout to actually finish and be Ready —
+   `kubectl -n kube-system rollout status daemonset/kube-multus-ds`.
+2. Delete the old (Failed) canary Pod — `kubectl -n downloads delete pod
+   vlan50-canary` (or via the Argo UI).
+3. Recreate it fresh — `kubectl apply -f manifests/multus/20-canary.yaml`
+   (or a selective Argo sync of just that resource).
+4. Record the **new** Pod UID (`kubectl -n downloads get pod vlan50-canary
+   -o jsonpath='{.metadata.uid}'`) and evaluate only logs/events tied to
+   that UID going forward — old aggregated events for the previous UID
+   are not evidence about this attempt.
+
+### Should this become a Deployment/Job instead, so it retries itself?
+
+Deliberately not, at least not while `ipam.type: static` has no
+allocator behind it (see above) and this canary is hardcoded to
+`10.10.50.100`. A controller that recreates the Pod automatically (a
+`Job`, a `Deployment` with `replicas: 1`) would retry through exactly the
+kind of transient window described above *without* a human confirming
+the previous attempt's netns was actually torn down cleanly first — two
+overlapping attempts (a slow-terminating old Pod and a freshly-started
+new one) both claiming `.100` on the same L2 segment is precisely the
+duplicate-address scenario the guard script's `arping -D` check now
+exists to catch, not something to engineer back in via unattended
+retries. Keeping this a manually recreated, controller-less Pod means
+recreation only ever happens when someone is actually watching the
+outcome — that property is worth more here than convenience. If this
+changes (e.g. a real per-IP allocator is introduced later), revisit.
 
 The managed switch accepts multiple source MAC addresses on nik-debian's
 port (confirmed in the session that authored this plan — see the

manifests/multus/vlan50-egress-guard-script.yaml

@@ -21,12 +21,26 @@
 # Required env on the init container: VLAN50_GATEWAY, TECHNITIUM_IP,
 # POD_CIDR, SERVICE_CIDR, NODE_IP (nik-debian's own node IP —
 # kubelet-originated probe traffic to the pod arrives via the primary
-# eth0 gateway and its replies must go back the same way, not out net1).
+# eth0 gateway and its replies must go back the same way, not out net1),
+# EXPECTED_VLAN50_IP (this workload's static VLAN 50 address, without a
+# prefix — .100 for the canary, .10/.11/.12 for qBittorrent/JDownloader/
+# the future browser-vpn-proxy — must match the /24 the NAD's static
+# IPAM assigned via the pod's own k8s.v1.cni.cncf.io/networks annotation,
+# checked below rather than trusted blindly).
 # Optional: EXTRA_ETH0_CIDR (space-separated, for any additional
 # narrowly-required cluster path — leave unset unless a specific need is
 # identified and reviewed; do not widen this to a blanket 10.0.0.0/8 or
 # similar).
 #
+# The init container needs NET_RAW in addition to NET_ADMIN (both, drop
+# ALL otherwise) — arping (used below for the duplicate-address and
+# gateway-reachability checks) builds raw ARP frames over an AF_PACKET
+# socket, which the kernel gates on CAP_NET_RAW specifically; NET_ADMIN
+# alone is not sufficient and arping fails immediately without it. This
+# is a hard runtime dependency of this script now, not optional
+# hardening — a pod wiring this ConfigMap in without also granting
+# NET_RAW will fail closed at the arping step every time.
+#
 # Corrected from an earlier version after code review: that version
 # deleted the eth0 default route and then routed pod/service CIDR
 # directly `dev eth0` with no gateway — remote pod addresses (on other
@@ -55,6 +69,18 @@ data:
     : "${POD_CIDR:?required}"
     : "${SERVICE_CIDR:?required}"
     : "${NODE_IP:?required}"
+    : "${EXPECTED_VLAN50_IP:?required}"
+
+    # Safe to print in full: interface/address/route state only, never
+    # touches the PIA token, WireGuard keys, or any other credential —
+    # those live entirely on minisforum's side of the tunnel, not in this
+    # pod's netns at all.
+    print_diagnostics() {
+      echo "[vlan50-egress-guard] diagnostics follow:" >&2
+      ip -details link show >&2 || true
+      ip -4 address show >&2 || true
+      ip -4 route show table all >&2 || true
+    }
 
     echo "[vlan50-egress-guard] waiting for net1"
     ready=0
@@ -67,13 +93,65 @@ data:
     done
     if [ "${ready}" -ne 1 ]; then
       echo "[vlan50-egress-guard] FATAL: net1 did not appear after 30s — refusing to start without the VLAN 50 attachment (Multus/NAD required, no eth0-only fallback)" >&2
+      print_diagnostics
       exit 1
     fi
 
+    echo "[vlan50-egress-guard] validating net1's address: expecting ${EXPECTED_VLAN50_IP}/24"
+    addr_ok=0
+    for _ in $(seq 1 15); do
+      if ip -4 -o address show dev net1 | awk '{print $4}' | grep -qx "${EXPECTED_VLAN50_IP}/24"; then
+        addr_ok=1
+        break
+      fi
+      sleep 1
+    done
+    if [ "${addr_ok}" -ne 1 ]; then
+      echo "[vlan50-egress-guard] FATAL: net1 does not carry the expected ${EXPECTED_VLAN50_IP}/24 after 15s (interface existing is not enough — the static IPAM result itself has to match what this workload is supposed to have). Actual net1 address(es):" >&2
+      ip -4 -o address show dev net1 >&2 || true
+      print_diagnostics
+      exit 1
+    fi
+    echo "[vlan50-egress-guard] net1 has ${EXPECTED_VLAN50_IP}/24"
+
+    # "Duplicate or tentative" for IPv4: the kernel's IFA_F_TENTATIVE flag
+    # (what `ip addr show` reports as "tentative") is an IPv6 DAD concept
+    # only — the static IPAM CNI plugin used here does no IPv4 DAD of its
+    # own, so there is no tentative *state* to check for IPv4. What
+    # actually matters — whether some OTHER host on VLAN 50 already holds
+    # this address — is checked directly with an ARP probe instead.
+    echo "[vlan50-egress-guard] checking ${EXPECTED_VLAN50_IP} is not already in use on VLAN 50 (ARP duplicate check)"
+    if ! arping -D -c 2 -w 2 -I net1 "${EXPECTED_VLAN50_IP}" >/tmp/arping-dup.log 2>&1; then
+      echo "[vlan50-egress-guard] FATAL: ${EXPECTED_VLAN50_IP} appears to already be in use on VLAN 50 — arping -D got a reply from another host, meaning this address is a duplicate. Refusing to proceed with a conflicting static IP. arping output:" >&2
+      cat /tmp/arping-dup.log >&2 || true
+      print_diagnostics
+      exit 1
+    fi
+    echo "[vlan50-egress-guard] no duplicate detected for ${EXPECTED_VLAN50_IP}"
+
+    # L2 (ARP) reachability, checked before this gateway is trusted with
+    # the default route below — deliberately not an ICMP ping instead:
+    # ARP resolution is a hard prerequisite for delivering any IP packet
+    # over Ethernet at all, so it is a reliable, low-false-positive proof
+    # the gateway is actually there; an ICMP probe would additionally
+    # depend on the gateway choosing to answer echo requests, which many
+    # routers disable for unrelated security reasons while still routing
+    # traffic completely normally — that would risk failing this check
+    # closed for a gateway that works fine.
+    echo "[vlan50-egress-guard] checking VLAN 50 gateway ${VLAN50_GATEWAY} answers ARP on net1 before trusting it"
+    if ! arping -c 3 -w 3 -I net1 "${VLAN50_GATEWAY}" >/tmp/arping-gw.log 2>&1; then
+      echo "[vlan50-egress-guard] FATAL: VLAN 50 gateway ${VLAN50_GATEWAY} did not answer ARP on net1 — refusing to install it as the default route. arping output:" >&2
+      cat /tmp/arping-gw.log >&2 || true
+      print_diagnostics
+      exit 1
+    fi
+    echo "[vlan50-egress-guard] gateway ${VLAN50_GATEWAY} is reachable"
+
     echo "[vlan50-egress-guard] capturing the original eth0 default gateway"
     ETH0_GATEWAY=$(ip route show default dev eth0 2>/dev/null | awk '/^default/ {print $3; exit}')
     if [ -z "${ETH0_GATEWAY}" ]; then
       echo "[vlan50-egress-guard] FATAL: could not identify eth0's original default gateway — refusing to proceed. Pod/service CIDR routes below need it explicitly (those destinations are not on-link on eth0), and continuing without it would either leave them unreachable or silently do nothing." >&2
+      print_diagnostics
       exit 1
     fi
     echo "[vlan50-egress-guard] eth0 gateway: ${ETH0_GATEWAY}"