pia_forward_bypass_chains held up against real production conditions,
not just a short recheck: 1528 healthy passes, 0 SAFETY DRIFT events,
0 safety_drift classifications since the fix was deployed
(2026-09-22 20:37 -> 2026-09-23 22:02). kube-router kept resyncing
throughout; the bypass tolerance handled it silently every time.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:
19:38:58 3rd unhealthy check -> restart
19:39:09 recovered - route open, handshake fresh
19:40:03 drift again -> emergency close -> route forced closed,
counter reset to 0
19:41:08 drift again -> emergency close again
VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.
This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.
Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.
Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.
pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live, 2026-09-22: after fixing pia-killswitch.sh's FORWARD-jump
repositioning and getting one pia-gateway register attempt past the
safety check, the very next attempt - roughly an hour later, no
reboot, no UFW reload, no script content change in between - hit the
identical "FORWARD jump to kill-switch chain is not first" failure
again. This disproves the earlier assumption that kube-router/dockerd
reassert their own position-1 jumps only at boot/restart scale:
kube-router resyncs on ordinary Kubernetes API events, and this
cluster's CronJobs (ca-sync, porkbun-ddns, gitea-backup) each
create/delete a Pod every few minutes - plausible triggers for
kube-router's own reconciliation loop, and therefore plausible
triggers for this drift, recurring on a timescale of minutes.
The reposition logic itself (previous commit) is correct - it just
only ran on the existing triggers (boot, UFW reload, an
Ansible-detected script content change), none of which fired between
the two attempts.
Adds pia-killswitch-reassert.timer/.service: re-runs the already fully
idempotent pia-killswitch.sh every pia_healthcheck_interval_sec (same
cadence as the health check), giving it regular chances to run instead
of only sporadic ones. Deliberately separate from
pia-gateway-recover.timer and NOT gated by pia_auto_recovery_enabled -
this is the kill switch continuing to assert its own already-
established, narrowly-scoped state (source rule, table default,
FORWARD jump position), the same kind of thing it already does
unconditionally at boot and after a UFW reload, just on a schedule
too - not the higher-level automatic recovery/registration logic the
plan was deliberately cautious about running unattended.
Unblocked live in the meantime with a manual
`sudo systemctl restart pia-killswitch.service`.
Offline checks: yamllint, ansible-playbook --syntax-check, rendered
templates (Jinja2 + bash -n on the updated script), ansible-lint
(diffed against main - no new finding categories), full pia_gateway.py
test suite (94/94, unaffected by this change) all pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live, 2026-09-22, on the addKey step of the first successful
kill-switch-check pass: every candidate failed with
kind=tls detail="certificate verification failed". Root-caused on
minisforum: PIA's own CA certificate (files/pia-ca.crt) has a CA:TRUE
Basic Constraints extension that isn't marked critical - a real RFC
5280 non-conformance, confirmed not a stale vendored copy (a fresh
fetch from pia-foss/manual-connections is byte-identical, same
sha256 already recorded in the file's own header, same defect).
Recent OpenSSL (3.2+; confirmed via a direct repro against
minisforum's 3.5.7 - SSLCertVerificationError, verify_message
"Basic Constraints of CA cert not marked critical") rejects this by
default, a check promoted from X509_STRICT-only into the baseline
chain-verification path. openssl s_client (LibreSSL, this dev
machine) and curl both accept the identical cert/chain/CA file - only
Python's ssl module, via this host's system OpenSSL, rejects it. This
went unnoticed until now because the three prior successful
registrations all went through curl --cacert (the legacy Ansible
path); this was the first time the Python ssl code path ran for real.
Python's stdlib ssl module has no custom verify-callback hook, so
there's no clean way to tolerate just this one known,
third-party-controlled defect while keeping full validation for
everything else. Rather than weakening verify_mode or adding a new
dependency (pyOpenSSL, for a real callback - considered, rejected as
a bigger lift than necessary) to route around a gap in a library we
don't control, addKey now shells out to curl
(HttpClient.get_pinned_curl, curl added to this role's apt list) -
the same tool PIA's own reference client already uses for this exact
endpoint, with --cacert and never -k/--insecure, so this is a
different, already-proven-correct TLS backend for the one endpoint
that needs it, not a downgrade. serverlist/token are unaffected
(public WebPKI certs) and still use the stdlib ssl path.
Verified: a real subprocess curl call against the actual failing
candidate (Server-11669-10a / 69.33.198.97) gets a genuine HTTP
response from PIA, proving TLS validation succeeds via this path.
Also found and handled: curl's own exit code for a --fail-triggered
rejection isn't fully portable (macOS/SecureTransport curl reports 56
where GNU/OpenSSL curl reports 22 for the same case) - classification
now also matches curl's own error text as a second signal, not just
the exit code.
Added 8 tests (94 total, all passing) covering the curl invocation
shape, stdin token delivery (no trailing newline, never in argv), and
exit-code + text-based error classification including the
cross-platform exit-code case found live.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_request()'s ssl.SSLCertVerificationError handler hardcoded "certificate
verification failed" regardless of cause, discarding e.verify_message/
e.reason (standard OpenSSL diagnostic text - never a secret) - exactly
what's needed to tell a wrong CA, expired cert, clock skew, and a
hostname mismatch apart. Found while diagnosing a live addKey TLS
failure on minisforum that doesn't reproduce from this machine (same
CA file, same pinned IP, same SNI hostname - openssl s_client and a
direct repro of get_pinned() both succeed here), so the next attempt
needs the real reason, not a collapsed generic string. Also stopped
discarding detail on the generic SSLError/OSError branches.
Verified: forcing a real verification failure (wrong CA) now reports
"unable to get local issuer certificate" instead of the old generic
text. Full test suite still passes (86/86).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live on the first deployment of the auto-recovery controller
(2026-09-22): pia-gateway register correctly refused with "FORWARD
jump to kill-switch chain is not first". Root cause confirmed on
minisforum via iptables -S FORWARD/-S KUBE-ROUTER-FORWARD/-S
DOCKER-USER/-S DOCKER-FORWARD: the PIA-VLAN50 jump was 4th, behind
kube-router's and dockerd's own jumps, which each independently
reassert position 1 on their own resyncs/restarts (kube-router had
even left a duplicate of its own rule behind). pia-killswitch.sh.j2
only checked whether its jump existed (iptables -C) before deciding
not to touch anything — existence isn't proof of position, so once
pushed down it stayed down.
Confirmed safe to reposition unconditionally by reading the actual
content of what's ahead of it rather than assuming: KUBE-ROUTER-FORWARD
only matches specific pod /32 addresses, DOCKER-USER was empty, and
DOCKER-FORWARD's one direct rule is scoped to -i docker0 (VLAN 50
arrives via enp1s0) - none can intercept VLAN 50 traffic regardless of
order. Fixed the script itself (delete + reinsert at position 1 if the
current first rule isn't ours) rather than loosening the Python
controller's forward_jump_first safety check, which was working
correctly when it refused to register against the drifted state.
Verified offline: rendered template passes bash -n, and the exact
live topology (rule present but 4th) reproduced against a bash mock
of iptables, confirming reposition-when-drifted, idempotency when
already first, and insert-when-absent all behave correctly. No real
iptables/wg/ip on this dev machine to test further; not yet
re-applied or re-verified against minisforum.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Serialize every mutating command (recover, renew, register,
rewrite-config, pause, resume, state-reset) on the shared lock,
acquired BEFORE state is read rather than after: reading first and
locking only around the write let a slower command silently clobber
whatever a faster one wrote in between — in particular, an operator
pause could be lost to an in-flight recovery pass that read state
before the pause ran and only saved its own copy back afterwards.
- Fix emergency_close() to reposition its VLAN 50 DROP to the first
effective FORWARD position when it exists but has been pushed down
by something else (e.g. an out-of-band ufw reload) — a bare `-C`
existence check said nothing about position.
- Tighten wait_for_verified()'s handshake freshness check from >= to >
since_epoch: wg show and since_epoch are both whole seconds, so a
handshake at exactly the same second as since_epoch is
indistinguishable from a stale, pre-activation one and must not
count as proof of a new one.
- Fix the Ansible config-validation task to fail on any exit code
outside status's three documented health exits (0/1/2), not just on
5 — verified live against the real program (no wg/ip/iptables on
this dev machine reproduces the "nothing configured yet" first-run
case; status correctly returns 1, and broken/missing config.json
correctly returns 5 and is now caught).
- Correct the live fault-injection drill in the role README: port
1337 is only the addKey registration API port, not the tunnel's own
WireGuard UDP endpoint (derive that from `wg show pia-wg endpoints`
instead), and make the packet-capture check direction-aware
(`tcpdump -Q out`) so it can't conflate a real egress leak with
VLAN 50's normal inbound arrival on enp1s0.
Added 6 tests (86 total, all passing): emergency-close repositioning
past a preceding ACCEPT, pause/resume blocking on an in-flight
lock holder and surviving into a subsequent recovery pass, and the
handshake boundary at exactly/one-second-either-side of since_epoch.
yamllint, ansible-playbook --syntax-check, and ansible-lint (diffed
against main — no new finding categories) all pass. No live changes;
nothing pushed to main.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Job hunt is over; point the portfolio domain at the personal site via a
302 Traefik RedirectRegex middleware instead of the rate-limit
middleware. Portfolio Deployment/Service/Certificate are untouched so
this is easy to revert.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same pattern as portfolio: Namespace, Deployment (pulls
gitea.nik4nao.com/nik/me.nik4nao.com:latest), Service, Let's Encrypt
Certificate, and Traefik IngressRoute for me.nik4nao.com, reconciled
by a new Argo CD Application. Includes the per-namespace registry
pull-secret script (run manually against the live cluster, not
committed as plaintext).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KS514CDC88M4wDwMLnS7u
Filebar sends its auth only on the legacy X-Emby-Authorization header,
with unquoted values (Client=Filebar instead of Client="Filebar"), and
never sends the Authorization header Jellyfin 12.0 requires to resolve
request.App -- causing every login to 400. Switches the debug proxy to
OpenResty and adds a Lua shim that rewrites just Filebar's requests into
the header format Jellyfin expects (matching SenPlayer's working
requests); everything else passes through unchanged.
This is a workaround for a Filebar bug, not a permanent fix -- see the
TODO in jellyfin-header-debug.yaml to remove it once Filebar ships a
corrected release.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016EQrGkfXFae4PZG9QfPRf4
Adds a transparent nginx passthrough that logs auth-related request
headers (Authorization, X-Emby-Authorization, X-Emby-Token,
X-MediaBrowser-Token, User-Agent) to diagnose why the Filebar iOS app
gets 400s on /Users/AuthenticateByName against Jellyfin 12.0. All
traffic still reaches the real jellyfin Service unchanged. Revert the
Ingress backend and remove jellyfin-header-debug.yaml once captured.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016EQrGkfXFae4PZG9QfPRf4
10.11.11 can't read the schema Jellyfin 12.0 already migrated the config
DB to (SQLite errors on every user query), which broke login entirely.
Pin to 12.0 instead of :latest to restore service and stop future
silent version jumps.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016EQrGkfXFae4PZG9QfPRf4
Enabling MSS clamping exposed a general hazard: systemctl restart
wg-quick@pia-wg tears down the live interface using whatever PreDown
rules are on disk right now, not what the interface was actually
brought up with. If Ansible had already rewritten pia-wg.conf with a
changed rule, the live interface (built under the old rules) won't
have whatever the new PreDown line tries to delete - iptables -D fails,
wg-quick's own set -e aborts the whole down/up sequence, and the
interface is left orphaned (never reaches its own built-in ip link
delete step), which then makes the following wg-quick up fail too with
"already exists".
Confirmed against wg-quick's real source (execute_hooks() runs each
hook via `(eval "$hook")`) that appending `2>/dev/null || true` to
each PreDown iptables -D line makes wg-quick treat it as succeeded
regardless of prior state - verified by rendering the template with
Jinja2 directly before trusting it against the live host. This
prevents the failure mode entirely for any future PostUp/PreDown
content change, not just this one.
Add INCIDENT-2026-08-25-vlan50-pia-qbittorrent.md covering all four
issues from this troubleshooting window: the zombie WireGuard tunnel,
a qBittorrent 5.2.0 stale-lock crash loop, the PMTU black hole this
fix addresses, and the restart failure hit while deploying it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
UFW allowed 53/udp+tcp from every other internal subnet but not from
10.10.0.0/24 (the WireGuard client subnet), so once the tunnel itself
was reachable, decrypted DNS queries still got dropped at minisforum's
own INPUT chain before reaching Technitium.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
AllowedIPs for the mac road-warrior peer was missing 10.10.40.0/24, so
DNS (10.10.40.53) and other home-LAN hosts were unreachable over the
tunnel. Also fixes the same string-vs-bool `when:` failure already
patched in e757850 (recent ansible-core rejects a `-e ...=true` CLI
string in a boolean conditional) for the client-config display tasks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the NodePort (30889) with type: LoadBalancer on the standard
SOCKS5 port 1080, at the user's request for a more natural port than a
30000+ NodePort value. loadBalancerIP: 10.10.40.20 (nik-debian) matches
values/gitea.yaml's gitea-ssh Service pattern exactly.
Documented the actual behavior after checking, not assuming: k3s's
ServiceLB (Klipper) binds the port on every node via a DaemonSet by
default - loadBalancerIP only sets the reported external IP, it
doesn't scope which nodes bind the port. That needs a separate
enablelb/lbpool node-label mechanism this repo doesn't use anywhere
yet (including on the existing gitea-ssh Service, which likely has the
same characteristic, unnoticed until now). So this change doesn't
narrow exposure to nik-debian only - it's still reachable at any node
IP on port 1080, the same multi-node reachability the NodePort version
already had, just a different port number.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
manifests/media/gluetun-proxy.yaml (the standalone Gluetun/OpenVPN
HTTP proxy, NodePort 30888) is now fully replaced by
manifests/multus/30-browser-vpn-proxy.yaml, live and validated
end-to-end (connectivity, DNS-through-proxy, PIA egress, fail-closed
kill switch). Removing the file from git - the live Deployment/Service
still need a separate kubectl delete, since media has prune: false.
Note gluetun.home.arpa (the dashy VPN-IP-banner hostname, and the
Certificate/IngressRoute in qbittorrent.yaml routing it to that pod's
ip-reporter sidecar) is unrelated and unaffected - it never pointed at
this standalone proxy.
Update the root README's VLAN 50 status table with browser-vpn-proxy
and this removal, including the exact commit to restore the old file
from if ever needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
sync-wave 3 (after the canary's wave 2) was a documentation convention,
not a real dependency, and it broke syncing entirely: this cluster's
argocd-cm ignores /status updates for reconciliation cluster-wide
(resource.customizations.ignoreResourceUpdates.all), so Argo's wave
orchestrator can never observe a freshly recreated Pod's Pending ->
Running transition happening during an active sync - exactly what the
canary does every test round, since it's deleted and recreated each
time. The sync hung forever on "waiting for healthy state of
/Pod/vlan50-canary" even once the pod was actually Running live.
Move browser-vpn-proxy to wave 1, alongside the NAD - the only thing
it actually needs to exist first. Document the root cause and the
kubectl patch to clear a stuck operation in the README, since this
risk applies to anything synced after the canary in this Application,
not just this one file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous commit (89524d4) only captured the file rename - a git
add with one stale/invalid pathspec aborted before staging the actual
content, and the resulting empty diff wasn't caught before committing.
This commit has the real changes described in that one's message:
EXPECTED_VLAN50_IP/NET_RAW on the guard init container, REQUIRE_AUTH
dropped to false with ALLOWED_IPS removed entirely (broken as drafted
- see the file's own header for why), sync-wave 3 annotation, and the
README updates dropping the now-unneeded secret-script references.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move manifests/multus/reserved/browser-vpn-proxy.yaml into the active
multus tree (sync-wave 3) now that qBittorrent and JDownloader have
both independently passed their canary, validation, and soak gates -
the precondition this file's own header already called for.
Add EXPECTED_VLAN50_IP/NET_RAW to its guard init container, matching
the fix qBittorrent/JDownloader/canary all needed for the arping-based
checks to run.
Deliberately no auth (REQUIRE_AUTH=false) and no ALLOWED_IPS, after
discussion: access restriction relies on the home network's own
firewall (Trusted/Lab -> Lab NodePort ALLOW, other VLANs DENY except
Technitium DNS), not an app-level control. Along the way, found and
avoided shipping a real bug in the original draft - go-socks5-proxy's
ALLOWED_IPS parses values with net.ParseIP (not CIDR-aware) and
matches with exact net.IP.Equal(), so the drafted "10.10.40.0/24"
value would have produced a nil whitelist entry that matches no real
client, rejecting every connection. This library has no way to express
a subnet allowlist at all - removed rather than left silently broken.
Also drops the now-unneeded browser-vpn-proxy-credentials Secret and
its companion script/README references.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
qbittorrent.yaml and jdownloader.yaml's own header comments still
pointed at manifests/media/rollback/*-gluetun.yaml and described the
migration as pending approval/preconditions - both are live and
validated now, and that directory is being removed (no kept on-disk
Gluetun copy). Rewrite both headers as a status note instead, pointing
rollback at git history (with the exact pre-migration commit) to match
the root README's rollback table, which gets the same exact-commit
treatment here instead of the vaguer reference from the previous
commit.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both the root README and manifests/README were still written as if the
whole VLAN 50 migration was drafted-but-unapplied; it's now fully live
and independently validated end to end (canary matrix, both media
workloads' own fail-closed kill-switch test). Update the status line
and PIA region accordingly.
Also drop references to manifests/media/rollback/*-gluetun.yaml, which
is being removed now that qBittorrent/JDownloader are confirmed
working - rollback guidance now points at git history instead of a
kept-file copy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The previous ip6tables->sysctl fix (26894d1) was itself wrong: writing
/proc/sys/net/ipv6/conf/all/disable_ipv6 failed live with "Read-only
file system" despite NET_ADMIN and a passing [ -w ] check - the
container runtime mounts /proc/sys read-only by default regardless of
capabilities, independent of file permission bits. Making it writable
needs either kubelet's securityContext.sysctls (and
net.ipv6.conf.*.disable_ipv6 isn't on its default safe-sysctls
allowlist, so that means a node-level --allowed-unsafe-sysctls flag)
or securityContext.procMount: Unmasked (which needs pod-level user
namespaces) - too much blast radius for one pod's IPv6 kill switch.
nft (nftables) is genuinely present in the same image (the "nftables"
apk package, installed alongside "iptables" but not "ip6tables") and
handles the ip6 address family without a separate binary, so it needs
no image or capability change. Verified by actually running the exact
commands against the real pinned nicolaka/netshoot:v0.11 image
(digest-matched to what's on nik-debian), not just against docs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
nicolaka/netshoot:v0.11 (the pinned tag actually in use) is built on
Alpine 3.18.0, where ip6tables is packaged separately from iptables;
this image's Dockerfile only installs the latter, so ip6tables does
not exist in the container at all (confirmed live: "ip6tables: not
found", and against Alpine's own v3.18 package index, not master's).
Write 1 to /proc/sys/net/ipv6/conf/all/disable_ipv6 instead - per the
kernel's own ip-sysctl docs this is equivalent to also setting
conf/default/disable_ipv6 and every existing per-interface
disable_ipv6 (lo/eth0/net1) at once, and is strictly stronger than an
iptables DROP policy would have been: no IPv6 activity happens on any
interface at all, not just filtered output. Needs no capability beyond
the NET_ADMIN the guard init container already has.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Diagnosed the vlan50-canary FailedCreatePodSandBox->net1-timeout failure
as a stale Pod artifact of the Multus DaemonSet rollout race (sandbox
created via a transient daemon/shim state mid-rollout; the current,
fully-settled daemon's own logs show no ADD for that UID, only a DEL).
No defect found in 02-daemonset.yaml or 10-nad-vlan50.yaml; both are
unchanged.
Independent of that diagnosis, harden the shared guard script per
review: print safe interface/address/route diagnostics before every
FATAL exit; validate net1 actually carries the workload's expected
static /24 address rather than just existing; add arping-based
duplicate-address and gateway-reachability checks before installing
the net1 default route (exit-code semantics verified against arping's
own source). Requires EXPECTED_VLAN50_IP and NET_RAW (for arping's raw
ARP sockets) on every consumer - wired into 20-canary.yaml now,
qbittorrent.yaml/jdownloader.yaml need the same when they're migrated.
Document the canary's Pod lifecycle: restartPolicy: Never means a
Failed canary never reruns on its own, and Argo "Synced" only reflects
manifest match, not runtime success - recreate it (new UID) after any
Multus/CNI change before trusting its result. Recommend keeping it a
manually recreated, controller-less Pod rather than a Job/Deployment,
since unattended auto-retry risks a duplicate-address race on its
static .100 IP - the exact class of bug this diagnosis just walked
through.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
k3s deliberately ships only the plugins its own flannel networking
needs (bandwidth, bridge, firewall, flannel, host-local, loopback,
portmap — confirmed against k3s's own build script; macvlan's absence
is an intentional upstream scope decision, k3s-io/k3s#9224). Multus's
macvlan delegate for the vlan50 NAD therefore had nothing to exec even
after the binDir/mount fixes.
Add a cni-plugins init container mirroring Rancher's own official
rke2-multus Helm chart (the fix k3s's own docs recommend for this
exact gap): rancher/hardened-cni-plugins, digest-pinned, writing into
our existing cnibin mount instead of the image's hardcoded
/opt/cni/bin, with SKIP_CNI_BINARIES=flannel so it doesn't overwrite
k3s's already-working flannel plugin.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
binDir alone (previous commit) wasn't sufficient: k3s's plugin bin dir
is a directory of symlinks into a content-hash-named sibling directory,
and the narrow cnibin mount didn't cover that sibling. FindInPath's
os.Stat follows symlinks, so it failed inside the container even though
the same path resolves fine on the real host. Mount the parent
(k3s-cni-data) into the main container instead so the symlinks resolve;
the init container keeps the narrower cnibin mount since it only writes
files and never follows them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Root cause: Multus's ChrootExec resolves a delegate plugin's absolute
path from the daemon container's own filesystem view, then execs that
exact path string after chrooting into the real host root. binDir
defaulted to /opt/cni/bin, which is valid inside the container (bind-
mounted to k3s's real bin dir) but doesn't exist on the actual host, so
every pod attaching to the VLAN 50 NAD failed FailedCreatePodSandBox
trying to delegate to flannel. Confirmed against multus-cni's source at
the exact pinned v4.3.0 tag, not assumed. Sets binDir explicitly to
k3s's real path and matches the cnibin mount path to it on both sides of
the chroot boundary (main container and the install_multus init
container).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CRD, RBAC, thick-plugin DaemonSet (nik-debian only, k3s CNI paths,
digest-pinned), the VLAN 50 NAD (macvlan bridge on enp1s0.50), the
shared pod egress-guard script, a temporary canary pod, and the
not-yet-deployed browser-vpn-proxy workload (kept in reserved/, outside
this Application's non-recursive source path).
Referenced by argocd/apps/multus.yaml (already pushed) but not deployed
by it — that Application has no syncPolicy.automated, so this still
needs an explicit selective sync per resource.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Points at manifests/multus/ — deliberately no syncPolicy.automated, so
app-of-apps picking this up only registers the Application; nothing
under it deploys until selectively synced by hand.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Creates enp1s0.50 on nik-debian (no L3 address, VLAN 40/node IP/default
route untouched) for Multus to later attach macvlan/ipvlan workloads to.
Verified live: enp1s0.50 is up with no IPv4 address (only the automatic
IPv6 link-local, which is expected and harmless), default route
unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Registers minisforum as a PIA WireGuard peer for VPN VLAN 50, with a
boot-ordered kill switch (dedicated PIA-VLAN50 iptables chain + a
terminal unreachable route in a dedicated routing table), multi-region
addKey fallback (Hong Kong -> Taiwan -> JP Tokyo, each region's full
server list, in order), and an observability-only health check.
Verified live against minisforum: registration succeeds, wg-quick@pia-wg
is up.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds explicit UFW routed-allow rules (10.42.0.0/16 pod-to-pod, pod-to-Technitium
DNS) since these nodes default their routed/FORWARD policy to DROP. Also brings
nik-debian's NFS (2049) and SMB (445) ports under Ansible-managed UFW rules via
the existing nfs-server role, scoped to the Lab/Trusted networks that need them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Set replicaCount: 0 for pihole and pihole-debian so Argo CD's selfHeal
doesn't scale them back up once its git connectivity recovers. Live
deployments already scaled to 0 directly; this just keeps git in sync
with that so it sticks.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
minisforum, debian, mac-mini, and gpu-node all moved from
192.168.7.0/24 to 10.10.40.0/24. Updates K3s server/agent config and
node IPs (including gpu-node's host_vars override), NFS export
allow-list and exports template, Pi-hole DNS records and
kube-vip/loadBalancerIP pins, WireGuard's pushed DNS/AllowedIPs, and
the NFS server IP baked into Jellyfin/Kavita/gitea-backup PVs and the
Ollama URL used by ai-gateway/Dashy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>