Found live, 2026-09-22, minutes after first enabling
pia-gateway-recover.timer (rollout step 3): a real, ~10-minute outage
loop. pia-gateway-recover and pia-killswitch-reassert are two
independently-scheduled 60s timers with no coordination. kube-router's
actual resync cadence turned out to be ~60-90s on this cluster - far
more frequent than the "boot-scale, rare" assumption behind the
previous reassert-timer fix - so a recover pass would periodically
sample iptables -S FORWARD in the window between a kube-router
push-down and the next reassert fix, correctly detect real (if
momentary) drift, and call emergency_close(), which unconditionally
forces table pia's route to unreachable AND resets
consecutive_failures to 0 in the same step. Journal timeline:
19:38:58 3rd unhealthy check -> restart
19:39:09 recovered - route open, handshake fresh
19:40:03 drift again -> emergency close -> route forced closed,
counter reset to 0
19:41:08 drift again -> emergency close again
VLAN 50 was open ~10s out of every 60-90s cycle. Confirmed via the
alert-bridge Discord notifications (working end to end, first real
trigger) and the full pia-gateway-recover/pia-killswitch-reassert
journals. Mitigated live: stopped pia-gateway-recover.timer, manually
restarted wg-quick@pia-wg to reopen the route.
This wasn't a detection bug - the drift was real. It was that treating
every momentary reordering as a full safety emergency was stricter
than the actual invariant required, given kube-router's own resync
cadence on this host - and we'd already proven (reading
KUBE-ROUTER-FORWARD's actual live content) that this specific,
recurring reordering cannot affect VLAN 50 traffic regardless of
position, since every rule in it is scoped to specific pod /32
addresses.
Adds pia_forward_bypass_chains (Classifier
._forward_jump_effectively_first): the FORWARD-jump safety check now
tolerates a configured, named chain sitting ahead of our jump, but
only for as long as that chain's OWN current content re-verifies -
live, on every single check, never a one-time trust - as structurally
unable to match 10.10.50.0/24 (every -s/-d value on every rule
disjoint from our subnet; a bare/unconditional rule, a negated match,
or any construct not explicitly recognized fails closed). Scoped to
KUBE-ROUTER-FORWARD only, not DOCKER-USER/DOCKER-FORWARD - dockerd
sets its hooks up once at daemon start, not on a recurring cycle, and
DOCKER-FORWARD itself jumps to three further nested chains never
fully content-verified, so it stays a hard drift trigger. An
unrecognized chain, or a bypass-listed chain whose content stops
verifying safe, still fails closed and emergency-closes exactly as
before.
Added 11 tests (105 total, all passing): the exact live topology
tolerated when bypass-configured and still blocked by default,
duplicate kube-router jumps (the exact live anomaly) tolerated, a
bypass chain picking up a matching source/bare rule/negated match
still blocking, an unrecognized chain still blocking even with a
bypass list configured, a FORWARD-level rule with a real matcher never
treated as a bypass candidate, a missing bypass chain blocking, and
two end-to-end cmd_recover tests confirming no emergency close on the
tolerated case and a real emergency close still firing on a genuine
drift.
pia-gateway-recover.timer is currently stopped on minisforum pending
redeployment and live re-verification of this fix (rollout step 3 is
not yet complete).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
38 lines
2.1 KiB
Django/Jinja
38 lines
2.1 KiB
Django/Jinja
{
|
|
"interface": {{ pia_wg_interface | to_json }},
|
|
"config_dir": {{ pia_wg_config_dir | to_json }},
|
|
"state_dir": {{ pia_gateway_state_dir | to_json }},
|
|
"credentials_file": {{ (pia_gateway_etc_dir + '/credentials.env') | to_json }},
|
|
"ca_file": {{ (pia_wg_config_dir + '/pia-ca.crt') | to_json }},
|
|
"lock_file": "/run/lock/pia-gateway.lock",
|
|
"route_table": {{ pia_route_table_name | to_json }},
|
|
"rule_priority": {{ pia_rule_priority | int }},
|
|
"vlan50_subnet": {{ pia_vlan50_subnet | to_json }},
|
|
"iptables_chain": {{ pia_iptables_chain | to_json }},
|
|
"lan_interface": {{ pia_lan_interface | to_json }},
|
|
"forward_bypass_chains": {{ pia_forward_bypass_chains | to_json }},
|
|
"mss_clamp_enabled": {{ pia_mss_clamp_enabled | bool | to_json }},
|
|
"region_candidates": {{ pia_region_candidates | to_json }},
|
|
"handshake_max_age_sec": {{ pia_healthcheck_handshake_max_age_sec | int }},
|
|
"failure_threshold": {{ pia_recovery_failure_threshold | int }},
|
|
"startup_grace_sec": {{ pia_recovery_startup_grace_sec | int }},
|
|
"restart_wait_sec": {{ pia_recovery_restart_wait_sec | int }},
|
|
"activation_wait_sec": {{ pia_recovery_activation_wait_sec | int }},
|
|
"recovery_max_duration_sec": {{ pia_recovery_max_duration_sec | int }},
|
|
"backoff_schedule_sec": {{ pia_recovery_backoff_schedule_sec | to_json }},
|
|
"backoff_jitter_fraction": 0.1,
|
|
"max_candidates_per_cycle": {{ pia_recovery_max_candidates_per_cycle | int }},
|
|
"max_rotation_candidates": {{ pia_recovery_max_rotation_candidates | int }},
|
|
"renewal_after_days": {{ pia_renewal_after_days | int }},
|
|
"http_connect_timeout_sec": {{ pia_recovery_http_connect_timeout_sec | int }},
|
|
"http_total_timeout_sec": {{ pia_recovery_http_total_timeout_sec | int }},
|
|
"clock_skew_allowance_sec": 60,
|
|
"backup_keep": {{ pia_recovery_backup_keep | int }},
|
|
"lock_wait_sec": {{ pia_recovery_lock_wait_sec | int }},
|
|
"alert_bridge_url": {{ pia_alert_bridge_url | to_json }},
|
|
"alert_source": {{ pia_alert_source | to_json }},
|
|
"metrics_file": {{ (pia_gateway_state_dir + '/metrics.prom') | to_json }},
|
|
"status_file": {{ (pia_gateway_state_dir + '/status.json') | to_json }},
|
|
"killswitch_unit": "pia-killswitch.service"
|
|
}
|