docs: record step 3 as verified live over a full 25.5-hour window
pia_forward_bypass_chains held up against real production conditions, not just a short recheck: 1528 healthy passes, 0 SAFETY DRIFT events, 0 safety_drift classifications since the fix was deployed (2026-09-22 20:37 -> 2026-09-23 22:02). kube-router kept resyncing throughout; the bypass tolerance handled it silently every time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
fb9ecffa9f
commit
80bf24cf0b
@ -599,11 +599,12 @@ external secret broker — not a 24-hour token.
|
||||
- `unknown` (a read command failing) is logged/alerted but never acted
|
||||
on, by design.
|
||||
|
||||
### Rollout status (2026-09-22)
|
||||
### Rollout status (2026-09-23)
|
||||
|
||||
Step 1 (baseline) and step 2 (deploy with timers off) are done and
|
||||
verified live. Step 3 (enable the recovery timer) was attempted, caused
|
||||
a real outage, and has been fixed but not yet re-verified live.
|
||||
Steps 1-3 are done and verified live. Step 3 required two attempts —
|
||||
the first caused a real (safely fail-closed) outage, root-caused and
|
||||
fixed same-day; the second was verified against a full day of real
|
||||
production conditions, not just a short window.
|
||||
|
||||
- **Step 2, attempt 1** failed: `pia-gateway register` correctly refused
|
||||
with `safety invariants failing ... FORWARD jump to kill-switch chain
|
||||
@ -645,10 +646,29 @@ a real outage, and has been fixed but not yet re-verified live.
|
||||
full `pia-gateway-recover`/`pia-killswitch-reassert` journals.
|
||||
Mitigated live: `sudo systemctl stop pia-gateway-recover.timer` +
|
||||
manual `sudo systemctl restart wg-quick@pia-wg`. Root-caused and fixed
|
||||
— see the "reassert timer alone wasn't the full fix" addendum above
|
||||
(`pia_forward_bypass_chains`, content-verified live on every check).
|
||||
**Not yet re-applied or re-verified live** — `pia-gateway-recover
|
||||
.timer` is currently stopped on minisforum pending that.
|
||||
same-day — see the "reassert timer alone wasn't the full fix" addendum
|
||||
above (`pia_forward_bypass_chains`, content-verified live on every
|
||||
check).
|
||||
- **Step 3, re-attempt**, with the bypass fix deployed (`ansible-playbook
|
||||
... -K -J`; config re-rendered with `forward_bypass_chains:
|
||||
["KUBE-ROUTER-FORWARD"]`, `rewrite-config` correctly re-rendered
|
||||
`pia-wg.conf` from the config-only change without a PIA call since a
|
||||
baseline already existed, recovery timer re-enabled). Left running
|
||||
unattended for a full **~25.5 hours** before re-checking, deliberately
|
||||
— the original incident took several minutes and multiple cycles to
|
||||
manifest, so a short recheck window would have proven nothing.
|
||||
Confirmed via exact counts, not inference: `journalctl -u
|
||||
pia-gateway-recover --since "2026-09-22 20:37:00"` showed **1528**
|
||||
`class=healthy` passes (matches the expected ~1530 at 60s cadence —
|
||||
the timer ran continuously the whole window), **0** `SAFETY DRIFT`
|
||||
lines, **0** `class=safety_drift` classifications. `pia-gateway
|
||||
status` at recheck time: `healthy`, `safety_ok: True`,
|
||||
`emergency_close_active: False`, `consecutive_failures: 0`,
|
||||
`last recovery: 1d ago restart` — the last restart was the one from
|
||||
*before* this fix; none has been needed since. **Step 3 is done.**
|
||||
kube-router did keep resyncing during this window (that's the whole
|
||||
premise of the fix); the bypass tolerance handled it silently every
|
||||
time, with no observable effect on the tunnel.
|
||||
|
||||
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
|
||||
— the original `head -3` suggestion is what hid the first issue; the
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user