docs: record step 3 as verified live over a full 25.5-hour window

pia_forward_bypass_chains held up against real production conditions,
not just a short recheck: 1528 healthy passes, 0 SAFETY DRIFT events,
0 safety_drift classifications since the fix was deployed
(2026-09-22 20:37 -> 2026-09-23 22:02). kube-router kept resyncing
throughout; the bypass tolerance handled it silently every time.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Nik Afiq 2026-09-23 22:09:13 +09:00
parent fb9ecffa9f
commit 80bf24cf0b

View File

@ -599,11 +599,12 @@ external secret broker — not a 24-hour token.
- `unknown` (a read command failing) is logged/alerted but never acted - `unknown` (a read command failing) is logged/alerted but never acted
on, by design. on, by design.
### Rollout status (2026-09-22) ### Rollout status (2026-09-23)
Step 1 (baseline) and step 2 (deploy with timers off) are done and Steps 1-3 are done and verified live. Step 3 required two attempts —
verified live. Step 3 (enable the recovery timer) was attempted, caused the first caused a real (safely fail-closed) outage, root-caused and
a real outage, and has been fixed but not yet re-verified live. fixed same-day; the second was verified against a full day of real
production conditions, not just a short window.
- **Step 2, attempt 1** failed: `pia-gateway register` correctly refused - **Step 2, attempt 1** failed: `pia-gateway register` correctly refused
with `safety invariants failing ... FORWARD jump to kill-switch chain with `safety invariants failing ... FORWARD jump to kill-switch chain
@ -645,10 +646,29 @@ a real outage, and has been fixed but not yet re-verified live.
full `pia-gateway-recover`/`pia-killswitch-reassert` journals. full `pia-gateway-recover`/`pia-killswitch-reassert` journals.
Mitigated live: `sudo systemctl stop pia-gateway-recover.timer` + Mitigated live: `sudo systemctl stop pia-gateway-recover.timer` +
manual `sudo systemctl restart wg-quick@pia-wg`. Root-caused and fixed manual `sudo systemctl restart wg-quick@pia-wg`. Root-caused and fixed
— see the "reassert timer alone wasn't the full fix" addendum above same-day — see the "reassert timer alone wasn't the full fix" addendum
(`pia_forward_bypass_chains`, content-verified live on every check). above (`pia_forward_bypass_chains`, content-verified live on every
**Not yet re-applied or re-verified live** — `pia-gateway-recover check).
.timer` is currently stopped on minisforum pending that. - **Step 3, re-attempt**, with the bypass fix deployed (`ansible-playbook
... -K -J`; config re-rendered with `forward_bypass_chains:
["KUBE-ROUTER-FORWARD"]`, `rewrite-config` correctly re-rendered
`pia-wg.conf` from the config-only change without a PIA call since a
baseline already existed, recovery timer re-enabled). Left running
unattended for a full **~25.5 hours** before re-checking, deliberately
— the original incident took several minutes and multiple cycles to
manifest, so a short recheck window would have proven nothing.
Confirmed via exact counts, not inference: `journalctl -u
pia-gateway-recover --since "2026-09-22 20:37:00"` showed **1528**
`class=healthy` passes (matches the expected ~1530 at 60s cadence —
the timer ran continuously the whole window), **0** `SAFETY DRIFT`
lines, **0** `class=safety_drift` classifications. `pia-gateway
status` at recheck time: `healthy`, `safety_ok: True`,
`emergency_close_active: False`, `consecutive_failures: 0`,
`last recovery: 1d ago restart` — the last restart was the one from
*before* this fix; none has been needed since. **Step 3 is done.**
kube-router did keep resyncing during this window (that's the whole
premise of the fix); the bypass tolerance handled it silently every
time, with no observable effect on the tunnel.
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below** **Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
— the original `head -3` suggestion is what hid the first issue; the — the original `head -3` suggestion is what hid the first issue; the