diff --git a/ansible/roles/pia-gateway/README.md b/ansible/roles/pia-gateway/README.md index d1ae8de..a61af3b 100644 --- a/ansible/roles/pia-gateway/README.md +++ b/ansible/roles/pia-gateway/README.md @@ -599,11 +599,12 @@ external secret broker — not a 24-hour token. - `unknown` (a read command failing) is logged/alerted but never acted on, by design. -### Rollout status (2026-09-22) +### Rollout status (2026-09-23) -Step 1 (baseline) and step 2 (deploy with timers off) are done and -verified live. Step 3 (enable the recovery timer) was attempted, caused -a real outage, and has been fixed but not yet re-verified live. +Steps 1-3 are done and verified live. Step 3 required two attempts — +the first caused a real (safely fail-closed) outage, root-caused and +fixed same-day; the second was verified against a full day of real +production conditions, not just a short window. - **Step 2, attempt 1** failed: `pia-gateway register` correctly refused with `safety invariants failing ... FORWARD jump to kill-switch chain @@ -645,10 +646,29 @@ a real outage, and has been fixed but not yet re-verified live. full `pia-gateway-recover`/`pia-killswitch-reassert` journals. Mitigated live: `sudo systemctl stop pia-gateway-recover.timer` + manual `sudo systemctl restart wg-quick@pia-wg`. Root-caused and fixed - — see the "reassert timer alone wasn't the full fix" addendum above - (`pia_forward_bypass_chains`, content-verified live on every check). - **Not yet re-applied or re-verified live** — `pia-gateway-recover - .timer` is currently stopped on minisforum pending that. + same-day — see the "reassert timer alone wasn't the full fix" addendum + above (`pia_forward_bypass_chains`, content-verified live on every + check). +- **Step 3, re-attempt**, with the bypass fix deployed (`ansible-playbook + ... -K -J`; config re-rendered with `forward_bypass_chains: + ["KUBE-ROUTER-FORWARD"]`, `rewrite-config` correctly re-rendered + `pia-wg.conf` from the config-only change without a PIA call since a + baseline already existed, recovery timer re-enabled). Left running + unattended for a full **~25.5 hours** before re-checking, deliberately + — the original incident took several minutes and multiple cycles to + manifest, so a short recheck window would have proven nothing. + Confirmed via exact counts, not inference: `journalctl -u + pia-gateway-recover --since "2026-09-22 20:37:00"` showed **1528** + `class=healthy` passes (matches the expected ~1530 at 60s cadence — + the timer ran continuously the whole window), **0** `SAFETY DRIFT` + lines, **0** `class=safety_drift` classifications. `pia-gateway + status` at recheck time: `healthy`, `safety_ok: True`, + `emergency_close_active: False`, `consecutive_failures: 0`, + `last recovery: 1d ago restart` — the last restart was the one from + *before* this fix; none has been needed since. **Step 3 is done.** + kube-router did keep resyncing during this window (that's the whole + premise of the fix); the bypass tolerance handled it silently every + time, with no observable effect on the tunnel. **Use `iptables -S FORWARD` (no `head`) for the baseline capture below** — the original `head -3` suggestion is what hid the first issue; the