feat: add periodic kill-switch reassertion timer

Found live, 2026-09-22: after fixing pia-killswitch.sh's FORWARD-jump
repositioning and getting one pia-gateway register attempt past the
safety check, the very next attempt - roughly an hour later, no
reboot, no UFW reload, no script content change in between - hit the
identical "FORWARD jump to kill-switch chain is not first" failure
again. This disproves the earlier assumption that kube-router/dockerd
reassert their own position-1 jumps only at boot/restart scale:
kube-router resyncs on ordinary Kubernetes API events, and this
cluster's CronJobs (ca-sync, porkbun-ddns, gitea-backup) each
create/delete a Pod every few minutes - plausible triggers for
kube-router's own reconciliation loop, and therefore plausible
triggers for this drift, recurring on a timescale of minutes.

The reposition logic itself (previous commit) is correct - it just
only ran on the existing triggers (boot, UFW reload, an
Ansible-detected script content change), none of which fired between
the two attempts.

Adds pia-killswitch-reassert.timer/.service: re-runs the already fully
idempotent pia-killswitch.sh every pia_healthcheck_interval_sec (same
cadence as the health check), giving it regular chances to run instead
of only sporadic ones. Deliberately separate from
pia-gateway-recover.timer and NOT gated by pia_auto_recovery_enabled -
this is the kill switch continuing to assert its own already-
established, narrowly-scoped state (source rule, table default,
FORWARD jump position), the same kind of thing it already does
unconditionally at boot and after a UFW reload, just on a schedule
too - not the higher-level automatic recovery/registration logic the
plan was deliberately cautious about running unattended.

Unblocked live in the meantime with a manual
`sudo systemctl restart pia-killswitch.service`.

Offline checks: yamllint, ansible-playbook --syntax-check, rendered
templates (Jinja2 + bash -n on the updated script), ansible-lint
(diffed against main - no new finding categories), full pia_gateway.py
test suite (94/94, unaffected by this change) all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Nik Afiq 2026-09-22 19:08:28 +09:00
parent 99f6ef84f0
commit 89d289addf
6 changed files with 122 additions and 9 deletions

View File

@ -189,6 +189,22 @@ real.
(`forward_jump_first`) was working as intended when it refused to
register against the drifted state — this is a fix to the thing it
checks, not a loosening of the check itself.
**This recurs far more often than "boot-scale," and the fix above only
helped once.** The reposition logic is correct, but it only *ran* on
the existing triggers (boot, a UFW reload, an Ansible-detected content
change to the script). Confirmed live: after that fix landed and got
one `pia-gateway register` attempt past this check, the very next
attempt — roughly an hour later, no reboot, no UFW reload, no script
change in between — hit the identical failure again. kube-router
resyncs on ordinary Kubernetes API events (this cluster's CronJobs —
`ca-sync`, `porkbun-ddns`, `gitea-backup` — each create and delete a
Pod every few minutes, each a plausible trigger), not just at
boot/restart scale as originally assumed. `pia-killswitch-reassert
.timer` (every `pia_healthcheck_interval_sec`, unconditional, not
gated by `pia_auto_recovery_enabled`) exists specifically to close this
gap by giving the already-correct, already-idempotent script regular
chances to run instead of only sporadic ones.
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
top of that closed baseline: default route via `pia-wg` in the `pia`
table, ACCEPT + established/related return inserted into the
@ -288,6 +304,7 @@ the host:
| `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) |
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
| `pia-killswitch-reassert.timer` → `.service` | every `pia_healthcheck_interval_sec`, re-runs `pia-killswitch.sh` (idempotent). **Not** gated by `pia_auto_recovery_enabled` — see the "FORWARD jump's position" note above and the service unit's own header for why |
### PIA's CA certificate and why addKey uses curl, not Python's `ssl`
@ -530,7 +547,7 @@ external secret broker — not a 24-hour token.
### Rollout status (2026-09-22)
Step 1 (baseline) and two attempts at step 2 have been run live.
Step 1 (baseline) and three attempts at step 2 have been run live.
- **Attempt 1** failed: `pia-gateway register` correctly refused with
`safety invariants failing ... FORWARD jump to kill-switch chain is
@ -542,7 +559,16 @@ Step 1 (baseline) and two attempts at step 2 have been run live.
failed differently: `addKey` rejected every candidate with
`kind=tls detail=certificate verification failed`. Root-caused and
fixed — see "PIA's CA certificate and why addKey uses curl, not
Python's `ssl`" above. Not yet re-applied or re-verified live.
Python's `ssl`" above.
- **Attempt 3**, roughly an hour after attempt 1's fix with no reboot,
UFW reload, or script change in between, hit the *identical* FORWARD-
jump failure again — proving the drift recurs on a timescale of
minutes under this cluster's normal CronJob/pod churn, not just at
boot-scale events as originally assumed. Unblocked live with a manual
`sudo systemctl restart pia-killswitch.service`; the durable fix is
`pia-killswitch-reassert.timer` (added, not yet applied or verified
live) — see the "FORWARD jump's position" note's "recurs far more
often" addendum above.
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
— the original `head -3` suggestion is what hid the first issue; the
@ -644,8 +670,10 @@ touching `wg0`, k3s, or the host default route):
```bash
sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-killswitch-reassert.timer pia-gateway-healthcheck.timer
sudo rm -f /etc/systemd/system/pia-killswitch.service \
/etc/systemd/system/pia-killswitch-reassert.service \
/etc/systemd/system/pia-killswitch-reassert.timer \
/etc/systemd/system/pia-gateway-healthcheck.service \
/etc/systemd/system/pia-gateway-healthcheck.timer \
/etc/systemd/system/pia-gateway-recover.service \

View File

@ -21,6 +21,9 @@
# killswitch script changed in the same run), pia-wg's own PostUp
# needs to run last so its "open" route is the final word, not
# killswitch's route-table seeding.
# 4. "Restart pia-killswitch reassertion timer" only restarts the
# *timer* (a scheduling change, e.g. its interval), never runs the
# script itself — no ordering constraint against the others.
- name: Reload systemd daemon
ansible.builtin.systemd:
@ -35,6 +38,11 @@
name: pia-killswitch.service
state: restarted
- name: Restart pia-killswitch reassertion timer
ansible.builtin.systemd:
name: pia-killswitch-reassert.timer
state: restarted
- name: Restart pia-wg
ansible.builtin.systemd:
name: "wg-quick@{{ pia_wg_interface }}"

View File

@ -40,3 +40,35 @@
name: pia-killswitch.service
enabled: true
state: started
- name: Deploy the kill-switch periodic reassertion systemd service
ansible.builtin.template:
src: pia-killswitch-reassert.service.j2
dest: /etc/systemd/system/pia-killswitch-reassert.service
mode: "0644"
owner: root
group: root
notify:
- Reload systemd daemon
- Restart pia-killswitch reassertion timer
- name: Deploy the kill-switch periodic reassertion systemd timer
ansible.builtin.template:
src: pia-killswitch-reassert.timer.j2
dest: /etc/systemd/system/pia-killswitch-reassert.timer
mode: "0644"
owner: root
group: root
notify:
- Reload systemd daemon
- Restart pia-killswitch reassertion timer
- name: Enable and start the kill-switch reassertion timer
ansible.builtin.systemd:
name: pia-killswitch-reassert.timer
enabled: true
state: started
daemon_reload: true
# Unconditional — not gated by pia_auto_recovery_enabled. See the
# service unit's own header for why this is scoped differently from
# the automatic recovery/registration timers.

View File

@ -0,0 +1,30 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Periodically re-runs the kill-switch seed script (pia-killswitch.sh) so
# its invariants — source rule, table default, and specifically the
# FORWARD jump's position — are re-asserted on a schedule, not only at
# the external trigger points that used to be the only times it ran
# (boot, a UFW reload, or an Ansible-detected content change to the
# script itself). Needed because other host components that also manage
# iptables — kube-router and dockerd — reassert their OWN position-1
# jumps independently, and confirmed live (2026-09-22) to do so more
# often than originally assumed: the FORWARD jump drifted twice within
# about an hour under this cluster's ordinary CronJob/pod churn (ca-sync,
# porkbun-ddns, gitea-backup, ...), not just at boot-scale events.
#
# The script itself is already fully idempotent (safe to run repeatedly
# — see its own header comment); this unit just gives it more chances to
# run. Deliberately separate from pia-gateway-recover.timer and NOT
# gated by pia_auto_recovery_enabled: this is the kill switch continuing
# to assert its own already-established, narrowly-scoped state — the
# same kind of thing it already does unconditionally at boot and after a
# UFW reload, just on a schedule too — not the higher-level automatic
# recovery/registration logic the plan was deliberately cautious about
# running unattended.
[Unit]
Description=Periodically re-assert the PIA gateway kill switch's FORWARD jump position
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/pia-killswitch.sh
LimitCORE=0

View File

@ -0,0 +1,11 @@
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
[Unit]
Description=Run the PIA gateway kill-switch reassertion every {{ pia_healthcheck_interval_sec }}s
[Timer]
OnBootSec=30
OnUnitActiveSec={{ pia_healthcheck_interval_sec }}
AccuracySec=5s
[Install]
WantedBy=timers.target

View File

@ -1,12 +1,16 @@
#!/bin/bash
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
#
# Runs once at boot, before wg-quick@{{ pia_wg_interface }}.service
# (enforced by the systemd drop-in in pia-killswitch.service's
# Before=/wg-quick's Requires=After=), and again any time UFW is
# reloaded (see handlers/main.yaml — "Reload ufw" fires before "Restart
# pia-killswitch" so a reload can never leave a permanent gap). Seeds the
# closed state that PIA gateway's kill switch relies on:
# Runs at boot, before wg-quick@{{ pia_wg_interface }}.service (enforced
# by the systemd drop-in in pia-killswitch.service's Before=/wg-quick's
# Requires=After=); any time UFW is reloaded (see handlers/main.yaml —
# "Reload ufw" fires before "Restart pia-killswitch" so a reload can
# never leave a permanent gap); and — since 2026-09-22, after the FORWARD
# jump was found drifting twice within about an hour under this
# cluster's ordinary CronJob/pod churn, far more often than a boot/reload
# -only cadence catches — every {{ pia_healthcheck_interval_sec }}s via
# pia-killswitch-reassert.timer. Seeds the closed state that PIA
# gateway's kill switch relies on:
#
# - a source rule sending {{ pia_vlan50_subnet }} to the dedicated
# "{{ pia_route_table_name }}" table, evaluated after the kernel's