feat: add periodic kill-switch reassertion timer
Found live, 2026-09-22: after fixing pia-killswitch.sh's FORWARD-jump repositioning and getting one pia-gateway register attempt past the safety check, the very next attempt - roughly an hour later, no reboot, no UFW reload, no script content change in between - hit the identical "FORWARD jump to kill-switch chain is not first" failure again. This disproves the earlier assumption that kube-router/dockerd reassert their own position-1 jumps only at boot/restart scale: kube-router resyncs on ordinary Kubernetes API events, and this cluster's CronJobs (ca-sync, porkbun-ddns, gitea-backup) each create/delete a Pod every few minutes - plausible triggers for kube-router's own reconciliation loop, and therefore plausible triggers for this drift, recurring on a timescale of minutes. The reposition logic itself (previous commit) is correct - it just only ran on the existing triggers (boot, UFW reload, an Ansible-detected script content change), none of which fired between the two attempts. Adds pia-killswitch-reassert.timer/.service: re-runs the already fully idempotent pia-killswitch.sh every pia_healthcheck_interval_sec (same cadence as the health check), giving it regular chances to run instead of only sporadic ones. Deliberately separate from pia-gateway-recover.timer and NOT gated by pia_auto_recovery_enabled - this is the kill switch continuing to assert its own already- established, narrowly-scoped state (source rule, table default, FORWARD jump position), the same kind of thing it already does unconditionally at boot and after a UFW reload, just on a schedule too - not the higher-level automatic recovery/registration logic the plan was deliberately cautious about running unattended. Unblocked live in the meantime with a manual `sudo systemctl restart pia-killswitch.service`. Offline checks: yamllint, ansible-playbook --syntax-check, rendered templates (Jinja2 + bash -n on the updated script), ansible-lint (diffed against main - no new finding categories), full pia_gateway.py test suite (94/94, unaffected by this change) all pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
99f6ef84f0
commit
89d289addf
@ -189,6 +189,22 @@ real.
|
|||||||
(`forward_jump_first`) was working as intended when it refused to
|
(`forward_jump_first`) was working as intended when it refused to
|
||||||
register against the drifted state — this is a fix to the thing it
|
register against the drifted state — this is a fix to the thing it
|
||||||
checks, not a loosening of the check itself.
|
checks, not a loosening of the check itself.
|
||||||
|
|
||||||
|
**This recurs far more often than "boot-scale," and the fix above only
|
||||||
|
helped once.** The reposition logic is correct, but it only *ran* on
|
||||||
|
the existing triggers (boot, a UFW reload, an Ansible-detected content
|
||||||
|
change to the script). Confirmed live: after that fix landed and got
|
||||||
|
one `pia-gateway register` attempt past this check, the very next
|
||||||
|
attempt — roughly an hour later, no reboot, no UFW reload, no script
|
||||||
|
change in between — hit the identical failure again. kube-router
|
||||||
|
resyncs on ordinary Kubernetes API events (this cluster's CronJobs —
|
||||||
|
`ca-sync`, `porkbun-ddns`, `gitea-backup` — each create and delete a
|
||||||
|
Pod every few minutes, each a plausible trigger), not just at
|
||||||
|
boot/restart scale as originally assumed. `pia-killswitch-reassert
|
||||||
|
.timer` (every `pia_healthcheck_interval_sec`, unconditional, not
|
||||||
|
gated by `pia_auto_recovery_enabled`) exists specifically to close this
|
||||||
|
gap by giving the already-correct, already-idempotent script regular
|
||||||
|
chances to run instead of only sporadic ones.
|
||||||
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
|
- `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on
|
||||||
top of that closed baseline: default route via `pia-wg` in the `pia`
|
top of that closed baseline: default route via `pia-wg` in the `pia`
|
||||||
table, ACCEPT + established/related return inserted into the
|
table, ACCEPT + established/related return inserted into the
|
||||||
@ -288,6 +304,7 @@ the host:
|
|||||||
| `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) |
|
| `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) |
|
||||||
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
|
| `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run |
|
||||||
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
|
| `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` |
|
||||||
|
| `pia-killswitch-reassert.timer` → `.service` | every `pia_healthcheck_interval_sec`, re-runs `pia-killswitch.sh` (idempotent). **Not** gated by `pia_auto_recovery_enabled` — see the "FORWARD jump's position" note above and the service unit's own header for why |
|
||||||
|
|
||||||
### PIA's CA certificate and why addKey uses curl, not Python's `ssl`
|
### PIA's CA certificate and why addKey uses curl, not Python's `ssl`
|
||||||
|
|
||||||
@ -530,7 +547,7 @@ external secret broker — not a 24-hour token.
|
|||||||
|
|
||||||
### Rollout status (2026-09-22)
|
### Rollout status (2026-09-22)
|
||||||
|
|
||||||
Step 1 (baseline) and two attempts at step 2 have been run live.
|
Step 1 (baseline) and three attempts at step 2 have been run live.
|
||||||
|
|
||||||
- **Attempt 1** failed: `pia-gateway register` correctly refused with
|
- **Attempt 1** failed: `pia-gateway register` correctly refused with
|
||||||
`safety invariants failing ... FORWARD jump to kill-switch chain is
|
`safety invariants failing ... FORWARD jump to kill-switch chain is
|
||||||
@ -542,7 +559,16 @@ Step 1 (baseline) and two attempts at step 2 have been run live.
|
|||||||
failed differently: `addKey` rejected every candidate with
|
failed differently: `addKey` rejected every candidate with
|
||||||
`kind=tls detail=certificate verification failed`. Root-caused and
|
`kind=tls detail=certificate verification failed`. Root-caused and
|
||||||
fixed — see "PIA's CA certificate and why addKey uses curl, not
|
fixed — see "PIA's CA certificate and why addKey uses curl, not
|
||||||
Python's `ssl`" above. Not yet re-applied or re-verified live.
|
Python's `ssl`" above.
|
||||||
|
- **Attempt 3**, roughly an hour after attempt 1's fix with no reboot,
|
||||||
|
UFW reload, or script change in between, hit the *identical* FORWARD-
|
||||||
|
jump failure again — proving the drift recurs on a timescale of
|
||||||
|
minutes under this cluster's normal CronJob/pod churn, not just at
|
||||||
|
boot-scale events as originally assumed. Unblocked live with a manual
|
||||||
|
`sudo systemctl restart pia-killswitch.service`; the durable fix is
|
||||||
|
`pia-killswitch-reassert.timer` (added, not yet applied or verified
|
||||||
|
live) — see the "FORWARD jump's position" note's "recurs far more
|
||||||
|
often" addendum above.
|
||||||
|
|
||||||
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
|
**Use `iptables -S FORWARD` (no `head`) for the baseline capture below**
|
||||||
— the original `head -3` suggestion is what hid the first issue; the
|
— the original `head -3` suggestion is what hid the first issue; the
|
||||||
@ -644,8 +670,10 @@ touching `wg0`, k3s, or the host default route):
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown
|
sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown
|
||||||
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer
|
sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-killswitch-reassert.timer pia-gateway-healthcheck.timer
|
||||||
sudo rm -f /etc/systemd/system/pia-killswitch.service \
|
sudo rm -f /etc/systemd/system/pia-killswitch.service \
|
||||||
|
/etc/systemd/system/pia-killswitch-reassert.service \
|
||||||
|
/etc/systemd/system/pia-killswitch-reassert.timer \
|
||||||
/etc/systemd/system/pia-gateway-healthcheck.service \
|
/etc/systemd/system/pia-gateway-healthcheck.service \
|
||||||
/etc/systemd/system/pia-gateway-healthcheck.timer \
|
/etc/systemd/system/pia-gateway-healthcheck.timer \
|
||||||
/etc/systemd/system/pia-gateway-recover.service \
|
/etc/systemd/system/pia-gateway-recover.service \
|
||||||
|
|||||||
@ -21,6 +21,9 @@
|
|||||||
# killswitch script changed in the same run), pia-wg's own PostUp
|
# killswitch script changed in the same run), pia-wg's own PostUp
|
||||||
# needs to run last so its "open" route is the final word, not
|
# needs to run last so its "open" route is the final word, not
|
||||||
# killswitch's route-table seeding.
|
# killswitch's route-table seeding.
|
||||||
|
# 4. "Restart pia-killswitch reassertion timer" only restarts the
|
||||||
|
# *timer* (a scheduling change, e.g. its interval), never runs the
|
||||||
|
# script itself — no ordering constraint against the others.
|
||||||
|
|
||||||
- name: Reload systemd daemon
|
- name: Reload systemd daemon
|
||||||
ansible.builtin.systemd:
|
ansible.builtin.systemd:
|
||||||
@ -35,6 +38,11 @@
|
|||||||
name: pia-killswitch.service
|
name: pia-killswitch.service
|
||||||
state: restarted
|
state: restarted
|
||||||
|
|
||||||
|
- name: Restart pia-killswitch reassertion timer
|
||||||
|
ansible.builtin.systemd:
|
||||||
|
name: pia-killswitch-reassert.timer
|
||||||
|
state: restarted
|
||||||
|
|
||||||
- name: Restart pia-wg
|
- name: Restart pia-wg
|
||||||
ansible.builtin.systemd:
|
ansible.builtin.systemd:
|
||||||
name: "wg-quick@{{ pia_wg_interface }}"
|
name: "wg-quick@{{ pia_wg_interface }}"
|
||||||
|
|||||||
@ -40,3 +40,35 @@
|
|||||||
name: pia-killswitch.service
|
name: pia-killswitch.service
|
||||||
enabled: true
|
enabled: true
|
||||||
state: started
|
state: started
|
||||||
|
|
||||||
|
- name: Deploy the kill-switch periodic reassertion systemd service
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: pia-killswitch-reassert.service.j2
|
||||||
|
dest: /etc/systemd/system/pia-killswitch-reassert.service
|
||||||
|
mode: "0644"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
notify:
|
||||||
|
- Reload systemd daemon
|
||||||
|
- Restart pia-killswitch reassertion timer
|
||||||
|
|
||||||
|
- name: Deploy the kill-switch periodic reassertion systemd timer
|
||||||
|
ansible.builtin.template:
|
||||||
|
src: pia-killswitch-reassert.timer.j2
|
||||||
|
dest: /etc/systemd/system/pia-killswitch-reassert.timer
|
||||||
|
mode: "0644"
|
||||||
|
owner: root
|
||||||
|
group: root
|
||||||
|
notify:
|
||||||
|
- Reload systemd daemon
|
||||||
|
- Restart pia-killswitch reassertion timer
|
||||||
|
|
||||||
|
- name: Enable and start the kill-switch reassertion timer
|
||||||
|
ansible.builtin.systemd:
|
||||||
|
name: pia-killswitch-reassert.timer
|
||||||
|
enabled: true
|
||||||
|
state: started
|
||||||
|
daemon_reload: true
|
||||||
|
# Unconditional — not gated by pia_auto_recovery_enabled. See the
|
||||||
|
# service unit's own header for why this is scoped differently from
|
||||||
|
# the automatic recovery/registration timers.
|
||||||
|
|||||||
@ -0,0 +1,30 @@
|
|||||||
|
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
|
||||||
|
#
|
||||||
|
# Periodically re-runs the kill-switch seed script (pia-killswitch.sh) so
|
||||||
|
# its invariants — source rule, table default, and specifically the
|
||||||
|
# FORWARD jump's position — are re-asserted on a schedule, not only at
|
||||||
|
# the external trigger points that used to be the only times it ran
|
||||||
|
# (boot, a UFW reload, or an Ansible-detected content change to the
|
||||||
|
# script itself). Needed because other host components that also manage
|
||||||
|
# iptables — kube-router and dockerd — reassert their OWN position-1
|
||||||
|
# jumps independently, and confirmed live (2026-09-22) to do so more
|
||||||
|
# often than originally assumed: the FORWARD jump drifted twice within
|
||||||
|
# about an hour under this cluster's ordinary CronJob/pod churn (ca-sync,
|
||||||
|
# porkbun-ddns, gitea-backup, ...), not just at boot-scale events.
|
||||||
|
#
|
||||||
|
# The script itself is already fully idempotent (safe to run repeatedly
|
||||||
|
# — see its own header comment); this unit just gives it more chances to
|
||||||
|
# run. Deliberately separate from pia-gateway-recover.timer and NOT
|
||||||
|
# gated by pia_auto_recovery_enabled: this is the kill switch continuing
|
||||||
|
# to assert its own already-established, narrowly-scoped state — the
|
||||||
|
# same kind of thing it already does unconditionally at boot and after a
|
||||||
|
# UFW reload, just on a schedule too — not the higher-level automatic
|
||||||
|
# recovery/registration logic the plan was deliberately cautious about
|
||||||
|
# running unattended.
|
||||||
|
[Unit]
|
||||||
|
Description=Periodically re-assert the PIA gateway kill switch's FORWARD jump position
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart=/usr/local/sbin/pia-killswitch.sh
|
||||||
|
LimitCORE=0
|
||||||
@ -0,0 +1,11 @@
|
|||||||
|
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
|
||||||
|
[Unit]
|
||||||
|
Description=Run the PIA gateway kill-switch reassertion every {{ pia_healthcheck_interval_sec }}s
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnBootSec=30
|
||||||
|
OnUnitActiveSec={{ pia_healthcheck_interval_sec }}
|
||||||
|
AccuracySec=5s
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@ -1,12 +1,16 @@
|
|||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
|
# Managed by Ansible (role: pia-gateway). Do not edit by hand.
|
||||||
#
|
#
|
||||||
# Runs once at boot, before wg-quick@{{ pia_wg_interface }}.service
|
# Runs at boot, before wg-quick@{{ pia_wg_interface }}.service (enforced
|
||||||
# (enforced by the systemd drop-in in pia-killswitch.service's
|
# by the systemd drop-in in pia-killswitch.service's Before=/wg-quick's
|
||||||
# Before=/wg-quick's Requires=After=), and again any time UFW is
|
# Requires=After=); any time UFW is reloaded (see handlers/main.yaml —
|
||||||
# reloaded (see handlers/main.yaml — "Reload ufw" fires before "Restart
|
# "Reload ufw" fires before "Restart pia-killswitch" so a reload can
|
||||||
# pia-killswitch" so a reload can never leave a permanent gap). Seeds the
|
# never leave a permanent gap); and — since 2026-09-22, after the FORWARD
|
||||||
# closed state that PIA gateway's kill switch relies on:
|
# jump was found drifting twice within about an hour under this
|
||||||
|
# cluster's ordinary CronJob/pod churn, far more often than a boot/reload
|
||||||
|
# -only cadence catches — every {{ pia_healthcheck_interval_sec }}s via
|
||||||
|
# pia-killswitch-reassert.timer. Seeds the closed state that PIA
|
||||||
|
# gateway's kill switch relies on:
|
||||||
#
|
#
|
||||||
# - a source rule sending {{ pia_vlan50_subnet }} to the dedicated
|
# - a source rule sending {{ pia_vlan50_subnet }} to the dedicated
|
||||||
# "{{ pia_route_table_name }}" table, evaluated after the kernel's
|
# "{{ pia_route_table_name }}" table, evaluated after the kernel's
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user