diff --git a/ansible/roles/pia-gateway/README.md b/ansible/roles/pia-gateway/README.md index 9c00e99..060c606 100644 --- a/ansible/roles/pia-gateway/README.md +++ b/ansible/roles/pia-gateway/README.md @@ -189,6 +189,22 @@ real. (`forward_jump_first`) was working as intended when it refused to register against the drifted state — this is a fix to the thing it checks, not a loosening of the check itself. + + **This recurs far more often than "boot-scale," and the fix above only + helped once.** The reposition logic is correct, but it only *ran* on + the existing triggers (boot, a UFW reload, an Ansible-detected content + change to the script). Confirmed live: after that fix landed and got + one `pia-gateway register` attempt past this check, the very next + attempt — roughly an hour later, no reboot, no UFW reload, no script + change in between — hit the identical failure again. kube-router + resyncs on ordinary Kubernetes API events (this cluster's CronJobs — + `ca-sync`, `porkbun-ddns`, `gitea-backup` — each create and delete a + Pod every few minutes, each a plausible trigger), not just at + boot/restart scale as originally assumed. `pia-killswitch-reassert + .timer` (every `pia_healthcheck_interval_sec`, unconditional, not + gated by `pia_auto_recovery_enabled`) exists specifically to close this + gap by giving the already-correct, already-idempotent script regular + chances to run instead of only sporadic ones. - `pia-wg.conf`'s own `PostUp`/`PreDown` open/close the narrower path on top of that closed baseline: default route via `pia-wg` in the `pia` table, ACCEPT + established/related return inserted into the @@ -288,6 +304,7 @@ the host: | `pia-gateway-renew.timer` → `.service` | daily 03:30 ±30 min, proactive same-key renewal (`renew`); **disabled by default** (`pia_renewal_enabled`) | | `pia-gateway-onfailure.service` | `OnFailure=` of the two above: posts an error alert if a unit crashes or refuses to run | | `pia-gateway-healthcheck.timer` | unchanged cadence, now runs `pia-gateway check` | +| `pia-killswitch-reassert.timer` → `.service` | every `pia_healthcheck_interval_sec`, re-runs `pia-killswitch.sh` (idempotent). **Not** gated by `pia_auto_recovery_enabled` — see the "FORWARD jump's position" note above and the service unit's own header for why | ### PIA's CA certificate and why addKey uses curl, not Python's `ssl` @@ -530,7 +547,7 @@ external secret broker — not a 24-hour token. ### Rollout status (2026-09-22) -Step 1 (baseline) and two attempts at step 2 have been run live. +Step 1 (baseline) and three attempts at step 2 have been run live. - **Attempt 1** failed: `pia-gateway register` correctly refused with `safety invariants failing ... FORWARD jump to kill-switch chain is @@ -542,7 +559,16 @@ Step 1 (baseline) and two attempts at step 2 have been run live. failed differently: `addKey` rejected every candidate with `kind=tls detail=certificate verification failed`. Root-caused and fixed — see "PIA's CA certificate and why addKey uses curl, not - Python's `ssl`" above. Not yet re-applied or re-verified live. + Python's `ssl`" above. +- **Attempt 3**, roughly an hour after attempt 1's fix with no reboot, + UFW reload, or script change in between, hit the *identical* FORWARD- + jump failure again — proving the drift recurs on a timescale of + minutes under this cluster's normal CronJob/pod churn, not just at + boot-scale events as originally assumed. Unblocked live with a manual + `sudo systemctl restart pia-killswitch.service`; the durable fix is + `pia-killswitch-reassert.timer` (added, not yet applied or verified + live) — see the "FORWARD jump's position" note's "recurs far more + often" addendum above. **Use `iptables -S FORWARD` (no `head`) for the baseline capture below** — the original `head -3` suggestion is what hid the first issue; the @@ -644,8 +670,10 @@ touching `wg0`, k3s, or the host default route): ```bash sudo systemctl disable --now pia-gateway-recover.timer pia-gateway-renew.timer # controller first, so it cannot react to the teardown -sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-gateway-healthcheck.timer +sudo systemctl disable --now wg-quick@pia-wg pia-killswitch.service pia-killswitch-reassert.timer pia-gateway-healthcheck.timer sudo rm -f /etc/systemd/system/pia-killswitch.service \ + /etc/systemd/system/pia-killswitch-reassert.service \ + /etc/systemd/system/pia-killswitch-reassert.timer \ /etc/systemd/system/pia-gateway-healthcheck.service \ /etc/systemd/system/pia-gateway-healthcheck.timer \ /etc/systemd/system/pia-gateway-recover.service \ diff --git a/ansible/roles/pia-gateway/handlers/main.yaml b/ansible/roles/pia-gateway/handlers/main.yaml index f99c132..a112273 100644 --- a/ansible/roles/pia-gateway/handlers/main.yaml +++ b/ansible/roles/pia-gateway/handlers/main.yaml @@ -21,6 +21,9 @@ # killswitch script changed in the same run), pia-wg's own PostUp # needs to run last so its "open" route is the final word, not # killswitch's route-table seeding. +# 4. "Restart pia-killswitch reassertion timer" only restarts the +# *timer* (a scheduling change, e.g. its interval), never runs the +# script itself — no ordering constraint against the others. - name: Reload systemd daemon ansible.builtin.systemd: @@ -35,6 +38,11 @@ name: pia-killswitch.service state: restarted +- name: Restart pia-killswitch reassertion timer + ansible.builtin.systemd: + name: pia-killswitch-reassert.timer + state: restarted + - name: Restart pia-wg ansible.builtin.systemd: name: "wg-quick@{{ pia_wg_interface }}" diff --git a/ansible/roles/pia-gateway/tasks/routing.yaml b/ansible/roles/pia-gateway/tasks/routing.yaml index 656e48b..e9b3d8d 100644 --- a/ansible/roles/pia-gateway/tasks/routing.yaml +++ b/ansible/roles/pia-gateway/tasks/routing.yaml @@ -40,3 +40,35 @@ name: pia-killswitch.service enabled: true state: started + +- name: Deploy the kill-switch periodic reassertion systemd service + ansible.builtin.template: + src: pia-killswitch-reassert.service.j2 + dest: /etc/systemd/system/pia-killswitch-reassert.service + mode: "0644" + owner: root + group: root + notify: + - Reload systemd daemon + - Restart pia-killswitch reassertion timer + +- name: Deploy the kill-switch periodic reassertion systemd timer + ansible.builtin.template: + src: pia-killswitch-reassert.timer.j2 + dest: /etc/systemd/system/pia-killswitch-reassert.timer + mode: "0644" + owner: root + group: root + notify: + - Reload systemd daemon + - Restart pia-killswitch reassertion timer + +- name: Enable and start the kill-switch reassertion timer + ansible.builtin.systemd: + name: pia-killswitch-reassert.timer + enabled: true + state: started + daemon_reload: true + # Unconditional — not gated by pia_auto_recovery_enabled. See the + # service unit's own header for why this is scoped differently from + # the automatic recovery/registration timers. diff --git a/ansible/roles/pia-gateway/templates/pia-killswitch-reassert.service.j2 b/ansible/roles/pia-gateway/templates/pia-killswitch-reassert.service.j2 new file mode 100644 index 0000000..e356492 --- /dev/null +++ b/ansible/roles/pia-gateway/templates/pia-killswitch-reassert.service.j2 @@ -0,0 +1,30 @@ +# Managed by Ansible (role: pia-gateway). Do not edit by hand. +# +# Periodically re-runs the kill-switch seed script (pia-killswitch.sh) so +# its invariants — source rule, table default, and specifically the +# FORWARD jump's position — are re-asserted on a schedule, not only at +# the external trigger points that used to be the only times it ran +# (boot, a UFW reload, or an Ansible-detected content change to the +# script itself). Needed because other host components that also manage +# iptables — kube-router and dockerd — reassert their OWN position-1 +# jumps independently, and confirmed live (2026-09-22) to do so more +# often than originally assumed: the FORWARD jump drifted twice within +# about an hour under this cluster's ordinary CronJob/pod churn (ca-sync, +# porkbun-ddns, gitea-backup, ...), not just at boot-scale events. +# +# The script itself is already fully idempotent (safe to run repeatedly +# — see its own header comment); this unit just gives it more chances to +# run. Deliberately separate from pia-gateway-recover.timer and NOT +# gated by pia_auto_recovery_enabled: this is the kill switch continuing +# to assert its own already-established, narrowly-scoped state — the +# same kind of thing it already does unconditionally at boot and after a +# UFW reload, just on a schedule too — not the higher-level automatic +# recovery/registration logic the plan was deliberately cautious about +# running unattended. +[Unit] +Description=Periodically re-assert the PIA gateway kill switch's FORWARD jump position + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/pia-killswitch.sh +LimitCORE=0 diff --git a/ansible/roles/pia-gateway/templates/pia-killswitch-reassert.timer.j2 b/ansible/roles/pia-gateway/templates/pia-killswitch-reassert.timer.j2 new file mode 100644 index 0000000..d21607e --- /dev/null +++ b/ansible/roles/pia-gateway/templates/pia-killswitch-reassert.timer.j2 @@ -0,0 +1,11 @@ +# Managed by Ansible (role: pia-gateway). Do not edit by hand. +[Unit] +Description=Run the PIA gateway kill-switch reassertion every {{ pia_healthcheck_interval_sec }}s + +[Timer] +OnBootSec=30 +OnUnitActiveSec={{ pia_healthcheck_interval_sec }} +AccuracySec=5s + +[Install] +WantedBy=timers.target diff --git a/ansible/roles/pia-gateway/templates/pia-killswitch.sh.j2 b/ansible/roles/pia-gateway/templates/pia-killswitch.sh.j2 index a63d079..98e5c7b 100644 --- a/ansible/roles/pia-gateway/templates/pia-killswitch.sh.j2 +++ b/ansible/roles/pia-gateway/templates/pia-killswitch.sh.j2 @@ -1,12 +1,16 @@ #!/bin/bash # Managed by Ansible (role: pia-gateway). Do not edit by hand. # -# Runs once at boot, before wg-quick@{{ pia_wg_interface }}.service -# (enforced by the systemd drop-in in pia-killswitch.service's -# Before=/wg-quick's Requires=After=), and again any time UFW is -# reloaded (see handlers/main.yaml — "Reload ufw" fires before "Restart -# pia-killswitch" so a reload can never leave a permanent gap). Seeds the -# closed state that PIA gateway's kill switch relies on: +# Runs at boot, before wg-quick@{{ pia_wg_interface }}.service (enforced +# by the systemd drop-in in pia-killswitch.service's Before=/wg-quick's +# Requires=After=); any time UFW is reloaded (see handlers/main.yaml — +# "Reload ufw" fires before "Restart pia-killswitch" so a reload can +# never leave a permanent gap); and — since 2026-09-22, after the FORWARD +# jump was found drifting twice within about an hour under this +# cluster's ordinary CronJob/pod churn, far more often than a boot/reload +# -only cadence catches — every {{ pia_healthcheck_interval_sec }}s via +# pia-killswitch-reassert.timer. Seeds the closed state that PIA +# gateway's kill switch relies on: # # - a source rule sending {{ pia_vlan50_subnet }} to the dedicated # "{{ pia_route_table_name }}" table, evaluated after the kernel's