refactor(navigation): enhance overlay handling and unify back navigation logic across tasks
This commit is contained in:
parent
c5873f4e58
commit
84790ecb27
@ -672,6 +672,7 @@ Current project state: mailbox, cafe, stamina, story_sweep, event_sweep, shop_co
|
||||
- the `scripts/` directory itself no longer exists
|
||||
- `ba_auto/driver.py` primitives are wired into all migrated task modules
|
||||
- existing driver primitives include `run_command`, `focus_game`, `click`, `move_mouse`, `scroll`, `keypress`, `screenshot`, `wait`, and `color_at`
|
||||
- `focus_game()` calls both `xdotool windowactivate` and `xdotool windowraise` on the Blue Archive window, not just the former — confirmed live (twice, first during `event_sweep.py` debugging, then again during a `bounty` run, see `plan.md`'s "Return-to-home audit follow-up #2") that a stray anti-cheat `XIGNCODE` window can render on top of the game and silently intercept clicks at fixed positions (notably the shared `navigation.BACK_BUTTON` coordinate), even though `xdotool getactivewindow` still reports `BlueArchive` as active throughout — `windowactivate` changes input focus, not window stacking order, so only `windowraise` actually clears it. `navigation.return_to_home` also calls `driver.focus_game()` once, partway through its retry budget, as a second escalation for the case the overlay appears mid-run rather than only at a task's own start
|
||||
- `ba_auto/navigation.py` has shared state probes used across tasks:
|
||||
- `is_on_subscreen`
|
||||
- `is_modal_open`
|
||||
|
||||
@ -364,7 +364,11 @@ EVENT_SWEEP_ROTATION_COUNT = "max"
|
||||
# module/shop/shop_utils.py get_item_position/ensure_choose/buy pattern);
|
||||
# see ba_auto/tasks/shop_utils.py for the shared control flow.
|
||||
SHOP_ICON = (1155, 1085) # bottom nav "ショップ" icon on the home screen
|
||||
SHOP_BACK_BUTTON = (85, 55)
|
||||
# SHOP_BACK_BUTTON removed 2026-07-12 (redundant with navigation.BACK_BUTTON,
|
||||
# the single source of truth for this shared coordinate -- both shop tasks'
|
||||
# own back-clicks now go through navigation.return_to_home instead of a
|
||||
# bare click, matching EVENT_BACK_BUTTON's earlier removal for the same
|
||||
# reason).
|
||||
SHOP_TAB_COMMON = (160, 208) # 通常アイテム tab (credit-point items)
|
||||
# 戦術対抗戦 tab (tactical-coin items). The reference reaches this via
|
||||
# goto_shop_by_name's OCR swipe-search over the shop-type tab list
|
||||
@ -447,7 +451,11 @@ TACTICAL_SHOP_TARGETS = [
|
||||
# paged arrows, this list's scroll position is deterministic (see
|
||||
# REGION_ROW_Y below), so there's nothing to OCR-locate.
|
||||
LESSON_ICON = (314, 1100) # bottom nav "スケジュール" icon on the home screen
|
||||
LESSON_BACK_BUTTON = (85, 55) # shared by both the region-list and per-region map screens
|
||||
# LESSON_BACK_BUTTON removed 2026-07-12 (redundant with navigation.BACK_
|
||||
# BUTTON, the single source of truth for this shared coordinate -- every
|
||||
# back-click in lesson.py now goes through navigation.click_back/
|
||||
# return_to_home instead of a bare click, matching EVENT_BACK_BUTTON's
|
||||
# earlier removal for the same reason).
|
||||
|
||||
# Clicking LESSON_ICON does not always land on the Location Select list --
|
||||
# confirmed live: the game remembers the last-viewed region and reopens
|
||||
|
||||
@ -25,6 +25,15 @@ def focus_game():
|
||||
if not window_ids:
|
||||
raise RuntimeError("Blue Archive window not found. Is the game running?")
|
||||
run_command(["xdotool", "windowactivate", window_ids[0]])
|
||||
# windowactivate alone isn't enough -- confirmed live (twice now, see
|
||||
# Handoff.md's original finding and plan.md's Bounty phase for a second
|
||||
# occurrence) that a stray anti-cheat XIGNCODE window can render
|
||||
# visibly on top of the game and intercept clicks at fixed positions
|
||||
# (notably the shared BACK_BUTTON coordinate) even while xdotool still
|
||||
# reports BlueArchive as the "active" window -- windowactivate changes
|
||||
# input focus, not stacking order, so it doesn't fix this by itself.
|
||||
# windowraise does. Cheap and harmless when no overlay is present.
|
||||
run_command(["xdotool", "windowraise", window_ids[0]])
|
||||
wait(0.5)
|
||||
|
||||
|
||||
|
||||
@ -68,11 +68,29 @@ def return_to_home(driver):
|
||||
mailbox/cafe/event's own stage modal) and falls back to clicking
|
||||
BACK_BUTTON if Escape didn't clear it, re-checking after each.
|
||||
|
||||
Halfway through the round budget, if neither has made any progress,
|
||||
re-raises the game window once via driver.focus_game() before
|
||||
continuing. Confirmed live (2026-07-12, a bounty run): a stray
|
||||
anti-cheat XIGNCODE window can render visibly on top of the game and
|
||||
silently intercept every Escape/BACK_BUTTON press at that exact screen
|
||||
position -- xdotool still reports BlueArchive as the "active" window
|
||||
throughout, so nothing here can detect the overlay directly, only that
|
||||
presses keep not working. windowactivate (already called at the start
|
||||
of every task via driver.focus_game()) doesn't fix this -- it changes
|
||||
input focus, not stacking order -- but windowraise does, confirmed live
|
||||
by reproducing the exact stuck state and watching return_to_home
|
||||
immediately succeed once the window was raised. driver.focus_game()
|
||||
itself now calls windowraise too, so this escalation is a second,
|
||||
later-in-the-budget safety net for the case an overlay appears or
|
||||
becomes input-stealing mid-run, after focus_game()'s own start-of-task
|
||||
call already ran.
|
||||
|
||||
Returns True once home is confirmed reached, False if still not home
|
||||
after RETURN_HOME_MAX_ROUNDS -- callers should treat False as "abort,
|
||||
don't guess further" rather than assume home was reached.
|
||||
"""
|
||||
for _ in range(RETURN_HOME_MAX_ROUNDS):
|
||||
escalated = False
|
||||
for round_num in range(1, RETURN_HOME_MAX_ROUNDS + 1):
|
||||
if not _not_home(driver):
|
||||
return True
|
||||
driver.keypress("Escape")
|
||||
@ -81,9 +99,47 @@ def return_to_home(driver):
|
||||
return True
|
||||
driver.click(*BACK_BUTTON)
|
||||
driver.wait(1)
|
||||
if not _not_home(driver):
|
||||
return True
|
||||
|
||||
if not escalated and round_num >= RETURN_HOME_MAX_ROUNDS // 2:
|
||||
escalated = True
|
||||
print("[navigation] still not home halfway through recovery -- re-raising the game window in case a stray overlay (e.g. XIGNCODE) is intercepting input")
|
||||
driver.focus_game()
|
||||
return not _not_home(driver)
|
||||
|
||||
|
||||
def click_back(driver, verify_fn, max_attempts=3):
|
||||
"""Click the shared BACK_BUTTON with retry, for callers that need to go
|
||||
back exactly ONE level (e.g. a per-region map -> its list screen) rather
|
||||
than return_to_home's "all the way to home" semantics -- so it can't
|
||||
just reuse return_to_home directly, even though it wants the same
|
||||
click-then-verify-then-retry robustness.
|
||||
|
||||
`verify_fn(driver)` should return True once the click's intended effect
|
||||
is confirmed (whatever that means for the caller's own screen
|
||||
transition). Escalates once, via driver.focus_game() (re-raising the
|
||||
game window), right before the FINAL attempt if every earlier one
|
||||
failed -- the same XIGNCODE-overlay defense return_to_home has (a stray
|
||||
anti-cheat window can silently intercept clicks at this exact shared
|
||||
coordinate even while the game reports itself as the active window; see
|
||||
return_to_home's own docstring for the live-confirmed history). Added
|
||||
2026-07-12 after an audit found lesson.py's own back-button clicks had
|
||||
no retry or overlay defense at all, unlike return_to_home.
|
||||
|
||||
Returns True once verify_fn confirms success, False if still
|
||||
unconfirmed after max_attempts.
|
||||
"""
|
||||
for attempt in range(1, max_attempts + 1):
|
||||
if attempt == max_attempts:
|
||||
driver.focus_game()
|
||||
driver.click(*BACK_BUTTON)
|
||||
driver.wait(1.5)
|
||||
if verify_fn(driver):
|
||||
return True
|
||||
return verify_fn(driver)
|
||||
|
||||
|
||||
def wait_for_state(driver, config, reactions, ends, max_iterations=30, poll_interval=1.0):
|
||||
"""Generic "watch the screen, react to anything recognized, stop once a
|
||||
recognized destination is reached" loop -- the local equivalent of the
|
||||
|
||||
@ -86,14 +86,20 @@ def _ensure_location_select_list(driver, config):
|
||||
only exists on the per-region screen, never on the list (confirmed by
|
||||
direct pixel sample: the same coordinate reads as plain dark background on
|
||||
the list screen).
|
||||
|
||||
Uses navigation.click_back (not a bare driver.click) since this is a
|
||||
BACK_BUTTON press like any other -- see that helper's docstring for why
|
||||
plain unverified back-clicks in this module were a real gap (an
|
||||
XIGNCODE overlay incident during a bounty run, 2026-07-12, found this
|
||||
exact click had no retry or overlay defense, unlike return_to_home).
|
||||
"""
|
||||
for attempt in range(1, OPEN_RETRIES + 1):
|
||||
if not _is_action_button_showing(driver, config, config.LESSON_ALL_SCHEDULES_BUTTON):
|
||||
return True
|
||||
print(f"[lesson] schedule screen resumed on a specific region's map instead of the Location Select list -- returning (attempt {attempt}/{OPEN_RETRIES})")
|
||||
driver.click(*config.LESSON_BACK_BUTTON)
|
||||
driver.wait(1.5)
|
||||
return not _is_action_button_showing(driver, config, config.LESSON_ALL_SCHEDULES_BUTTON)
|
||||
if not _is_action_button_showing(driver, config, config.LESSON_ALL_SCHEDULES_BUTTON):
|
||||
return True
|
||||
print("[lesson] schedule screen resumed on a specific region's map instead of the Location Select list -- returning")
|
||||
return navigation.click_back(
|
||||
driver, lambda d: not _is_action_button_showing(d, config, config.LESSON_ALL_SCHEDULES_BUTTON),
|
||||
max_attempts=OPEN_RETRIES,
|
||||
)
|
||||
|
||||
|
||||
def _open_schedule_screen(driver, config):
|
||||
@ -209,10 +215,19 @@ def _scan_open_grid_cells(driver, config):
|
||||
|
||||
|
||||
def _close_region_grid(driver, config):
|
||||
# Called up to 12x per run (once per region during the scan, again
|
||||
# during execution) -- previously a single unverified driver.click, the
|
||||
# highest-risk gap the XIGNCODE-overlay audit found (2026-07-12): a
|
||||
# silently-missed click here leaves the next region's _open_region_grid
|
||||
# call starting from the wrong screen, which then fails its own
|
||||
# retries too and silently skips that region. navigation.click_back
|
||||
# adds the retry + overlay-recovery escalation this always needed.
|
||||
_close_grid_modal(driver, config)
|
||||
driver.wait(0.5)
|
||||
driver.click(*config.LESSON_BACK_BUTTON)
|
||||
driver.wait(1.5)
|
||||
if not navigation.click_back(
|
||||
driver, lambda d: not _is_action_button_showing(d, config, config.LESSON_ALL_SCHEDULES_BUTTON),
|
||||
):
|
||||
print("[lesson] warning: could not confirm return to the Location Select list after closing the region grid")
|
||||
|
||||
|
||||
def _scan_all_regions(driver, config):
|
||||
@ -360,6 +375,11 @@ def run(driver, config):
|
||||
print(f"[lesson] priority queue: {triple_count} triple(s), {double_count} double(s), {single_count} single(s)")
|
||||
_run_queue(driver, config, queue, tickets)
|
||||
|
||||
driver.click(*config.LESSON_BACK_BUTTON)
|
||||
driver.wait(1.5)
|
||||
# navigation.return_to_home over a bare click here too -- final cleanup
|
||||
# click, lower-risk than the mid-run ones above since ba_daily.py's own
|
||||
# wrapper calls return_to_home again regardless, but no reason not to
|
||||
# use the verified+overlay-safe path directly instead of an unverified
|
||||
# click first.
|
||||
if not navigation.return_to_home(driver):
|
||||
print("[lesson] warning: could not confirm return to home screen")
|
||||
print("[lesson] Done.")
|
||||
|
||||
@ -22,6 +22,10 @@ def run(driver, config):
|
||||
currency_label="credits",
|
||||
)
|
||||
|
||||
driver.click(*config.SHOP_BACK_BUTTON)
|
||||
driver.wait(1.5)
|
||||
# navigation.return_to_home over a bare click -- verified + carries the
|
||||
# XIGNCODE-overlay recovery escalation return_to_home has (see its
|
||||
# docstring); an unverified click here was a real gap found in a
|
||||
# 2026-07-12 audit of every direct BACK_BUTTON usage in this project.
|
||||
if not navigation.return_to_home(driver):
|
||||
print("[shop_common] warning: could not confirm return to home screen")
|
||||
print("[shop_common] Done.")
|
||||
|
||||
@ -29,6 +29,10 @@ def run(driver, config):
|
||||
currency_label="tactical coin",
|
||||
)
|
||||
|
||||
driver.click(*config.SHOP_BACK_BUTTON)
|
||||
driver.wait(1.5)
|
||||
# navigation.return_to_home over a bare click -- verified + carries the
|
||||
# XIGNCODE-overlay recovery escalation return_to_home has (see its
|
||||
# docstring); an unverified click here was a real gap found in a
|
||||
# 2026-07-12 audit of every direct BACK_BUTTON usage in this project.
|
||||
if not navigation.return_to_home(driver):
|
||||
print("[shop_tactical] warning: could not confirm return to home screen")
|
||||
print("[shop_tactical] Done.")
|
||||
|
||||
14
plan.md
14
plan.md
@ -626,6 +626,20 @@ The existing `is_modal_open` primitive (`MODAL_DIM_PROBE`, `(960,200)`), already
|
||||
|
||||
First pass made the pre-task `return_to_home` call a hard gate: if it failed, `_run_task` printed an error and skipped the task entirely without attempting it. Per explicit user correction — "I don't want to abort, I want it to be able to self heal. Return to home first, then proceed with the script." — this was reverted in favor of a self-healing design: a new `_ensure_home(name)` retries `navigation.return_to_home` across `PRE_TASK_HOME_RETRIES` (3) bounded attempts, `PRE_TASK_HOME_RETRY_WAIT` (5s) apart, giving a genuinely transient condition (a screen still mid-transition when first checked, a slow network hiccup dialog) real time to clear rather than hammering the same check back-to-back. Even if all 3 attempts still can't confirm home, the task is attempted anyway — never hard-aborted. Each task's own individual navigation already verifies its own state before acting (e.g. mailbox's own click-then-check on `MAILBOX_ICON`), so a task started from an unconfirmed state still fails safely at its own first step rather than cascading blindly — the same protection every task already has against a single missed click mid-run, just relied on one level higher up instead of gating on it. Confirmed live: normal case (game already home) still runs cleanly through the real CLI with no behavior change.
|
||||
|
||||
#### Return-to-home audit follow-up #2: the `XIGNCODE` overlay gets an automatic recovery (2026-07-12)
|
||||
|
||||
**Reported**: a real `bounty` run correctly detected `not_sweepable` (0/6 tickets — expected, already spent that day) and aborted cleanly without spending anything, but then both `bounty.py`'s own `navigation.return_to_home` call and `ba_daily.py`'s wrapper's post-task call failed to confirm reaching home — the user asked directly: "it couldn't press Esc key(?) to go to home."
|
||||
|
||||
This is a recurrence of a gotcha first found during `event_sweep.py`'s original live debugging (documented only in the now-superseded `Handoff.md`, never in `plan.md`/`CLAUDE.md`): an anti-cheat `XIGNCODE` window can render visibly on top of the game and intercept clicks at a fixed screen position, even though `xdotool getactivewindow` still reports `BlueArchive` as active throughout — `windowactivate` changes input focus, not window stacking order, so it doesn't fix this by itself. That first occurrence was written up as "not yet turned into an automatic recovery... since this was only seen once, incidentally, not reproduced deliberately." It has now recurred, in a completely different task, confirming it's a real recurring environment condition rather than a one-off fluke, so it earned the automatic recovery this time.
|
||||
|
||||
**Confirmed live via screenshot**: the game was sitting cleanly on the bounty Location Select screen (0/6 tickets, correctly not_sweepable, no stuck game modal at all) — but a small stray window rendered in the top-left corner, directly overlapping `navigation.BACK_BUTTON`'s `(85,55)` coordinate. `xdotool search --name XIGNCODE` confirmed the window existed; `getactivewindow` still reported `BlueArchive`, reproducing the exact misleading symptom from the original incident. Manually running `xdotool windowactivate` + `xdotool windowraise` on the BlueArchive window ID made the overlay disappear and `navigation.return_to_home` immediately succeeded afterward — confirming the diagnosis and the fix in one step, at zero further cost (no ticket, no click, purely a window-stacking operation).
|
||||
|
||||
**Fix, in two layers**: (1) `driver.focus_game()` — already called at the start of every single task — now calls `xdotool windowraise` right after `windowactivate`, preemptively clearing the overlay before any task's own navigation begins. (2) `navigation.return_to_home` gained a second, later-in-the-budget escalation: if still not home halfway through `RETURN_HOME_MAX_ROUNDS`, it calls `driver.focus_game()` once (re-raising the window) before continuing the remaining rounds — covering the case (exactly what happened here) where the overlay appears or becomes input-stealing *during* a run, after the task's own start-of-run `focus_game()` call already completed.
|
||||
|
||||
**Confirmed live end-to-end**: re-ran `./ba_dailies.sh bounty` for real (still 0/6 tickets, so free) after deploying the fix, with the overlay already cleared from the manual diagnosis step — no "could not confirm return to home screen" warning on either the internal or wrapper-level call this time, and a direct check confirmed the game genuinely on the home screen afterward. The fix hasn't yet been proven to self-recover *while the overlay is actively present* (the manual clearing during diagnosis happened to also fix it before the automated code path was exercised) — the next time this overlay appears during a live run, watch for `"[navigation] still not home halfway through recovery -- re-raising the game window..."` in the log to confirm the escalation path itself fires and works, not just that `focus_game()`'s own preemptive raise is enough.
|
||||
|
||||
**Follow-up, same session — "could the same error happen in other scripts?"** The two-layer fix above only covers `navigation.return_to_home`/`driver.focus_game()`. Auditing every direct use of the shared `(85,55)` coordinate found 5 places that clicked it *without* going through either: `shop_common.py`/`shop_tactical.py`'s end-of-run cleanup clicks, and — the real risk — `lesson.py`'s `_ensure_location_select_list` (had its own retry loop, but no overlay escalation) and `_close_region_grid` (called up to 12x per run, zero retry or verification at all; a silently-missed click here leaves the next region's `_open_region_grid` starting from the wrong screen and silently skipping that region). Fixed with a new shared `navigation.click_back(driver, verify_fn, max_attempts=3)` — the same retry + windowraise-escalation pattern as `return_to_home`, but parameterized by a caller-supplied verification check instead of hardcoding "reached home," for callers (like `lesson.py`'s per-region-map → list transition) that only want to go back ONE level. Applied to both `lesson.py` call sites; the three lower-risk end-of-run cleanup clicks (`lesson.py`, `shop_common.py`, `shop_tactical.py`) were switched to call `navigation.return_to_home` directly instead, since "go all the way home" is exactly their intent anyway. `config.SHOP_BACK_BUTTON`/`LESSON_BACK_BUTTON` were then removed as dead code (redundant with `navigation.BACK_BUTTON`, matching `EVENT_BACK_BUTTON`'s earlier removal for the identical reason). **Confirmed live**: `lesson` (0 tickets, so the no-op path — confirmed clean return home, no warnings), `shop_common` (a real purchase, 8 items/1,211,500 credits — confirmed clean return home), and `shop_tactical` (a real purchase, 2 items/45 tactical coin — confirmed clean return home) all ran successfully through the real CLI with the new navigation. `lesson.py`'s two hardened mid-run call sites (`_close_region_grid`/`_ensure_location_select_list`) weren't actually exercised by this test, though, since 0 tickets meant the scan/execute phase never ran — they'll get real coverage the next time `lesson` runs with tickets available.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### OCR
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user