From b2ed5719085445e12ecda8607d0df7b0faf8e178 Mon Sep 17 00:00:00 2001 From: Nik Afiq Date: Mon, 6 Jul 2026 01:58:01 +0900 Subject: [PATCH] feat(cafe): handle bond rank-up cutscene during pat loop and improve state verification --- CLAUDE.md | 3 ++- ba_auto/config.py | 8 ++++++++ ba_auto/reference_notes/mapping.md | 2 +- ba_auto/tasks/cafe.py | 19 +++++++++++++++++++ plan.md | 9 ++++++++- 5 files changed, 38 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 93b9711..11a262e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -448,7 +448,8 @@ Current project state (mailbox, cafe, stamina, and story_sweep all migrated to r - `ba_auto/driver.py` primitives (`run_command`, `focus_game`, `click`, `move_mouse`, `scroll`, `keypress`, `screenshot`, `wait`, `color_at`) are wired into all four task modules - `ba_auto/navigation.py` has two shared state probes used across tasks: `is_on_subscreen` (any mailbox/cafe/shop-style panel vs. the home screen) and `is_modal_open` (a dimmed dialog overlay), plus `wait_for_state()` (a scoped port of the reference's `core/picture.py::co_detect` — watch for any of several named states, react to known non-terminal ones, stop on a recognized terminal one; see `plan.md` Phase 10). `story_sweep.py` additionally has its own local modal probes/close logic because the stage-info modal is wide enough to break `is_modal_open`'s default probe point, and does not close on Escape at all (see `plan.md` Phase 9); its modal also renders at least two different internal layouts (a plain one for the bonus "-A" stage, a taller tabbed one for regular numbered stages) whose button coordinates differ, discovered live in Phase 10 - Live testing found both the mailbox-icon and cafe-icon fixed coordinates were flaky (missed the first click, worked on retry) and that neither task verified anything before proceeding, so a missed click cascaded into blind actions and could reach an unverified Escape press on the home screen — which triggers Blue Archive's own "exit the game?" confirmation. See `plan.md` Phases 5–6 for the full writeup. This is the concrete reason every task now verifies state before acting rather than trusting fixed coordinates or a single click blindly -- Not yet verified for cafe: rank-up popups mid-pat-loop, and whether camera zoom/pan can drift over a long unattended run (the reference project zooms out before detecting; ours does not, and testing didn't reproduce a failure from skipping it — see `plan.md` Phase 6 "Not verified" list) +- Rank-up popups mid-pat-loop are now handled: a pat that crosses an affection-rank threshold shows a full-screen "絆ランクアップ!" cutscene with no cafe header at all, which `find_cafe_sparkle()` can never recognize (it's nothing like the sparkle template) — the loop used to just spin uselessly against it for the rest of the room's click budget. `cafe.py`'s `_dismiss_rank_up_if_shown()` reuses the existing `navigation.is_on_subscreen` header-brightness probe (confirmed against `screenshots/cafe/student/01-02`: it reads r<200 during the cutscene vs. r>200 on the normal cafe screen) to detect it and press Enter until it clears, porting the reference's own `to_cafe()`-navigation handling of `relationship_rank_up` (`module/cafe_reward.py`). Not yet live-confirmed against a real rank-up trigger (it's semi-random, tied to hitting an affection threshold) — the fix is grounded in real captured screenshots, not a live end-to-end run; treat as implemented-but-unverified until one happens naturally during a real run +- Not yet verified for cafe: whether camera zoom/pan can drift over a long unattended run (the reference project zooms out before detecting; ours does not, and testing didn't reproduce a failure from skipping it — see `plan.md` Phase 6 "Not verified" list) Migration goal: diff --git a/ba_auto/config.py b/ba_auto/config.py index 78c485b..73aede5 100644 --- a/ba_auto/config.py +++ b/ba_auto/config.py @@ -31,6 +31,14 @@ CAFE_INCOME = (1780, 1105) # nothing and the loop keeps polling rather than giving up after one miss). CAFE_MAX_CLICKS_PER_ROOM = 15 CAFE_SPARKLE_TEMPLATE = os.path.join(ASSET_DIR, "cafe_sparkle.png") +# A pat that crosses an affection-rank threshold shows a full-screen "絆ラン +# クアップ!" (Bond Rank Up!) cutscene with no cafe header visible at all -- +# confirmed against screenshots/cafe/student/01-02: navigation.is_on_subscreen's +# header probe reads (183,220,240) there (r<200, fails) vs (248,249,250) on +# the normal cafe screen (r>200, passes), so the existing header-brightness +# check already tells the two apart. Bounds how many Enter presses +# _dismiss_rank_up_if_shown will try before giving up. +CAFE_RANK_UP_DISMISS_RETRIES = 5 # Home -> お仕事 (Work hub) -> 任務 (Task) card -> Normal/Hard story region browser. WORK_ICON = (1793, 1138) diff --git a/ba_auto/reference_notes/mapping.md b/ba_auto/reference_notes/mapping.md index f62dc1a..6cbc68d 100644 --- a/ba_auto/reference_notes/mapping.md +++ b/ba_auto/reference_notes/mapping.md @@ -5,7 +5,7 @@ Maps each local feature to the corresponding `~/repo/baas-reference/module/...` | Local feature | Reference file | Reference functions/classes | Local file | Backend replacements | Status | |---|---|---|---|---|---| | Mailbox | `module/mail.py` | `to_mail`, `implement` | `ba_auto/tasks/mailbox.py` | tap/click via xdotool, screenshot via scrot, `color.rgb_in_range` → `driver.color_at` pixel-probe check | Migrated: real Python, state-verified via color probe (no legacy bridge) | -| Cafe | `module/cafe_reward.py` | `to_cafe`, `interaction_for_cafe_solve_method3`, `collect` | `ba_auto/tasks/cafe.py` | `picture.co_detect`/`color.rgb_in_range` → `driver.color_at` pixel-probe checks; sparkle template match ported in-process into `ba_auto/detector.py` (`find_cafe_sparkle`, now multi-scale) | Migrated: real Python, state-verified via color probes (no legacy bridge). Pat loop now polls for the full attempt budget instead of stopping on the first miss (see `plan.md` Phase 6 follow-up) — not yet confirmed against a live sparkle since none was available during testing | +| Cafe | `module/cafe_reward.py` | `to_cafe` (its `relationship_rank_up` popup-handling now also ported, see below), `interaction_for_cafe_solve_method3`, `collect` | `ba_auto/tasks/cafe.py` | `picture.co_detect`/`color.rgb_in_range` → `driver.color_at` pixel-probe checks; sparkle template match ported in-process into `ba_auto/detector.py` (`find_cafe_sparkle`, now multi-scale) | Migrated: real Python, state-verified via color probes (no legacy bridge). Pat loop now polls for the full attempt budget instead of stopping on the first miss (see `plan.md` Phase 6 follow-up) — not yet confirmed against a live sparkle since none was available during testing. `_dismiss_rank_up_if_shown` reuses `navigation.is_on_subscreen` to detect and clear the full-screen bond-rank-up cutscene after a pat (see `plan.md` Phase 6 follow-up: rank-up popups) — not yet live-confirmed against a real trigger | | Stamina/AP | `module/collect_daily_free_power.py`, `module/collect_daily_task_power.py` | `to_tasks`/`implement` (task-power, ported); `to_purchase_pyroxenes_menu` (free-power, not ported) | `ba_auto/tasks/stamina.py` | `color.rgb_in_range` → `driver.color_at`; reference's per-tab claim loop → live UI's single "一括受取" bulk-claim button + Enter | Partially migrated: Mission-panel claim done (see `plan.md` Phase 8). Daily Free Power (real-money purchase menu) deliberately not automated | | Normal/Hard story AP sweep | `module/explore_tasks/sweep_task.py`, `module/explore_tasks/task_utils.py` | `to_region` (ported: OCR region-number readout + delta-click), a scoped-down `swipe_search_target_str` (ported: OCR stage-row label matching), `start_sweep`'s named-outcome contract (ported via `navigation.wait_for_state`, this project's scoped `co_detect` port) | `ba_auto/tasks/story_sweep.py` | OCR region/stage-name matching, ported for real (Phase 10) — replaces Phase 9's "next-region arrow stops advancing, then random stage" heuristic; MAX click verified via `SWEEP_MINUS_BUTTON_PROBE` color check (reused, still correct), modal closed via its own X button (Escape doesn't close it; X-button position re-calibrated per stage-layout variant, see Phase 10) | Done (see `plan.md` Phase 10, supersedes Phase 9). Config-driven exact `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS`), not latest-region/random-stage. Opt-in only, not in default flow | | Group/Club AP | `module/group.py` | Need to inspect | `ba_auto/tasks/group.py` | fixed click + state check via local driver | Not started | diff --git a/ba_auto/tasks/cafe.py b/ba_auto/tasks/cafe.py index 11cda52..8e7a2e7 100644 --- a/ba_auto/tasks/cafe.py +++ b/ba_auto/tasks/cafe.py @@ -33,6 +33,22 @@ def _enter_room(driver, coords): return False +def _dismiss_rank_up_if_shown(driver, config): + # The reference's own to_cafe() navigation (module/cafe_reward.py) treats + # 'relationship_rank_up' as a recognized, reactively-dismissed popup + # after every pat round -- this loop's original port had no equivalent, + # so a rank-up cutscene just sat there while find_cafe_sparkle() kept + # returning None against it (a full-screen character portrait, nothing + # like the sparkle template) for the rest of the room's click budget. + # That's the "freeze" -- not a timing fluke, a genuinely unhandled state. + for _ in range(config.CAFE_RANK_UP_DISMISS_RETRIES): + if navigation.is_on_subscreen(driver): + return True + driver.keypress("Return") + driver.wait(1.5) + return navigation.is_on_subscreen(driver) + + def _pat_room(driver, config): # Sparkles appear on a per-student cooldown, so most single checks find # nothing -- the old Bash loop (and an earlier version of this one) gave @@ -51,6 +67,9 @@ def _pat_room(driver, config): # park the cursor away from the sparkle area so it can't occlude the # next detection screenshot (see screenshots/cafe/sparkle/02_*_cursor_on_head.png) driver.move_mouse(10, 1190) + if not _dismiss_rank_up_if_shown(driver, config): + print("[cafe] warning: cafe screen not confirmed after a pat (rank-up cutscene stuck?) -- stopping this room's pat loop rather than clicking blindly") + break patted += 1 print(f"[cafe] patted sparkle at ({x}, {y}), score={score:.3f}") print(f"[cafe] patted {patted} sparkle(s)" if patted else "[cafe] no sparkle found") diff --git a/plan.md b/plan.md index f0142ed..18cc5fc 100644 --- a/plan.md +++ b/plan.md @@ -272,9 +272,16 @@ Verified live (two full runs against the real game, plus a manual step-by-step r Not verified / open risks: -- rank-up popups: not observed during testing (no student ranked up while testing), still unhandled if one appears mid-loop - whether zoom/pan state could drift over a long unattended run and eventually break sparkle detection (see above — no evidence of this yet, but the reference project treats it as necessary) +#### Phase 6 follow-up: rank-up popups mid-pat-loop ("it will freeze a bit") + +A user report during real usage: a pat that causes a bond-rank-up makes the loop "freeze a bit." Confirmed as a real, previously-unhandled gap, not a timing fluke — `find_cafe_sparkle()` was being asked to recognize a full-screen "絆ランクアップ!" cutscene (no cafe header, no chrome at all — see `screenshots/cafe/student/01`/`02`) as if it were the sparkle template, which it obviously never matches, so the loop just spun `driver.wait(1)` uselessly for the rest of the room's click budget. The reference's own `to_cafe()` navigation (`module/cafe_reward.py`) already treats `relationship_rank_up` as a recognized, reactively-dismissed popup checked after every pat round — this project's port had never carried that over. + +Fix: `cafe.py`'s `_dismiss_rank_up_if_shown()`, called after every pat (click + Enter + move-mouse), reuses the *existing* `navigation.is_on_subscreen` header-brightness probe rather than adding a new one — directly confirmed against the user-provided screenshots: the header probe point reads `(183, 220, 240)` during the cutscene (r<200, fails the check) vs. `(248, 249, 250)` on the normal cafe screen (r>200, passes). Presses Enter (bounded, `config.CAFE_RANK_UP_DISMISS_RETRIES = 5`) until `is_on_subscreen` confirms the cafe room is back, rather than assuming one Enter is enough; if it never clears, the pat loop stops rather than continuing to click blindly. + +Not yet live-confirmed against a real rank-up trigger — it's semi-random (tied to hitting an affection threshold) and didn't happen to occur during this session's testing. The fix is grounded in the user's own captured screenshots (a real observed state, precisely measured), not a guess, but a live run actually hitting this path and recovering cleanly is still open. + #### Phase 6 follow-up: "farming affection doesn't happen" report A later report claimed pats weren't landing at all, with the original `ba_dailies.sh` `do_cafe_room`/`do_cafe` pasted as the expected-behavior reference. Re-reading that Bash carefully changed the diagnosis: the original `detect_and_click.py` did one screenshot → detect → click per invocation and the Bash loop only kept calling it back-to-back while hits kept landing, breaking immediately on the first miss (`grep -q "^MATCH" || break`) — i.e. give-up-on-first-miss was the *original design*, not a regression introduced by the Python port. Detection math (mask, threshold `0.97`, click offset `(75, 47)`) ported over byte-for-byte identical.