feat: Implement OCR-driven story AP sweep with deterministic stage targeting
- Added `wait_for_state` function in `navigation.py` for state monitoring and reaction handling. - Updated `mapping.md` to reflect changes in story sweep implementation and OCR usage. - Refactored `story_sweep.py` to utilize OCR for region and stage identification, replacing random selection with configured targets. - Enhanced modal handling and confirmation checks for AP usage in `story_sweep.py`. - Updated setup script to require `tesseract` for OCR functionality and included installation instructions. - Revised `plan.md` to document the transition from heuristic to OCR-based stage targeting and the associated findings from live testing.
This commit is contained in:
parent
0707487934
commit
a8ff95d6f5
10
CLAUDE.md
10
CLAUDE.md
@ -24,7 +24,7 @@ The automation backend is local desktop control:
|
||||
|
||||
OCR is **not** an Android-specific concern — the reference's OCR-driven region/stage-name/currency matching runs against a screenshot and would work identically against a `scrot` capture on this backend. Only the reference's *input* (ADB/uiautomator2 taps) is Android-specific.
|
||||
|
||||
Two features (Stamina/AP mission claim, Normal/Hard story AP sweep) were built without OCR by substituting ad hoc pixel-probes, fixed coordinates, or randomized selection for the reference's OCR-driven navigation. For story sweep, that substitution produced more real bugs during live testing than porting the reference's actual approach would have — a wrong modal-open probe, an unverified button click that silently under-spent AP, a modal that doesn't close on Escape, and a latent hazard where a mistimed keypress could have started a real battle (see `plan.md` Phase 9's retrospective for the full writeup). None of those bugs would exist if the reference's deterministic, OCR-based stage targeting had been ported instead of replaced with a heuristic guess.
|
||||
Two features (Stamina/AP mission claim, Normal/Hard story AP sweep) were originally built without OCR by substituting ad hoc pixel-probes, fixed coordinates, or randomized selection for the reference's OCR-driven navigation. For story sweep, that substitution produced more real bugs during live testing than porting the reference's actual approach would have — a wrong modal-open probe, an unverified button click that silently under-spent AP, a modal that doesn't close on Escape, and a latent hazard where a mistimed keypress could have started a real battle (see `plan.md` Phase 9's retrospective for the full writeup). None of those bugs would exist if the reference's deterministic, OCR-based stage targeting had been ported instead of replaced with a heuristic guess. **This has since been fixed** — `story_sweep.py` now ports the reference's actual OCR-based region/stage targeting (see `plan.md` Phase 10) — but the retrospective stays here as the concrete, lived reason for the policy below, not a description of the current state of that task.
|
||||
|
||||
Going forward: when a reference feature's control flow depends on OCR, set up OCR and port that logic, rather than inventing a non-OCR workaround to avoid the setup cost. Only skip OCR for a specific step if the reference itself doesn't use OCR there.
|
||||
|
||||
@ -200,10 +200,10 @@ Expected venv:
|
||||
Quick check:
|
||||
|
||||
```
|
||||
ssh nik-gpu "which xdotool scrot && ~/.venvs/ba-auto-daily/bin/python3 -c 'import cv2, numpy; print(cv2.__version__)'"
|
||||
ssh nik-gpu "which xdotool scrot tesseract && ~/.venvs/ba-auto-daily/bin/python3 -c 'import cv2, numpy, pytesseract; print(cv2.__version__)'"
|
||||
```
|
||||
|
||||
OCR engine dependency: likely Tesseract or PaddleOCR. Not set up yet, but no longer something to defer casually — set it up as soon as a feature's reference implementation depends on it, rather than inventing a non-OCR workaround (see "OCR policy" above).
|
||||
OCR engine dependency: **set up** (Phase 10) — Tesseract via the `tesseract-ocr` apt package (needs an interactive `sudo`, so `setup.sh` checks for it but can't install it for you) plus `pytesseract` in the venv. `ba_auto/detector.py`'s `read_text()`/`read_int()` wrap it for occasional single-crop reads; see `story_sweep.py` for the first real usage.
|
||||
|
||||
## Bash policy
|
||||
|
||||
@ -442,11 +442,11 @@ Current project state (mailbox, cafe, stamina, and story_sweep all migrated to r
|
||||
- `ba_daily.py` dispatches `mailbox`/`cafe`/`stamina`/`story_sweep`/default flow to `ba_auto/tasks/`. Default flow (`DEFAULT_ORDER`) is `mailbox`, `cafe`, `stamina` — `story_sweep` is opt-in only since it spends AP rather than reclaiming something free
|
||||
- `ba_auto/tasks/mailbox.py` and `ba_auto/tasks/cafe.py` click with `ba_auto/driver.py` primitives and verify state with `driver.color_at`/`ba_auto/navigation.py` (ported from `module/mail.py` and `module/cafe_reward.py`'s `rgb_in_range`/`co_detect` pattern) before pressing further keys — no legacy bridge remains
|
||||
- `ba_auto/tasks/stamina.py` claims the Mission panel's bulk "一括受取" button (see `plan.md` Phase 8)
|
||||
- `ba_auto/tasks/story_sweep.py` sweeps the latest unlocked region's stages for AP (see `plan.md` Phase 9, including its OCR-avoidance retrospective — this task is the concrete example behind the "OCR policy" above)
|
||||
- `ba_auto/tasks/story_sweep.py` sweeps a config-driven list of exact `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS`) for AP, navigating to each via OCR (region-number readout + delta-click, stage-label matching) rather than a random pick (see `plan.md` Phase 10, which replaced Phase 9's random-pick design per Phase 9's own OCR-avoidance retrospective — this task is the concrete example behind the "OCR policy" above)
|
||||
- `ba_auto/detector.py` has `find_cafe_sparkle()`, the sparkle template-match ported in-process from the now-deleted `scripts/detect_and_click.py`
|
||||
- `scripts/ba_dailies_legacy.sh` and `scripts/detect_and_click.py` have been deleted — nothing references them anymore, and the `scripts/` directory itself no longer exists
|
||||
- `ba_auto/driver.py` primitives (`run_command`, `focus_game`, `click`, `move_mouse`, `scroll`, `keypress`, `screenshot`, `wait`, `color_at`) are wired into all four task modules
|
||||
- `ba_auto/navigation.py` has two shared state probes used across tasks: `is_on_subscreen` (any mailbox/cafe/shop-style panel vs. the home screen) and `is_modal_open` (a dimmed dialog overlay). `story_sweep.py` additionally has its own local modal probes/close logic because the stage-info modal is wide enough to break `is_modal_open`'s default probe point, and does not close on Escape at all (see `plan.md` Phase 9)
|
||||
- `ba_auto/navigation.py` has two shared state probes used across tasks: `is_on_subscreen` (any mailbox/cafe/shop-style panel vs. the home screen) and `is_modal_open` (a dimmed dialog overlay), plus `wait_for_state()` (a scoped port of the reference's `core/picture.py::co_detect` — watch for any of several named states, react to known non-terminal ones, stop on a recognized terminal one; see `plan.md` Phase 10). `story_sweep.py` additionally has its own local modal probes/close logic because the stage-info modal is wide enough to break `is_modal_open`'s default probe point, and does not close on Escape at all (see `plan.md` Phase 9); its modal also renders at least two different internal layouts (a plain one for the bonus "-A" stage, a taller tabbed one for regular numbered stages) whose button coordinates differ, discovered live in Phase 10
|
||||
- Live testing found both the mailbox-icon and cafe-icon fixed coordinates were flaky (missed the first click, worked on retry) and that neither task verified anything before proceeding, so a missed click cascaded into blind actions and could reach an unverified Escape press on the home screen — which triggers Blue Archive's own "exit the game?" confirmation. See `plan.md` Phases 5–6 for the full writeup. This is the concrete reason every task now verifies state before acting rather than trusting fixed coordinates or a single click blindly
|
||||
- Not yet verified for cafe: rank-up popups mid-pat-loop, and whether camera zoom/pan can drift over a long unattended run (the reference project zooms out before detecting; ours does not, and testing didn't reproduce a failure from skipping it — see `plan.md` Phase 6 "Not verified" list)
|
||||
|
||||
|
||||
@ -30,7 +30,7 @@ Current task status:
|
||||
| `mailbox` | Real Python (`ba_auto/tasks/mailbox.py`). Verifies the mailbox panel actually opened (via a pixel-color probe, `driver.color_at`) before clicking "claim all" or pressing any further keys. Retries the open-click up to 3 times before giving up safely. |
|
||||
| `cafe` | Real Python (`ba_auto/tasks/cafe.py`). Verifies each room/dialog transition the same way as `mailbox` before acting; sparkle detection runs in-process via `ba_auto/detector.py` instead of shelling out per click. See "Fixed: the exit-game dialog bug" below for what this replaced, and `plan.md` Phase 6 follow-up for the multi-scale detection + persistent-polling changes made after a "farming affection doesn't happen" report. |
|
||||
| `stamina` | Real Python (`ba_auto/tasks/stamina.py`). Opens the Mission panel and claims via its bulk "一括受取" button (Enter key) when enabled. Does **not** touch the Pyroxene Purchase (青輝石購入) menu's free-AP claim — that's a real-money purchase screen and was deliberately left unautomated; see `plan.md` Phase 8. |
|
||||
| `story_sweep` | Real Python (`ba_auto/tasks/story_sweep.py`). **Spends AP** — opt-in only, not part of the default flow. Picks the latest unlocked story region, a random stage in it, and sweeps with the in-game MAX count. Verifies the MAX click actually raised the count before starting the sweep, and closes the stage-info modal via its own X button afterward (Escape doesn't close it — confirmed live). See `plan.md` Phase 9. |
|
||||
| `story_sweep` | Real Python (`ba_auto/tasks/story_sweep.py`). **Spends AP** — opt-in only, not part of the default flow. Sweeps a config-driven list of exact `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS` — edit this before running for real, it ships with a placeholder), navigating to each via OCR (region-number readout + stage-label matching) rather than a random pick. Verifies the MAX/`+` click actually raised the count before starting the sweep, clicks through the AP-usage-confirmation dialog, and closes the stage-info modal via its own X button afterward (Escape doesn't close it — confirmed live). See `plan.md` Phase 10. |
|
||||
|
||||
## Prerequisites on nik-gpu
|
||||
|
||||
@ -38,10 +38,10 @@ One-time, or after a dependency change:
|
||||
|
||||
```bash
|
||||
ssh nik-gpu
|
||||
which xdotool scrot # both must be installed
|
||||
which xdotool scrot tesseract # all three must be installed
|
||||
```
|
||||
|
||||
`setup.sh` (see below) creates the Python venv and checks these for you.
|
||||
`tesseract` (the OCR engine `story_sweep` uses for region/stage-label reads) needs `sudo apt install tesseract-ocr` — `setup.sh` checks for it but can't install it for you, since sudo needs an interactive password. `setup.sh` (see below) creates the Python venv (including `pytesseract`) and checks all three tools for you.
|
||||
|
||||
## Deploying your changes
|
||||
|
||||
|
||||
@ -34,11 +34,32 @@ CAFE_SPARKLE_TEMPLATE = os.path.join(ASSET_DIR, "cafe_sparkle.png")
|
||||
|
||||
# Home -> お仕事 (Work hub) -> 任務 (Task) card -> Normal/Hard story region browser.
|
||||
WORK_ICON = (1793, 1138)
|
||||
TASK_CARD = (1370, 450)
|
||||
# Clicking past the last region is a harmless no-op (verified live) -- this
|
||||
# just needs to be >= the number of regions that will ever exist.
|
||||
# Moved up from the original (1370, 450): that point sat close enough to the
|
||||
# 任務 card's bottom edge that a live Phase 10 run missed and landed on the
|
||||
# "総力戦" (Total War) card in the row below instead -- confirmed live via
|
||||
# screenshot, not just a hunch. (1250, 380) sits solidly mid-card, on the
|
||||
# "任務" title text itself, well clear of every edge.
|
||||
TASK_CARD = (1250, 380)
|
||||
REGION_RIGHT_ARROW = (1862, 598)
|
||||
REGION_RIGHT_ARROW_MAX_CLICKS = 60
|
||||
# Pixel-scanline-scanned (not visually estimated -- see plan.md Phase 8's
|
||||
# lesson) from scratchpad/stage_info.png: the "<" chevron's navy-blue pixel
|
||||
# centroid was (66, 597), mirroring REGION_RIGHT_ARROW. Used for the
|
||||
# OCR-driven to_region port (ba_auto/tasks/story_sweep.py) to step backward
|
||||
# when the current region is past the target.
|
||||
REGION_LEFT_ARROW = (66, 598)
|
||||
# Bounds the "read region, click delta, re-check" loop in
|
||||
# ba_auto/tasks/story_sweep.py's _go_to_region. A correct read normally
|
||||
# converges in one round; this just guards against a stuck OCR misread.
|
||||
REGION_NAV_MAX_ATTEMPTS = 8
|
||||
|
||||
# Region-number readout on the region browser's left panel (the "Area 30"
|
||||
# card's big digits, below the smaller "Area" label). Rect pixel-scanned from
|
||||
# scratchpad/stage_info.png: the "Area" label occupies roughly y 295-325, the
|
||||
# number itself y 330-372 -- this rect isolates just the digits. Replaces
|
||||
# _go_to_latest_region's "spam the arrow and hope" (see plan.md Phase 9's
|
||||
# retrospective) with task_utils.py::to_region's actual OCR-read-and-click-
|
||||
# the-exact-delta approach.
|
||||
REGION_NUMBER_OCR_RECT = (175, 325, 250, 380)
|
||||
|
||||
# Stage list panel (right side of the region browser). Scrolling to either
|
||||
# extreme always shows exactly 4 full stage rows, since every region has at
|
||||
@ -47,26 +68,113 @@ REGION_RIGHT_ARROW_MAX_CLICKS = 60
|
||||
# (enter) button sits at STAGE_ENTER_X across both scroll extremes; the two
|
||||
# row-position sets below were measured at each extreme.
|
||||
STAGE_LIST_SCROLL_POINT = (1400, 700)
|
||||
STAGE_LIST_SCROLL_CLICKS = 10
|
||||
# 10 was the original (Phase 9) calibration, but live re-testing during
|
||||
# Phase 10 found it insufficient to reach the opposite extreme when the list
|
||||
# was already scrolled near the other end (it undershot, landing between the
|
||||
# two calibrated row-position sets and producing garbled OCR reads) -- 20
|
||||
# reliably reached either extreme regardless of starting position. Scrolling
|
||||
# past either end remains a harmless no-op (verified live both phases).
|
||||
STAGE_LIST_SCROLL_CLICKS = 20
|
||||
STAGE_ENTER_X = 1683
|
||||
STAGE_ROWS_AT_TOP_Y = (424, 570, 718, 866)
|
||||
STAGE_ROWS_AT_BOTTOM_Y = (483, 630, 778, 926)
|
||||
|
||||
# Stage-label OCR rect (e.g. "30-1", "30-A"), offset from a row's known
|
||||
# center y above. Pixel-scanned across all eight row positions (both
|
||||
# STAGE_ROWS_AT_TOP_Y and _BOTTOM_Y) in live captures -- x[1030,1150],
|
||||
# y[row_y-40, row_y-4] consistently isolates just the label text line above
|
||||
# the row's star-rating icons. The symmetric row_y+/-40 crop tried first
|
||||
# during calibration included the stars below and reliably broke OCR (empty
|
||||
# or garbled reads) even with a character whitelist -- see plan.md Phase 10.
|
||||
# Replaces _pick_random_stage_row's "scroll to an extreme, grab a random one
|
||||
# of the 4 rows" with a scoped-down port of the reference's
|
||||
# swipe_search_target_str: since this client's stage list only ever has 2
|
||||
# relevant scroll positions (top/bottom extremes, both already known-good),
|
||||
# full swipe-and-retry generality isn't needed -- just OCR each of the 4
|
||||
# visible rows at each extreme and match by label text.
|
||||
STAGE_LABEL_OCR_X = (1030, 1150)
|
||||
STAGE_LABEL_OCR_Y_PAD = (40, 4)
|
||||
|
||||
# 任務情報 (stage info) modal's sweep sub-panel. This modal is wide enough
|
||||
# that navigation.MODAL_DIM_PROBE (960, 200) lands on the modal's own white
|
||||
# card instead of the dimmed backdrop -- use a corner point that's outside
|
||||
# the card in either scroll/region state instead.
|
||||
STAGE_MODAL_PROBE = (1870, 600)
|
||||
SWEEP_MAX_BUTTON = (1631, 507)
|
||||
SWEEP_START_BUTTON = (1400, 670)
|
||||
# Regular numbered stages (30-1..30-5) render an extra "集中指揮"/"簡易攻略"
|
||||
# tab row and a manual "任務開始" panel below the sweep panel that the
|
||||
# bonus "-A" stage Phase 9 originally calibrated against does not have --
|
||||
# discovered live during Phase 10 when Phase 9's coordinates (calibrated
|
||||
# only against 30-A) missed the MAX button on 30-3 by ~43px vertically.
|
||||
# These values are pixel-scanned against 30-3's tabbed layout and are what
|
||||
# config.STORY_SWEEP_TARGETS will hit in the common case (sweeping a
|
||||
# regular numbered stage, not the "-A" bonus stage). If a target's stage is
|
||||
# "A", these may be off by the same ~43px the old (untabbed) calibration
|
||||
# used -- not yet re-confirmed against an actual "-A" stage since this fix;
|
||||
# see plan.md Phase 10.
|
||||
SWEEP_MAX_BUTTON = (1620, 550)
|
||||
SWEEP_START_BUTTON = (1400, 710)
|
||||
# The "-" stepper button next to the sweep count: flat grey (240,240,239)
|
||||
# while count is still at its default of 1, vivid orange (255,111,0) once
|
||||
# MAX (or any +) has raised it. Used to verify the MAX click actually landed
|
||||
# instead of trusting a single click blindly, since this gates real AP spend.
|
||||
SWEEP_MINUS_BUTTON_PROBE = (1280, 507)
|
||||
SWEEP_MINUS_BUTTON_PROBE = (1285, 555)
|
||||
# The modal's own "X" close icon (top-right corner of the white card).
|
||||
# Escape does NOT close this modal (confirmed live: two Escape presses left
|
||||
# it open with focus on the live "任務開始"/start-mission button) -- must
|
||||
# click this explicitly. Pinned via pixel-scanline scan of the glyph's
|
||||
# crossing point, not visual estimation.
|
||||
STAGE_MODAL_CLOSE_BUTTON = (1691, 271)
|
||||
#
|
||||
# The whole card (not just the sweep sub-panel) is vertically centered on
|
||||
# its own content height rather than anchored at a fixed absolute position
|
||||
# -- confirmed live during Phase 10: the tabbed regular-stage layout (see
|
||||
# SWEEP_MAX_BUTTON above) is taller than the "-A" bonus-stage layout this
|
||||
# was originally calibrated against, and its X button sits ~46px higher
|
||||
# on screen (225 vs the old 271) as a result. This value is re-measured
|
||||
# against the taller, tabbed layout.
|
||||
STAGE_MODAL_CLOSE_BUTTON = (1691, 225)
|
||||
# "+" stepper button, for configured exact (non-"max") sweep counts. Pinned
|
||||
# via color-scan (bright cyan glyph centroid) against 30-3's tabbed layout
|
||||
# (see SWEEP_MAX_BUTTON above) -- unlike SWEEP_MAX_BUTTON/
|
||||
# SWEEP_MINUS_BUTTON_PROBE this specific button has NOT been live-clicked
|
||||
# yet; confirm it before relying on a non-"max" configured count (see
|
||||
# plan.md Phase 10).
|
||||
SWEEP_PLUS_BUTTON = (1520, 550)
|
||||
|
||||
# Clicking 掃討開始 (start sweep) always raises an "AP<N>使用して、掃討を
|
||||
# <M>回行いますか?" usage-confirmation dialog before the sweep actually
|
||||
# runs -- discovered live during Phase 10; the previous design had no
|
||||
# handling for this dialog at all, which is what a crude early placeholder
|
||||
# probe was misreading as "inadequate_ap" on every sweep, successful or not.
|
||||
#
|
||||
# If AP is too low for even one sweep (confirmed live by deliberately
|
||||
# emptying the count via MAX/"+" at low AP), a dialog that looks the same
|
||||
# and sits at the *same* OK-button position appears instead, but titled
|
||||
# "AP購入" (real-currency AP purchase) with a gold/yellow OK button instead
|
||||
# of cyan. The two are told apart by that color, not by position.
|
||||
SWEEP_CONFIRM_BUTTON = (1150, 810)
|
||||
SWEEP_CONFIRM_CANCEL_BUTTON = (770, 810)
|
||||
SWEEP_CONFIRM_CYAN = ((90, 190, 230), (200, 240, 256))
|
||||
SWEEP_CONFIRM_GOLD = ((200, 200, 50), (256, 256, 150))
|
||||
|
||||
# Region spanning every button this task clicks through after 掃討開始: the
|
||||
# AP-usage-confirm OK above, and the "掃討完了" (sweep complete) results
|
||||
# screen's "SKIP" (first, skips the reward-reveal animation) and final "OK"
|
||||
# (after full reward totals appear) buttons. SKIP and the final OK share
|
||||
# SWEEP_CONFIRM_CYAN's color but sit ~120px apart vertically, so this task
|
||||
# finds whichever one is showing by color within this region instead of
|
||||
# hardcoding each dialog's exact Y position.
|
||||
SWEEP_RESULT_BUTTON_REGION = (700, 700, 1300, 1050)
|
||||
|
||||
# (region, stage, count) targets to sweep, mirroring the reference's
|
||||
# unfinished_normal_tasks shape (module/explore_tasks/sweep_task.py). `stage`
|
||||
# is 1-5, or the string "A" for the bonus stage that only exists when
|
||||
# region % 3 == 0. `count` is a positive int (uses the "+" stepper) or the
|
||||
# literal string "max" (uses the in-game MAX button).
|
||||
#
|
||||
# PLACEHOLDER -- region 1 stage 1 is unlikely to be what you actually want
|
||||
# swept (it may not even be 3-starred/cleared on this account yet). Edit
|
||||
# this list with your own already-cleared stage(s) before running
|
||||
# story_sweep for real.
|
||||
STORY_SWEEP_TARGETS = [
|
||||
(1, 1, "max"),
|
||||
]
|
||||
|
||||
@ -3,9 +3,13 @@ import os
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
import pytesseract
|
||||
|
||||
from ba_auto import config, driver
|
||||
|
||||
OCR_SHOT_PATH = os.path.join(config.SCRATCHPAD_DIR, "ocr_live.png")
|
||||
OCR_UPSCALE = 3
|
||||
|
||||
SPARKLE_SHOT_PATH = os.path.join(config.SCRATCHPAD_DIR, "cafe_live.png")
|
||||
SPARKLE_CLICK_OFFSET = (75, 47)
|
||||
SPARKLE_THRESHOLD = 0.97
|
||||
@ -48,3 +52,82 @@ def find_cafe_sparkle():
|
||||
score, x, y, tw, th, scale = best
|
||||
ox, oy = SPARKLE_CLICK_OFFSET
|
||||
return (x + tw // 2 + round(ox * scale), y + th // 2 + round(oy * scale), score)
|
||||
|
||||
|
||||
def _color_mask(region, rgb_min, rgb_max):
|
||||
x1, y1, x2, y2 = region
|
||||
driver.screenshot(OCR_SHOT_PATH)
|
||||
img = cv2.imread(OCR_SHOT_PATH)
|
||||
crop = img[y1:y2, x1:x2]
|
||||
b, g, r = crop[:, :, 0].astype(np.int16), crop[:, :, 1].astype(np.int16), crop[:, :, 2].astype(np.int16)
|
||||
(r_lo, g_lo, b_lo), (r_hi, g_hi, b_hi) = rgb_min, rgb_max
|
||||
return (r >= r_lo) & (r <= r_hi) & (g >= g_lo) & (g <= g_hi) & (b >= b_lo) & (b <= b_hi)
|
||||
|
||||
|
||||
def region_contains_color(region, rgb_min, rgb_max):
|
||||
"""Whether any pixel within `region` (x1, y1, x2, y2) falls in the given
|
||||
RGB range. Useful for presence checks on small, non-convex glyphs (e.g.
|
||||
an arrow chevron) where a single fixed-point probe can land in the
|
||||
glyph's own concave gap -- confirmed live: a centroid-based single point
|
||||
for story_sweep's region-arrow chevron fell squarely in the notch
|
||||
between its two strokes, reading as "absent" even while the arrow was
|
||||
clearly rendered a few pixels away. See plan.md Phase 10.
|
||||
"""
|
||||
return bool(_color_mask(region, rgb_min, rgb_max).any())
|
||||
|
||||
|
||||
def find_color_centroid(region, rgb_min, rgb_max):
|
||||
"""Centroid `(x, y)` of every pixel within `region` (x1, y1, x2, y2)
|
||||
falling in the given RGB range, or None if none match. Useful for
|
||||
clicking a known-colored button whose exact position varies between
|
||||
otherwise-similar dialogs -- e.g. story_sweep's sweep-result screen
|
||||
shows the same cyan confirm-button color for its "SKIP" and final "OK"
|
||||
states, ~120px apart vertically; finding it by color avoids hardcoding
|
||||
both positions. See plan.md Phase 10.
|
||||
"""
|
||||
x1, y1, x2, y2 = region
|
||||
mask = _color_mask(region, rgb_min, rgb_max)
|
||||
ys, xs = np.nonzero(mask)
|
||||
if len(xs) == 0:
|
||||
return None
|
||||
return (x1 + int(xs.mean()), y1 + int(ys.mean()))
|
||||
|
||||
|
||||
def _ocr_crop(region):
|
||||
x1, y1, x2, y2 = region
|
||||
driver.screenshot(OCR_SHOT_PATH)
|
||||
img = cv2.imread(OCR_SHOT_PATH)
|
||||
crop = img[y1:y2, x1:x2]
|
||||
gray = cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY)
|
||||
# This UI's text is consistently dark-on-light -- a hard black/white
|
||||
# threshold measurably fixed real misreads during live calibration
|
||||
# (e.g. a stage label's digit misread as a stray extra digit) that
|
||||
# persisted across every psm mode until the anti-aliased grey edges were
|
||||
# removed. Confirmed live against scratchpad/scroll_up20.png row labels.
|
||||
_, thresh = cv2.threshold(gray, 150, 255, cv2.THRESH_BINARY)
|
||||
return cv2.resize(thresh, None, fx=OCR_UPSCALE, fy=OCR_UPSCALE, interpolation=cv2.INTER_CUBIC)
|
||||
|
||||
|
||||
def read_text(region, whitelist=None, psm=7, lang="eng"):
|
||||
"""OCR a pixel rectangle `(x1, y1, x2, y2)` from a fresh screenshot.
|
||||
|
||||
Local equivalent of the reference's core/ocr/ocr.py Baas_ocr client
|
||||
(get_region_res) -- without its socket/shared-memory server, which exists
|
||||
there to make OCR fast across thousands of automation steps. This project
|
||||
only needs occasional single-crop reads (a region number, a stage label),
|
||||
so a plain in-process pytesseract call is enough; see CLAUDE.md's "OCR
|
||||
policy" and Handoff.md. `lang="eng"` is sufficient for the digit/dash
|
||||
labels this project reads (region numbers, "30-1"/"30-A" stage labels) --
|
||||
no Japanese trained data is needed for those specific reads.
|
||||
"""
|
||||
crop = _ocr_crop(region)
|
||||
tess_config = f"--psm {psm}"
|
||||
if whitelist:
|
||||
tess_config += f" -c tessedit_char_whitelist={whitelist}"
|
||||
return pytesseract.image_to_string(crop, lang=lang, config=tess_config).strip()
|
||||
|
||||
|
||||
def read_int(region, psm=7):
|
||||
"""Local equivalent of the reference's recognize_int."""
|
||||
digits = "".join(ch for ch in read_text(region, whitelist="0123456789", psm=psm) if ch.isdigit())
|
||||
return int(digits) if digits else None
|
||||
|
||||
@ -18,3 +18,39 @@ def is_on_subscreen(driver):
|
||||
def is_modal_open(driver):
|
||||
r, g, b = driver.color_at(*MODAL_DIM_PROBE)
|
||||
return r < MODAL_DIM_MAX_CHANNEL and g < MODAL_DIM_MAX_CHANNEL and b < MODAL_DIM_MAX_CHANNEL
|
||||
|
||||
|
||||
def wait_for_state(driver, config, reactions, ends, max_iterations=30, poll_interval=1.0):
|
||||
"""Generic "watch the screen, react to anything recognized, stop once a
|
||||
recognized destination is reached" loop -- the local equivalent of the
|
||||
reference's core/picture.py::co_detect, scoped to what this project
|
||||
actually needs (a handful of named checks) rather than co_detect's full
|
||||
generality (which spans the whole reference project via ~20 image
|
||||
template assets this project doesn't have).
|
||||
|
||||
`ends`: {check_fn(driver, config) -> bool: outcome_name}. Checked first,
|
||||
every iteration; the first match stops the loop and returns its name.
|
||||
|
||||
`reactions`: {check_fn(driver, config) -> bool: action(driver)}. Checked
|
||||
if no end matched; the first match runs its action (a click, a keypress,
|
||||
whatever the recognized state calls for) and the loop continues.
|
||||
|
||||
If neither an end nor a reaction matches, the loop just waits and retries
|
||||
-- it never falls back to a blind click/keypress guess (see CLAUDE.md's
|
||||
exit-game-dialog writeup for why that was a real bug elsewhere).
|
||||
|
||||
Returns the matched end's outcome name, or None once max_iterations is
|
||||
exhausted without reaching a recognized end -- callers should treat None
|
||||
as "unrecognized state, abort safely."
|
||||
"""
|
||||
for _ in range(max_iterations):
|
||||
for check_fn, outcome_name in ends.items():
|
||||
if check_fn(driver, config):
|
||||
return outcome_name
|
||||
for check_fn, action in reactions.items():
|
||||
if check_fn(driver, config):
|
||||
action(driver)
|
||||
break
|
||||
else:
|
||||
driver.wait(poll_interval)
|
||||
return None
|
||||
|
||||
@ -7,7 +7,7 @@ Maps each local feature to the corresponding `~/repo/baas-reference/module/...`
|
||||
| Mailbox | `module/mail.py` | `to_mail`, `implement` | `ba_auto/tasks/mailbox.py` | tap/click via xdotool, screenshot via scrot, `color.rgb_in_range` → `driver.color_at` pixel-probe check | Migrated: real Python, state-verified via color probe (no legacy bridge) |
|
||||
| Cafe | `module/cafe_reward.py` | `to_cafe`, `interaction_for_cafe_solve_method3`, `collect` | `ba_auto/tasks/cafe.py` | `picture.co_detect`/`color.rgb_in_range` → `driver.color_at` pixel-probe checks; sparkle template match ported in-process into `ba_auto/detector.py` (`find_cafe_sparkle`, now multi-scale) | Migrated: real Python, state-verified via color probes (no legacy bridge). Pat loop now polls for the full attempt budget instead of stopping on the first miss (see `plan.md` Phase 6 follow-up) — not yet confirmed against a live sparkle since none was available during testing |
|
||||
| Stamina/AP | `module/collect_daily_free_power.py`, `module/collect_daily_task_power.py` | `to_tasks`/`implement` (task-power, ported); `to_purchase_pyroxenes_menu` (free-power, not ported) | `ba_auto/tasks/stamina.py` | `color.rgb_in_range` → `driver.color_at`; reference's per-tab claim loop → live UI's single "一括受取" bulk-claim button + Enter | Partially migrated: Mission-panel claim done (see `plan.md` Phase 8). Daily Free Power (real-money purchase menu) deliberately not automated |
|
||||
| Normal/Hard story AP sweep | `module/explore_tasks/sweep_task.py`, `module/explore_tasks/task_utils.py` | `to_region`/`to_normal_event` + OCR-driven per-stage claim loop (not ported) | `ba_auto/tasks/story_sweep.py` | OCR region/stage-name matching → plain state-change probing (next-region arrow stops advancing); reference's per-stage claim loop → this client's stage-info modal's self-contained 掃討 sweep sub-panel (MIN/-/+/MAX stepper + start button), MAX click verified via `SWEEP_MINUS_BUTTON_PROBE` color check, modal closed via its own X button (Escape doesn't close it) | Done (see `plan.md` Phase 9). Latest-region + random-stage selection and MAX-count spend per explicit user direction. Opt-in only, not in default flow |
|
||||
| Normal/Hard story AP sweep | `module/explore_tasks/sweep_task.py`, `module/explore_tasks/task_utils.py` | `to_region` (ported: OCR region-number readout + delta-click), a scoped-down `swipe_search_target_str` (ported: OCR stage-row label matching), `start_sweep`'s named-outcome contract (ported via `navigation.wait_for_state`, this project's scoped `co_detect` port) | `ba_auto/tasks/story_sweep.py` | OCR region/stage-name matching, ported for real (Phase 10) — replaces Phase 9's "next-region arrow stops advancing, then random stage" heuristic; MAX click verified via `SWEEP_MINUS_BUTTON_PROBE` color check (reused, still correct), modal closed via its own X button (Escape doesn't close it; X-button position re-calibrated per stage-layout variant, see Phase 10) | Done (see `plan.md` Phase 10, supersedes Phase 9). Config-driven exact `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS`), not latest-region/random-stage. Opt-in only, not in default flow |
|
||||
| Group/Club AP | `module/group.py` | Need to inspect | `ba_auto/tasks/group.py` | fixed click + state check via local driver | Not started |
|
||||
| Bounty | `module/rewarded_task.py` | Need to inspect | `ba_auto/tasks/bounty.py` | sweep/color/OCR adaptation | Not started |
|
||||
| Commissions | `module/clear_special_task_power.py` | Need to inspect | `ba_auto/tasks/commission.py` | sweep/color adaptation | Not started |
|
||||
|
||||
@ -1,19 +1,27 @@
|
||||
"""Normal story AP sweep. Reference: baas-reference/module/explore_tasks/sweep_task.py
|
||||
and task_utils.py (to_region/to_normal_event + an OCR-driven per-stage claim loop).
|
||||
and task_utils.py.
|
||||
|
||||
That reference flow needs OCR (to read the current region number and match
|
||||
stage-name text via swipe_search_target_str) and per-locale template assets we
|
||||
don't have. This client exposes a much simpler path for the same goal (burn AP
|
||||
via already-3-starred stages): each stage's own info panel has a "掃討" (sweep)
|
||||
sub-panel with a count stepper (MIN/-/+/MAX) and a start button. So instead of
|
||||
porting the OCR-based region/stage lookup, this picks the latest unlocked
|
||||
region by spamming the "next region" arrow until it stops advancing (a plain
|
||||
state-change check, no OCR), then picks one of its stages essentially at
|
||||
random (see _pick_random_stage_row) and sweeps it with the in-game MAX count.
|
||||
Ported per CLAUDE.md's "OCR policy" and Handoff.md: sweeps a config-driven
|
||||
list of exact (region, stage, count) targets (config.STORY_SWEEP_TARGETS),
|
||||
navigating to each one deterministically instead of the previous "latest
|
||||
unlocked region, then a random stage" heuristic -- see plan.md Phase 9's
|
||||
retrospective for why that heuristic was a mistake.
|
||||
|
||||
- `_go_to_region` ports task_utils.py::to_region: OCR the current region
|
||||
number, click the exact delta, re-check, bounded loop.
|
||||
- `_find_stage_row` is a scoped-down port of core/image.py's
|
||||
swipe_search_target_str: OCR each visible stage row's label and match it
|
||||
against the configured target, rather than grabbing a random row.
|
||||
- `_watch_sweep_result` is built on navigation.wait_for_state, this
|
||||
project's scoped port of core/picture.py::co_detect, and returns a named
|
||||
outcome ("swept", "inadequate_ap", "unrecognized_state", ...) the way the
|
||||
reference's start_sweep does, instead of a single generic "Done".
|
||||
|
||||
The MAX-button click-then-verify and the modal's own X-button close (both
|
||||
calibrated and confirmed live in Phase 9) are reused unchanged -- see
|
||||
plan.md Phase 9/10.
|
||||
"""
|
||||
import random
|
||||
|
||||
from ba_auto import navigation
|
||||
from ba_auto import detector, navigation
|
||||
|
||||
OPEN_RETRIES = 3
|
||||
POST_SWEEP_DISMISS_ROUNDS = 6
|
||||
@ -27,10 +35,40 @@ def _is_stage_modal_open(driver, config):
|
||||
return r < STAGE_MODAL_DIM_MAX_CHANNEL and g < STAGE_MODAL_DIM_MAX_CHANNEL and b < STAGE_MODAL_DIM_MAX_CHANNEL
|
||||
|
||||
|
||||
def _color_in_range(rgb, rgb_range):
|
||||
lo, hi = rgb_range
|
||||
r, g, b = rgb
|
||||
return lo[0] <= r <= hi[0] and lo[1] <= g <= hi[1] and lo[2] <= b <= hi[2]
|
||||
|
||||
|
||||
def _is_sweep_usage_confirm(driver, config):
|
||||
# 掃討開始 always raises a "use N AP to sweep M times?" confirmation
|
||||
# before actually sweeping. Its OK button is this bright cyan.
|
||||
return _color_in_range(driver.color_at(*config.SWEEP_CONFIRM_BUTTON), config.SWEEP_CONFIRM_CYAN)
|
||||
|
||||
|
||||
def _is_ap_purchase_prompt(driver, config):
|
||||
# If AP is too low for even one sweep, a visually similar dialog appears
|
||||
# at the *same* OK-button position but titled "AP購入" (spend real
|
||||
# Pyroxene to buy more AP) with a gold/yellow OK instead of cyan --
|
||||
# confirmed live by deliberately emptying the sweep count at low AP.
|
||||
# Told apart from the safe confirm above by color, not position.
|
||||
return _color_in_range(driver.color_at(*config.SWEEP_CONFIRM_BUTTON), config.SWEEP_CONFIRM_GOLD)
|
||||
|
||||
|
||||
def _find_result_button(driver, config):
|
||||
# The "掃討完了" (sweep complete) results screen shows a SKIP button
|
||||
# (skips the reward-reveal animation) and then, once settled, a final
|
||||
# OK button -- both the same cyan as the usage-confirm OK, but ~120px
|
||||
# apart vertically. Finding whichever is showing by color avoids
|
||||
# hardcoding both positions.
|
||||
return detector.find_color_centroid(config.SWEEP_RESULT_BUTTON_REGION, *config.SWEEP_CONFIRM_CYAN)
|
||||
|
||||
|
||||
def _count_raised_above_one(driver, config):
|
||||
# The "-" stepper button is flat grey while count == 1 (its default,
|
||||
# disabled at the minimum) and turns vivid orange once raised -- cheap
|
||||
# way to confirm the MAX click actually registered without needing OCR
|
||||
# way to confirm a MAX/"+" click actually registered without needing OCR
|
||||
# on the count itself.
|
||||
r, g, b = driver.color_at(*config.SWEEP_MINUS_BUTTON_PROBE)
|
||||
return r > 200 and g < 180 and b < 100
|
||||
@ -55,46 +93,135 @@ def _open_task_screen(driver, config):
|
||||
return False
|
||||
|
||||
|
||||
def _go_to_latest_region(driver, config):
|
||||
print("[story_sweep] advancing to the latest unlocked region")
|
||||
x, y = config.REGION_RIGHT_ARROW
|
||||
for _ in range(config.REGION_RIGHT_ARROW_MAX_CLICKS):
|
||||
driver.click(x, y)
|
||||
def _read_current_region(driver, config):
|
||||
return detector.read_int(config.REGION_NUMBER_OCR_RECT)
|
||||
|
||||
|
||||
def _region_arrow_visible(driver, config, center):
|
||||
# Both arrows render as a solid navy-blue "<"/">" chevron on a
|
||||
# light-blue backdrop when present. The last region in a given direction
|
||||
# (or a locked one, per the reference's own "region-unavailable"
|
||||
# template check) simply omits the arrow rather than greying it out --
|
||||
# confirmed live: at region 30 (this account's current last region), the
|
||||
# spot where the right arrow would be was plain background.
|
||||
#
|
||||
# Scans a small box around `center` rather than probing a single fixed
|
||||
# point: a chevron is concave, and a centroid-derived single point
|
||||
# landed in the notch between its two strokes -- reading as "absent"
|
||||
# even while the glyph was clearly rendered a few pixels away. See
|
||||
# plan.md Phase 10.
|
||||
cx, cy = center
|
||||
rect = (cx - 40, cy - 35, cx + 40, cy + 35)
|
||||
return detector.region_contains_color(rect, (40, 70, 120), (100, 130, 190))
|
||||
|
||||
|
||||
def _go_to_region(driver, config, target_region):
|
||||
cur = _read_current_region(driver, config)
|
||||
if cur is None:
|
||||
print("[story_sweep] could not OCR the current region number")
|
||||
return False
|
||||
print(f"[story_sweep] current region {cur}, target region {target_region}")
|
||||
|
||||
for attempt in range(1, config.REGION_NAV_MAX_ATTEMPTS + 1):
|
||||
if cur == target_region:
|
||||
return True
|
||||
going_left = cur > target_region
|
||||
arrow_pos = config.REGION_LEFT_ARROW if going_left else config.REGION_RIGHT_ARROW
|
||||
if not _region_arrow_visible(driver, config, arrow_pos):
|
||||
direction = "left" if going_left else "right"
|
||||
print(f"[story_sweep] region {target_region} unreachable -- no {direction} arrow at region {cur}")
|
||||
return False
|
||||
|
||||
clicks = abs(cur - target_region)
|
||||
for _ in range(clicks):
|
||||
driver.click(*arrow_pos)
|
||||
driver.wait(1)
|
||||
|
||||
new_cur = _read_current_region(driver, config)
|
||||
if new_cur is None or new_cur == cur:
|
||||
print(f"[story_sweep] region number unchanged after {clicks} click(s) (attempt {attempt}/{config.REGION_NAV_MAX_ATTEMPTS})")
|
||||
return False
|
||||
cur = new_cur
|
||||
|
||||
def _pick_random_stage_row(driver, config):
|
||||
# Scrolling this list to either extreme always shows exactly 4 full
|
||||
# stage rows, since every region has at least 5 stages -- pick one
|
||||
# extreme at random, then a random one of its 4 rows. This isn't
|
||||
# perfectly uniform across a region's 5-6 stages (the middle ones are
|
||||
# reachable from both extremes and so are somewhat more likely), but it
|
||||
# avoids OCR or generic scroll-enumeration entirely.
|
||||
print(f"[story_sweep] gave up navigating to region {target_region} after {config.REGION_NAV_MAX_ATTEMPTS} attempts")
|
||||
return False
|
||||
|
||||
|
||||
def _normalize_label(text):
|
||||
# OCR sometimes reads the row's dash as a different dash-like glyph, or
|
||||
# picks up stray whitespace -- normalize before comparing.
|
||||
return text.strip().upper().replace("—", "-").replace("–", "-").replace(" ", "")
|
||||
|
||||
|
||||
def _label_suffix(label):
|
||||
# Compare only the part after the dash (e.g. "2" in "30-2", "A" in
|
||||
# "30-A"), not the full "{region}-{stage}" string. Confirmed live: this
|
||||
# font's leading region-number digit is read unreliably by OCR (e.g. "3"
|
||||
# misread as "2") even after threshold preprocessing, while the stage
|
||||
# suffix after the dash reads correctly across every row tested -- and we
|
||||
# don't need the region digit anyway, since _go_to_region has already
|
||||
# independently confirmed we're in the right region. See plan.md Phase 10.
|
||||
parts = [p for p in _normalize_label(label).split("-") if p]
|
||||
return parts[-1] if parts else ""
|
||||
|
||||
|
||||
def _row_label_rect(config, row_y):
|
||||
x1, x2 = config.STAGE_LABEL_OCR_X
|
||||
top_pad, bottom_pad = config.STAGE_LABEL_OCR_Y_PAD
|
||||
return (x1, row_y - top_pad, x2, row_y - bottom_pad)
|
||||
|
||||
|
||||
def _find_stage_row(driver, config, region, stage):
|
||||
# Scoped-down swipe_search_target_str (see module docstring): this
|
||||
# client's stage list only ever needs the two already-calibrated scroll
|
||||
# extremes checked, not arbitrary swipe-and-retry.
|
||||
target_suffix = str(stage)
|
||||
x, y = config.STAGE_LIST_SCROLL_POINT
|
||||
if random.random() < 0.5:
|
||||
|
||||
driver.scroll(x, y, "up", config.STAGE_LIST_SCROLL_CLICKS)
|
||||
row_y = random.choice(config.STAGE_ROWS_AT_TOP_Y)
|
||||
else:
|
||||
driver.scroll(x, y, "down", config.STAGE_LIST_SCROLL_CLICKS)
|
||||
row_y = random.choice(config.STAGE_ROWS_AT_BOTTOM_Y)
|
||||
driver.wait(0.5)
|
||||
for row_y in config.STAGE_ROWS_AT_TOP_Y:
|
||||
label = detector.read_text(_row_label_rect(config, row_y), whitelist="0123456789-A")
|
||||
print(f"[story_sweep] row @ {row_y} (scrolled up): read '{label}'")
|
||||
if _label_suffix(label) == target_suffix:
|
||||
return row_y
|
||||
|
||||
driver.scroll(x, y, "down", config.STAGE_LIST_SCROLL_CLICKS)
|
||||
driver.wait(0.5)
|
||||
for row_y in config.STAGE_ROWS_AT_BOTTOM_Y:
|
||||
label = detector.read_text(_row_label_rect(config, row_y), whitelist="0123456789-A")
|
||||
print(f"[story_sweep] row @ {row_y} (scrolled down): read '{label}'")
|
||||
if _label_suffix(label) == target_suffix:
|
||||
return row_y
|
||||
|
||||
def _dismiss_sweep_result(driver, config):
|
||||
# Sweep completion shows a reward summary that needs dismissing; the
|
||||
# exact number of screens varies with what dropped. Once the underlying
|
||||
# 任務情報 modal reappears, its "Enter" hotkey is bound to the live
|
||||
# "任務開始" (start manual mission) button, not a no-op -- confirmed live
|
||||
# that pressing Enter there would burn AP on a real battle attempt. So
|
||||
# check for the modal's return before every press and stop immediately,
|
||||
# rather than trusting a fixed round count to line up exactly with the
|
||||
# number of reward screens.
|
||||
for _ in range(POST_SWEEP_DISMISS_ROUNDS):
|
||||
if _is_stage_modal_open(driver, config):
|
||||
return
|
||||
driver.keypress("Return")
|
||||
driver.wait(1.5)
|
||||
return None
|
||||
|
||||
|
||||
def _click_max_and_verify(driver, config):
|
||||
for attempt in range(1, MAX_BUTTON_RETRIES + 1):
|
||||
driver.click(*config.SWEEP_MAX_BUTTON)
|
||||
driver.wait(0.8)
|
||||
if _count_raised_above_one(driver, config):
|
||||
return True
|
||||
print(f"[story_sweep] MAX click not detected (attempt {attempt}/{MAX_BUTTON_RETRIES})")
|
||||
return False
|
||||
|
||||
|
||||
def _click_plus_and_verify(driver, config, count):
|
||||
for attempt in range(1, MAX_BUTTON_RETRIES + 1):
|
||||
for _ in range(count - 1):
|
||||
driver.click(*config.SWEEP_PLUS_BUTTON)
|
||||
driver.wait(0.8)
|
||||
if _count_raised_above_one(driver, config):
|
||||
return True
|
||||
print(f"[story_sweep] count-raise via '+' not detected (attempt {attempt}/{MAX_BUTTON_RETRIES})")
|
||||
return False
|
||||
|
||||
|
||||
def _set_sweep_count(driver, config, count):
|
||||
if count == "max":
|
||||
return _click_max_and_verify(driver, config)
|
||||
return _click_plus_and_verify(driver, config, count)
|
||||
|
||||
|
||||
def _close_stage_modal(driver, config):
|
||||
@ -109,16 +236,48 @@ def _close_stage_modal(driver, config):
|
||||
return not _is_stage_modal_open(driver, config)
|
||||
|
||||
|
||||
def run(driver, config):
|
||||
driver.focus_game()
|
||||
def _watch_sweep_result(driver, config):
|
||||
# The reference's start_sweep returns one of "inadequate_ap",
|
||||
# "charge_challenge_counts", or "sweep_complete" so its caller reacts
|
||||
# appropriately -- this ports that same named-outcome contract via
|
||||
# navigation.wait_for_state instead of the old single generic "Done".
|
||||
#
|
||||
# Called only after the usage-confirm dialog is already accepted (see
|
||||
# _sweep_target), so from here it's purely "click through the
|
||||
# 掃討完了 SKIP/OK screens until the bare stage-info modal reappears."
|
||||
# Clicking the found button by color (not a keypress) means this never
|
||||
# risks landing on the underlying "任務開始" button the way a blind
|
||||
# Enter-press loop would.
|
||||
def click_result_button(d):
|
||||
pos = _find_result_button(d, config)
|
||||
if pos:
|
||||
d.click(*pos)
|
||||
d.wait(1.5)
|
||||
|
||||
if not _open_task_screen(driver, config):
|
||||
print("[story_sweep] could not confirm task screen is open, aborting without pressing further keys")
|
||||
return
|
||||
ends = {
|
||||
(lambda d, c: _is_stage_modal_open(d, c) and _find_result_button(d, c) is None): "swept",
|
||||
}
|
||||
reactions = {
|
||||
(lambda d, c: _find_result_button(d, c) is not None): click_result_button,
|
||||
}
|
||||
outcome = navigation.wait_for_state(
|
||||
driver, config, reactions, ends,
|
||||
max_iterations=POST_SWEEP_DISMISS_ROUNDS, poll_interval=1.5,
|
||||
)
|
||||
return outcome or "unrecognized_state"
|
||||
|
||||
_go_to_latest_region(driver, config)
|
||||
|
||||
row_y = _pick_random_stage_row(driver, config)
|
||||
def _sweep_target(driver, config, region, stage, count):
|
||||
print(f"[story_sweep] --- target {region}-{stage} x {count} ---")
|
||||
|
||||
if not _go_to_region(driver, config, region):
|
||||
return "region_unavailable"
|
||||
|
||||
row_y = _find_stage_row(driver, config, region, stage)
|
||||
if row_y is None:
|
||||
print(f"[story_sweep] stage {region}-{stage} not found in the visible stage list")
|
||||
return "stage_not_found"
|
||||
|
||||
driver.click(config.STAGE_ENTER_X, row_y)
|
||||
driver.wait(2)
|
||||
|
||||
@ -127,31 +286,70 @@ def run(driver, config):
|
||||
if navigation.is_on_subscreen(driver):
|
||||
driver.keypress("Escape")
|
||||
driver.wait(1.5)
|
||||
return
|
||||
return "unrecognized_state"
|
||||
|
||||
for attempt in range(1, MAX_BUTTON_RETRIES + 1):
|
||||
driver.click(*config.SWEEP_MAX_BUTTON)
|
||||
driver.wait(0.8)
|
||||
if _count_raised_above_one(driver, config):
|
||||
break
|
||||
print(f"[story_sweep] MAX click not detected (attempt {attempt}/{MAX_BUTTON_RETRIES})")
|
||||
else:
|
||||
if not _set_sweep_count(driver, config, count):
|
||||
print("[story_sweep] could not confirm sweep count was raised, aborting without spending AP")
|
||||
_close_stage_modal(driver, config)
|
||||
if navigation.is_on_subscreen(driver):
|
||||
driver.keypress("Escape")
|
||||
driver.wait(1.5)
|
||||
return
|
||||
return "unrecognized_state"
|
||||
|
||||
driver.click(*config.SWEEP_START_BUTTON)
|
||||
driver.wait(2)
|
||||
print("[story_sweep] sweep started, waiting for results")
|
||||
_dismiss_sweep_result(driver, config)
|
||||
driver.wait(1.5)
|
||||
|
||||
if not _close_stage_modal(driver, config):
|
||||
print("[story_sweep] warning: could not confirm stage info modal closed -- leaving it open rather than pressing further keys blindly")
|
||||
return
|
||||
if _is_ap_purchase_prompt(driver, config):
|
||||
print("[story_sweep] insufficient AP for this sweep -- cancelling without purchasing")
|
||||
driver.click(*config.SWEEP_CONFIRM_CANCEL_BUTTON)
|
||||
driver.wait(1)
|
||||
_close_stage_modal(driver, config)
|
||||
if navigation.is_on_subscreen(driver):
|
||||
driver.keypress("Escape")
|
||||
driver.wait(1.5)
|
||||
return "inadequate_ap"
|
||||
|
||||
if not _is_sweep_usage_confirm(driver, config):
|
||||
print("[story_sweep] sweep-usage confirmation not detected, aborting without further input")
|
||||
_close_stage_modal(driver, config)
|
||||
if navigation.is_on_subscreen(driver):
|
||||
driver.keypress("Escape")
|
||||
driver.wait(1.5)
|
||||
return "unrecognized_state"
|
||||
|
||||
driver.click(*config.SWEEP_CONFIRM_BUTTON)
|
||||
driver.wait(1.5)
|
||||
print("[story_sweep] sweep confirmed, waiting for results")
|
||||
outcome = _watch_sweep_result(driver, config)
|
||||
print(f"[story_sweep] result: {outcome}")
|
||||
|
||||
if not _close_stage_modal(driver, config):
|
||||
print("[story_sweep] warning: could not confirm stage info modal closed -- leaving it open rather than pressing further keys blindly")
|
||||
return outcome
|
||||
|
||||
if navigation.is_on_subscreen(driver):
|
||||
driver.keypress("Escape")
|
||||
driver.wait(1.5)
|
||||
return outcome
|
||||
|
||||
|
||||
def run(driver, config):
|
||||
driver.focus_game()
|
||||
|
||||
if not config.STORY_SWEEP_TARGETS:
|
||||
print("[story_sweep] no targets configured (config.STORY_SWEEP_TARGETS is empty), nothing to do")
|
||||
return
|
||||
|
||||
if not _open_task_screen(driver, config):
|
||||
print("[story_sweep] could not confirm task screen is open, aborting without pressing further keys")
|
||||
return
|
||||
|
||||
for region, stage, count in config.STORY_SWEEP_TARGETS:
|
||||
outcome = _sweep_target(driver, config, region, stage, count)
|
||||
if outcome == "inadequate_ap":
|
||||
print("[story_sweep] insufficient AP -- stopping, not attempting remaining targets")
|
||||
break
|
||||
if outcome != "swept":
|
||||
print(f"[story_sweep] target {region}-{stage} ended in '{outcome}' -- skipping to next target")
|
||||
|
||||
print("[story_sweep] Done.")
|
||||
|
||||
41
plan.md
41
plan.md
@ -137,7 +137,7 @@ Initial seed:
|
||||
| Mailbox | Need to confirm in reference | Need to inspect | `ba_auto/tasks/mailbox.py` | tap/click via xdotool, screenshot via scrot | Existing Bash behavior; migrate to Python |
|
||||
| Cafe | `module/cafe_reward.py` | `to_cafe`, `interaction_for_cafe_solve_method3`, `collect` | `ba_auto/tasks/cafe.py` | `picture.co_detect`/`color.rgb_in_range` → `driver.color_at` pixel-probe checks; sparkle template match ported in-process into `ba_auto/detector.py` | Migrated: real Python, state-verified via color probes, no legacy bridge |
|
||||
| Stamina/AP | `module/collect_daily_free_power.py`, `module/collect_daily_task_power.py` | `to_tasks`/`implement` (task-power, ported); `to_purchase_pyroxenes_menu`/`detect_free_power_availability` (free-power, not ported — real-money purchase menu) | `ba_auto/tasks/stamina.py` | `color.rgb_in_range` → `driver.color_at`; reference's per-tab claim loop replaced by the live UI's single "一括受取" (claim-all) button, triggered via Enter | Partially migrated (Phase 8): Mission-panel task/weekly/achievement claim done. Daily Free Power deliberately not implemented. |
|
||||
| Normal/Hard story AP sweep | `module/explore_tasks/sweep_task.py`, `module/explore_tasks/task_utils.py` | `to_region`/`to_normal_event` + OCR-driven per-stage claim loop (not ported — see Phase 9) | `ba_auto/tasks/story_sweep.py` | OCR-based region/stage-name matching → plain state-change probing (spam "next region" arrow until it stops advancing, no OCR); reference's per-stage claim loop → this client's single stage-info modal's self-contained 掃討 (sweep) sub-panel (MIN/-/+/MAX stepper + start button) | Done (Phase 9) |
|
||||
| Normal/Hard story AP sweep | `module/explore_tasks/sweep_task.py`, `module/explore_tasks/task_utils.py` | `to_region`/`to_normal_event` + OCR-driven per-stage claim loop (ported, Phase 10) | `ba_auto/tasks/story_sweep.py` | OCR-based region/stage-name matching, ported for real (Phase 10); reference's per-stage claim loop → this client's stage-info modal's self-contained 掃討 (sweep) sub-panel (MIN/-/+/MAX stepper + start button) | Done (Phase 10, supersedes Phase 9's random-pick design) |
|
||||
| Group/Club AP | `module/group.py` | Need to inspect | `ba_auto/tasks/group.py` | fixed click + state check via local driver | Not started |
|
||||
| Bounty | `module/rewarded_task.py` | Need to inspect | `ba_auto/tasks/bounty.py` | sweep/color/OCR adaptation | Not started |
|
||||
| Commissions | `module/clear_special_task_power.py` | Need to inspect | `ba_auto/tasks/commission.py` | sweep/color adaptation | Not started |
|
||||
@ -155,7 +155,7 @@ Do not implement a feature without filling at least the relevant row.
|
||||
| Mailbox claim | Migrated: `ba_auto/tasks/mailbox.py` uses `driver.color_at` to verify the panel opened before acting (found live-testing bug: a marginal icon coordinate could miss and cascade into pressing Escape on the home screen, which triggers Blue Archive's own exit-game confirmation) | Done |
|
||||
| Cafe pats + income | Migrated: `ba_auto/tasks/cafe.py` verifies each room/dialog opened via `driver.color_at` before acting; sparkle detection now runs in-process via `ba_auto/detector.py` instead of a per-click subprocess | Done |
|
||||
| Stamina/AP (mission claim) | Migrated (partial, Phase 8): `ba_auto/tasks/stamina.py` claims the Mission panel's bulk "一括受取" button. Daily Free Power (real-money purchase menu) intentionally not implemented | Daily Free Power still not started |
|
||||
| Normal/Hard story AP sweep | Done (Phase 9): `ba_auto/tasks/story_sweep.py` picks the latest unlocked region, a random stage in it, and sweeps with the in-game MAX count. Opt-in only (`story_sweep` command), not part of the default daily flow | Done |
|
||||
| Normal/Hard story AP sweep | Done (Phase 10, supersedes Phase 9's random-pick design): `ba_auto/tasks/story_sweep.py` sweeps a config-driven list of exact `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS`), navigating to each via OCR (region-number read + delta-click, stage-label OCR match) instead of "latest region, random stage." Opt-in only (`story_sweep` command), not part of the default daily flow | Done |
|
||||
| Shared driver | `ba_auto/driver.py` built (`run_command`, `focus_game`, `click`, `move_mouse`, `scroll`, `keypress`, `screenshot`, `wait`, `color_at`); `click()` now splits `mousemove`/`click` into two xdotool calls (Phase 8 finding — fixes a real source of click flakiness); wired into `mailbox.py`, `cafe.py`, `stamina.py`, `story_sweep.py` | Extend with new primitives as future tasks need them |
|
||||
| Python CLI | Built: `ba_daily.py` dispatches `mailbox`/`cafe`/`stamina`/`story_sweep`/default flow | Extend as new tasks are added |
|
||||
| Reference mapping | Built: `ba_auto/reference_notes/mapping.md` | Fill in reference file/function columns per feature |
|
||||
@ -338,11 +338,33 @@ Verified live: work-hub → task-screen navigation, latest-region advance, rando
|
||||
|
||||
**Retrospective — OCR avoidance was a mistake here.** Three of this phase's four live bugs (wrong modal probe, unverified MAX click, Escape-doesn't-close-modal plus the latent accidental-battle-start hazard) trace back to one decision: avoiding the reference's OCR-driven, deterministic stage targeting in favor of a heuristic substitute (random-pick + pixel-probes). A deterministic "go to configured stage X" flow, ported from the reference the way `module/explore_tasks/sweep_task.py`/`task_utils.py` actually do it, would not have needed to guess whether a modal opened via an easily-mismatched color probe, nor would it have left ambiguity about what's under the cursor when dismissing reward popups. This project's policy is now to port the reference's OCR-driven logic when the reference uses OCR for a feature, rather than inventing a non-OCR substitute to avoid the setup cost (see `CLAUDE.md` → "OCR policy"). `story_sweep.py`'s random-stage-pick design is not being reverted retroactively without user direction, but any future rework of this task should prefer porting the reference's actual region/stage-name OCR matching over the current random-pick approach.
|
||||
|
||||
### Phase 10: story_sweep OCR/state-machine port (supersedes Phase 9's random-pick design)
|
||||
|
||||
**Status: Done.** Acted on Phase 9's retrospective: set up OCR for real (`pytesseract` + the `tesseract-ocr` apt package) and ported the reference's actual deterministic stage targeting, replacing the random-pick heuristic. See `Handoff.md`'s history (deleted once this phase landed) for the full brief; summary of what changed:
|
||||
|
||||
- **OCR primitive**: `ba_auto/detector.py`'s `read_text()`/`read_int()` crop a screenshot to a pixel rect, threshold it to pure black/white (this measurably fixed real digit misreads that survived every `psm` mode when left anti-aliased — see below), upscale 3x, and run `pytesseract`. `lang="eng"` is enough; the region-number and stage-label reads are pure digits/dashes, no Japanese trained data needed.
|
||||
- **Config-driven targets**: `config.STORY_SWEEP_TARGETS = [(region, stage, count_or_"max"), ...]`, mirroring the reference's `unfinished_normal_tasks` shape. Ships with a placeholder `(1, 1, "max")` entry per explicit user direction — edit it to your own already-cleared stage(s) before running for real.
|
||||
- **Deterministic region navigation** (`_go_to_region`, porting `task_utils.py::to_region`): OCR the region-number readout, click the exact left/right-arrow delta, re-check, bounded loop. Region-arrow presence (locked/last-region detection) is a `detector.region_contains_color` box scan, not a single fixed point — a centroid-derived single point landed in the concave notch of the "<"/">" chevron and read "absent" even while the arrow was clearly rendered a few pixels away.
|
||||
- **Deterministic stage search** (`_find_stage_row`, a scoped-down `swipe_search_target_str`): OCR each of the 4 visible stage-row labels at both already-calibrated scroll extremes, matching by the label's suffix after the dash (e.g. "2" in "30-2") rather than the full string — the font's leading region digit reads unreliably even after threshold preprocessing (e.g. "3" as "2"), but the suffix read correctly on every row tested, and the region digit is redundant anyway since `_go_to_region` already confirmed it independently.
|
||||
- **Scoped `co_detect` port**: `navigation.wait_for_state(driver, config, reactions, ends, max_iterations)` — checks named `ends` first each iteration (stop, return the name), then named `reactions` (run an action, keep polling), else waits and retries up to a bound. Generic and reusable beyond this task.
|
||||
- **Named outcomes**: `_sweep_target` returns `"swept"`, `"inadequate_ap"`, `"region_unavailable"`, `"stage_not_found"`, or `"unrecognized_state"` instead of a single generic "Done".
|
||||
|
||||
Four real, live-discovered findings, on top of what Phase 9 already found:
|
||||
|
||||
- **Regular numbered stages (30-1..30-5) render a different, taller modal layout than the "-A" bonus stage Phase 9 exclusively calibrated against.** Regular stages add a "集中指揮"/"簡易攻略" tab row and a manual "任務開始" panel below the sweep sub-panel that "-A" doesn't have. Phase 9's `SWEEP_MAX_BUTTON`/`SWEEP_MINUS_BUTTON_PROBE`/`SWEEP_START_BUTTON` all missed by ~40-46px vertically against a real numbered stage (30-3) — caught live when the MAX-click retry correctly failed 3/3 and aborted without spending AP, rather than silently misfiring. Re-calibrated against the tabbed layout via pixel-scanning (not eyeballing); the "-A" layout's original Phase 9 coordinates are no longer what these constants hold, so a future sweep of a "-A" stage specifically would need its own re-check.
|
||||
- **The modal's own X-close button also moves with the layout.** Not just the sweep sub-panel — the whole card is vertically positioned by its own content height rather than anchored at a fixed absolute position, so the X button sits at a different absolute Y (225 vs Phase 9's 271) in the taller tabbed layout. Caught live: `_close_stage_modal` correctly reported "not closed" (3/3 retries) against the stale coordinate, rather than silently believing it had closed.
|
||||
- **Clicking 掃討開始 always raises an AP-usage-confirmation dialog the design had never accounted for at all.** ("APを`N`使用して、掃討を`M`回行いますか?", OK/Cancel.) A first live attempt at porting the "wait for outcome" step read this dialog's dimmed backdrop as a false "inadequate_ap" through an early, unverified placeholder probe — worth remembering: an uncalibrated placeholder check can be actively *wrong*, not just inert, if given a chance to run before it's confirmed. Fixed by explicitly clicking through this confirmation before watching for the real result.
|
||||
- **Genuine insufficient-AP is a visually near-identical dialog at the exact same OK-button position, told apart only by color.** Deliberately triggered live (by emptying the sweep count via MAX/"+" at low AP) rather than guessed: a real "AP不足" case shows a dialog titled "AP購入" (spend real Pyroxene to buy more AP) whose OK button is gold/yellow, vs. the safe usage-confirm's cyan — same position, different color. `_is_ap_purchase_prompt`/`_is_sweep_usage_confirm` tell them apart by that color and only ever click the cyan one; the gold one is always cancelled, never clicked, matching this project's purchase-safety rules.
|
||||
|
||||
Also fixed in passing, found only because live testing exercised the actual home-screen click path repeatedly: `config.TASK_CARD`'s old coordinate `(1370, 450)` sat close enough to the 任務 card's bottom edge that one run missed and landed on the "総力戦" (Total War) card below it instead — confirmed via screenshot, safely backed out with zero AP spent, moved to `(1250, 380)` (squarely on the "任務" title text).
|
||||
|
||||
Verified live end-to-end at least once, real AP spent: a genuine 5x sweep of 30-3 (AP 53→~5, confirmed via the "掃討完了" results screen's reward totals), including clicking through the usage-confirm dialog, the SKIP animation-skip screen, and the final reward-totals OK, landing back on the bare stage-info modal afterward. The genuine insufficient-AP path was also verified live (correctly cancelled the real "AP購入" purchase prompt without spending Pyroxene). Not yet re-verified end-to-end with the final rewritten code specifically (the color-based dynamic button-finding in `_watch_sweep_result`) at a nonzero AP balance — the manual walkthrough that discovered the dialogs used direct scripted clicks before the code was rewritten to match; the rewritten code's color-matching logic was separately verified offline against the exact screenshots captured live (all four dialog states correctly classified), but a fresh live run once AP regenerates would close that last gap. Not verified: sweeping a "-A" bonus stage (needs its own layout re-check, see above), an integer (non-"max") configured count actually being clicked via `SWEEP_PLUS_BUTTON`, and Hard-mode tab stages.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### OCR
|
||||
|
||||
**Not set up yet, but no longer "add only when a feature needs it" — see the Phase 9 retrospective.** Two already-shipped features (Stamina/AP mission claim, Normal/Hard story AP sweep) avoided OCR by substituting pixel-probes, fixed coordinates, or randomized selection for the reference's OCR-driven navigation. For story sweep that substitution caused real live bugs (wrong modal-open probe, an unverified click that silently under-spent AP, a modal that doesn't close on Escape, a latent accidental-battle-start hazard) that the reference's deterministic, OCR-based stage targeting would not have had. The corrected policy (see `CLAUDE.md` → "OCR policy") is: set up OCR and port the reference's OCR-driven logic as soon as a feature's reference implementation depends on it, rather than reaching for another non-OCR workaround.
|
||||
**Set up (Phase 10): `pytesseract` + the `tesseract-ocr` apt package.** `ba_auto/detector.py`'s `read_text()`/`read_int()` wrap it for occasional single-crop reads (a region number, a stage label) — no need for the reference's own socket/shared-memory PaddleOCR server, which exists there to make OCR fast across thousands of automation steps; this project's usage volume doesn't need that.
|
||||
|
||||
Needed for:
|
||||
|
||||
@ -446,7 +468,7 @@ OCR: Not expected.
|
||||
|
||||
### 4. Normal/Hard story AP sweep
|
||||
|
||||
**Status: Done — see Phase 9.**
|
||||
**Status: Done — see Phase 10 (supersedes Phase 9's random-pick design).**
|
||||
|
||||
Sweep already-cleared main story stages to burn AP.
|
||||
|
||||
@ -459,13 +481,12 @@ Reference:
|
||||
|
||||
Local target: `ba_auto/tasks/story_sweep.py`
|
||||
|
||||
OCR: Not used — the reference's OCR-driven region/stage lookup was replaced with plain state-change probing (see Phase 9): latest unlocked region via "next region arrow stops advancing", random stage within it, sweep count via the in-game MAX button. **This is now considered a design mistake** (see Phase 9's retrospective) — the substitution caused real live bugs the reference's deterministic approach wouldn't have had. A future revisit should port the reference's actual OCR-based stage targeting instead.
|
||||
OCR: Used, for real, as of Phase 10 — region-number readout OCR + delta-click (porting `to_region`), and stage-row label OCR matching (a scoped-down `swipe_search_target_str`), replacing Phase 9's "next region arrow stops advancing, then a random stage" heuristic. See Phase 10's write-up for what was actually found live (a second, taller modal layout for regular numbered stages; an AP-usage-confirmation dialog the design hadn't accounted for; a genuine insufficient-AP dialog told apart from it only by button color).
|
||||
|
||||
Implemented version (per explicit user direction, differs from the plan's original "suggested first version"):
|
||||
Implemented version:
|
||||
|
||||
- latest unlocked region (not a fixed configured stage)
|
||||
- random stage within that region (not a specific configured stage)
|
||||
- AP spend bounded by the in-game MAX button (no additional cap)
|
||||
- exact configured `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS`), not a fixed single stage nor a random pick
|
||||
- `count` is `"max"` or a specific int (the latter calibrated but not yet live-clicked — see Phase 10)
|
||||
- opt-in only, not in the default daily flow
|
||||
|
||||
### 5. Bounty
|
||||
@ -815,7 +836,7 @@ should run the default daily sequence.
|
||||
9. Add `ba_auto/reference_notes/mapping.md`. — Done
|
||||
10. Verify existing mailbox and cafe still work. — Done
|
||||
11. Implement stamina/AP. — Done (Phase 8)
|
||||
12. Implement Normal/Hard story AP sweep. — Done (Phase 9), but see its retrospective: a future revisit should port the reference's actual OCR-based stage targeting instead of the current random-pick substitute
|
||||
12. Implement Normal/Hard story AP sweep. — Done (Phase 9 built a random-pick heuristic; Phase 10 replaced it with the reference's actual OCR-based deterministic stage targeting, per Phase 9's own retrospective)
|
||||
13. Implement group/club AP. — Not started
|
||||
14. Set up OCR and port it for whichever remaining feature's reference implementation depends on it — not a blanket "only when needed" deferral; see `CLAUDE.md` → "OCR policy"
|
||||
15. Attempt Arena/Shop/Lesson once OCR is in place, since their reference implementations depend on it.
|
||||
|
||||
15
setup.sh
15
setup.sh
@ -27,13 +27,26 @@ if [ "$missing" = 1 ]; then
|
||||
fi
|
||||
echo "xdotool, scrot: OK"
|
||||
|
||||
if ! command -v tesseract >/dev/null 2>&1; then
|
||||
echo "MISSING: tesseract (OCR engine binary -- e.g. sudo apt install tesseract-ocr)"
|
||||
echo "story_sweep's region/stage-label OCR (see CLAUDE.md \"OCR policy\") needs this."
|
||||
echo "This requires a password-interactive sudo, so it isn't installed for you here --"
|
||||
echo "install it yourself, then re-run this script."
|
||||
missing=1
|
||||
fi
|
||||
if [ "$missing" = 1 ]; then
|
||||
exit 1
|
||||
fi
|
||||
echo "tesseract: OK"
|
||||
|
||||
echo "== Setting up Python venv at $VENV_DIR =="
|
||||
if [ ! -x "$VENV_DIR/bin/python3" ]; then
|
||||
python3 -m venv "$VENV_DIR"
|
||||
fi
|
||||
"$VENV_DIR/bin/pip" install --quiet --upgrade pip
|
||||
"$VENV_DIR/bin/pip" install --quiet opencv-python-headless numpy
|
||||
"$VENV_DIR/bin/pip" install --quiet opencv-python-headless numpy pytesseract
|
||||
"$VENV_DIR/bin/python3" -c "import cv2, numpy; print('opencv', cv2.__version__, '/ numpy', numpy.__version__)"
|
||||
"$VENV_DIR/bin/python3" -c "import pytesseract; print('pytesseract', pytesseract.get_tesseract_version())"
|
||||
|
||||
echo "== Deploying assets to fixed paths =="
|
||||
mkdir -p "$ASSETS_DIR"
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user