Compare commits

..

10 Commits

Author SHA1 Message Date
9f4d8f4361 feat(story_sweep): add daily-rotating target for story sweeps and update configuration 2026-07-08 23:55:00 +09:00
51ad297f69 feat: Implement Lesson/Schedule task
- Added `lesson.py` to handle the scheduling of lessons based on affection values.
- Integrated OCR functionality to read affection counts from heart badges.
- Updated `README.md` to include the new lesson task in the task status section.
- Enhanced `config.py` with necessary configurations for the lesson task.
- Modified `detector.py` to include a new function for reading heart badge values accurately.
- Updated `mapping.md` to reflect the new lesson task implementation.
- Adjusted `ba_daily.py` to include the lesson task in the command dispatch.
- Updated `plan.md` to document the completion of the lesson task and its testing outcomes.
- Modified `setup.sh` to include the lesson task in the run command instructions.
2026-07-08 23:45:18 +09:00
1bb2ccdd6f feat: Add Common Shop and Tactical Shop tasks with shared purchasing logic 2026-07-07 23:08:31 +09:00
4842d575a5 docs: Update CLAUDE.md to clarify project goals, OCR policy, and migration details 2026-07-07 22:26:37 +09:00
b2ed571908 feat(cafe): handle bond rank-up cutscene during pat loop and improve state verification 2026-07-06 01:58:01 +09:00
a8ff95d6f5 feat: Implement OCR-driven story AP sweep with deterministic stage targeting
- Added `wait_for_state` function in `navigation.py` for state monitoring and reaction handling.
- Updated `mapping.md` to reflect changes in story sweep implementation and OCR usage.
- Refactored `story_sweep.py` to utilize OCR for region and stage identification, replacing random selection with configured targets.
- Enhanced modal handling and confirmation checks for AP usage in `story_sweep.py`.
- Updated setup script to require `tesseract` for OCR functionality and included installation instructions.
- Revised `plan.md` to document the transition from heuristic to OCR-based stage targeting and the associated findings from live testing.
2026-07-06 01:41:58 +09:00
0707487934 Enhance automation project with story sweep feature and OCR policy updates
- Introduced `story_sweep.py` for Normal/Hard story AP sweeping, allowing users to spend AP on randomly selected stages.
- Updated `CLAUDE.md` to clarify the OCR policy, emphasizing the need to port OCR-driven logic from the reference implementation rather than substituting with non-OCR methods.
- Modified `README.md` and `plan.md` to reflect the new story sweep feature and its operational details.
- Adjusted `setup.sh` to include the new command for running the story sweep.
- Enhanced `driver.py` with a new scroll function for better interaction with the game UI.
- Updated configuration mappings in `config.py` to support the new story sweep functionality.
- Refined existing task modules to ensure consistent state verification and error handling.
2026-07-05 20:19:35 +09:00
0b16399948 Refactor directory labels and update task migration status
- Changed directory labels from ".scratchpad/" to "scratchpad/" in graph.json and related documentation for consistency.
- Updated plan.md to reflect the current status of the Stamina/AP task migration, including details on the mission claim process and identified bugs.
- Modified setup.sh to include 'stamina' in the run command options for the daily script.
2026-07-05 19:15:32 +09:00
a04ef5d0c5 Refactor scratchpad directory references and update Phase 6 follow-up report
- Changed references from './scratchpad' to '.scratchpad/' in graph.json and plan.md for consistency.
- Expanded Phase 6 follow-up section in plan.md to clarify changes made to the pat detection logic:
  - Updated `find_cafe_sparkle()` to utilize multiple template scales for improved detection.
  - Modified `_pat_room` to allow polling for maximum clicks instead of breaking on the first miss.
  - Added mouse movement after each pat to prevent cursor occlusion of sparkles.
- Verified that room entry and modal state checks function correctly, but end-to-end pat success remains untested due to lack of available interactions.
2026-07-05 17:57:26 +09:00
3a4534f205 Migrate cafe and mailbox tasks to Python: replace legacy Bash scripts, implement state verification, and integrate sparkle detection. 2026-07-05 16:31:26 +09:00
23 changed files with 2224 additions and 417 deletions

1
.gitignore vendored
View File

@ -3,3 +3,4 @@ graphify-out/cost.json
graphify-out/cache/
__pycache__/
*.pyc
scratchpad/

425
CLAUDE.md
View File

@ -8,19 +8,55 @@ This repository is a personal Blue Archive JP daily-automation project.
It controls the real Blue Archive PC/Steam/Proton client running on a Linux machine named `nik-gpu`. The development machine is a MacBook named `nik-macbookair`.
**The original goal is a backend port, not a rewrite.** The reference project (`~/repo/baas-reference/`) only works against an Android emulator via ADB/uiautomator2, typically run on a Windows PC. This project's job is to make that same automation logic work against the real JP client running under Proton on Linux instead — porting the reference's decision logic, navigation, and state-detection approach, including its use of OCR, as faithfully as practical, and replacing only the parts that are genuinely Android-specific with local desktop equivalents.
Android-specific parts include:
- ADB input
- uiautomator2 taps
- Android UI-object-tree queries
- emulator-specific assumptions
Local desktop equivalents include:
- `xdotool`
- `scrot`
- OpenCV
- OCR against screenshots
- local image/color/template detection
Inventing a different, simpler approach to avoid porting a piece of reference logic is a deviation from the project goal, not a shortcut toward it.
The automation backend is local desktop control:
- `xdotool` for mouse/keyboard/window control
- `scrot` for screenshots
- Python/OpenCV for image matching and color/template detection
- OCR later, when needed
- OCR, using Tesseract or PaddleOCR, wherever the reference implementation uses OCR for a feature
- no Android emulator control
- no ADB
- no uiautomator2
### OCR policy
OCR is **not** an Android-specific concern. The reference's OCR-driven region, stage-name, and currency matching runs against a screenshot and can be ported to this backend using `scrot` captures. Only the reference's input mechanism is Android-specific.
Two features, Stamina/AP mission claim and Normal/Hard story AP sweep, were originally built without OCR by substituting ad hoc pixel probes, fixed coordinates, or randomized selection for the reference's OCR-driven navigation. For story sweep, that substitution produced more real bugs during live testing than porting the reference's actual approach would have:
- a wrong modal-open probe
- an unverified button click that silently under-spent AP
- a modal that does not close on Escape
- a latent hazard where a mistimed keypress could have started a real battle
See `plan.md` Phase 9's retrospective for the full writeup.
That issue has since been fixed: `story_sweep.py` now ports the reference's actual OCR-based region/stage targeting. See `plan.md` Phase 10. The retrospective stays here as the concrete, lived reason for the OCR policy, not as a description of the current state of that task.
Going forward: when a reference feature's control flow depends on OCR, set up OCR and port that logic. Do not invent a non-OCR workaround just to avoid the setup cost. Only skip OCR for a specific step if the reference itself does not use OCR there.
The reference project is located at:
```
```text
~/repo/baas-reference/
```
@ -38,7 +74,7 @@ All feature logic must live in Python.
The intended structure is:
```
```text
ba-auto-daily/
├── ba_dailies.sh
├── ba_daily.py
@ -59,12 +95,14 @@ ba-auto-daily/
│ └── mapping.md
├── assets/
├── screenshots/
├── scripts/
├── scratchpad/
├── setup.sh
├── plan.md
└── CLAUDE.md
```
`scripts/` previously held one-off Bash/Python helpers. It was deleted once mailbox and cafe migrated off it. Do not recreate `scripts/` as a place to stash feature logic.
The exact layout can evolve, but the architectural rule should not change:
- Bash launches Python.
@ -76,7 +114,7 @@ The exact layout can evolve, but the architectural rule should not change:
Before implementing any new feature, inspect the matching reference implementation in:
```
```text
~/repo/baas-reference/module/
```
@ -94,7 +132,7 @@ For every feature, first identify:
Then implement the feature in Python under:
```
```text
ba_auto/tasks/
```
@ -117,16 +155,30 @@ The reference repository is **read-only**.
- edit files in `~/repo/baas-reference/`
- reimplement a reference feature from scratch in Bash
- create a local solution that ignores the reference flow when a reference implementation already exists
- substitute a pixel probe, fixed coordinate, or randomized shortcut for reference logic that uses OCR, just to avoid setting up OCR
## Two-machine architecture
Development happens on: `nik-macbookair`
Development happens on:
Runtime happens on: `nik-gpu`
```text
nik-macbookair
```
Runtime happens on:
```text
nik-gpu
```
The Blue Archive client and X display live on `nik-gpu`.
Assume: (`DISPLAY=:0`, GDM `XAUTHORITY` under `/run/user/1000`)
Assume:
```text
DISPLAY=:0
GDM XAUTHORITY under /run/user/1000
```
The game runs under Steam/Proton on the Linux desktop.
@ -138,7 +190,7 @@ During iteration, files are pushed from `nik-macbookair` to `nik-gpu`.
Typical paths on `nik-gpu`:
```
```text
~/ba_dailies.sh
~/ba_daily.py
~/ba_auto/
@ -148,7 +200,7 @@ Typical paths on `nik-gpu`:
The current setup may still contain older paths such as:
```
```text
~/ba_scripts/detect_and_click.py
```
@ -158,12 +210,28 @@ When refactoring, prefer consolidating Python code into `ba_auto/`.
For a fresh checkout on `nik-gpu`, run from the repo root:
```
```bash
./setup.sh
```
After initial setup, individual changes may be pushed with `scp` or `rsync`.
Prefer syncing the project from the repository root:
```bash
rsync -av \
--exclude='.git/' \
--exclude='__pycache__/' \
--exclude='*.pyc' \
--exclude='.claude/settings.local.json' \
--exclude='graphify-out/' \
--exclude='screenshots/' \
--exclude='scratchpad/' \
./ nik-gpu:~/repo/ba-auto-daily/
```
Do not sync Claude Code internal temporary folders.
## Runtime dependencies on nik-gpu
These are host-level dependencies. Confirm they exist before assuming a bug is in the project code.
@ -178,22 +246,28 @@ Required Python packages:
- `opencv-python` or `opencv-python-headless`
- `numpy`
- `pytesseract`
Expected venv:
```
```text
~/.venvs/ba-auto-daily/bin/python3
```
Quick check:
```
ssh nik-gpu "which xdotool scrot && ~/.venvs/ba-auto-daily/bin/python3 -c 'import cv2, numpy; print(cv2.__version__)'"
```bash
ssh nik-gpu "which xdotool scrot tesseract && ~/.venvs/ba-auto-daily/bin/python3 -c 'import cv2, numpy, pytesseract; print(cv2.__version__)'"
```
Future dependency: OCR engine, likely Tesseract or PaddleOCR.
OCR engine dependency is set up as of Phase 10:
Do not introduce OCR casually. Add it only when implementing a feature that actually needs OCR.
- Tesseract via the `tesseract-ocr` apt package
- `pytesseract` in the venv
Installing `tesseract-ocr` needs interactive `sudo`, so `setup.sh` should check for it but should not assume it can install it automatically.
`ba_auto/detector.py`'s `read_text()` and `read_int()` wrap OCR for occasional single-crop reads. See `story_sweep.py` for the first real usage.
## Bash policy
@ -204,8 +278,10 @@ Preferred shape:
```bash
#!/usr/bin/env bash
set -euo pipefail
VENV_PYTHON="${VENV_PYTHON:-$HOME/.venvs/ba-auto-daily/bin/python3}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
exec "$VENV_PYTHON" "$SCRIPT_DIR/ba_daily.py" "$@"
```
@ -234,36 +310,44 @@ If an existing Bash function exists, migrate it to Python rather than extending
The intended Python CLI entry point is:
```
```text
ba_daily.py
```
It should support commands such as:
```
```bash
./ba_dailies.sh
./ba_dailies.sh mailbox
./ba_dailies.sh cafe
./ba_dailies.sh stamina
./ba_dailies.sh story_sweep
./ba_dailies.sh group
```
No argument should run the default daily flow.
Example default flow:
Current default flow in `ba_daily.py`'s `DEFAULT_ORDER`:
1. focus game
2. mailbox
3. cafe
4. future daily tasks
4. stamina
5. future daily tasks
The CLI should dispatch into task modules under `ba_auto/tasks/`.
`story_sweep` spends AP rather than reclaiming something free, so it is deliberately excluded from the default flow. It must be invoked explicitly.
The CLI should dispatch into task modules under:
```text
ba_auto/tasks/
```
## Driver layer
Create and maintain a driver layer in:
```
```text
ba_auto/driver.py
```
@ -271,14 +355,15 @@ The driver layer should wrap the PC/Steam/Proton backend.
It should provide reusable primitives such as:
```
```python
focus_game()
screenshot()
click(x, y)
double_click(x, y)
drag/swipe(start, end, duration)
drag(start, end, duration)
swipe(start, end, duration)
keypress(key)
sleep/wait
wait(seconds)
wait_until(...)
color_at(...)
region_average_color(...)
@ -286,7 +371,7 @@ template_match(...)
find_and_click_template(...)
```
Feature modules should not directly shell out to `xdotool` or `scrot` unless a driver primitive is missing and being added.
Feature modules should not directly shell out to `xdotool` or `scrot` unless a driver primitive is missing and is being added.
Prefer:
@ -297,16 +382,22 @@ driver.click(x, y)
over:
```python
subprocess.run(["xdotool", "click", ...])
subprocess.run(["xdotool", "click", "1"])
```
This keeps the project close to the reference architecture while replacing only the backend-control layer.
## Detector layer
Image matching and color matching should live in `ba_auto/detector.py` or in clearly named helper classes/functions.
Image matching, color matching, and OCR wrappers should live in:
Existing logic from `scripts/detect_and_click.py` should be migrated into reusable Python functions.
```text
ba_auto/detector.py
```
or in clearly named helper classes/functions.
The cafe sparkle-matching logic formerly in `scripts/detect_and_click.py` has been migrated into `ba_auto/detector.py` as `find_cafe_sparkle()`. It should be called in-process rather than as a per-click subprocess.
The detector should support:
@ -315,13 +406,18 @@ The detector should support:
- threshold tuning
- masked matching
- click-offset handling
- debug image output to `./scratchpad`
- OCR crop reads
- debug image output to `scratchpad/`
Avoid one Python cold start per click attempt where possible. Prefer long-running Python task logic that can take repeated screenshots and click repeatedly from one process.
## Navigation layer
Common navigation should live in `ba_auto/navigation.py`.
Common navigation should live in:
```text
ba_auto/navigation.py
```
Use this for shared flows such as:
@ -333,31 +429,37 @@ Use this for shared flows such as:
- opening lesson/schedule
- closing popups
- generic back/escape handling
- waiting for known UI states
Do not duplicate navigation click sequences inside every task if they can be shared.
## Task modules
Each feature should have a task module: `ba_auto/tasks/<feature>.py`
Example:
Each feature should have a task module:
```text
ba_auto/tasks/<feature>.py
```
Examples:
```text
ba_auto/tasks/mailbox.py
ba_auto/tasks/cafe.py
ba_auto/tasks/group.py
ba_auto/tasks/stamina.py
ba_auto/tasks/story_sweep.py
ba_auto/tasks/group.py
```
Each task module should expose a clear function such as:
```
```python
run(driver, config)
```
or:
```
```python
run_mailbox(driver, config)
```
@ -367,7 +469,7 @@ Keep task files feature-focused.
Maintain a mapping file at:
```
```text
ba_auto/reference_notes/mapping.md
```
@ -375,74 +477,239 @@ Before or during implementation of a feature, update the mapping.
Use this format:
```markdown
| Local feature | Reference file | Reference functions/classes | Local file | Backend replacements | Status |
|---|---|---|---|---|---|
| Cafe | `module/cafe.py` or relevant file | `...` | `ba_auto/tasks/cafe.py` | uiautomator2 tap -> xdotool click, screenshot -> scrot/OpenCV | In progress |
```
## Working conventions
Use `./scratchpad` for temporary or intermediate files.
Always work from the repository root:
```bash
cd ~/repo/ba-auto-daily
```
Use `scratchpad/` in the project root for all temporary, generated, diagnostic, calibration, and investigation files.
Examples:
- cropped calibration images
- debug screenshots
- annotated match results
- temporary Python probes
- temporary investigation notes
- one-off image-analysis scripts
- temporary OCR experiments
- screenshots copied back from `nik-gpu` for analysis
Do not use `/tmp` or `/private/tmp` unless there is a strong reason.
Create it if missing:
```bash
mkdir -p scratchpad
```
Do **not** use Claude Code's internal temporary directories such as:
```text
/private/tmp/claude-*
/tmp/claude-*
```
Do **not** run probes from `/private/tmp/claude-*` or write temporary scripts there. These paths are outside the repository and often trigger extra permission prompts, interrupting the development flow.
When a temporary Python probe is needed, write it into `scratchpad/` first, then run it from the repository root.
Preferred pattern:
```bash
cat > scratchpad/probe_image.py <<'PY'
from PIL import Image
img = Image.open("scratchpad/shop_probe_04_checked.png")
print(img.size)
points = [
("checkbox_checked_center", (972, 297)),
("checkbox_checked_bg", (975, 300)),
("card_border_left", (940, 380)),
("card_border_unselected", (1165, 380)),
("buy_button", (1751, 1112)),
("cancel_button", (1525, 1112)),
]
for name, xy in points:
print(name, xy, img.getpixel(xy))
PY
python3 scratchpad/probe_image.py
```
Avoid this pattern:
```bash
cd /private/tmp/claude-*/scratchpad && python3 -c "..."
```
Also avoid large multiline `python3 -c "..."` commands when they contain comments or complex quoting. Prefer a temporary script under `scratchpad/` because it is easier to inspect, rerun, and keep within workspace permissions.
`screenshots/` contains human reference captures. They are useful for calibration and documentation, but they are not necessarily automated test fixtures.
`assets/` contains local template images. These should be captured from the local game setup where possible.
## Temporary probe policy
Temporary probes are allowed, but they must be workspace-local.
For non-trivial investigation, use:
```text
scratchpad/<short_descriptive_name>.py
```
Examples:
```text
scratchpad/probe_shop_checkbox.py
scratchpad/probe_cafe_income.py
scratchpad/probe_story_modal.py
scratchpad/probe_ocr_region.py
```
A temporary probe should:
- run from the repository root
- read inputs from `scratchpad/`, `screenshots/`, or `assets/`
- write outputs to `scratchpad/`
- avoid `/tmp`, `/private/tmp`, and Claude Code internal paths
- avoid relying on absolute `/private/tmp/claude-*` paths
- be deletable once the investigation is complete
Prefer:
```bash
python3 scratchpad/probe_shop_checkbox.py
```
Do not prefer:
```bash
python3 -c "large multiline script..."
```
For very small one-liners, `python3 -c` is acceptable only when it does not require changing directories into `/private/tmp/claude-*`.
## Testing and checks
Because the game only runs on `nik-gpu`, local macOS testing is limited.
Before deploying, run static/syntax checks locally:
```
```bash
bash -n ba_dailies.sh
python3 -m py_compile ba_daily.py
python3 -m py_compile ba_auto/*.py
python3 -m py_compile ba_auto/tasks/*.py
```
For temporary investigation scripts, use:
```bash
python3 scratchpad/<script_name>.py
```
Do not run temporary investigation scripts from Claude Code's internal `/private/tmp/claude-*` workspace.
On `nik-gpu`, run real integration tests against the live game.
Example:
```
ssh nik-gpu "~/ba_dailies.sh cafe"
```bash
ssh nik-gpu "~/repo/ba-auto-daily/ba_dailies.sh cafe"
```
When debugging image matching, write debug images to `./scratchpad`.
When debugging image matching, write debug images to:
```text
scratchpad/
```
When debugging on `nik-gpu`, copy relevant screenshots or debug images back into the local project `scratchpad/` if they need local analysis.
## Existing features
Current project state (Phase 1 skeleton in place, mailbox migrated):
Current project state: mailbox, cafe, stamina, story_sweep, shop_common, shop_tactical, and lesson are all migrated to real Python. No Bash feature logic remains.
- `ba_dailies.sh` is a thin launcher that execs `ba_daily.py`
- `ba_daily.py` dispatches `mailbox`/`cafe`/default flow to `ba_auto/tasks/`
- `ba_auto/tasks/mailbox.py` is real Python: it clicks with `ba_auto/driver.py` primitives and verifies state with `driver.color_at` (ported from `module/mail.py`'s `rgb_in_range` pattern) before pressing further keys — no legacy bridge
- `ba_auto/tasks/cafe.py` still bridges to `scripts/ba_dailies_legacy.sh` (the pre-migration click sequence), marked with a TODO — the real logic has not moved into Python yet
- cafe's bridge still shells out to `scripts/detect_and_click.py` for OpenCV sparkle detection, unchanged
- `ba_auto/driver.py` has the primitives (`run_command`, `focus_game`, `click`, `keypress`, `screenshot`, `wait`, `color_at`) wired into `mailbox.py`; not yet used by `cafe.py`
- `ba_auto/detector.py` and `ba_auto/navigation.py` are placeholders
- Live testing found the old fixed mailbox-icon coordinate was marginal and could miss, cascading into an unverified Escape press that triggers Blue Archive's own "exit the game?" confirmation — see `plan.md` Phase 5 for details. This is the concrete reason `mailbox.py` verifies state before acting rather than trusting fixed coordinates blindly.
- `ba_daily.py` dispatches `mailbox`, `cafe`, `stamina`, `story_sweep`, `shop_common`, `shop_tactical`, `lesson`, and default flow to `ba_auto/tasks/`
- default flow is `mailbox`, `cafe`, `stamina`
- `story_sweep`, `shop_common`, `shop_tactical`, and `lesson` are opt-in only since they spend AP/credits/tactical coin/lesson tickets rather than reclaiming something free
- `ba_auto/tasks/mailbox.py` and `ba_auto/tasks/cafe.py` click with `ba_auto/driver.py` primitives and verify state with `driver.color_at` and `ba_auto/navigation.py`
- mailbox and cafe were ported from the reference patterns around `module/mail.py` and `module/cafe_reward.py`
- no legacy bridge remains
- `ba_auto/tasks/stamina.py` claims the Mission panel's bulk `一括受取` button
- see `plan.md` Phase 8 for stamina details
- `ba_auto/tasks/story_sweep.py` sweeps a config-driven list of exact `(region, stage, count)` targets from `config.STORY_SWEEP_TARGETS`, plus one daily-rotating target (`config.STORY_SWEEP_ROTATION_*`) that cycles through a fixed region's stages one per day
- story sweep navigates via OCR, using region-number readout and stage-label matching, rather than random selection
- see `plan.md` Phase 10 for the OCR-based story sweep port, and its "Phase 10 follow-up" for the rotation-target fix
- `ba_auto/tasks/shop_common.py` and `ba_auto/tasks/shop_tactical.py` share a checkbox-grid-then-bulk-buy flow (`ba_auto/tasks/shop_utils.py`) against config-driven `(row, col, name, expected_price)` targets (`config.COMMON_SHOP_TARGETS` / `config.TACTICAL_SHOP_TARGETS`)
- item identification is by fixed grid position, not per-item OCR — the reference's own `get_item_position` indexes an external static price table this repo doesn't have, so a locally pixel-scanned position table is the faithful port, not an OCR-avoidance shortcut; price-digit OCR is layered on top as an extra catalog-drift safety check the reference doesn't even do per-item
- both shops were live-tested with real purchases (see `plan.md` Phase 11), which also surfaced a real, previously-unknown per-refresh-cycle purchase cap on these items (not shown as a visible counter) — the task correctly detected the now-unselectable items and safely declined rather than misfiring
- `ba_auto/tasks/lesson.py` sweeps every unlocked region's schedule grid (a scrollable list of 12 named regions, each opening a grid modal of up to 9 location cards), picking the highest-affection available lesson each time until lesson tickets or lessons run out
- per-cell affection is read via `detector.read_int_on_heart_badge`, a dedicated OCR path for the pink/magenta heart-shaped badge — the project's normal grayscale-threshold OCR (`read_int`) misreads it, because the badge's own outline stroke survives the same threshold as the digit glyph; a "done today" portrait keeps its number and gets a green checkmark added alongside it rather than losing the number, so done-ness is checked via that checkmark's color, not inferred from a failed OCR read
- lesson was live-tested with real tickets spent (see `plan.md` Phase 12), which surfaced two real bugs from that assumption gap plus an OCR contamination issue — both fixed; see Phase 12 for the full writeup
- `ba_auto/detector.py` has `find_cafe_sparkle()`, the sparkle template-match ported in-process from the now-deleted `scripts/detect_and_click.py`
- `scripts/ba_dailies_legacy.sh` and `scripts/detect_and_click.py` have been deleted
- the `scripts/` directory itself no longer exists
- `ba_auto/driver.py` primitives are wired into all migrated task modules
- existing driver primitives include `run_command`, `focus_game`, `click`, `move_mouse`, `scroll`, `keypress`, `screenshot`, `wait`, and `color_at`
- `ba_auto/navigation.py` has shared state probes used across tasks:
- `is_on_subscreen`
- `is_modal_open`
- `wait_for_state()`
Migration goal:
`wait_for_state()` is a scoped port of the reference's `core/picture.py::co_detect`: watch for any of several named states, react to known non-terminal ones, and stop on a recognized terminal one.
- preserve existing behavior
- move feature logic from Bash into Python
- keep `ba_dailies.sh` as launcher only
- make detection logic reusable
- prepare for future features by copying the reference project's structure where appropriate
`story_sweep.py` additionally has its own local modal probes and close logic because the stage-info modal is wide enough to break `is_modal_open`'s default probe point, and it does not close on Escape. See `plan.md` Phase 9.
The story sweep modal renders at least two different internal layouts:
- a plain one for the bonus `-A` stage
- a taller tabbed one for regular numbered stages
Their button coordinates differ. This was discovered live in Phase 10.
Live testing found both the mailbox-icon and cafe-icon fixed coordinates were flaky. They missed the first click and worked on retry. Neither task originally verified state before proceeding, so a missed click cascaded into blind actions and could reach an unverified Escape press on the home screen, which triggers Blue Archive's own "exit the game?" confirmation.
See `plan.md` Phases 56 for the full writeup. This is the concrete reason every task now verifies state before acting rather than trusting fixed coordinates or a single click blindly.
Rank-up popups mid-pat-loop are now handled.
A pat that crosses an affection-rank threshold shows a full-screen `絆ランクアップ!` cutscene with no cafe header. `find_cafe_sparkle()` can never recognize this because it is nothing like the sparkle template. The loop used to spin uselessly against it for the rest of the room's click budget.
`cafe.py`'s `_dismiss_rank_up_if_shown()` reuses the existing `navigation.is_on_subscreen` header-brightness probe. This was confirmed against `screenshots/cafe/student/01-02`:
- rank-up cutscene: header probe reads `r < 200`
- normal cafe screen: header probe reads `r > 200`
It presses Enter until the cutscene clears, porting the reference's own `to_cafe()` navigation handling of `relationship_rank_up` from `module/cafe_reward.py`.
This is not yet live-confirmed against a real cafe-pat rank-up trigger specifically. It is grounded in real captured screenshots, but treat it as implemented-but-unverified until one happens naturally during a real cafe run.
The same full-screen `絆ランクアップ!` cutscene *was* confirmed live during Lesson/Schedule work (see `plan.md` Phase 12), and that work found `is_on_subscreen`'s fixed header probe reads inconsistently across different characters' cutscene art — bright for one, which would make cafe's "keep pressing Enter while NOT on subscreen" loop think the cutscene had already cleared when it hadn't. Not necessarily hazardous on its own (it would just stop dismissing early and let the pat loop retry against a still-showing cutscene), but worth a more robust probe if cafe's rank-up path ever misbehaves live.
Not yet verified for cafe:
- whether camera zoom/pan can drift over a long unattended run
- the reference project zooms out before detecting
- this project currently does not
- testing did not reproduce a failure from skipping it
- see `plan.md` Phase 6 "Not verified" list
## Known cafe gaps to verify during migration
When migrating cafe logic, verify these manually against the actual current behavior:
When migrating or modifying cafe logic, verify these manually against the actual current behavior:
- whether student rotation popups are handled
- whether rank-up popups are handled
@ -454,13 +721,23 @@ When migrating cafe logic, verify these manually against the actual current beha
Do not assume the old Bash implementation handles these correctly.
## Migration goal
The migration goal is:
- preserve existing behavior
- move feature logic from Bash into Python
- keep `ba_dailies.sh` as launcher only
- make detection logic reusable
- prepare for future features by copying the reference project's structure where appropriate
## Anti-patterns
Do not:
- add new `do_<feature>` Bash functions
- build a giant Bash automation script
- recreate reference logic as fixed coordinate Bash click chains
- recreate reference logic as fixed-coordinate Bash click chains
- skip reading the reference module before implementing a feature
- put OCR in Bash
- put OpenCV state machines in Bash
@ -469,6 +746,13 @@ Do not:
- copy reference image assets blindly
- edit `~/repo/baas-reference/`
- implement event-specific features before the generic reusable machinery exists
- invent a pixel probe, fixed-coordinate, or randomized substitute for reference logic that uses OCR, just to avoid setting up OCR
- treat OCR as an Android-specific concern that can be deferred indefinitely
- run temporary probes from `/private/tmp/claude-*` instead of the repository `scratchpad/`
- create investigation scripts outside the repository when they can live in `scratchpad/`
- use multiline `python3 -c "..."` commands for non-trivial probes
- rely on Claude Code internal temporary paths in committed code, notes, commands, or debugging workflow
- write generated debug files into the repository root when they belong in `scratchpad/`
## Feature implementation workflow
@ -476,7 +760,7 @@ For every new feature:
1. Read the relevant `~/repo/baas-reference/module/...` file.
2. Summarize the upstream feature flow in notes or comments.
3. Add/update the reference mapping table.
3. Add or update the reference mapping table.
4. Identify missing local driver primitives.
5. Implement or improve those primitives in `ba_auto/driver.py` or `ba_auto/detector.py`.
6. Implement the feature in `ba_auto/tasks/<feature>.py`.
@ -486,17 +770,20 @@ For every new feature:
10. Deploy to `nik-gpu`.
11. Test against the live game.
12. Update `plan.md` status.
13. Move temporary investigation files into `scratchpad/`, or delete them if they are no longer useful.
## Priority when uncertain
When uncertain, prefer this order:
1. Preserve existing working behavior.
2. Follow the reference project's control flow.
3. Keep logic in Python.
4. Add reusable driver primitives instead of feature-specific hacks.
5. Use local screenshots/assets only when backend differences require it.
6. Avoid large rewrites that do not move the project closer to reference-driven Python architecture.
2. Follow the reference project's control flow, including its use of OCR.
3. Do not deprioritize OCR-based navigation/matching in favor of a simpler non-OCR substitute.
4. Keep logic in Python.
5. Add reusable driver primitives instead of feature-specific hacks.
6. Use local screenshots/assets only when backend differences require it.
7. Use `scratchpad/` for temporary work and diagnostics.
8. Avoid large rewrites that do not move the project closer to reference-driven Python architecture.
## User preference
@ -504,4 +791,6 @@ The user wants this project to be as close to the original Blue Archive Auto Scr
The user specifically does not want Claude Code to keep converting feature work into Bash.
Respect that preference.
The user also wants Claude Code to avoid using `/private/tmp/claude-*` for development probes when a repository-local `scratchpad/` directory can be used instead, because those paths trigger permission prompts and interrupt the development flow.
Respect those preferences.

View File

@ -28,7 +28,12 @@ Current task status:
| Command | Implementation |
|---|---|
| `mailbox` | Real Python (`ba_auto/tasks/mailbox.py`). Verifies the mailbox panel actually opened (via a pixel-color probe, `driver.color_at`) before clicking "claim all" or pressing any further keys. Retries the open-click up to 3 times before giving up safely. |
| `cafe` | Still a bridge: `ba_auto/tasks/cafe.py` shells out to `scripts/ba_dailies_legacy.sh cafe`, the original unverified fixed-coordinate click sequence, unchanged. Not yet hardened — see "Known issue" below. |
| `cafe` | Real Python (`ba_auto/tasks/cafe.py`). Verifies each room/dialog transition the same way as `mailbox` before acting; sparkle detection runs in-process via `ba_auto/detector.py` instead of shelling out per click. See "Fixed: the exit-game dialog bug" below for what this replaced, and `plan.md` Phase 6 follow-up for the multi-scale detection + persistent-polling changes made after a "farming affection doesn't happen" report. |
| `stamina` | Real Python (`ba_auto/tasks/stamina.py`). Opens the Mission panel and claims via its bulk "一括受取" button (Enter key) when enabled. Does **not** touch the Pyroxene Purchase (青輝石購入) menu's free-AP claim — that's a real-money purchase screen and was deliberately left unautomated; see `plan.md` Phase 8. |
| `story_sweep` | Real Python (`ba_auto/tasks/story_sweep.py`). **Spends AP** — opt-in only, not part of the default flow. Sweeps a config-driven list of exact `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS` — edit this before running for real, it ships with a placeholder), navigating to each via OCR (region-number readout + stage-label matching) rather than a random pick. Verifies the MAX/`+` click actually raised the count before starting the sweep, clicks through the AP-usage-confirmation dialog, and closes the stage-info modal via its own X button afterward (Escape doesn't close it — confirmed live). See `plan.md` Phase 10. |
| `shop_common` | Real Python (`ba_auto/tasks/shop_common.py`). **Spends credits** — opt-in only, not part of the default flow. Buys a config-driven list of exact `(row, col, name, expected_price)` targets (`config.COMMON_SHOP_TARGETS`) from the 通常アイテム tab: OCR-verifies each item's price before checking its box, then clicks the bulk 購入 button and confirms through the purchase dialog. Live-tested with real purchases. See `plan.md` Phase 11. |
| `shop_tactical` | Real Python (`ba_auto/tasks/shop_tactical.py`). **Spends tactical coin** — opt-in only, not part of the default flow. Same flow as `shop_common` against the 戦術対抗戦 tab and `config.TACTICAL_SHOP_TARGETS`. Live-tested with real purchases. See `plan.md` Phase 11. |
| `lesson` | Real Python (`ba_auto/tasks/lesson.py`). **Spends lesson tickets** — opt-in only, not part of the default flow. Sweeps every unlocked region's schedule grid, picking the highest-affection available lesson each time (via `detector.read_int_on_heart_badge` OCR on each portrait's heart-shaped badge) until tickets or lessons run out. Live-tested with real tickets spent. See `plan.md` Phase 12. |
## Prerequisites on nik-gpu
@ -36,10 +41,10 @@ One-time, or after a dependency change:
```bash
ssh nik-gpu
which xdotool scrot # both must be installed
which xdotool scrot tesseract # all three must be installed
```
`setup.sh` (see below) creates the Python venv and checks these for you.
`tesseract` (the OCR engine `story_sweep` uses for region/stage-label reads) needs `sudo apt install tesseract-ocr``setup.sh` checks for it but can't install it for you, since sudo needs an interactive password. `setup.sh` (see below) creates the Python venv (including `pytesseract`) and checks all three tools for you.
## Deploying your changes
@ -55,16 +60,21 @@ rsync -av --delete \
ssh nik-gpu "cd ~/repo/ba-auto-daily && ./setup.sh"
```
`setup.sh` copies the synced files into the fixed runtime paths (`~/ba_dailies.sh`, `~/ba_daily.py`, `~/ba_auto/`, `~/ba_scripts/`, `~/ba_assets/`) and (re)installs the venv. Re-run both commands any time you change code — there is no auto-deploy.
`setup.sh` copies the synced files into the fixed runtime paths (`~/ba_dailies.sh`, `~/ba_daily.py`, `~/ba_auto/`, `~/ba_assets/`) and (re)installs the venv. Re-run both commands any time you change code — there is no auto-deploy.
## Running it
The game must already be running on `nik-gpu` (window titled `BlueArchive`). Then, on `nik-gpu`:
```bash
~/ba_dailies.sh # default: mailbox, then cafe
~/ba_dailies.sh # default: mailbox, cafe, then stamina
~/ba_dailies.sh mailbox # just mailbox
~/ba_dailies.sh cafe # just cafe
~/ba_dailies.sh stamina # just the Mission-panel bulk claim
~/ba_dailies.sh story_sweep # spends AP -- not in the default flow, run explicitly
~/ba_dailies.sh shop_common # spends credits -- not in the default flow, run explicitly
~/ba_dailies.sh shop_tactical # spends tactical coin -- not in the default flow, run explicitly
~/ba_dailies.sh lesson # spends lesson tickets -- not in the default flow, run explicitly
```
You can also run these remotely without a separate `ssh` login step:
@ -73,7 +83,7 @@ You can also run these remotely without a separate `ssh` login step:
ssh nik-gpu "~/ba_dailies.sh mailbox"
```
Exit code `0` means the script ran to completion — it does **not** by itself guarantee the in-game action succeeded (`mailbox` checks and logs this explicitly; `cafe` currently does not).
Exit code `0` means the script ran to completion without a Python exception — it does **not** by itself guarantee the in-game action succeeded. `mailbox`, `cafe`, and `stamina` all log what they actually detected and did (or why they safely aborted), so check the log output, not just the exit code.
## How to watch/verify it
@ -86,11 +96,13 @@ Exit code `0` means the script ran to completion — it does **not** by itself g
```
- **Read the log output.** Each task prints what it's doing, e.g. `mailbox` prints `panel not detected after click (attempt N/3)` if a click misses, and `nothing to claim` vs `claiming all` depending on what it found.
## Known issue: don't trust `cafe` unattended yet
## Fixed: the exit-game dialog bug
Live-testing `mailbox` surfaced a real bug in the *original*, unverified click sequence (see `plan.md` Phase 5 for the full writeup): a slightly-off icon coordinate caused a missed click, and the fixed sequence blindly kept going — clicking, pressing Enter, pressing Escape — with no idea whether any of it landed. The trailing Escape ended up hitting the bare home screen, which triggers Blue Archive's own **"Exit the game?"** confirmation dialog.
Live-testing both `mailbox` and `cafe` surfaced the same real bug in the *original*, unverified click sequences (see `plan.md` Phases 56 for the full writeup): a slightly-off icon coordinate occasionally caused a missed click, and the fixed sequence blindly kept going — clicking, pressing Enter, pressing Escape — with no idea whether any of it landed. The trailing Escape ended up hitting the bare home screen, which triggers Blue Archive's own **"Exit the game?"** confirmation dialog. `cafe`'s sequence is much longer than `mailbox`'s (open → pat loop ×15 → switch room → pat loop ×15 → claim income → exit), so a missed click there had more room to cascade.
`mailbox.py` now guards against this (verifies state before acting, retries, aborts safely instead of guessing). **`cafe` still uses the old unverified bridge and has the same failure mode.** If you run `~/ba_dailies.sh cafe` and something looks off, check the screen before pressing anything — if you see an unexpected confirmation dialog, press **Escape/Cancel**, never Enter/OK, until you've confirmed what it's asking.
Both `mailbox.py` and `cafe.py` now guard against this: they verify state via pixel-color probes before acting, retry a bounded number of times on a missed click, and abort safely (no further keypresses) instead of guessing. This was verified against the live game, including an actual income claim (gold and AP increased as expected) and a real sparkle detect-and-click.
That said, this is one round of live testing, not exhaustive coverage — see `plan.md` Phase 6 "Not verified" for open risks (rank-up popups mid-loop, long-run camera zoom/pan drift). If you ever see an unexpected confirmation dialog while running either command, press **Escape/Cancel**, never Enter/OK, until you've confirmed what it's asking.
## Local checks (nik-macbookair)
@ -98,7 +110,6 @@ The game can't run here, so this only catches syntax errors, not behavior:
```bash
bash -n ba_dailies.sh
bash -n scripts/ba_dailies_legacy.sh
python3 -m py_compile ba_daily.py ba_auto/*.py ba_auto/tasks/*.py
```
@ -107,5 +118,5 @@ Real verification only happens by actually running against the live game on `nik
## More detail
- `CLAUDE.md` — architecture rules and conventions for this repo.
- `plan.md` — feature-by-feature migration status and backlog, including the mailbox bug writeup.
- `plan.md` — feature-by-feature migration status and backlog, including the mailbox/cafe exit-game-dialog bug writeup.
- `ba_auto/reference_notes/mapping.md` — maps each feature to its `baas-reference` source.

View File

@ -6,11 +6,368 @@ XAUTHORITY = "/run/user/1000/gdm/Xauthority"
ENV = {**os.environ, "DISPLAY": DISPLAY, "XAUTHORITY": XAUTHORITY}
WINDOW_NAME = "BlueArchive"
ASSET_DIR = os.path.expanduser("~/ba_assets")
# TODO: remove once cafe is ported off the legacy Bash bridge (mailbox no longer uses this).
LEGACY_SCRIPT = os.path.expanduser("~/ba_scripts/ba_dailies_legacy.sh")
# Runtime working files (probe/detection screenshots); never /tmp, per CLAUDE.md.
SCRATCHPAD_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "scratchpad")
os.makedirs(SCRATCHPAD_DIR, exist_ok=True)
# Refined from (1726, 60): that coordinate sat on the edge of the icon's
# hitbox and intermittently missed during live testing.
MAILBOX_ICON = (1732, 50)
CLAIM_ALL = (1691, 1128)
MISSION_ICON = (75, 350)
# Background pixel inside the "一括受取" (claim all) button on the Mission
# panel: bright yellow (r~250, r-b~180+) when something is claimable, flat
# grey (r-b<20) when not. Calibrated live at 1920x1200.
MISSION_CLAIM_PROBE = (1600, 1100)
CAFE_ICON = (165, 1100)
CAFE_ROOM_SWITCH = (190, 160)
CAFE_INCOME = (1780, 1105)
# Max sparkle-detection attempts per room (hits and misses both count --
# sparkles are on a per-student cooldown, so most checks legitimately find
# nothing and the loop keeps polling rather than giving up after one miss).
CAFE_MAX_CLICKS_PER_ROOM = 15
CAFE_SPARKLE_TEMPLATE = os.path.join(ASSET_DIR, "cafe_sparkle.png")
# A pat that crosses an affection-rank threshold shows a full-screen "絆ラン
# クアップ!" (Bond Rank Up!) cutscene with no cafe header visible at all --
# confirmed against screenshots/cafe/student/01-02: navigation.is_on_subscreen's
# header probe reads (183,220,240) there (r<200, fails) vs (248,249,250) on
# the normal cafe screen (r>200, passes), so the existing header-brightness
# check already tells the two apart. Bounds how many Enter presses
# _dismiss_rank_up_if_shown will try before giving up.
CAFE_RANK_UP_DISMISS_RETRIES = 5
# Home -> お仕事 (Work hub) -> 任務 (Task) card -> Normal/Hard story region browser.
WORK_ICON = (1793, 1138)
# Moved up from the original (1370, 450): that point sat close enough to the
# 任務 card's bottom edge that a live Phase 10 run missed and landed on the
# "総力戦" (Total War) card in the row below instead -- confirmed live via
# screenshot, not just a hunch. (1250, 380) sits solidly mid-card, on the
# "任務" title text itself, well clear of every edge.
TASK_CARD = (1250, 380)
REGION_RIGHT_ARROW = (1862, 598)
# Pixel-scanline-scanned (not visually estimated -- see plan.md Phase 8's
# lesson) from scratchpad/stage_info.png: the "<" chevron's navy-blue pixel
# centroid was (66, 597), mirroring REGION_RIGHT_ARROW. Used for the
# OCR-driven to_region port (ba_auto/tasks/story_sweep.py) to step backward
# when the current region is past the target.
REGION_LEFT_ARROW = (66, 598)
# Bounds the "read region, click delta, re-check" loop in
# ba_auto/tasks/story_sweep.py's _go_to_region. A correct read normally
# converges in one round; this just guards against a stuck OCR misread.
REGION_NAV_MAX_ATTEMPTS = 8
# Region-number readout on the region browser's left panel (the "Area 30"
# card's big digits, below the smaller "Area" label). Rect pixel-scanned from
# scratchpad/stage_info.png: the "Area" label occupies roughly y 295-325, the
# number itself y 330-372 -- this rect isolates just the digits. Replaces
# _go_to_latest_region's "spam the arrow and hope" (see plan.md Phase 9's
# retrospective) with task_utils.py::to_region's actual OCR-read-and-click-
# the-exact-delta approach.
REGION_NUMBER_OCR_RECT = (175, 325, 250, 380)
# Stage list panel (right side of the region browser). Scrolling to either
# extreme always shows exactly 4 full stage rows, since every region has at
# least 5 stages -- scrolling past either end is a harmless no-op (verified
# live), so a generous bounded click count is safe. Each stage row's "入場"
# (enter) button sits at STAGE_ENTER_X across both scroll extremes; the two
# row-position sets below were measured at each extreme.
STAGE_LIST_SCROLL_POINT = (1400, 700)
# 10 was the original (Phase 9) calibration, but live re-testing during
# Phase 10 found it insufficient to reach the opposite extreme when the list
# was already scrolled near the other end (it undershot, landing between the
# two calibrated row-position sets and producing garbled OCR reads) -- 20
# reliably reached either extreme regardless of starting position. Scrolling
# past either end remains a harmless no-op (verified live both phases).
STAGE_LIST_SCROLL_CLICKS = 20
STAGE_ENTER_X = 1683
STAGE_ROWS_AT_TOP_Y = (424, 570, 718, 866)
STAGE_ROWS_AT_BOTTOM_Y = (483, 630, 778, 926)
# Stage-label OCR rect (e.g. "30-1", "30-A"), offset from a row's known
# center y above. Pixel-scanned across all eight row positions (both
# STAGE_ROWS_AT_TOP_Y and _BOTTOM_Y) in live captures -- x[1030,1150],
# y[row_y-40, row_y-4] consistently isolates just the label text line above
# the row's star-rating icons. The symmetric row_y+/-40 crop tried first
# during calibration included the stars below and reliably broke OCR (empty
# or garbled reads) even with a character whitelist -- see plan.md Phase 10.
# Replaces _pick_random_stage_row's "scroll to an extreme, grab a random one
# of the 4 rows" with a scoped-down port of the reference's
# swipe_search_target_str: since this client's stage list only ever has 2
# relevant scroll positions (top/bottom extremes, both already known-good),
# full swipe-and-retry generality isn't needed -- just OCR each of the 4
# visible rows at each extreme and match by label text.
STAGE_LABEL_OCR_X = (1030, 1150)
STAGE_LABEL_OCR_Y_PAD = (40, 4)
# 任務情報 (stage info) modal's sweep sub-panel. This modal is wide enough
# that navigation.MODAL_DIM_PROBE (960, 200) lands on the modal's own white
# card instead of the dimmed backdrop -- use a corner point that's outside
# the card in either scroll/region state instead.
STAGE_MODAL_PROBE = (1870, 600)
# Regular numbered stages (30-1..30-5) render an extra "集中指揮"/"簡易攻略"
# tab row and a manual "任務開始" panel below the sweep panel that the
# bonus "-A" stage Phase 9 originally calibrated against does not have --
# discovered live during Phase 10 when Phase 9's coordinates (calibrated
# only against 30-A) missed the MAX button on 30-3 by ~43px vertically.
# These values are pixel-scanned against 30-3's tabbed layout and are what
# config.STORY_SWEEP_TARGETS will hit in the common case (sweeping a
# regular numbered stage, not the "-A" bonus stage). If a target's stage is
# "A", these may be off by the same ~43px the old (untabbed) calibration
# used -- not yet re-confirmed against an actual "-A" stage since this fix;
# see plan.md Phase 10.
SWEEP_MAX_BUTTON = (1620, 550)
SWEEP_START_BUTTON = (1400, 710)
# The "-" stepper button next to the sweep count: flat grey (240,240,239)
# while count is still at its default of 1, vivid orange (255,111,0) once
# MAX (or any +) has raised it. Used to verify the MAX click actually landed
# instead of trusting a single click blindly, since this gates real AP spend.
SWEEP_MINUS_BUTTON_PROBE = (1285, 555)
# The modal's own "X" close icon (top-right corner of the white card).
# Escape does NOT close this modal (confirmed live: two Escape presses left
# it open with focus on the live "任務開始"/start-mission button) -- must
# click this explicitly. Pinned via pixel-scanline scan of the glyph's
# crossing point, not visual estimation.
#
# The whole card (not just the sweep sub-panel) is vertically centered on
# its own content height rather than anchored at a fixed absolute position
# -- confirmed live during Phase 10: the tabbed regular-stage layout (see
# SWEEP_MAX_BUTTON above) is taller than the "-A" bonus-stage layout this
# was originally calibrated against, and its X button sits ~46px higher
# on screen (225 vs the old 271) as a result. This value is re-measured
# against the taller, tabbed layout.
STAGE_MODAL_CLOSE_BUTTON = (1691, 225)
# "+" stepper button, for configured exact (non-"max") sweep counts. Pinned
# via color-scan (bright cyan glyph centroid) against 30-3's tabbed layout
# (see SWEEP_MAX_BUTTON above) -- unlike SWEEP_MAX_BUTTON/
# SWEEP_MINUS_BUTTON_PROBE this specific button has NOT been live-clicked
# yet; confirm it before relying on a non-"max" configured count (see
# plan.md Phase 10).
SWEEP_PLUS_BUTTON = (1520, 550)
# Clicking 掃討開始 (start sweep) always raises an "AP<N>使用して、掃討を
# <M>回行いますか?" usage-confirmation dialog before the sweep actually
# runs -- discovered live during Phase 10; the previous design had no
# handling for this dialog at all, which is what a crude early placeholder
# probe was misreading as "inadequate_ap" on every sweep, successful or not.
#
# If AP is too low for even one sweep (confirmed live by deliberately
# emptying the count via MAX/"+" at low AP), a dialog that looks the same
# and sits at the *same* OK-button position appears instead, but titled
# "AP購入" (real-currency AP purchase) with a gold/yellow OK button instead
# of cyan. The two are told apart by that color, not by position.
SWEEP_CONFIRM_BUTTON = (1150, 810)
SWEEP_CONFIRM_CANCEL_BUTTON = (770, 810)
SWEEP_CONFIRM_CYAN = ((90, 190, 230), (200, 240, 256))
SWEEP_CONFIRM_GOLD = ((200, 200, 50), (256, 256, 150))
# Region spanning every button this task clicks through after 掃討開始: the
# AP-usage-confirm OK above, and the "掃討完了" (sweep complete) results
# screen's "SKIP" (first, skips the reward-reveal animation) and final "OK"
# (after full reward totals appear) buttons. SKIP and the final OK share
# SWEEP_CONFIRM_CYAN's color but sit ~120px apart vertically, so this task
# finds whichever one is showing by color within this region instead of
# hardcoding each dialog's exact Y position.
SWEEP_RESULT_BUTTON_REGION = (700, 700, 1300, 1050)
# (region, stage, count) targets to sweep, mirroring the reference's
# unfinished_normal_tasks shape (module/explore_tasks/sweep_task.py). `stage`
# is 1-5, or the string "A" for the bonus stage that only exists when
# region % 3 == 0. `count` is a positive int (uses the "+" stepper) or the
# literal string "max" (uses the in-game MAX button).
#
# Empty by default -- the earlier placeholder here was (1, 1, "max"), which
# a real run then dutifully swept region 1 stage 1 instead of the account's
# actual last region, since story_sweep has no "find the latest region"
# heuristic anymore (see plan.md Phase 9's retrospective for why that
# heuristic was removed). Add entries here for any additional fixed targets
# you want swept every run, on top of the daily rotation target below.
STORY_SWEEP_TARGETS = []
# Daily-rotating target: sweeps a single region, cycling through its stages
# one per day rather than grinding the same stage every run, per explicit
# user direction. `ROTATION_STAGE_COUNT` is how many stages that region has
# (1..N); which one runs today is `today's date -> N` via a plain date
# ordinal modulo, not the calendar day-of-year, so the cycle doesn't skip or
# repeat around a year boundary. Set STORY_SWEEP_ROTATION_REGION to None to
# disable this and only sweep STORY_SWEEP_TARGETS.
STORY_SWEEP_ROTATION_REGION = 30
STORY_SWEEP_ROTATION_STAGE_COUNT = 6
STORY_SWEEP_ROTATION_COUNT = "max"
# Common Shop / Tactical Challenge Shop. Both tabs share the same underlying
# checkbox-grid-then-bulk-buy UI (the live equivalent of the reference's
# module/shop/shop_utils.py get_item_position/ensure_choose/buy pattern);
# see ba_auto/tasks/shop_utils.py for the shared control flow.
SHOP_ICON = (1155, 1085) # bottom nav "ショップ" icon on the home screen
SHOP_BACK_BUTTON = (85, 55)
SHOP_TAB_COMMON = (160, 208) # 通常アイテム tab (credit-point items)
# 戦術対抗戦 tab (tactical-coin items). The reference reaches this via
# goto_shop_by_name's OCR swipe-search over the shop-type tab list
# (module/shop/shop_utils.py) because that list can require scrolling on
# some accounts/versions. Confirmed live here: this account's tab list is
# only 7 entries and all fit on screen with no scroll needed, so a fixed
# click is faithful (there is nothing to search for), not a shortcut around
# the OCR the reference would otherwise need.
SHOP_TAB_TACTICAL = (160, 915)
# Item grid checkbox top-left-ish click point per (row, col), pixel-scanned
# live against the shop's real catalog (see plan.md's shop phase). Confirmed
# identical across both shop tabs -- it's the same shared UI component.
SHOP_ITEM_COL_X = [972, 1197, 1423, 1649]
SHOP_ITEM_ROW_Y = [297, 674]
# The checkbox glyph renders this vivid yellow-green only once checked
# (plain white/grey otherwise) -- pixel-sampled from a live checked vs.
# unchecked capture of the same card.
SHOP_CHECKED_RGB = ((60, 130, 80), (220, 245, 115))
# Each item's price-digit crop, as an (x1, y1, x2, y2) offset added to that
# item's own (col_x, row_y). Pixel-scanned and OCR-tested live against all 8
# configured targets' actual prices (12,500 up to 500,000) -- wide enough
# for the largest configured price without bleeding into the neighboring
# column's card.
SHOP_PRICE_OCR_OFFSET = (48, 166, 145, 195)
SHOP_BUY_BUTTON = (1751, 1112)
SHOP_CANCEL_BUTTON = (1525, 1112)
# A corner point that both the purchase-confirm dialog and the post-purchase
# "報酬獲得!" (reward acquired) banner dim away from pure white as they
# cover the screen -- confirmed live across both, and confirmed to stay pure
# white with no dialog open (across tab switches and scrolling). Cheaper and
# more robust than tracking each dialog's own layout individually.
SHOP_OVERLAY_PROBE = (100, 600)
SHOP_OVERLAY_IDLE_MIN_CHANNEL = 200
# Bounds shop_utils.confirm_purchase's "press Enter until idle" loop.
# Confirmed live: exactly 2 presses clears both the confirm dialog and the
# reward banner in one purchase; this leaves headroom for any additional
# one-time popup (e.g. a first-time notice) without spinning forever if the
# game is ever in a state this project doesn't recognize.
SHOP_PURCHASE_MAX_ENTER_PRESSES = 6
# Top status bar credit-point balance -- present on every screen, not
# shop-specific. Rect excludes the currency icon on the left, which OCR
# otherwise misreads as a spurious leading digit (confirmed live).
CREDIT_BALANCE_OCR_RECT = (1040, 15, 1290, 55)
# The Tactical Challenge Shop's own coin-balance readout, shown inline above
# the item grid on that tab specifically (not in the top status bar).
TACTICAL_COIN_OCR_RECT = (1090, 100, 1290, 150)
# (row, col, item name (log/debug only -- identification is by grid
# position, see shop_utils.py's module docstring for why), expected
# credit-point price). Buy-list confirmed with the user; all 8 are visible
# without scrolling.
COMMON_SHOP_TARGETS = [
(0, 0, "初級レポート", 12500),
(0, 1, "中級レポート", 125000),
(0, 2, "上級レポート", 300000),
(0, 3, "最上級レポート", 500000),
(1, 0, "初級強化珠", 10000),
(1, 1, "中級強化珠", 40000),
(1, 2, "上級強化珠", 96000),
(1, 3, "最上級強化珠", 128000),
]
# (row, col, item name, expected tactical-coin price). Buy-list confirmed
# with the user; both visible without scrolling.
TACTICAL_SHOP_TARGETS = [
(0, 0, "初級栄養ドリンク(AP30)", 15),
(0, 1, "中級栄養ドリンク(AP60)", 30),
]
# Lesson/Schedule (module/lesson.py). This client renders the reference's
# paged single-region view as a scrollable "Location Select" list of 12
# named regions instead -- these names are the reference's own
# lesson_region_name.JP list (core/config/default_config.py), embedded there
# directly rather than fetched externally, so hardcoding it locally isn't an
# external-data problem the way the shop price table was. They're used only
# for logging here, not for OCR identification -- unlike the reference's
# paged arrows, this list's scroll position is deterministic (see
# REGION_ROW_Y below), so there's nothing to OCR-locate.
LESSON_ICON = (314, 1100) # bottom nav "スケジュール" icon on the home screen
LESSON_BACK_BUTTON = (85, 55) # shared by both the region-list and per-region map screens
LESSON_REGION_NAMES = [
"シャーレオフィス", "シャーレ居住区", "ゲヘナ学園・中央区", "アビドス高等学校",
"ミレニアム・スタディーエリア", "トリニティ・スクエア", "レッドウインター連邦学園",
"百鬼夜行中心部", "D.U.シラトリ区", "山海経中央特区", "春葉原", "ワイルドハント総合芸術地区",
]
# The region list only ever settles at two scroll positions -- scrolled fully
# to top (regions 0-5 visible) or fully to bottom (regions 6-11) -- confirmed
# live: 15 scroll-down clicks always lands on the same bottom state, it
# doesn't keep scrolling past it. Each row's card is clickable at this
# center-ish Y regardless of which of the two scroll states is showing.
LESSON_REGION_ROW_Y = [265, 420, 585, 745, 900, 1060]
LESSON_REGION_LIST_SCROLL_POINT = (1400, 700)
LESSON_REGION_LIST_SCROLL_CLICKS = 15
LESSON_REGION_ROW_X = 1400
# Per-region isometric map screen -- opens the "全てのスケジュール" grid
# modal (this client's rendering of the reference's per-region 3x3
# get_lesson_each_region_status/get_lesson_relationship_counts grid).
LESSON_ALL_SCHEDULES_BUTTON = (1770, 1118)
LESSON_GRID_MODAL_CLOSE_BUTTON = (1710, 211)
# Grid modal cell layout: up to 3 columns x 3 rows of location cards, each
# showing up to 3 student portraits with a heart-shaped affection-count
# badge at bottom-right. Pixel-scanned live against two different regions'
# modals (Gehenna Central: 8 cells, Schale Office: 7 cells) -- consistent
# across both. A region with more than 9 currently-unlocked locations would
# need scrolling inside this modal, which isn't implemented (not yet seen
# live on this account; see plan.md).
LESSON_GRID_COL_X = [270, 786, 1302] # portrait-slot-0 center per column
LESSON_GRID_ROW_HEADER_Y = [380, 608, 836] # click target to open a cell's info panel
LESSON_GRID_ROW_PORTRAIT_Y = [486, 713, 940]
LESSON_GRID_PORTRAIT_STEP_X = 109 # slot 1/2 center = slot 0 center + N * this
# Badge center sits below-right of each portrait-slot's own center point.
LESSON_GRID_BADGE_OFFSET = (38, 24)
LESSON_GRID_BADGE_OCR_HALF_SIZE = (26, 20) # crop half-width/half-height around badge center
# A green checkmark can appear at top-right of a portrait ALONGSIDE its
# unchanged heart badge number, not instead of it (confirmed live -- an
# earlier assumption that "done" always blanks the number was wrong) --
# this is the only reliable "already done today" signal, so it's checked
# separately rather than inferred from the badge OCR.
LESSON_GRID_CHECKMARK_OFFSET = (41, -31)
LESSON_GRID_CHECKMARK_RGB = ((130, 190, 60), (220, 255, 160))
LESSON_GRID_CHECKMARK_HALF_SIZE = (14, 13)
# A masked, digit-only OCR read of the heart badge can still occasionally
# fuse a stray leading digit from portrait art bleeding into the crop's left
# edge (confirmed live: "13" -> "413", "18" -> "418", reproducible across
# every psm mode -- see detector.read_int_on_heart_badge). Real affection
# values never reach this range in practice, so treat anything this large
# as contamination and discard it rather than trust it.
LESSON_GRID_BADGE_MAX_PLAUSIBLE = 99
# "保有チケット N/M" readout, top-left of the region-list/map screens.
LESSON_TICKET_OCR_RECT = (295, 133, 385, 172)
# Per-cell info panel ("スケジュール情報"), opened by clicking a grid cell.
# Its Start button and the post-schedule report's OK button (below) are the
# same bright-cyan gradient button at nearly the same position -- confirmed
# by direct pixel sample, sharing one color range.
LESSON_ACTION_BUTTON_RGB = ((90, 195, 235), (150, 240, 255))
LESSON_INFO_START_BUTTON = (960, 890)
LESSON_INFO_CLOSE_BUTTON = (1435, 240)
# After clicking Start, a variable sequence of intermediate screens can
# appear before settling back on the grid modal -- a bond-rank-up full-
# screen cutscene (confirmed live, see screenshots/lesson/ -- its art is
# character-dependent, so no fixed color/position reliably identifies it)
# and/or a "スケジュールレポート" results modal. A transient "run the
# schedule twice" campaign multiplier was also observed live, doubling how
# many of these screens appear in a row -- rather than special-case any of
# this, `lesson.py` presses Enter in a bounded loop and re-checks two fixed
# markers each round:
#
# - the grid modal's own title underline (this exact yellow-gold, confirmed
# live) is visible ONLY when the grid modal is frontmost and idle -- both
# the report modal and the cutscene cover it, confirmed live against all
# three states.
LESSON_GRID_IDLE_PROBE_RECT = (790, 244, 1130, 252)
LESSON_GRID_IDLE_RGB = ((235, 220, 70), (255, 250, 130))
# - the report modal's OK button -- same color as LESSON_ACTION_BUTTON_RGB,
# its own confirmed fixed position -- clicked directly rather than folded
# into the blind Enter-press fallback, since we can verify it precisely.
LESSON_REPORT_OK_BUTTON = (960, 895)
LESSON_POST_SCHEDULE_MAX_ENTER_PRESSES = 8

View File

@ -1,3 +1,162 @@
"""Image/color matching helpers (OpenCV-based); not wired up yet."""
"""Image/color matching helpers (OpenCV-based). Ported from scripts/detect_and_click.py."""
import os
# TODO: migrate scripts/detect_and_click.py sparkle-matching logic here.
import cv2
import numpy as np
import pytesseract
from ba_auto import config, driver
OCR_SHOT_PATH = os.path.join(config.SCRATCHPAD_DIR, "ocr_live.png")
OCR_UPSCALE = 3
SPARKLE_SHOT_PATH = os.path.join(config.SCRATCHPAD_DIR, "cafe_live.png")
SPARKLE_CLICK_OFFSET = (75, 47)
SPARKLE_THRESHOLD = 0.97
# The cafe camera's zoom level isn't reset before farming, and the sparkle
# icon's on-screen size scales with it (see screenshots/cafe/sparkle/
# 01_sparke_zoomed_centered.png vs 03_sparkle_zoomed_out.png) -- a single
# fixed-scale template match misses whenever the camera isn't at the exact
# zoom the template was captured at. Try a spread of scales instead.
SPARKLE_SCALES = (0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2)
def _masked_template(template):
b, g, r = cv2.split(template.astype(np.int16))
yellow_white = ((r > 180) & (g > 140) & (r - b > 60)) | ((r > 200) & (g > 200) & (b > 200))
mask_plane = (yellow_white.astype(np.uint8)) * 255
return cv2.merge([mask_plane, mask_plane, mask_plane])
def find_cafe_sparkle():
driver.screenshot(SPARKLE_SHOT_PATH)
template_full = cv2.imread(config.CAFE_SPARKLE_TEMPLATE)
img = cv2.imread(SPARKLE_SHOT_PATH)
th0, tw0 = template_full.shape[:2]
best = None
for scale in SPARKLE_SCALES:
tw, th = max(1, round(tw0 * scale)), max(1, round(th0 * scale))
template = cv2.resize(template_full, (tw, th))
mask = _masked_template(template)
result = cv2.matchTemplate(img, template, cv2.TM_CCORR_NORMED, mask=mask)
_, max_val, _, max_loc = cv2.minMaxLoc(result)
if max_val >= SPARKLE_THRESHOLD and (best is None or max_val > best[0]):
best = (max_val, max_loc[0], max_loc[1], tw, th, scale)
if best is None:
return None
score, x, y, tw, th, scale = best
ox, oy = SPARKLE_CLICK_OFFSET
return (x + tw // 2 + round(ox * scale), y + th // 2 + round(oy * scale), score)
def _color_mask(region, rgb_min, rgb_max):
x1, y1, x2, y2 = region
driver.screenshot(OCR_SHOT_PATH)
img = cv2.imread(OCR_SHOT_PATH)
crop = img[y1:y2, x1:x2]
b, g, r = crop[:, :, 0].astype(np.int16), crop[:, :, 1].astype(np.int16), crop[:, :, 2].astype(np.int16)
(r_lo, g_lo, b_lo), (r_hi, g_hi, b_hi) = rgb_min, rgb_max
return (r >= r_lo) & (r <= r_hi) & (g >= g_lo) & (g <= g_hi) & (b >= b_lo) & (b <= b_hi)
def region_contains_color(region, rgb_min, rgb_max):
"""Whether any pixel within `region` (x1, y1, x2, y2) falls in the given
RGB range. Useful for presence checks on small, non-convex glyphs (e.g.
an arrow chevron) where a single fixed-point probe can land in the
glyph's own concave gap -- confirmed live: a centroid-based single point
for story_sweep's region-arrow chevron fell squarely in the notch
between its two strokes, reading as "absent" even while the arrow was
clearly rendered a few pixels away. See plan.md Phase 10.
"""
return bool(_color_mask(region, rgb_min, rgb_max).any())
def find_color_centroid(region, rgb_min, rgb_max):
"""Centroid `(x, y)` of every pixel within `region` (x1, y1, x2, y2)
falling in the given RGB range, or None if none match. Useful for
clicking a known-colored button whose exact position varies between
otherwise-similar dialogs -- e.g. story_sweep's sweep-result screen
shows the same cyan confirm-button color for its "SKIP" and final "OK"
states, ~120px apart vertically; finding it by color avoids hardcoding
both positions. See plan.md Phase 10.
"""
x1, y1, x2, y2 = region
mask = _color_mask(region, rgb_min, rgb_max)
ys, xs = np.nonzero(mask)
if len(xs) == 0:
return None
return (x1 + int(xs.mean()), y1 + int(ys.mean()))
def _ocr_crop(region):
x1, y1, x2, y2 = region
driver.screenshot(OCR_SHOT_PATH)
img = cv2.imread(OCR_SHOT_PATH)
crop = img[y1:y2, x1:x2]
gray = cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY)
# This UI's text is consistently dark-on-light -- a hard black/white
# threshold measurably fixed real misreads during live calibration
# (e.g. a stage label's digit misread as a stray extra digit) that
# persisted across every psm mode until the anti-aliased grey edges were
# removed. Confirmed live against scratchpad/scroll_up20.png row labels.
_, thresh = cv2.threshold(gray, 150, 255, cv2.THRESH_BINARY)
return cv2.resize(thresh, None, fx=OCR_UPSCALE, fy=OCR_UPSCALE, interpolation=cv2.INTER_CUBIC)
def read_text(region, whitelist=None, psm=7, lang="eng"):
"""OCR a pixel rectangle `(x1, y1, x2, y2)` from a fresh screenshot.
Local equivalent of the reference's core/ocr/ocr.py Baas_ocr client
(get_region_res) -- without its socket/shared-memory server, which exists
there to make OCR fast across thousands of automation steps. This project
only needs occasional single-crop reads (a region number, a stage label),
so a plain in-process pytesseract call is enough; see CLAUDE.md's "OCR
policy" and Handoff.md. `lang="eng"` is sufficient for the digit/dash
labels this project reads (region numbers, "30-1"/"30-A" stage labels) --
no Japanese trained data is needed for those specific reads.
"""
crop = _ocr_crop(region)
tess_config = f"--psm {psm}"
if whitelist:
tess_config += f" -c tessedit_char_whitelist={whitelist}"
return pytesseract.image_to_string(crop, lang=lang, config=tess_config).strip()
def read_int(region, psm=7):
"""Local equivalent of the reference's recognize_int."""
digits = "".join(ch for ch in read_text(region, whitelist="0123456789", psm=psm) if ch.isdigit())
return int(digits) if digits else None
def read_int_on_heart_badge(region, psm=7):
"""OCR a small dark-navy digit rendered on lesson.py's pink/magenta
heart-shaped affection badge.
read_int's plain grayscale threshold (tuned for this UI's normal
dark-text-on-light-card look) misreads these: the heart's own outline
stroke is a darker, saturated magenta whose grayscale value happens to
fall on the same side of the threshold as the digit glyph, so it
survives as stray black marks tesseract sometimes fuses into extra
digits -- confirmed live, "13" -> "113", "19" -> "119". The digit color
is reliably R < G (navy/blue-grey) while every pink/magenta badge tone
sampled (fill and outline, light and dark) is R > G, so masking on that
channel relationship instead of raw brightness cleanly drops the badge
shape and keeps just the glyph.
"""
x1, y1, x2, y2 = region
driver.screenshot(OCR_SHOT_PATH)
img = cv2.imread(OCR_SHOT_PATH)
crop = img[y1:y2, x1:x2]
b, g, r = crop[:, :, 0].astype(np.int16), crop[:, :, 1].astype(np.int16), crop[:, :, 2].astype(np.int16)
ink = (r < g) & (np.maximum(np.maximum(r, g), b) < 170)
binary = np.where(ink, 0, 255).astype(np.uint8)
upscaled = cv2.resize(binary, None, fx=OCR_UPSCALE, fy=OCR_UPSCALE, interpolation=cv2.INTER_CUBIC)
tess_config = f"--psm {psm} -c tessedit_char_whitelist=0123456789"
text = pytesseract.image_to_string(upscaled, lang="eng", config=tess_config).strip()
digits = "".join(ch for ch in text if ch.isdigit())
return int(digits) if digits else None

View File

@ -1,4 +1,5 @@
"""Local PC/Steam/Proton control backend (xdotool/scrot wrappers)."""
import os
import subprocess
import time
@ -6,7 +7,7 @@ import cv2
from ba_auto import config
PROBE_SHOT_PATH = "/tmp/ba_auto_probe.png"
PROBE_SHOT_PATH = os.path.join(config.SCRATCHPAD_DIR, "probe.png")
def run_command(args, **kwargs):
@ -28,10 +29,33 @@ def focus_game():
def click(x, y):
run_command(["xdotool", "mousemove", str(x), str(y), "click", "1"])
# A combined "mousemove X Y click 1" invocation is unreliable here --
# empirically (see plan.md's click-flakiness writeups) the game's input
# handler sometimes misses clicks fired before it's processed the move.
# Splitting into separate commands with a short pause between fixes it.
run_command(["xdotool", "mousemove", str(x), str(y)])
wait(0.2)
run_command(["xdotool", "click", "1"])
wait(0.5)
def move_mouse(x, y):
run_command(["xdotool", "mousemove", str(x), str(y)])
def scroll(x, y, direction, clicks=1):
# xdotool button 4/5 = scroll wheel up/down. A drag (mousedown/move/mouseup)
# does not register as a list-scroll gesture in this Proton client; the
# wheel does.
button = "4" if direction == "up" else "5"
run_command(["xdotool", "mousemove", str(x), str(y)])
wait(0.2)
for _ in range(clicks):
run_command(["xdotool", "click", button])
wait(0.15)
wait(0.3)
def keypress(key):
run_command(["xdotool", "key", key])
wait(0.5)

View File

@ -1,3 +1,56 @@
"""Shared navigation helpers (home, menu, popups, back/escape); not wired up yet."""
"""Shared navigation helpers (home, menu, popups, back/escape)."""
# TODO: extract common navigation flows here as tasks are ported off Bash.
# The mailbox/cafe/shop-style header bar renders a plain light background
# here; the home screen shows character art instead.
SUBSCREEN_HEADER_PROBE = (500, 10)
SUBSCREEN_HEADER_MIN_CHANNEL = 200
# Any modal dialog dims the screen behind it to roughly this darkness.
MODAL_DIM_PROBE = (960, 200)
MODAL_DIM_MAX_CHANNEL = 150
def is_on_subscreen(driver):
r, g, b = driver.color_at(*SUBSCREEN_HEADER_PROBE)
return r > SUBSCREEN_HEADER_MIN_CHANNEL and g > SUBSCREEN_HEADER_MIN_CHANNEL and b > SUBSCREEN_HEADER_MIN_CHANNEL
def is_modal_open(driver):
r, g, b = driver.color_at(*MODAL_DIM_PROBE)
return r < MODAL_DIM_MAX_CHANNEL and g < MODAL_DIM_MAX_CHANNEL and b < MODAL_DIM_MAX_CHANNEL
def wait_for_state(driver, config, reactions, ends, max_iterations=30, poll_interval=1.0):
"""Generic "watch the screen, react to anything recognized, stop once a
recognized destination is reached" loop -- the local equivalent of the
reference's core/picture.py::co_detect, scoped to what this project
actually needs (a handful of named checks) rather than co_detect's full
generality (which spans the whole reference project via ~20 image
template assets this project doesn't have).
`ends`: {check_fn(driver, config) -> bool: outcome_name}. Checked first,
every iteration; the first match stops the loop and returns its name.
`reactions`: {check_fn(driver, config) -> bool: action(driver)}. Checked
if no end matched; the first match runs its action (a click, a keypress,
whatever the recognized state calls for) and the loop continues.
If neither an end nor a reaction matches, the loop just waits and retries
-- it never falls back to a blind click/keypress guess (see CLAUDE.md's
exit-game-dialog writeup for why that was a real bug elsewhere).
Returns the matched end's outcome name, or None once max_iterations is
exhausted without reaching a recognized end -- callers should treat None
as "unrecognized state, abort safely."
"""
for _ in range(max_iterations):
for check_fn, outcome_name in ends.items():
if check_fn(driver, config):
return outcome_name
for check_fn, action in reactions.items():
if check_fn(driver, config):
action(driver)
break
else:
driver.wait(poll_interval)
return None

View File

@ -5,14 +5,15 @@ Maps each local feature to the corresponding `~/repo/baas-reference/module/...`
| Local feature | Reference file | Reference functions/classes | Local file | Backend replacements | Status |
|---|---|---|---|---|---|
| Mailbox | `module/mail.py` | `to_mail`, `implement` | `ba_auto/tasks/mailbox.py` | tap/click via xdotool, screenshot via scrot, `color.rgb_in_range``driver.color_at` pixel-probe check | Migrated: real Python, state-verified via color probe (no legacy bridge) |
| Cafe | Need to confirm in reference | Need to inspect | `ba_auto/tasks/cafe.py` | template matching via OpenCV, click via xdotool | Bridged to `scripts/ba_dailies_legacy.sh` + `scripts/detect_and_click.py`; Python port pending (plan.md Phase 6) |
| Stamina/AP | `module/collect_daily_free_power.py`, `module/collect_daily_task_power.py` | Need to inspect | `ba_auto/tasks/stamina.py` | color checks/clicks via local driver | Not started |
| Cafe | `module/cafe_reward.py` | `to_cafe` (its `relationship_rank_up` popup-handling now also ported, see below), `interaction_for_cafe_solve_method3`, `collect` | `ba_auto/tasks/cafe.py` | `picture.co_detect`/`color.rgb_in_range``driver.color_at` pixel-probe checks; sparkle template match ported in-process into `ba_auto/detector.py` (`find_cafe_sparkle`, now multi-scale) | Migrated: real Python, state-verified via color probes (no legacy bridge). Pat loop now polls for the full attempt budget instead of stopping on the first miss (see `plan.md` Phase 6 follow-up) — not yet confirmed against a live sparkle since none was available during testing. `_dismiss_rank_up_if_shown` reuses `navigation.is_on_subscreen` to detect and clear the full-screen bond-rank-up cutscene after a pat (see `plan.md` Phase 6 follow-up: rank-up popups) — not yet live-confirmed against a real trigger |
| Stamina/AP | `module/collect_daily_free_power.py`, `module/collect_daily_task_power.py` | `to_tasks`/`implement` (task-power, ported); `to_purchase_pyroxenes_menu` (free-power, not ported) | `ba_auto/tasks/stamina.py` | `color.rgb_in_range``driver.color_at`; reference's per-tab claim loop → live UI's single "一括受取" bulk-claim button + Enter | Partially migrated: Mission-panel claim done (see `plan.md` Phase 8). Daily Free Power (real-money purchase menu) deliberately not automated |
| Normal/Hard story AP sweep | `module/explore_tasks/sweep_task.py`, `module/explore_tasks/task_utils.py` | `to_region` (ported: OCR region-number readout + delta-click), a scoped-down `swipe_search_target_str` (ported: OCR stage-row label matching), `start_sweep`'s named-outcome contract (ported via `navigation.wait_for_state`, this project's scoped `co_detect` port) | `ba_auto/tasks/story_sweep.py` | OCR region/stage-name matching, ported for real (Phase 10) — replaces Phase 9's "next-region arrow stops advancing, then random stage" heuristic; MAX click verified via `SWEEP_MINUS_BUTTON_PROBE` color check (reused, still correct), modal closed via its own X button (Escape doesn't close it; X-button position re-calibrated per stage-layout variant, see Phase 10) | Done (see `plan.md` Phase 10, supersedes Phase 9). Config-driven exact `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS`), not latest-region/random-stage. Opt-in only, not in default flow |
| Group/Club AP | `module/group.py` | Need to inspect | `ba_auto/tasks/group.py` | fixed click + state check via local driver | Not started |
| Bounty | `module/rewarded_task.py` | Need to inspect | `ba_auto/tasks/bounty.py` | sweep/color/OCR adaptation | Not started |
| Commissions | `module/clear_special_task_power.py` | Need to inspect | `ba_auto/tasks/commission.py` | sweep/color adaptation | Not started |
| Arena | `module/arena.py` | Need to inspect | `ba_auto/tasks/arena.py` | auto-fight + OCR + local driver | Not started |
| Common Shop | `module/shop/common_shop.py`, `module/shop/shop_utils.py` | Need to inspect | `ba_auto/tasks/shop_common.py` | OCR + tab navigation + local clicks | Not started |
| Tactical Shop | `module/shop/tactical_challenge_shop.py`, `module/shop/shop_utils.py` | Need to inspect | `ba_auto/tasks/shop_tactical.py` | OCR + tab navigation + local clicks | Not started |
| Lesson/Schedule | `module/lesson.py` | Need to inspect | `ba_auto/tasks/lesson.py` | OCR + template/portrait search + local driver | Not started |
| Common Shop | `module/shop/common_shop.py`, `module/shop/shop_utils.py` | `implement`, `to_common_shop`, `get_item_position`/`ensure_choose`/`buy` (shared, see Tactical Shop row) | `ba_auto/tasks/shop_common.py`, `ba_auto/tasks/shop_utils.py` | `get_item_position`'s color+template item-state scan → fixed grid-position targets (`config.COMMON_SHOP_TARGETS`) + price-digit OCR verify, since the reference's own item-identification here indexes an external static price table (`self.static_config.common_shop_price_list`, fetched from a remote resource) this repo doesn't have — not per-item OCR, so this isn't an OCR-avoidance shortcut. Purchase-confirm dialog + reward-acquired banner handled via a single overlay-darkness probe (`config.SHOP_OVERLAY_PROBE`) instead of tracking each dialog's own layout | Done. Live-tested with real purchases (all 8 configured targets bought, cost matched exactly). Discovered live: these items have a per-refresh-cycle purchase cap not shown as a visible counter (unlike the 青輝石 tab's "あと1回購入可能" labels) — confirmed by re-running the task after purchase and observing it correctly detect the now-unselectable items (checkbox + individual 購入 button both unresponsive) and safely decline rather than guess. A fresh, everything-available run hasn't been re-verified since the account had already exhausted this cycle's purchases via that same test |
| Tactical Shop | `module/shop/tactical_challenge_shop.py`, `module/shop/shop_utils.py` | `implement`, `goto_shop_by_name`, shared `get_item_position`/`ensure_choose`/`buy` | `ba_auto/tasks/shop_tactical.py`, `ba_auto/tasks/shop_utils.py` | `goto_shop_by_name`'s OCR swipe-search over the shop-tab list → fixed click (`config.SHOP_TAB_TACTICAL`): this account's tab list is only 7 entries and fits on screen with no scroll needed, confirmed live, so there's nothing to search for — not an OCR-avoidance shortcut. Same grid-position + price-OCR-verify + overlay-probe design as Common Shop, sharing `shop_utils.run_shop_tab` | Done. Live-tested with real purchases (both configured AP-recovery drinks bought; AP and tactical-coin balance changes matched exactly) |
| Lesson/Schedule | `module/lesson.py` | `implement`, `to_lesson_location_select`/`to_select_location`/`to_all_locations` (nav state machine), `get_lesson_region_num`/`switch_lesson_region_page`/`to_lesson_region` (paged region nav), `get_lesson_each_region_status`+`check_region_availability` (per-cell status via isometric-parallelogram pixel scan), `get_lesson_relationship_counts` (per-cell affection pip count via color count), `choose_lesson` (selection policy), `execute_lesson`/`to_location_info`/`start_lesson` (click cell -> info panel -> start -> result) | `ba_auto/tasks/lesson.py` | `picture.co_detect` -> `navigation.wait_for_state`-style bounded Enter-press loop (see below); the reference's paged-arrow region nav (needing OCR to know current position) -> this client renders the 12 regions as a scrollable list instead, which only ever settles at two scroll positions (`config.LESSON_REGION_ROW_Y`), so navigation is direct index-based clicking with nothing to OCR-locate; the reference's isometric `Parallelogram`/`Triangle` per-cell scan (tuned to the reference's own screen layout) -> reading each portrait's heart-shaped affection badge via a dedicated OCR path (`detector.read_int_on_heart_badge`) needs no isometric geometry at all | Done. Config-driven scope only in the sense of the *policy* (affection-first selection, sweep every unlocked region until tickets/lessons run out, no ticket purchasing, no favor-student targeting) -- unlike shop, no user-specific target list was needed since the reference's own `lesson_region_name.JP` (embedded directly in its `default_config.py`, not externally fetched) already names all 12 regions, used here only for logging. Live-tested for real: 5 real tickets spent across 3 regions with correct outcomes (ticket count, cleanup navigation, home-screen return all verified). Two real bugs were found and fixed from that run -- see below and `plan.md`'s Lesson phase |
Do not implement a feature without filling at least the relevant row.

View File

@ -1,6 +1,124 @@
"""Cafe daily task."""
"""Cafe daily task. Ported from baas-reference module/cafe_reward.py's state-probe pattern."""
from ba_auto import detector, navigation
ROOM_OPEN_RETRIES = 3
# "受取" (claim) renders as flat grey when there is nothing to collect yet.
CLAIM_PROBE = (960, 850)
CLAIM_DISABLED_RGB = (218, 218, 218)
CLAIM_DISABLED_TOLERANCE = 15
def _claim_disabled(driver):
r, g, b = driver.color_at(*CLAIM_PROBE)
tr, tg, tb = CLAIM_DISABLED_RGB
return (
abs(r - tr) <= CLAIM_DISABLED_TOLERANCE
and abs(g - tg) <= CLAIM_DISABLED_TOLERANCE
and abs(b - tb) <= CLAIM_DISABLED_TOLERANCE
)
def _enter_room(driver, coords):
for attempt in range(1, ROOM_OPEN_RETRIES + 1):
driver.click(*coords)
driver.wait(3)
if navigation.is_on_subscreen(driver):
# dismiss the "visited student list" notice shown on room entry
driver.keypress("Return")
driver.wait(1)
return True
print(f"[cafe] room not detected after click (attempt {attempt}/{ROOM_OPEN_RETRIES})")
return False
def _dismiss_rank_up_if_shown(driver, config):
# The reference's own to_cafe() navigation (module/cafe_reward.py) treats
# 'relationship_rank_up' as a recognized, reactively-dismissed popup
# after every pat round -- this loop's original port had no equivalent,
# so a rank-up cutscene just sat there while find_cafe_sparkle() kept
# returning None against it (a full-screen character portrait, nothing
# like the sparkle template) for the rest of the room's click budget.
# That's the "freeze" -- not a timing fluke, a genuinely unhandled state.
for _ in range(config.CAFE_RANK_UP_DISMISS_RETRIES):
if navigation.is_on_subscreen(driver):
return True
driver.keypress("Return")
driver.wait(1.5)
return navigation.is_on_subscreen(driver)
def _pat_room(driver, config):
# Sparkles appear on a per-student cooldown, so most single checks find
# nothing -- the old Bash loop (and an earlier version of this one) gave
# up on the very first miss, which meant it essentially never farmed.
# Keep polling for the full budget instead of bailing early.
patted = 0
for _ in range(config.CAFE_MAX_CLICKS_PER_ROOM):
match = detector.find_cafe_sparkle()
if match is None:
driver.wait(1)
continue
x, y, score = match
driver.click(x, y)
driver.wait(1)
driver.keypress("Return")
# park the cursor away from the sparkle area so it can't occlude the
# next detection screenshot (see screenshots/cafe/sparkle/02_*_cursor_on_head.png)
driver.move_mouse(10, 1190)
if not _dismiss_rank_up_if_shown(driver, config):
print("[cafe] warning: cafe screen not confirmed after a pat (rank-up cutscene stuck?) -- stopping this room's pat loop rather than clicking blindly")
break
patted += 1
print(f"[cafe] patted sparkle at ({x}, {y}), score={score:.3f}")
print(f"[cafe] patted {patted} sparkle(s)" if patted else "[cafe] no sparkle found")
def _claim_income(driver, config):
driver.click(*config.CAFE_INCOME)
driver.wait(2)
if not navigation.is_modal_open(driver):
print("[cafe] income panel not detected, skipping claim")
return
if _claim_disabled(driver):
print("[cafe] nothing to claim")
else:
print("[cafe] claiming income")
driver.keypress("Return")
driver.wait(2)
driver.keypress("Return")
driver.wait(2)
if navigation.is_modal_open(driver):
driver.keypress("Escape")
driver.wait(1.5)
def run(driver, config):
# TODO: migrate old Bash cafe logic into this Python task.
driver.run_command([config.LEGACY_SCRIPT, "cafe"])
driver.focus_game()
if not _enter_room(driver, config.CAFE_ICON):
print("[cafe] could not confirm cafe is open, aborting without pressing further keys")
return
print("[cafe] room 1: farming affection")
_pat_room(driver, config)
if not _enter_room(driver, config.CAFE_ROOM_SWITCH):
print("[cafe] could not confirm room switch, stopping before income claim")
if navigation.is_on_subscreen(driver):
driver.keypress("Escape")
driver.wait(1.5)
return
print("[cafe] room 2: farming affection")
_pat_room(driver, config)
_claim_income(driver, config)
if navigation.is_on_subscreen(driver):
driver.keypress("Escape")
driver.wait(1.5)
print("[cafe] Done.")

247
ba_auto/tasks/lesson.py Normal file
View File

@ -0,0 +1,247 @@
"""Lesson/Schedule. Reference: baas-reference/module/lesson.py.
Scoped for v1 per plan.md's "Suggested first version" and explicit user
direction: affection-first selection (mirrors the reference's
lesson_relationship_first=True), sweep every unlocked region in a fixed
order until either lesson tickets or scoreable lessons run out, no
lesson-ticket purchasing (same real-currency-adjacent caution as Daily Free
Power / the shop's manual refresh button -- see CLAUDE.md), no favor-student
targeting (deferred, matching plan.md).
This client renders the reference's paged single-region view + isometric
3x3 status grid (get_lesson_each_region_status/get_lesson_relationship_counts,
built on an isometric Parallelogram/Triangle pixel scan tuned to the
reference's own screen layout) as a two-level UI instead: a scrollable list
of the same 12 named regions (config.LESSON_REGION_NAMES, taken directly
from the reference's own lesson_region_name.JP list -- used here only for
logging, since unlike the reference's paged arrows this list's scroll
position is deterministic and needs no OCR to locate), each opening a grid
modal of up to 9 location cards. Each card shows up to 3 student portraits
with a heart-shaped affection-count badge -- reading those via OCR (this
project's local equivalent of the reference's pip-counting
get_lesson_relationship_counts) needs no isometric geometry at all. A locked
location isn't listed in the grid modal at all, so there's nothing to do
there; a "no relationship yet" portrait has no badge and its OCR read comes
back empty, scoring as a non-candidate. An "already done today" portrait
was initially assumed to also blank its badge, but live testing showed that
assumption was wrong: it keeps showing its (unchanged) number and gets a
green checkmark added at top-right instead -- so done-ness is checked
separately via that checkmark's color, not inferred from the OCR read.
Reading the badge itself also needed its own OCR path
(detector.read_int_on_heart_badge): a plain grayscale-threshold read (this
project's usual approach) misreads it, because the heart's own outline
stroke happens to survive the same threshold as the digit glyph and
tesseract sometimes fuses the two into extra digits (see that function's
docstring, and config.py's LESSON_GRID_BADGE_MAX_PLAUSIBLE for the
plausibility-cap backstop this project keeps as a second line of defense).
"""
from ba_auto import detector, navigation
OPEN_RETRIES = 3
REGIONS_PER_SCREEN = 6
TOTAL_REGIONS = 12
GRID_ROWS = 3
GRID_COLS = 3
GRID_SLOTS = 3
def _read_ticket_count(driver, config):
text = detector.read_text(config.LESSON_TICKET_OCR_RECT, whitelist="0123456789/", psm=7)
head = text.split("/")[0] if "/" in text else text
digits = "".join(ch for ch in head if ch.isdigit())
return int(digits) if digits else None
def _open_schedule_screen(driver, config):
for attempt in range(1, OPEN_RETRIES + 1):
driver.click(*config.LESSON_ICON)
driver.wait(2)
if navigation.is_on_subscreen(driver):
return True
print(f"[lesson] schedule screen not detected after click (attempt {attempt}/{OPEN_RETRIES})")
return False
def _scroll_region_list(driver, config, to_bottom):
x, y = config.LESSON_REGION_LIST_SCROLL_POINT
direction = "down" if to_bottom else "up"
driver.scroll(x, y, direction, config.LESSON_REGION_LIST_SCROLL_CLICKS)
driver.wait(0.8)
def _is_grid_idle(driver, config):
lo, hi = config.LESSON_GRID_IDLE_RGB
return detector.region_contains_color(config.LESSON_GRID_IDLE_PROBE_RECT, lo, hi)
def _is_action_button_showing(driver, config, position):
r, g, b = driver.color_at(*position)
lo, hi = config.LESSON_ACTION_BUTTON_RGB
return lo[0] <= r <= hi[0] and lo[1] <= g <= hi[1] and lo[2] <= b <= hi[2]
def _open_region_grid(driver, config, region_index):
# Retries the whole click-row -> click-all-schedules sequence from
# scratch rather than trying to separately verify "did the isometric map
# open" -- that screen shares the same bright header as the plain list
# (navigation.is_on_subscreen can't tell them apart), so the grid modal
# actually opening is the only reliable signal either step worked.
screen_bottom = region_index >= REGIONS_PER_SCREEN
row = region_index % REGIONS_PER_SCREEN
for attempt in range(1, OPEN_RETRIES + 1):
_scroll_region_list(driver, config, to_bottom=screen_bottom)
driver.click(config.LESSON_REGION_ROW_X, config.LESSON_REGION_ROW_Y[row])
driver.wait(1.5)
driver.click(*config.LESSON_ALL_SCHEDULES_BUTTON)
driver.wait(1.2)
if _is_grid_idle(driver, config):
return True
print(f"[lesson] schedule grid not detected for region index {region_index} (attempt {attempt}/{OPEN_RETRIES})")
return False
def _close_grid_modal(driver, config):
driver.click(*config.LESSON_GRID_MODAL_CLOSE_BUTTON)
driver.wait(1)
def _slot_center(config, row, col, slot):
cx = config.LESSON_GRID_COL_X[col] + slot * config.LESSON_GRID_PORTRAIT_STEP_X
cy = config.LESSON_GRID_ROW_PORTRAIT_Y[row]
return cx, cy
def _badge_rect(config, row, col, slot):
cx, cy = _slot_center(config, row, col, slot)
ox, oy = config.LESSON_GRID_BADGE_OFFSET
hx, hy = config.LESSON_GRID_BADGE_OCR_HALF_SIZE
cx, cy = cx + ox, cy + oy
return (cx - hx, cy - hy, cx + hx, cy + hy)
def _checkmark_rect(config, row, col, slot):
cx, cy = _slot_center(config, row, col, slot)
ox, oy = config.LESSON_GRID_CHECKMARK_OFFSET
hx, hy = config.LESSON_GRID_CHECKMARK_HALF_SIZE
cx, cy = cx + ox, cy + oy
return (cx - hx, cy - hy, cx + hx, cy + hy)
def _is_slot_already_done(driver, config, row, col, slot):
lo, hi = config.LESSON_GRID_CHECKMARK_RGB
return detector.region_contains_color(_checkmark_rect(config, row, col, slot), lo, hi)
def _read_slot_affection(driver, config, row, col, slot):
if _is_slot_already_done(driver, config, row, col, slot):
return None
value = detector.read_int_on_heart_badge(_badge_rect(config, row, col, slot))
if value is not None and value > config.LESSON_GRID_BADGE_MAX_PLAUSIBLE:
# Contamination from portrait art bleeding into the crop's edge,
# not a real affection value -- see config.py's comment.
return None
return value
def _find_best_cell(driver, config):
best_score, best_cell = -1, None
for row in range(GRID_ROWS):
for col in range(GRID_COLS):
cell_score = -1
for slot in range(GRID_SLOTS):
value = _read_slot_affection(driver, config, row, col, slot)
if value is not None and value > cell_score:
cell_score = value
if cell_score > best_score:
best_score, best_cell = cell_score, (row, col)
return best_cell, best_score
def _click_cell(driver, config, row, col):
driver.click(config.LESSON_GRID_COL_X[col], config.LESSON_GRID_ROW_HEADER_Y[row])
driver.wait(1.2)
def _run_one_schedule(driver, config, row, col):
"""Click a grid cell through to a completed schedule. Returns True once
settled back at the idle grid modal, False if it never got there."""
_click_cell(driver, config, row, col)
if not _is_action_button_showing(driver, config, config.LESSON_INFO_START_BUTTON):
print("[lesson] info panel start button not detected, aborting this cell without pressing further keys")
return False
driver.click(*config.LESSON_INFO_START_BUTTON)
driver.wait(1.5)
# A variable number of intermediate screens can follow (a bond-rank-up
# cutscene, a results modal, occasionally both twice under a "2x
# schedule" campaign multiplier -- see config.py's comment). Neither
# intermediate screen has a reliable fixed marker of its own except the
# results modal's OK button, so anything that isn't "back at the idle
# grid" or "OK button showing" gets a plain Enter press, bounded.
for _ in range(config.LESSON_POST_SCHEDULE_MAX_ENTER_PRESSES):
if _is_grid_idle(driver, config):
return True
if _is_action_button_showing(driver, config, config.LESSON_REPORT_OK_BUTTON):
driver.click(*config.LESSON_REPORT_OK_BUTTON)
else:
driver.keypress("Return")
driver.wait(1.5)
return _is_grid_idle(driver, config)
def _sweep_region(driver, config, region_index, remaining_tickets):
name = config.LESSON_REGION_NAMES[region_index]
if not _open_region_grid(driver, config, region_index):
print(f"[lesson] could not confirm schedule grid opened for {name}, skipping")
return remaining_tickets
while remaining_tickets > 0:
cell, score = _find_best_cell(driver, config)
if cell is None:
print(f"[lesson] no schedulable lesson found in {name}")
break
row, col = cell
print(f"[lesson] {name}: best available affection {score} at row {row} col {col}")
if not _run_one_schedule(driver, config, row, col):
print(f"[lesson] warning: could not confirm return to the schedule grid after {name} row {row} col {col} -- stopping this region")
break
new_count = _read_ticket_count(driver, config)
remaining_tickets = new_count if new_count is not None else remaining_tickets - 1
print(f"[lesson] tickets remaining: {remaining_tickets}")
_close_grid_modal(driver, config)
driver.wait(0.5)
driver.click(*config.LESSON_BACK_BUTTON)
driver.wait(1.5)
return remaining_tickets
def run(driver, config):
driver.focus_game()
if not _open_schedule_screen(driver, config):
print("[lesson] could not confirm schedule screen is open, aborting without pressing further keys")
return
tickets = _read_ticket_count(driver, config)
if tickets is None:
print("[lesson] could not OCR ticket count, aborting without pressing further keys")
return
print(f"[lesson] starting tickets: {tickets}")
if tickets <= 0:
print("[lesson] no lesson tickets available, nothing to do")
else:
for region_index in range(TOTAL_REGIONS):
if tickets <= 0:
print("[lesson] out of lesson tickets -- stopping")
break
tickets = _sweep_region(driver, config, region_index, tickets)
driver.click(*config.LESSON_BACK_BUTTON)
driver.wait(1.5)
print("[lesson] Done.")

View File

@ -1,9 +1,6 @@
"""Mailbox daily task. Ported from baas-reference module/mail.py's color-probe pattern."""
# (500, 10) sits on the mailbox panel's plain header background; the home
# screen shows character art there instead, so this tells open vs not-open.
HEADER_PROBE = (500, 10)
HEADER_OPEN_MIN_CHANNEL = 200
from ba_auto import navigation
# "Claim all" renders as flat grey (153,153,153) when there is nothing to claim.
CLAIM_ALL_PROBE = (1710, 1128)
@ -13,11 +10,6 @@ CLAIM_ALL_DISABLED_TOLERANCE = 12
OPEN_RETRIES = 3
def _is_open(driver):
r, g, b = driver.color_at(*HEADER_PROBE)
return r > HEADER_OPEN_MIN_CHANNEL and g > HEADER_OPEN_MIN_CHANNEL and b > HEADER_OPEN_MIN_CHANNEL
def _claim_all_disabled(driver):
r, g, b = driver.color_at(*CLAIM_ALL_PROBE)
tr, tg, tb = CLAIM_ALL_DISABLED_RGB
@ -35,7 +27,7 @@ def run(driver, config):
for attempt in range(1, OPEN_RETRIES + 1):
driver.click(*config.MAILBOX_ICON)
driver.wait(1.5)
if _is_open(driver):
if navigation.is_on_subscreen(driver):
opened = True
break
print(f"[mailbox] panel not detected after click (attempt {attempt}/{OPEN_RETRIES})")

View File

@ -0,0 +1,27 @@
"""Common Shop (通常アイテム) daily task. Ported from baas-reference
module/shop/common_shop.py's implement() -- see shop_utils.py for the
shared checkbox-grid-then-bulk-buy control flow both shop tabs use."""
from ba_auto import navigation
from ba_auto.tasks import shop_utils
def run(driver, config):
driver.focus_game()
driver.click(*config.SHOP_ICON)
driver.wait(2)
if not navigation.is_on_subscreen(driver):
print("[shop_common] could not confirm shop opened, aborting without pressing further keys")
return
shop_utils.run_shop_tab(
driver, config,
tab_button=config.SHOP_TAB_COMMON,
targets=config.COMMON_SHOP_TARGETS,
balance_rect=config.CREDIT_BALANCE_OCR_RECT,
currency_label="credits",
)
driver.click(*config.SHOP_BACK_BUTTON)
driver.wait(1.5)
print("[shop_common] Done.")

View File

@ -0,0 +1,34 @@
"""Tactical Challenge Shop (戦術対抗戦) daily task. Ported from baas-reference
module/shop/tactical_challenge_shop.py's implement() -- see shop_utils.py for
the shared checkbox-grid-then-bulk-buy control flow both shop tabs use.
The reference reaches this tab via goto_shop_by_name's OCR swipe-search over
the shop-type tab list (module/shop/shop_utils.py), needed there because that
list can require scrolling. This account's tab list is only 7 entries and
fits on screen with no scrolling, confirmed live -- config.SHOP_TAB_TACTICAL
is a fixed click on the already-visible tab, not a shortcut around OCR (there
is nothing to search for)."""
from ba_auto import navigation
from ba_auto.tasks import shop_utils
def run(driver, config):
driver.focus_game()
driver.click(*config.SHOP_ICON)
driver.wait(2)
if not navigation.is_on_subscreen(driver):
print("[shop_tactical] could not confirm shop opened, aborting without pressing further keys")
return
shop_utils.run_shop_tab(
driver, config,
tab_button=config.SHOP_TAB_TACTICAL,
targets=config.TACTICAL_SHOP_TARGETS,
balance_rect=config.TACTICAL_COIN_OCR_RECT,
currency_label="tactical coin",
)
driver.click(*config.SHOP_BACK_BUTTON)
driver.wait(1.5)
print("[shop_tactical] Done.")

127
ba_auto/tasks/shop_utils.py Normal file
View File

@ -0,0 +1,127 @@
"""Shared Common Shop / Tactical Shop grid logic.
Ported from baas-reference module/shop/shop_utils.py's get_item_position/
ensure_choose/buy pattern -- the checkbox-select-then-bulk-buy grid shared by
both shop tabs in the live client. Adapted from the reference's
color+template item-state scan to this project's fixed-grid-position +
price-OCR-verify design: unlike story_sweep, the reference's own
item-identification here isn't OCR at all -- it indexes into
self.static_config.common_shop_price_list, a table sourced from an external
resource this repo doesn't have. So a locally pixel-scanned, name-commented
position table (config.COMMON_SHOP_TARGETS / config.TACTICAL_SHOP_TARGETS) is
the faithful local equivalent, not a shortcut around OCR the reference uses.
Price-digit OCR -- something the reference doesn't even do per-item -- is
layered on top as an extra safety net against catalog drift, consistent with
this project's verify-before-spend pattern elsewhere (mailbox/cafe/story_sweep).
"""
from ba_auto import detector, navigation
def _checkbox_center(config, row, col):
return config.SHOP_ITEM_COL_X[col], config.SHOP_ITEM_ROW_Y[row]
def _checkbox_region(config, row, col):
x, y = _checkbox_center(config, row, col)
return (x - 15, y - 15, x + 15, y + 15)
def _price_rect(config, row, col):
x, y = _checkbox_center(config, row, col)
x1, y1, x2, y2 = config.SHOP_PRICE_OCR_OFFSET
return (x + x1, y + y1, x + x2, y + y2)
def _is_checked(driver, config, row, col):
lo, hi = config.SHOP_CHECKED_RGB
return detector.region_contains_color(_checkbox_region(config, row, col), lo, hi)
def select_targets(driver, config, targets):
"""OCR-verify each target's price, then click its checkbox and confirm
it registers as checked. Returns (selected, skipped); a price mismatch
or an unconfirmed checkbox skips just that target rather than aborting
the whole run (mirrors story_sweep's per-target skip-not-abort design).
"""
selected = []
skipped = []
for row, col, name, expected_price in targets:
price = detector.read_int(_price_rect(config, row, col))
if price != expected_price:
print(f"[shop] '{name}' price read as {price}, expected {expected_price} -- skipping (catalog may have changed)")
skipped.append((name, "price_mismatch"))
continue
x, y = _checkbox_center(config, row, col)
driver.click(x, y)
driver.wait(0.3)
if not _is_checked(driver, config, row, col):
print(f"[shop] '{name}' checkbox did not register as checked -- skipping")
skipped.append((name, "checkbox_not_confirmed"))
continue
selected.append((name, expected_price))
return selected, skipped
def confirm_purchase(driver, config):
"""Click the bulk Buy button, then click through the purchase-confirm
dialog and the reward-acquired banner. Both dim
config.SHOP_OVERLAY_PROBE away from pure white; press Enter until it
reads idle again, bounded by config.SHOP_PURCHASE_MAX_ENTER_PRESSES --
this project doesn't press keys past a bound against an unrecognized
state (see navigation.wait_for_state). Returns True once idle, False if
it never clears within the bound.
"""
driver.click(*config.SHOP_BUY_BUTTON)
driver.wait(1.5)
min_ch = config.SHOP_OVERLAY_IDLE_MIN_CHANNEL
for _ in range(config.SHOP_PURCHASE_MAX_ENTER_PRESSES):
r, g, b = driver.color_at(*config.SHOP_OVERLAY_PROBE)
if r > min_ch and g > min_ch and b > min_ch:
return True
driver.keypress("Return")
driver.wait(1.5)
return False
def run_shop_tab(driver, config, *, tab_button, targets, balance_rect, currency_label):
"""Shared control flow for both shop tabs: open the tab, verify enough
currency for everything configured, select+buy, report the outcome.
Ported from the reference's common_shop.py/tactical_challenge_shop.py
implement() -- both are this same shape (read assets, calculate cost,
buy, verify), differing only in tab entry point and currency.
"""
if not targets:
print(f"[shop] no targets configured for {currency_label}, nothing to do")
return "no_targets"
driver.click(*tab_button)
driver.wait(1.5)
if not navigation.is_on_subscreen(driver):
print("[shop] could not confirm shop tab opened, aborting without pressing further keys")
return "navigation_failed"
balance = detector.read_int(balance_rect)
if balance is None:
print(f"[shop] could not read {currency_label} balance, aborting without spending")
return "balance_unreadable"
total_cost = sum(price for _, _, _, price in targets)
if balance < total_cost:
print(f"[shop] insufficient {currency_label}: have {balance}, need {total_cost} -- stopping, nothing bought")
return "inadequate_assets"
selected, skipped = select_targets(driver, config, targets)
if not selected:
print("[shop] nothing selected (all targets skipped), cancelling")
return "nothing_selected"
if not confirm_purchase(driver, config):
print("[shop] purchase confirmation did not resolve to an idle screen within the retry bound -- stopping, check the game manually")
return "unrecognized_state"
spent = sum(price for _, price in selected)
msg = f"[shop] bought {len(selected)} item(s) for {spent} {currency_label}"
if skipped:
msg += f", skipped {len(skipped)}"
print(msg)
return "purchased"

59
ba_auto/tasks/stamina.py Normal file
View File

@ -0,0 +1,59 @@
"""Mission/task-menu AP+pyroxene claim. Ported from baas-reference module/collect_daily_task_power.py's claim-loop pattern (its rgb_in_range checks replaced with driver.color_at probes)."""
from ba_auto import navigation
OPEN_RETRIES = 3
# One claim-all round is normally enough (button goes grey right after), but
# bound it in case multiple reward reveals need dismissing in sequence.
CLAIM_MAX_ROUNDS = 5
# Yellow "one-click claim all" button vs. its own flat-grey disabled state --
# same background-pixel-probe idea as mailbox.CLAIM_ALL_PROBE/cafe.CLAIM_PROBE,
# but distinguished by hue (yellow has a big r-b gap; grey doesn't) since the
# button's grey isn't a single fixed RGB the way mailbox/cafe's are.
CLAIM_ENABLED_MIN_R = 230
CLAIM_ENABLED_MIN_RB_GAP = 100
def _claim_enabled(driver, config):
r, g, b = driver.color_at(*config.MISSION_CLAIM_PROBE)
return r > CLAIM_ENABLED_MIN_R and (r - b) > CLAIM_ENABLED_MIN_RB_GAP
def run(driver, config):
driver.focus_game()
opened = False
for attempt in range(1, OPEN_RETRIES + 1):
driver.focus_game()
driver.click(*config.MISSION_ICON)
driver.wait(2)
if navigation.is_on_subscreen(driver):
opened = True
break
print(f"[stamina] mission panel not detected after click (attempt {attempt}/{OPEN_RETRIES})")
if not opened:
print("[stamina] could not confirm mission panel is open, aborting without pressing further keys")
return
claimed_any = False
for _ in range(CLAIM_MAX_ROUNDS):
if not _claim_enabled(driver, config):
break
print("[stamina] claiming mission rewards")
driver.keypress("Return")
driver.wait(1.5)
# dismiss the reward-reveal popup ("TOUCH" prompt); harmless no-op if
# nothing is actually showing, same assumption as cafe's room-entry dismiss
driver.keypress("Return")
driver.wait(1.5)
claimed_any = True
if not claimed_any:
print("[stamina] nothing to claim")
if navigation.is_on_subscreen(driver):
driver.keypress("Escape")
driver.wait(1.5)
print("[stamina] Done.")

View File

@ -0,0 +1,379 @@
"""Normal story AP sweep. Reference: baas-reference/module/explore_tasks/sweep_task.py
and task_utils.py.
Ported per CLAUDE.md's "OCR policy" and Handoff.md: sweeps a config-driven
list of exact (region, stage, count) targets (config.STORY_SWEEP_TARGETS),
navigating to each one deterministically instead of the previous "latest
unlocked region, then a random stage" heuristic -- see plan.md Phase 9's
retrospective for why that heuristic was a mistake.
`_rotation_target` adds a second, date-derived target on top of that static
list, per explicit user direction: rather than grinding one fixed stage in
their current last region every run, it cycles through that region's stages
one per day (a plain date-ordinal modulo, so the cycle doesn't skip or
repeat around a year boundary). See config.py's STORY_SWEEP_ROTATION_*
constants.
- `_go_to_region` ports task_utils.py::to_region: OCR the current region
number, click the exact delta, re-check, bounded loop.
- `_find_stage_row` is a scoped-down port of core/image.py's
swipe_search_target_str: OCR each visible stage row's label and match it
against the configured target, rather than grabbing a random row.
- `_watch_sweep_result` is built on navigation.wait_for_state, this
project's scoped port of core/picture.py::co_detect, and returns a named
outcome ("swept", "inadequate_ap", "unrecognized_state", ...) the way the
reference's start_sweep does, instead of a single generic "Done".
The MAX-button click-then-verify and the modal's own X-button close (both
calibrated and confirmed live in Phase 9) are reused unchanged -- see
plan.md Phase 9/10.
"""
import datetime
from ba_auto import detector, navigation
OPEN_RETRIES = 3
POST_SWEEP_DISMISS_ROUNDS = 6
STAGE_MODAL_DIM_MAX_CHANNEL = 150
MAX_BUTTON_RETRIES = 3
MODAL_CLOSE_RETRIES = 3
def _is_stage_modal_open(driver, config):
r, g, b = driver.color_at(*config.STAGE_MODAL_PROBE)
return r < STAGE_MODAL_DIM_MAX_CHANNEL and g < STAGE_MODAL_DIM_MAX_CHANNEL and b < STAGE_MODAL_DIM_MAX_CHANNEL
def _color_in_range(rgb, rgb_range):
lo, hi = rgb_range
r, g, b = rgb
return lo[0] <= r <= hi[0] and lo[1] <= g <= hi[1] and lo[2] <= b <= hi[2]
def _is_sweep_usage_confirm(driver, config):
# 掃討開始 always raises a "use N AP to sweep M times?" confirmation
# before actually sweeping. Its OK button is this bright cyan.
return _color_in_range(driver.color_at(*config.SWEEP_CONFIRM_BUTTON), config.SWEEP_CONFIRM_CYAN)
def _is_ap_purchase_prompt(driver, config):
# If AP is too low for even one sweep, a visually similar dialog appears
# at the *same* OK-button position but titled "AP購入" (spend real
# Pyroxene to buy more AP) with a gold/yellow OK instead of cyan --
# confirmed live by deliberately emptying the sweep count at low AP.
# Told apart from the safe confirm above by color, not position.
return _color_in_range(driver.color_at(*config.SWEEP_CONFIRM_BUTTON), config.SWEEP_CONFIRM_GOLD)
def _find_result_button(driver, config):
# The "掃討完了" (sweep complete) results screen shows a SKIP button
# (skips the reward-reveal animation) and then, once settled, a final
# OK button -- both the same cyan as the usage-confirm OK, but ~120px
# apart vertically. Finding whichever is showing by color avoids
# hardcoding both positions.
return detector.find_color_centroid(config.SWEEP_RESULT_BUTTON_REGION, *config.SWEEP_CONFIRM_CYAN)
def _count_raised_above_one(driver, config):
# The "-" stepper button is flat grey while count == 1 (its default,
# disabled at the minimum) and turns vivid orange once raised -- cheap
# way to confirm a MAX/"+" click actually registered without needing OCR
# on the count itself.
r, g, b = driver.color_at(*config.SWEEP_MINUS_BUTTON_PROBE)
return r > 200 and g < 180 and b < 100
def _open_task_screen(driver, config):
for attempt in range(1, OPEN_RETRIES + 1):
driver.click(*config.WORK_ICON)
driver.wait(2)
if navigation.is_on_subscreen(driver):
break
print(f"[story_sweep] work hub not detected after click (attempt {attempt}/{OPEN_RETRIES})")
else:
return False
for attempt in range(1, OPEN_RETRIES + 1):
driver.click(*config.TASK_CARD)
driver.wait(2)
if navigation.is_on_subscreen(driver):
return True
print(f"[story_sweep] task screen not detected after click (attempt {attempt}/{OPEN_RETRIES})")
return False
def _read_current_region(driver, config):
return detector.read_int(config.REGION_NUMBER_OCR_RECT)
def _region_arrow_visible(driver, config, center):
# Both arrows render as a solid navy-blue "<"/">" chevron on a
# light-blue backdrop when present. The last region in a given direction
# (or a locked one, per the reference's own "region-unavailable"
# template check) simply omits the arrow rather than greying it out --
# confirmed live: at region 30 (this account's current last region), the
# spot where the right arrow would be was plain background.
#
# Scans a small box around `center` rather than probing a single fixed
# point: a chevron is concave, and a centroid-derived single point
# landed in the notch between its two strokes -- reading as "absent"
# even while the glyph was clearly rendered a few pixels away. See
# plan.md Phase 10.
cx, cy = center
rect = (cx - 40, cy - 35, cx + 40, cy + 35)
return detector.region_contains_color(rect, (40, 70, 120), (100, 130, 190))
def _go_to_region(driver, config, target_region):
cur = _read_current_region(driver, config)
if cur is None:
print("[story_sweep] could not OCR the current region number")
return False
print(f"[story_sweep] current region {cur}, target region {target_region}")
for attempt in range(1, config.REGION_NAV_MAX_ATTEMPTS + 1):
if cur == target_region:
return True
going_left = cur > target_region
arrow_pos = config.REGION_LEFT_ARROW if going_left else config.REGION_RIGHT_ARROW
if not _region_arrow_visible(driver, config, arrow_pos):
direction = "left" if going_left else "right"
print(f"[story_sweep] region {target_region} unreachable -- no {direction} arrow at region {cur}")
return False
clicks = abs(cur - target_region)
for _ in range(clicks):
driver.click(*arrow_pos)
driver.wait(1)
new_cur = _read_current_region(driver, config)
if new_cur is None or new_cur == cur:
print(f"[story_sweep] region number unchanged after {clicks} click(s) (attempt {attempt}/{config.REGION_NAV_MAX_ATTEMPTS})")
return False
cur = new_cur
print(f"[story_sweep] gave up navigating to region {target_region} after {config.REGION_NAV_MAX_ATTEMPTS} attempts")
return False
def _normalize_label(text):
# OCR sometimes reads the row's dash as a different dash-like glyph, or
# picks up stray whitespace -- normalize before comparing.
return text.strip().upper().replace("", "-").replace("", "-").replace(" ", "")
def _label_suffix(label):
# Compare only the part after the dash (e.g. "2" in "30-2", "A" in
# "30-A"), not the full "{region}-{stage}" string. Confirmed live: this
# font's leading region-number digit is read unreliably by OCR (e.g. "3"
# misread as "2") even after threshold preprocessing, while the stage
# suffix after the dash reads correctly across every row tested -- and we
# don't need the region digit anyway, since _go_to_region has already
# independently confirmed we're in the right region. See plan.md Phase 10.
parts = [p for p in _normalize_label(label).split("-") if p]
return parts[-1] if parts else ""
def _row_label_rect(config, row_y):
x1, x2 = config.STAGE_LABEL_OCR_X
top_pad, bottom_pad = config.STAGE_LABEL_OCR_Y_PAD
return (x1, row_y - top_pad, x2, row_y - bottom_pad)
def _find_stage_row(driver, config, region, stage):
# Scoped-down swipe_search_target_str (see module docstring): this
# client's stage list only ever needs the two already-calibrated scroll
# extremes checked, not arbitrary swipe-and-retry.
target_suffix = str(stage)
x, y = config.STAGE_LIST_SCROLL_POINT
driver.scroll(x, y, "up", config.STAGE_LIST_SCROLL_CLICKS)
driver.wait(0.5)
for row_y in config.STAGE_ROWS_AT_TOP_Y:
label = detector.read_text(_row_label_rect(config, row_y), whitelist="0123456789-A")
print(f"[story_sweep] row @ {row_y} (scrolled up): read '{label}'")
if _label_suffix(label) == target_suffix:
return row_y
driver.scroll(x, y, "down", config.STAGE_LIST_SCROLL_CLICKS)
driver.wait(0.5)
for row_y in config.STAGE_ROWS_AT_BOTTOM_Y:
label = detector.read_text(_row_label_rect(config, row_y), whitelist="0123456789-A")
print(f"[story_sweep] row @ {row_y} (scrolled down): read '{label}'")
if _label_suffix(label) == target_suffix:
return row_y
return None
def _click_max_and_verify(driver, config):
for attempt in range(1, MAX_BUTTON_RETRIES + 1):
driver.click(*config.SWEEP_MAX_BUTTON)
driver.wait(0.8)
if _count_raised_above_one(driver, config):
return True
print(f"[story_sweep] MAX click not detected (attempt {attempt}/{MAX_BUTTON_RETRIES})")
return False
def _click_plus_and_verify(driver, config, count):
for attempt in range(1, MAX_BUTTON_RETRIES + 1):
for _ in range(count - 1):
driver.click(*config.SWEEP_PLUS_BUTTON)
driver.wait(0.8)
if _count_raised_above_one(driver, config):
return True
print(f"[story_sweep] count-raise via '+' not detected (attempt {attempt}/{MAX_BUTTON_RETRIES})")
return False
def _set_sweep_count(driver, config, count):
if count == "max":
return _click_max_and_verify(driver, config)
return _click_plus_and_verify(driver, config, count)
def _close_stage_modal(driver, config):
# Escape does not close this modal (confirmed live: it stayed open with
# focus on the live "任務開始" button after two Escape presses) -- the
# only reliable close path is clicking its own X button, verified.
for _ in range(MODAL_CLOSE_RETRIES):
if not _is_stage_modal_open(driver, config):
return True
driver.click(*config.STAGE_MODAL_CLOSE_BUTTON)
driver.wait(1)
return not _is_stage_modal_open(driver, config)
def _watch_sweep_result(driver, config):
# The reference's start_sweep returns one of "inadequate_ap",
# "charge_challenge_counts", or "sweep_complete" so its caller reacts
# appropriately -- this ports that same named-outcome contract via
# navigation.wait_for_state instead of the old single generic "Done".
#
# Called only after the usage-confirm dialog is already accepted (see
# _sweep_target), so from here it's purely "click through the
# 掃討完了 SKIP/OK screens until the bare stage-info modal reappears."
# Clicking the found button by color (not a keypress) means this never
# risks landing on the underlying "任務開始" button the way a blind
# Enter-press loop would.
def click_result_button(d):
pos = _find_result_button(d, config)
if pos:
d.click(*pos)
d.wait(1.5)
ends = {
(lambda d, c: _is_stage_modal_open(d, c) and _find_result_button(d, c) is None): "swept",
}
reactions = {
(lambda d, c: _find_result_button(d, c) is not None): click_result_button,
}
outcome = navigation.wait_for_state(
driver, config, reactions, ends,
max_iterations=POST_SWEEP_DISMISS_ROUNDS, poll_interval=1.5,
)
return outcome or "unrecognized_state"
def _sweep_target(driver, config, region, stage, count):
print(f"[story_sweep] --- target {region}-{stage} x {count} ---")
if not _go_to_region(driver, config, region):
return "region_unavailable"
row_y = _find_stage_row(driver, config, region, stage)
if row_y is None:
print(f"[story_sweep] stage {region}-{stage} not found in the visible stage list")
return "stage_not_found"
driver.click(config.STAGE_ENTER_X, row_y)
driver.wait(2)
if not _is_stage_modal_open(driver, config):
print("[story_sweep] stage info panel not detected, aborting")
if navigation.is_on_subscreen(driver):
driver.keypress("Escape")
driver.wait(1.5)
return "unrecognized_state"
if not _set_sweep_count(driver, config, count):
print("[story_sweep] could not confirm sweep count was raised, aborting without spending AP")
_close_stage_modal(driver, config)
if navigation.is_on_subscreen(driver):
driver.keypress("Escape")
driver.wait(1.5)
return "unrecognized_state"
driver.click(*config.SWEEP_START_BUTTON)
driver.wait(1.5)
if _is_ap_purchase_prompt(driver, config):
print("[story_sweep] insufficient AP for this sweep -- cancelling without purchasing")
driver.click(*config.SWEEP_CONFIRM_CANCEL_BUTTON)
driver.wait(1)
_close_stage_modal(driver, config)
if navigation.is_on_subscreen(driver):
driver.keypress("Escape")
driver.wait(1.5)
return "inadequate_ap"
if not _is_sweep_usage_confirm(driver, config):
print("[story_sweep] sweep-usage confirmation not detected, aborting without further input")
_close_stage_modal(driver, config)
if navigation.is_on_subscreen(driver):
driver.keypress("Escape")
driver.wait(1.5)
return "unrecognized_state"
driver.click(*config.SWEEP_CONFIRM_BUTTON)
driver.wait(1.5)
print("[story_sweep] sweep confirmed, waiting for results")
outcome = _watch_sweep_result(driver, config)
print(f"[story_sweep] result: {outcome}")
if not _close_stage_modal(driver, config):
print("[story_sweep] warning: could not confirm stage info modal closed -- leaving it open rather than pressing further keys blindly")
return outcome
if navigation.is_on_subscreen(driver):
driver.keypress("Escape")
driver.wait(1.5)
return outcome
def _rotation_target(config):
region = getattr(config, "STORY_SWEEP_ROTATION_REGION", None)
if not region:
return None
stage_count = config.STORY_SWEEP_ROTATION_STAGE_COUNT
stage = (datetime.date.today().toordinal() % stage_count) + 1
return (region, stage, config.STORY_SWEEP_ROTATION_COUNT)
def run(driver, config):
driver.focus_game()
targets = list(config.STORY_SWEEP_TARGETS)
rotation = _rotation_target(config)
if rotation:
print(f"[story_sweep] today's rotation target: region {rotation[0]} stage {rotation[1]}")
targets.append(rotation)
if not targets:
print("[story_sweep] no targets configured (config.STORY_SWEEP_TARGETS is empty and rotation is disabled), nothing to do")
return
if not _open_task_screen(driver, config):
print("[story_sweep] could not confirm task screen is open, aborting without pressing further keys")
return
for region, stage, count in targets:
outcome = _sweep_target(driver, config, region, stage, count)
if outcome == "inadequate_ap":
print("[story_sweep] insufficient AP -- stopping, not attempting remaining targets")
break
if outcome != "swept":
print(f"[story_sweep] target {region}-{stage} ended in '{outcome}' -- skipping to next target")
print("[story_sweep] Done.")

View File

@ -3,13 +3,22 @@
import sys
from ba_auto import config, driver
from ba_auto.tasks import cafe, mailbox
from ba_auto.tasks import cafe, lesson, mailbox, shop_common, shop_tactical, stamina, story_sweep
TASKS = {
"mailbox": mailbox.run,
"cafe": cafe.run,
"stamina": stamina.run,
"story_sweep": story_sweep.run,
"shop_common": shop_common.run,
"shop_tactical": shop_tactical.run,
"lesson": lesson.run,
}
DEFAULT_ORDER = ["mailbox", "cafe"]
# story_sweep, both shop tasks, and lesson are opt-in only (not in the
# default flow): they spend AP/credits/tactical coin/lesson tickets on an
# automated choice rather than reclaiming something free, which is a real
# resource decision the default unattended run shouldn't make blindly.
DEFAULT_ORDER = ["mailbox", "cafe", "stamina"]
def main(argv):

File diff suppressed because one or more lines are too long

View File

@ -348,7 +348,7 @@
"norm_label": "screenshots/ reference-capture directory"
},
{
"label": "./scratchpad working-file convention",
"label": "scratchpad/ working-file convention",
"file_type": "concept",
"source_file": "CLAUDE.md",
"source_location": "L38",
@ -358,7 +358,7 @@
"contributor": null,
"id": "claude_scratchpad_dir",
"community": 0,
"norm_label": "./scratchpad working-file convention"
"norm_label": "scratchpad/ working-file convention"
},
{
"label": "~/.venvs/ba-auto-daily Python venv (opencv-python, numpy)",

308
plan.md
View File

@ -8,7 +8,7 @@ This project controls the PC/Steam/Proton Blue Archive client running on `nik-gp
- scrot
- Python
- OpenCV
- later OCR when needed
- OCR, ported wherever the reference implementation uses it for a feature (see `CLAUDE.md` → "OCR policy" — this is no longer a "later, when needed" deferral)
Development happens on `nik-macbookair`.
@ -76,13 +76,12 @@ This is the intended direction. It does not have to be completed all at once.
| `~/repo/ba-auto-daily/ba_dailies.sh` | Thin launcher only. It should call the Python entry point. Do not add new feature logic here. |
| `~/repo/ba-auto-daily/ba_daily.py` | Main Python CLI entry point. Dispatches tasks such as mailbox, cafe, stamina, group, etc. |
| `~/repo/ba-auto-daily/ba_auto/driver.py` | Local PC/Steam/Proton control backend. Wraps xdotool, scrot, waits, clicks, swipes, keypresses, screenshots, and window focus. |
| `~/repo/ba-auto-daily/ba_auto/detector.py` | OpenCV/template/color matching helpers. Existing `scripts/detect_and_click.py` logic should be migrated here. |
| `~/repo/ba-auto-daily/ba_auto/navigation.py` | Shared navigation helpers: home, menu, close popup, back, open feature screens. |
| `~/repo/ba-auto-daily/ba_auto/detector.py` | OpenCV/template/color matching helpers. Currently has `find_cafe_sparkle()`, ported in-process from the retired `scripts/detect_and_click.py`. |
| `~/repo/ba-auto-daily/ba_auto/navigation.py` | Shared navigation/state-probe helpers: `is_on_subscreen`, `is_modal_open`, used by both `mailbox.py` and `cafe.py`. |
| `~/repo/ba-auto-daily/ba_auto/tasks/` | Feature implementations. Each task should adapt the relevant `baas-reference/module/...` logic where possible. |
| `~/repo/ba-auto-daily/ba_auto/reference_notes/mapping.md` | Reference mapping table: local feature → reference module → local implementation → driver gaps. |
| `~/repo/ba-auto-daily/assets/` | Locally captured template images, such as cafe sparkle. Do not blindly copy assets from the reference repo. |
| `~/repo/ba-auto-daily/screenshots/` | Human reference screenshots, mostly Moonlight/game captures, used for calibration and debugging. |
| `~/repo/ba-auto-daily/scripts/` | Transitional scripts. Long-term reusable Python logic should move into `ba_auto/`. |
| `~/repo/ba-auto-daily/setup.sh` | Bootstrap/deploy helper for `nik-gpu`. Should install/check dependencies and copy runtime files. |
| `~/repo/baas-reference/` | Read-only GPL-3.0 reference clone. Study and adapt. Never edit. |
@ -98,11 +97,7 @@ Preferred runtime layout:
| `nik-gpu:~/ba_assets/` | Runtime assets/templates. |
| `nik-gpu:~/.venvs/ba-auto-daily/` | Python virtual environment. |
Current older layout may include:
| Path | What it is |
|---|---|
| `nik-gpu:~/ba_scripts/detect_and_click.py` | Old standalone detector helper. Should eventually be replaced by `ba_auto/detector.py`. |
`nik-gpu:~/ba_scripts/` may still contain `detect_and_click.py` and `ba_dailies_legacy.sh` left over from before both mailbox and cafe were migrated off them. Neither is deployed or referenced by anything anymore (`setup.sh` stopped copying them once Phase 6 landed) — safe to delete manually on `nik-gpu`, just not automated here.
## Implementation strategy
@ -140,8 +135,9 @@ Initial seed:
| Local feature | Reference file | Reference functions/classes | Local file | Backend replacements | Status |
|---|---|---|---|---|---|
| Mailbox | Need to confirm in reference | Need to inspect | `ba_auto/tasks/mailbox.py` | tap/click via xdotool, screenshot via scrot | Existing Bash behavior; migrate to Python |
| Cafe | Need to confirm in reference | Need to inspect | `ba_auto/tasks/cafe.py` | template matching via OpenCV, click via xdotool | Existing Bash + helper behavior; migrate to Python |
| Stamina/AP | `module/collect_daily_free_power.py`, `module/collect_daily_task_power.py` | Need to inspect | `ba_auto/tasks/stamina.py` | color checks/clicks via local driver | Not started |
| Cafe | `module/cafe_reward.py` | `to_cafe`, `interaction_for_cafe_solve_method3`, `collect` | `ba_auto/tasks/cafe.py` | `picture.co_detect`/`color.rgb_in_range``driver.color_at` pixel-probe checks; sparkle template match ported in-process into `ba_auto/detector.py` | Migrated: real Python, state-verified via color probes, no legacy bridge |
| Stamina/AP | `module/collect_daily_free_power.py`, `module/collect_daily_task_power.py` | `to_tasks`/`implement` (task-power, ported); `to_purchase_pyroxenes_menu`/`detect_free_power_availability` (free-power, not ported — real-money purchase menu) | `ba_auto/tasks/stamina.py` | `color.rgb_in_range``driver.color_at`; reference's per-tab claim loop replaced by the live UI's single "一括受取" (claim-all) button, triggered via Enter | Partially migrated (Phase 8): Mission-panel task/weekly/achievement claim done. Daily Free Power deliberately not implemented. |
| Normal/Hard story AP sweep | `module/explore_tasks/sweep_task.py`, `module/explore_tasks/task_utils.py` | `to_region`/`to_normal_event` + OCR-driven per-stage claim loop (ported, Phase 10) | `ba_auto/tasks/story_sweep.py` | OCR-based region/stage-name matching, ported for real (Phase 10); reference's per-stage claim loop → this client's stage-info modal's self-contained 掃討 (sweep) sub-panel (MIN/-/+/MAX stepper + start button) | Done (Phase 10, supersedes Phase 9's random-pick design) |
| Group/Club AP | `module/group.py` | Need to inspect | `ba_auto/tasks/group.py` | fixed click + state check via local driver | Not started |
| Bounty | `module/rewarded_task.py` | Need to inspect | `ba_auto/tasks/bounty.py` | sweep/color/OCR adaptation | Not started |
| Commissions | `module/clear_special_task_power.py` | Need to inspect | `ba_auto/tasks/commission.py` | sweep/color adaptation | Not started |
@ -157,9 +153,13 @@ Do not implement a feature without filling at least the relevant row.
| Feature | Current status | Target status |
|---|---|---|
| Mailbox claim | Migrated: `ba_auto/tasks/mailbox.py` uses `driver.color_at` to verify the panel opened before acting (found live-testing bug: a marginal icon coordinate could miss and cascade into pressing Escape on the home screen, which triggers Blue Archive's own exit-game confirmation) | Done |
| Cafe pats + income | Bridged: `ba_auto/tasks/cafe.py` calls `scripts/ba_dailies_legacy.sh cafe` (which still shells out to `scripts/detect_and_click.py`) | Migrate to `ba_auto/tasks/cafe.py` and `ba_auto/detector.py` (Phase 6) |
| Shared driver | `ba_auto/driver.py` built (`run_command`, `focus_game`, `click`, `keypress`, `screenshot`, `wait`); not yet wired into tasks | Wire into mailbox/cafe as they migrate off the Bash bridge |
| Python CLI | Built: `ba_daily.py` dispatches `mailbox`/`cafe`/default flow | Extend as new tasks are added |
| Cafe pats + income | Migrated: `ba_auto/tasks/cafe.py` verifies each room/dialog opened via `driver.color_at` before acting; sparkle detection now runs in-process via `ba_auto/detector.py` instead of a per-click subprocess | Done |
| Stamina/AP (mission claim) | Migrated (partial, Phase 8): `ba_auto/tasks/stamina.py` claims the Mission panel's bulk "一括受取" button. Daily Free Power (real-money purchase menu) intentionally not implemented | Daily Free Power still not started |
| Normal/Hard story AP sweep | Done (Phase 10, supersedes Phase 9's random-pick design): `ba_auto/tasks/story_sweep.py` sweeps a config-driven list of exact `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS`), navigating to each via OCR (region-number read + delta-click, stage-label OCR match) instead of "latest region, random stage." Opt-in only (`story_sweep` command), not part of the default daily flow | Done |
| Common Shop / Tactical Shop | Done (Phase 11): `ba_auto/tasks/shop_common.py` / `shop_tactical.py` share a checkbox-grid-then-bulk-buy flow (`ba_auto/tasks/shop_utils.py`) against config-driven `(row, col, name, expected_price)` targets, price-OCR-verified before each click. Live-tested with real purchases in both shops. Opt-in only (`shop_common`/`shop_tactical` commands), not part of the default daily flow | Done |
| Lesson/Schedule | Done (Phase 12): `ba_auto/tasks/lesson.py` sweeps every unlocked region's schedule grid, picking the highest-affection available lesson each time via a dedicated heart-badge OCR read (`detector.read_int_on_heart_badge`) until tickets or lessons run out. Live-tested with real tickets spent; two real bugs (checkmark-doesn't-blank-the-number, badge OCR misreads) found and fixed. Opt-in only (`lesson` command), not part of the default daily flow | Done |
| Shared driver | `ba_auto/driver.py` built (`run_command`, `focus_game`, `click`, `move_mouse`, `scroll`, `keypress`, `screenshot`, `wait`, `color_at`); `click()` now splits `mousemove`/`click` into two xdotool calls (Phase 8 finding — fixes a real source of click flakiness); wired into `mailbox.py`, `cafe.py`, `stamina.py`, `story_sweep.py`, `shop_common.py`, `shop_tactical.py`, `lesson.py` | Extend with new primitives as future tasks need them |
| Python CLI | Built: `ba_daily.py` dispatches `mailbox`/`cafe`/`stamina`/`story_sweep`/`shop_common`/`shop_tactical`/`lesson`/default flow | Extend as new tasks are added |
| Reference mapping | Built: `ba_auto/reference_notes/mapping.md` | Fill in reference file/function columns per feature |
| Everything else | Not started | Implement reference-first in Python |
@ -210,7 +210,7 @@ Keep compatibility with:
### Phase 3: Driver extraction
**Status: Primitives added to `ba_auto/driver.py`, not yet wired into task modules.**
**Status: Done.** Primitives (including `color_at`, added during the mailbox/cafe hardening work) are wired into both `ba_auto/tasks/mailbox.py` and `ba_auto/tasks/cafe.py`.
Move shell interactions into `ba_auto/driver.py`.
@ -229,11 +229,9 @@ wait_until(...)
### Phase 4: Detector extraction
**Status: Not started — `ba_auto/detector.py` is currently a placeholder.**
**Status: Done (scoped).** `scripts/detect_and_click.py`'s sparkle-matching logic (masked template match against `assets/cafe_sparkle.png`) was ported into `ba_auto/detector.py` as `find_cafe_sparkle()`, called in-process from `ba_auto/tasks/cafe.py` — this removed the old per-click Python cold start (a fresh `cv2`/`numpy` import per subprocess call) that CLAUDE.md's driver-layer guidance specifically warns against. `scripts/detect_and_click.py` had no remaining callers once this landed, so it was deleted rather than kept as a compatibility wrapper. The more generic primitives listed below (`load_template`, `match_template`, etc.) have not been built — only the one concrete sparkle-matching function needed so far exists; generalize when a second detector use case actually needs it.
Move `scripts/detect_and_click.py` logic into `ba_auto/detector.py`.
Detector primitives should include:
Detector primitives, generalize later if needed:
```
load_template(...)
@ -244,8 +242,6 @@ color_mask(...)
debug_write_match(...)
```
The old script may remain as a compatibility wrapper temporarily, but the reusable logic should live under `ba_auto/`.
### Phase 5: Mailbox migration
**Status: Done.** Live testing surfaced a real bug: the old `MAILBOX_ICON` coordinate `(1726, 60)` sat on the edge of the icon's hitbox and intermittently missed, and the fixed click sequence had no way to notice — it cascaded into pressing Escape on the bare home screen, which triggers Blue Archive's own "exit the game?" confirmation (dismissed safely with Cancel during testing; no game state was lost). The Python port in `ba_auto/tasks/mailbox.py` fixes the coordinate and, following `module/mail.py`'s `rgb_in_range` pattern, verifies the panel actually opened (and whether "claim all" is disabled) via `driver.color_at` before pressing any further keys, with a bounded retry and a safe abort if the panel never appears.
@ -260,27 +256,54 @@ The CLI should call it through Python.
### Phase 6: Cafe migration
**Status: Not started — `ba_auto/tasks/cafe.py` currently bridges to `scripts/ba_dailies_legacy.sh`.**
**Status: Done.** Same root cause as the mailbox bug (Phase 5), confirmed by step-by-step live replay with screenshots: `CAFE_ICON` clicks are flaky (missed on the first attempt, worked on retry at the identical coordinate — this is xdotool/Proton click-registration flakiness, not a coordinate-precision problem), and the old sequence had zero verification across its ~12 steps (open → dismiss notice → pat loop ×15 → switch room → dismiss notice → pat loop ×15 → claim income ×2 Enter → ×2 Escape). A missed click anywhere cascades into blind actions on whatever screen is actually showing, which — same as mailbox — very likely ends with an unverified Escape hitting the home screen and triggering Blue Archive's own exit-game confirmation.
Move cafe logic from Bash to:
`ba_auto/tasks/cafe.py` now verifies state at every transition using `driver.color_at`, following `module/cafe_reward.py`'s `picture.co_detect`/`rgb_in_range` pattern:
```
ba_auto/tasks/cafe.py
```
- opening the cafe icon and the room-switch button both retry (bounded) and confirm the panel actually opened via the same subscreen-header probe as mailbox, now shared in `ba_auto/navigation.is_on_subscreen`
- the "visited student list" notice that appears on every room entry is dismissed with Enter; this is harmless as a no-op if no popup is actually present (verified live), so no separate presence check was needed there
- the income dialog's own dimmed-overlay backdrop is checked (`navigation.is_modal_open`) before pressing Enter to claim, and the "receive" button's disabled-grey color is checked before attempting to claim at all (mirrors `collect()`'s `rgb_in_range` gate in the reference)
- the closing Escape(s) only fire when a subscreen/modal is confirmed still open, never blindly
During migration, verify:
Verified live (two full runs against the real game, plus a manual step-by-step replay of every transition):
- both rooms still work
- sparkle detection still works
- cafe income claim still works
- rank-up popups are handled or explicitly documented as not handled
- student rotation popups are handled or explicitly documented as not handled
- view centering/zoom state is robust
- repeated detection does not suffer from Python cold-start delay
- both rooms open and pat correctly
- sparkle detection still works and now runs in-process (see Phase 4) instead of shelling out per click
- cafe income claim works (confirmed gold +81,251 / AP +61 on an actual claim) and correctly no-ops when there's nothing to collect
- the reference's `zoom_out` step (camera zoom before sparkle detection — CLAUDE.md's "view centering/zoom" gap) was **not** ported: detection matched at 0.99 confidence without it in live testing, so it wasn't reproducibly broken here. Left as a documented open risk below rather than added speculatively.
Not verified / open risks:
- whether zoom/pan state could drift over a long unattended run and eventually break sparkle detection (see above — no evidence of this yet, but the reference project treats it as necessary)
#### Phase 6 follow-up: rank-up popups mid-pat-loop ("it will freeze a bit")
A user report during real usage: a pat that causes a bond-rank-up makes the loop "freeze a bit." Confirmed as a real, previously-unhandled gap, not a timing fluke — `find_cafe_sparkle()` was being asked to recognize a full-screen "絆ランクアップ!" cutscene (no cafe header, no chrome at all — see `screenshots/cafe/student/01`/`02`) as if it were the sparkle template, which it obviously never matches, so the loop just spun `driver.wait(1)` uselessly for the rest of the room's click budget. The reference's own `to_cafe()` navigation (`module/cafe_reward.py`) already treats `relationship_rank_up` as a recognized, reactively-dismissed popup checked after every pat round — this project's port had never carried that over.
Fix: `cafe.py`'s `_dismiss_rank_up_if_shown()`, called after every pat (click + Enter + move-mouse), reuses the *existing* `navigation.is_on_subscreen` header-brightness probe rather than adding a new one — directly confirmed against the user-provided screenshots: the header probe point reads `(183, 220, 240)` during the cutscene (r<200, fails the check) vs. `(248, 249, 250)` on the normal cafe screen (r>200, passes). Presses Enter (bounded, `config.CAFE_RANK_UP_DISMISS_RETRIES = 5`) until `is_on_subscreen` confirms the cafe room is back, rather than assuming one Enter is enough; if it never clears, the pat loop stops rather than continuing to click blindly.
Not yet live-confirmed against a real rank-up trigger — it's semi-random (tied to hitting an affection threshold) and didn't happen to occur during this session's testing. The fix is grounded in the user's own captured screenshots (a real observed state, precisely measured), not a guess, but a live run actually hitting this path and recovering cleanly is still open.
#### Phase 6 follow-up: "farming affection doesn't happen" report
A later report claimed pats weren't landing at all, with the original `ba_dailies.sh` `do_cafe_room`/`do_cafe` pasted as the expected-behavior reference. Re-reading that Bash carefully changed the diagnosis: the original `detect_and_click.py` did one screenshot → detect → click per invocation and the Bash loop only kept calling it back-to-back while hits kept landing, breaking immediately on the first miss (`grep -q "^MATCH" || break`) — i.e. give-up-on-first-miss was the *original design*, not a regression introduced by the Python port. Detection math (mask, threshold `0.97`, click offset `(75, 47)`) ported over byte-for-byte identical.
Changes made this round:
- `ba_auto/detector.py`: `find_cafe_sparkle()` now tries multiple template scales (`SPARKLE_SCALES`) instead of one fixed size, since the cafe camera's zoom isn't reset before farming and isn't guaranteed to match whatever zoom the template was captured at. Strictly more permissive than the original single-scale match — no observed downside — but not confirmed as the actual root cause of the report (no live zoom-mismatch case was reproduced/observed).
- `ba_auto/tasks/cafe.py`: `_pat_room` now polls for the full `CAFE_MAX_CLICKS_PER_ROOM` budget with a 1s wait between misses instead of breaking on the very first miss. This is a deliberate deviation from the original design (see above) — cheap (adds at most ~15s per room when nothing is available) and covers the case where a screenshot lands mid-animation right after the room transition.
- `driver.move_mouse` added and called after each pat to park the cursor away from the sparkle area, per `screenshots/cafe/sparkle/02_*_cursor_on_head.png` showing the cursor can occlude the icon.
What was directly verified live after these changes:
- the room-entry/no-modal state probes (`navigation.is_on_subscreen`, `is_modal_open`) read correctly on real captured frames from both rooms
- neither room had a visible sparkle on any student at the time of testing (confirmed by eye on the actual screenshots, not inferred from the "no sparkle found" log) — this is the most likely explanation for why a same-session automated run kept reporting no matches: this session's own manual+automated testing had already consumed the available per-student affection interactions, which regenerate on a real-world cooldown far longer than one room visit
**Not yet verified**: an actual end-to-end pat (detect → click → affection-up dialog dismissed) succeeding after this round's changes, because no interactable sparkle was available during testing to exercise it against. Re-run `~/ba_dailies.sh cafe` once interactions have had time to regenerate and confirm `[cafe] patted N sparkle(s)` appears with N > 0.
### Phase 7: setup.sh update
**Status: Done — `setup.sh` now deploys `ba_daily.py`, `ba_auto/`, and `scripts/ba_dailies_legacy.sh`.**
**Status: Done — `setup.sh` deploys `ba_daily.py` and `ba_auto/`.** `scripts/ba_dailies_legacy.sh` and `scripts/detect_and_click.py` were deleted once mailbox and cafe both migrated off them (Phases 56); `setup.sh` no longer references either.
Update `setup.sh` so it deploys:
@ -293,11 +316,103 @@ assets/
to the expected runtime paths on `nik-gpu`.
### Phase 8: Stamina/AP mission claim
**Status: Partially done.** Read `module/collect_daily_task_power.py` (the "Tasks" menu claim loop — `to_tasks` + `rgb_in_range` checks against two fixed pixel pairs, click, dismiss, repeat) and `module/collect_daily_free_power.py` (a `picture.co_detect` state-machine walk into the Pyroxene Purchase menu's Package tab to claim a genuinely free 10 AP item). Live reconnaissance on the home screen found the direct local equivalents: a `ミッション` (Mission) icon opening a panel with per-tab claim buttons *and* a single bulk "一括受取" (claim all) button whose keyboard shortcut is literally Enter — much simpler than porting the reference's per-tab color-probe loop. `ba_auto/tasks/stamina.py` opens the Mission panel, checks whether "一括受取" is enabled (bright yellow vs. flat grey background probe at `config.MISSION_CLAIM_PROBE`), and if so presses Enter to claim, Enter again to dismiss the reward-reveal card (same "harmless no-op if absent" assumption as cafe's room-entry dismiss), bounded to a few rounds in case multiple rewards queue up.
The 青輝石購入 (Pyroxene Purchase) icon — reference's Daily Free Power entry point — was opened once to confirm the free-claim flow's location, but turned out to be a real-money purchase menu (¥3,000¥4,900 package buttons visible immediately) with the genuinely-free item buried in a further tab. Given `plan.md`'s own purchase-safety rules ("avoid unbounded spending", "avoid buying unknown items"), this was **deliberately not automated this round** — the dialog was closed without navigating further. Treat this as a separate, explicitly-confirmed piece of future work, not an oversight.
Two real bugs found and fixed during live calibration, both worth remembering for future coordinate-hunting:
- **Visual gridline coordinate estimates were wrong twice in a row.** Reading icon bounds off a scaled/annotated screenshot crop by eye put the Mission icon's center at `(146, 352)` — which is actually in the dead space between the Mission and Pyroxene-Purchase icons, close enough to the latter's edge that clicks there landed on Pyroxene Purchase instead. The fix was sampling actual pixel colors along a scanline (`img.getpixel`) to find each icon's true left/right edge against the background, rather than eyeballing gridlines — this put the real center at `(75, 350)`, squarely inside the icon graphic, confirmed live. Lesson: for icon coordinates, prefer a pixel-boundary scan over a visual grid-overlay estimate.
- **`driver.click()`'s combined `xdotool mousemove X Y click 1` invocation is unreliable; splitting it fixed a chunk of this project's long-documented click flakiness.** Repeated single-click tests at a *verified-correct* coordinate still missed intermittently until the mousemove and click were issued as two separate `xdotool` calls with a short (0.2s) pause between them — after that, every subsequent click registered. This plausibly explains some of the "icon click missed on the first attempt, worked on retry" flakiness documented in Phases 56 (mailbox/cafe icons). Applied to `driver.click()` itself (project-wide, since all tasks share it) rather than special-cased in `stamina.py`; regression-tested live against `mailbox` and `cafe` after the change — both still work.
Not yet done: Group/Club AP, and Daily Free Power (see above).
### Phase 9: Normal/Hard story AP sweep
**Status: Done.** Read `module/explore_tasks/sweep_task.py` and `module/explore_tasks/task_utils.py` — the reference flow reads the current region number and matches stage-name text via OCR (`swipe_search_target_str`) to navigate to a configured target stage, then runs a per-stage claim loop. This client exposes a much simpler path to the same goal (burn AP via already-3-starred stages) that avoids porting the OCR-based lookup entirely: each stage's own 任務情報 (task info) modal has a self-contained 掃討 (sweep) sub-panel with a MIN/-/+/MAX count stepper and a start button.
Per explicit user direction on target selection: rather than a fixed configured stage (the plan's original "suggested first version"), `ba_auto/tasks/story_sweep.py` gets the *latest unlocked* region by spamming the "next region" arrow until it stops advancing (a plain state-change check, no OCR — clicking past the last region is a harmless no-op, verified live), then picks one of that region's stages essentially at random (`_pick_random_stage_row`: scroll the stage list to one of its two extremes at random, then pick a random one of the 4 visible rows there — not perfectly uniform since middle stages are reachable from both extremes, but avoids OCR/generic scroll-enumeration). AP spend is bounded by the in-game MAX button per explicit user direction (no additional cap layered on top).
Three real bugs were found and fixed during live calibration, all specific to the fact that this task spends real AP (unlike every other task so far, which only claims free rewards):
- **`navigation.is_modal_open`'s default probe `(960, 200)` false-negatives on this modal.** The 任務情報 modal is wide enough that `(960, 200)` lands on the modal's own white card, not the dimmed backdrop. Fixed with a task-specific `config.STAGE_MODAL_PROBE = (1870, 600)` and a local `_is_stage_modal_open()` check. First live test aborted safely on this false negative (correctly spent 0 AP) before the fix.
- **The MAX button click was never verified, and silently under-delivered.** A live run completed without error but only spent ~10 AP (one sweep) instead of the ~190 AP a real MAX (19 sweeps) should cost — diagnosed by comparing the actual AP/gold delta against the AP preview text seen during manual calibration. Root cause: the same general click-flakiness documented in Phase 8, just unverified here because nothing checked it. Fixed with `_count_raised_above_one()`: the sweep count's "-" stepper button is flat grey at the default count of 1 and turns vivid orange once raised, so probing `config.SWEEP_MINUS_BUTTON_PROBE` after the MAX click cheaply confirms it landed, without needing OCR on the count itself. Wrapped in a bounded retry (`MAX_BUTTON_RETRIES = 3`), aborting with zero AP spent if it never confirms. A subsequent live test hit 0/3 on this retry (a real flakiness cluster, not a logic bug) and correctly aborted without spending; the very next live run succeeded on attempt 1 with a genuine MAX (count 1→19, AP 191→1 confirmed by screenshot), so the retry+verify mechanism does its job on both sides — safe abort on failure, correct spend on success.
- **Escape does not close this modal, and the fallback dismiss loop was a latent hazard.** After a real sweep, the post-sweep dismiss loop pressed Enter a fixed number of times to clear reward-summary popups; live testing showed that once those popups run out, the *same* underlying 任務情報 modal reappears — and its Enter hotkey is bound to the live "任務開始" (start manual mission) button, not a no-op. The original fixed round count (3) happened to land exactly on the modal's reappearance without going further, but a different reward-popup count on another run could just as easily have pressed one Enter too many and started a real manual battle attempt. Two fixes: `_dismiss_sweep_result` now checks `_is_stage_modal_open` before every Enter press and stops immediately once the modal reappears, instead of trusting a fixed count; and closing now happens via a new `_close_stage_modal()` that clicks the modal's own X button (`config.STAGE_MODAL_CLOSE_BUTTON`, pinned via pixel-scanline scan of the glyph, not visual estimate) with a bounded retry+verify, since two Escape presses were confirmed live to leave the modal open. If the X-click ever fails to confirm closed, the task logs a warning and stops rather than pressing any further keys blindly.
Verified live: work-hub → task-screen navigation, latest-region advance, random stage pick, stage-modal-open detection, MAX click+verify, a genuine MAX sweep (19 runs, AP 191→1, gold +9,144), and the modal-close-via-X-button fix (confirmed via direct scripted click that it reliably closes and returns to the stage list). Not yet re-verified end-to-end in one single run: the fixed dismiss-loop-then-X-close sequence together, since AP was down to 1/240 after the successful test and there wasn't a further real sweep available to test against before the fix was deployed — each half was verified independently instead. Re-run `~/ba_dailies.sh story_sweep` once AP has regenerated to confirm the full sequence end-to-end.
`story_sweep` is deliberately **not** in `ba_daily.py`'s `DEFAULT_ORDER` — it spends AP on a randomly-picked stage rather than reclaiming something free, which is a real resource decision the default unattended run shouldn't make blindly. It must be invoked explicitly (`~/ba_dailies.sh story_sweep`).
**Retrospective — OCR avoidance was a mistake here.** Three of this phase's four live bugs (wrong modal probe, unverified MAX click, Escape-doesn't-close-modal plus the latent accidental-battle-start hazard) trace back to one decision: avoiding the reference's OCR-driven, deterministic stage targeting in favor of a heuristic substitute (random-pick + pixel-probes). A deterministic "go to configured stage X" flow, ported from the reference the way `module/explore_tasks/sweep_task.py`/`task_utils.py` actually do it, would not have needed to guess whether a modal opened via an easily-mismatched color probe, nor would it have left ambiguity about what's under the cursor when dismissing reward popups. This project's policy is now to port the reference's OCR-driven logic when the reference uses OCR for a feature, rather than inventing a non-OCR substitute to avoid the setup cost (see `CLAUDE.md` → "OCR policy"). `story_sweep.py`'s random-stage-pick design is not being reverted retroactively without user direction, but any future rework of this task should prefer porting the reference's actual region/stage-name OCR matching over the current random-pick approach.
### Phase 10: story_sweep OCR/state-machine port (supersedes Phase 9's random-pick design)
**Status: Done.** Acted on Phase 9's retrospective: set up OCR for real (`pytesseract` + the `tesseract-ocr` apt package) and ported the reference's actual deterministic stage targeting, replacing the random-pick heuristic. See `Handoff.md`'s history (deleted once this phase landed) for the full brief; summary of what changed:
- **OCR primitive**: `ba_auto/detector.py`'s `read_text()`/`read_int()` crop a screenshot to a pixel rect, threshold it to pure black/white (this measurably fixed real digit misreads that survived every `psm` mode when left anti-aliased — see below), upscale 3x, and run `pytesseract`. `lang="eng"` is enough; the region-number and stage-label reads are pure digits/dashes, no Japanese trained data needed.
- **Config-driven targets**: `config.STORY_SWEEP_TARGETS = [(region, stage, count_or_"max"), ...]`, mirroring the reference's `unfinished_normal_tasks` shape. Ships with a placeholder `(1, 1, "max")` entry per explicit user direction — edit it to your own already-cleared stage(s) before running for real.
- **Deterministic region navigation** (`_go_to_region`, porting `task_utils.py::to_region`): OCR the region-number readout, click the exact left/right-arrow delta, re-check, bounded loop. Region-arrow presence (locked/last-region detection) is a `detector.region_contains_color` box scan, not a single fixed point — a centroid-derived single point landed in the concave notch of the "<"/">" chevron and read "absent" even while the arrow was clearly rendered a few pixels away.
- **Deterministic stage search** (`_find_stage_row`, a scoped-down `swipe_search_target_str`): OCR each of the 4 visible stage-row labels at both already-calibrated scroll extremes, matching by the label's suffix after the dash (e.g. "2" in "30-2") rather than the full string — the font's leading region digit reads unreliably even after threshold preprocessing (e.g. "3" as "2"), but the suffix read correctly on every row tested, and the region digit is redundant anyway since `_go_to_region` already confirmed it independently.
- **Scoped `co_detect` port**: `navigation.wait_for_state(driver, config, reactions, ends, max_iterations)` — checks named `ends` first each iteration (stop, return the name), then named `reactions` (run an action, keep polling), else waits and retries up to a bound. Generic and reusable beyond this task.
- **Named outcomes**: `_sweep_target` returns `"swept"`, `"inadequate_ap"`, `"region_unavailable"`, `"stage_not_found"`, or `"unrecognized_state"` instead of a single generic "Done".
Four real, live-discovered findings, on top of what Phase 9 already found:
- **Regular numbered stages (30-1..30-5) render a different, taller modal layout than the "-A" bonus stage Phase 9 exclusively calibrated against.** Regular stages add a "集中指揮"/"簡易攻略" tab row and a manual "任務開始" panel below the sweep sub-panel that "-A" doesn't have. Phase 9's `SWEEP_MAX_BUTTON`/`SWEEP_MINUS_BUTTON_PROBE`/`SWEEP_START_BUTTON` all missed by ~40-46px vertically against a real numbered stage (30-3) — caught live when the MAX-click retry correctly failed 3/3 and aborted without spending AP, rather than silently misfiring. Re-calibrated against the tabbed layout via pixel-scanning (not eyeballing); the "-A" layout's original Phase 9 coordinates are no longer what these constants hold, so a future sweep of a "-A" stage specifically would need its own re-check.
- **The modal's own X-close button also moves with the layout.** Not just the sweep sub-panel — the whole card is vertically positioned by its own content height rather than anchored at a fixed absolute position, so the X button sits at a different absolute Y (225 vs Phase 9's 271) in the taller tabbed layout. Caught live: `_close_stage_modal` correctly reported "not closed" (3/3 retries) against the stale coordinate, rather than silently believing it had closed.
- **Clicking 掃討開始 always raises an AP-usage-confirmation dialog the design had never accounted for at all.** ("APを`N`使用して、掃討を`M`回行いますか?", OK/Cancel.) A first live attempt at porting the "wait for outcome" step read this dialog's dimmed backdrop as a false "inadequate_ap" through an early, unverified placeholder probe — worth remembering: an uncalibrated placeholder check can be actively *wrong*, not just inert, if given a chance to run before it's confirmed. Fixed by explicitly clicking through this confirmation before watching for the real result.
- **Genuine insufficient-AP is a visually near-identical dialog at the exact same OK-button position, told apart only by color.** Deliberately triggered live (by emptying the sweep count via MAX/"+" at low AP) rather than guessed: a real "AP不足" case shows a dialog titled "AP購入" (spend real Pyroxene to buy more AP) whose OK button is gold/yellow, vs. the safe usage-confirm's cyan — same position, different color. `_is_ap_purchase_prompt`/`_is_sweep_usage_confirm` tell them apart by that color and only ever click the cyan one; the gold one is always cancelled, never clicked, matching this project's purchase-safety rules.
Also fixed in passing, found only because live testing exercised the actual home-screen click path repeatedly: `config.TASK_CARD`'s old coordinate `(1370, 450)` sat close enough to the 任務 card's bottom edge that one run missed and landed on the "総力戦" (Total War) card below it instead — confirmed via screenshot, safely backed out with zero AP spent, moved to `(1250, 380)` (squarely on the "任務" title text).
Verified live end-to-end at least once, real AP spent: a genuine 5x sweep of 30-3 (AP 53→~5, confirmed via the "掃討完了" results screen's reward totals), including clicking through the usage-confirm dialog, the SKIP animation-skip screen, and the final reward-totals OK, landing back on the bare stage-info modal afterward. The genuine insufficient-AP path was also verified live (correctly cancelled the real "AP購入" purchase prompt without spending Pyroxene). Not yet re-verified end-to-end with the final rewritten code specifically (the color-based dynamic button-finding in `_watch_sweep_result`) at a nonzero AP balance — the manual walkthrough that discovered the dialogs used direct scripted clicks before the code was rewritten to match; the rewritten code's color-matching logic was separately verified offline against the exact screenshots captured live (all four dialog states correctly classified), but a fresh live run once AP regenerates would close that last gap. Not verified: sweeping a "-A" bonus stage (needs its own layout re-check, see above), an integer (non-"max") configured count actually being clicked via `SWEEP_PLUS_BUTTON`, and Hard-mode tab stages.
**Phase 10 follow-up (user-reported):** the user ran `story_sweep` for real and it spent AP on region 1 stage 1 instead of region 30 (their actual current last region). Not a code bug — `config.STORY_SWEEP_TARGETS` still held the literal placeholder `(1, 1, "max")` shipped with this phase, and the user hadn't edited it yet. Rather than just filling in one static `(30, N, "max")` entry, the user asked for the stage within region 30 to rotate daily across all 6 of that region's stages instead of grinding one fixed stage every run. Added `config.STORY_SWEEP_ROTATION_REGION`/`STORY_SWEEP_ROTATION_STAGE_COUNT`/`STORY_SWEEP_ROTATION_COUNT` and `story_sweep._rotation_target()`, which computes `(region, stage, count)` from `datetime.date.today().toordinal() % stage_count` — a plain date-ordinal modulo rather than calendar day-of-year, so the 6-day cycle doesn't skip or repeat around a year boundary. This target is appended to (not a replacement for) whatever's in `STORY_SWEEP_TARGETS`, which is now empty by default. Verified the computed target offline (region 30, stage 6, on the date this was fixed) but not yet re-run against the live game since AP hadn't regenerated.
### Phase 11: Common Shop + Tactical Shop
**Status: Done.** Ported `module/shop/common_shop.py` / `module/shop/tactical_challenge_shop.py`'s `implement()` and the shared `module/shop/shop_utils.py` (`to_common_shop`, `get_item_position`/`ensure_choose`/`buy`) to `ba_auto/tasks/shop_common.py` / `shop_tactical.py`, sharing control flow through a new `ba_auto/tasks/shop_utils.py`.
Key design call, made after live-capturing both shop tabs before writing any code: the reference's own item identification inside the grid is **not** OCR-based. `get_item_position` scans fixed pixel columns and matches a purchasable-state color plus a currency-icon template, then maps that grid position to an item identity via `self.static_config.common_shop_price_list` — a table sourced from an external resource this repo doesn't contain (the reference dataclass just declares the field; the actual values are fetched elsewhere, at BAAS's own runtime). So porting this feature couldn't mean "OCR the item names" (the reference doesn't do that either) — it meant building our own local equivalent of that static table by live-capturing the real catalog and letting the user pick their buy list from it, same shape as `STORY_SWEEP_TARGETS`. `config.COMMON_SHOP_TARGETS` / `config.TACTICAL_SHOP_TARGETS` are `(row, col, item name (comment only), expected price)` tuples; identification is by fixed grid position, with price-digit OCR (something the reference doesn't even do per-item) layered on as an extra live-catalog-drift safety net, consistent with this project's verify-before-spend pattern elsewhere.
What was found live, captured before writing any code (see the session's live-capture screenshots, not kept in the repo):
- **Both shop tabs share one UI**: a 4-column checkbox grid per item, then a single bulk "購入" (Buy) button that appears once ≥1 item is checked, rather than the reference's per-item purchase flow. Checked state renders a distinct vivid yellow-green on the checkbox glyph, easily told apart from the plain white/grey unchecked state by `detector.region_contains_color` — no template matching needed.
- **The tactical shop's tab list (7 entries) fits on screen with no scrolling**, so `config.SHOP_TAB_TACTICAL` is a fixed click rather than a port of the reference's `goto_shop_by_name` OCR swipe-search — there's nothing to search for on this account, so a fixed click is the faithful choice here, not a shortcut around OCR (see `CLAUDE.md`'s OCR policy: only skip OCR where the reference's own need for it doesn't apply).
- **Price-digit OCR needed real calibration**, same as Phase 10's stage labels: a rect wide enough for the widest configured price (500,000) without also catching the neighboring column's card, and narrow enough on the left to exclude the currency icon (which OCR otherwise misreads as a spurious leading digit — confirmed live, e.g. the top-bar credit balance read "454920755" instead of "154920755" until the icon was excluded).
- **One shared corner-pixel probe (`config.SHOP_OVERLAY_PROBE`, a bottom-left point) detects both the purchase-confirm dialog and the post-purchase "報酬獲得!" (reward acquired) banner** — both dim it away from pure white; it stays pure white with nothing open, confirmed stable across tab switches and scrolling. `shop_utils.confirm_purchase` presses Enter in a bounded loop until it reads idle again, rather than tracking each dialog's own layout individually.
- **Live-tested with real purchases, not just offline-verified.** With the user's explicit buy lists confirmed first (8 Common Shop items: 初級/中級/上級/最上級レポート + 初級/中級/上級/最上級強化珠; 2 Tactical Shop items: 初級/中級栄養ドリンク), both shops were run for real: Common Shop total cost matched the pre-calculated 1,211,500 credits exactly (confirmed against the game's own confirmation-dialog total); Tactical Shop's AP gain (+90) and coin spend (-45) matched exactly. Both purchases resolved cleanly back to an idle screen via the overlay-probe loop.
- **Discovered live, not anticipated going in: these shop items have a per-refresh-cycle purchase cap that isn't shown as a visible counter** (unlike the Pyroxene-shop tab's explicit "あと1回購入可能" labels — Common Shop items just look normal until you've already bought them, then go quietly unresponsive). Found by re-running the actual `shop_common` CLI task shortly after the manual purchase above: every configured item's checkbox failed to register as checked, and a direct test of the individual per-item "購入" button and the page's own "全て選択" (select-all) control confirmed those specific items are genuinely non-interactive right now (select-all successfully picked up *other*, not-yet-purchased items further down the list), while credits stayed unchanged throughout. The task correctly reported "nothing selected, cancelling" and spent nothing, rather than misfiring — a real edge case the price-verify + checkbox-confirm design caught safely, not a bug in it. Net effect: a fully "fresh, everything-available" unattended run hasn't been re-verified end-to-end today, since this account's cycle allowance for these specific items was already spent via the same session's manual calibration. Next run after the shop's own refresh timer (~5h, shown in-game as "更新まで") should exercise that path for real.
- **Not verified**: non-fully-visible target rows requiring a scroll (both current buy lists happen to be fully visible without scrolling, so `shop_utils` has no scroll/pagination logic yet — would need porting `buy()`'s `last_checked_idx`/swipe-diff tracking if a future buy list needs it); the pre-purchase "insufficient assets" abort path (both currencies were comfortably sufficient this run); the paid manual shop-refresh flow (`更新` button) — deliberately not automated for v1, same reasoning as Daily Free Power (real-currency spend needs explicit human intent, not a default unattended path).
### Phase 12: Lesson/Schedule
**Status: Done.** Ported `module/lesson.py`'s control flow (`implement`, the `to_*` navigation state machine, `get_lesson_each_region_status`/`get_lesson_relationship_counts`, `choose_lesson`, `execute_lesson`) to `ba_auto/tasks/lesson.py`.
Scope, decided with the user before writing any code (see the two `AskUserQuestion` answers): affection-first selection (mirrors `lesson_relationship_first=True` — matches this feature's own "affection farming" framing, not raw reward-tier grinding), sweep every unlocked region in a fixed order until either lesson tickets or scoreable lessons run out (no per-region target list needed from the user, unlike shop's per-item buy list), no lesson-ticket purchasing and no favor-student targeting (both deferred, matching plan.md's original "Suggested first version").
Key finding before writing any code: **this client's UI is structurally the same two-level layout the reference describes (12 named regions, each with up to 9 individual lesson locations) but rendered completely differently** — a scrollable list of regions instead of the reference's paged single-region swipe view, and a clean "すべてのスケジュール" grid-card modal instead of the reference's raw isometric map + `Parallelogram`/`Triangle` pixel-scanned status grid. Two consequences:
- **No OCR is needed for region navigation at all.** The reference OCRs the current region name because its paged arrows leave position ambiguous. This client's region list only ever settles at two scroll positions (scrolled to top: regions 0-5; scrolled to bottom: regions 6-11, confirmed live — repeated scroll-down clicks don't keep scrolling past this), so navigation is just "scroll to the right state, click the row at a fixed Y" — deterministic, nothing to locate. The reference's own 12 JP region names (`core/config/default_config.py`'s `lesson_region_name.JP` — embedded directly there, not fetched externally like the shop price table) are kept locally purely for log readability, confirmed to match this account's list exactly, row for row.
- **No isometric geometry is needed for per-cell status either.** Each grid-modal card shows up to 3 student portraits with a heart-shaped affection-count badge — reading that number via OCR needs no geometry at all, replacing the reference's isometric pixel-scan outright.
Three real bugs were found and fixed via a live test run (5 real tickets spent across 3 regions, ticket accounting and cleanup navigation both confirmed correct throughout):
- **A checkmark does not blank the badge number — it renders alongside it.** The initial design assumed an "already done today" portrait would fail the digit-whitelisted OCR read (no number left to read), the same way a "no relationship yet" portrait does, and treated both as identically "not a candidate." Live testing showed this was wrong: a done portrait keeps its unchanged number and gets a small green checkmark added at top-right instead. Without separately checking for that checkmark, the same already-done cell could be re-picked immediately after being completed. Fixed by detecting the checkmark directly via its own fixed color/offset (`config.LESSON_GRID_CHECKMARK_*`, `lesson._is_slot_already_done`) rather than inferring done-ness from the OCR read.
- **The badge's own OCR read was unreliable, and not for the reason first suspected.** Reusing the project's existing generic `detector.read_int` (grayscale + hard threshold, tuned for this UI's normal dark-text-on-light-card look) on the pink/magenta heart badge produced wildly oversized results live — "13" read as "113", "19" as "119", "18" as "418". Root cause, confirmed by rendering the exact same threshold step locally: the heart's own darker outline stroke has a grayscale value that happens to land on the same side of the threshold as the digit glyph, surviving as stray black marks that tesseract sometimes fuses into extra leading digits. Fixed with a dedicated `detector.read_int_on_heart_badge`, which masks on the color relationship "R < G" (true for the navy digit glyph in every sample, false for every pink/magenta badge tone, light or dark) instead of raw brightness this alone fixed most cases. A few slots still occasionally fuse a stray digit from the character's own portrait art bleeding into the crop's edge (this is character-art-dependent, not fixable by OCR config alone confirmed reproducible across every psm mode tried). Rather than chase perfect crop geometry per-character, `config.LESSON_GRID_BADGE_MAX_PLAUSIBLE` discards any reading of 100+ as certain contamination (real affection values never reach that range in practice) a second line of defense, not the primary fix.
- **A transient "2x schedule" campaign event (active on this account, ~6 days remaining) doubles how many intermediate screens appear after starting a lesson**, and a bond-rank-up cutscene can appear too (same full-screen "絆ランクアップ!" style already handled in `cafe.py`, confirmed live here for the first time against a real trigger). Rather than special-case either, `lesson._run_one_schedule` presses Enter in a bounded loop, checking two fixed markers each round: the grid modal's own title underline color (visible only when it's frontmost and idle — both the results modal and the cutscene cover it, confirmed live against screenshots of all three states) as the "done" signal, and the results modal's OK button color (shared with the info panel's Start button, confirmed by direct pixel sample) as the one intermediate state worth clicking precisely rather than folding into the blind-Enter fallback.
Also confirmed live: the "保有チケット N/M" ticket counter is directly visible on the region-list screen (no submenu needed, unlike the reference's `to_purchase_lesson_ticket`/OCR-a-modal approach) — `lesson._read_ticket_count` just OCRs it directly and re-reads it after every schedule to track spend, rather than assuming exactly 1 ticket per schedule (the 2x campaign event doesn't change the ticket cost, only the reward/cutscene count, but re-reading rather than assuming keeps this robust either way).
**Not verified**: a region with more than 9 currently-unlocked locations (would need scroll support inside the grid modal — not implemented, not yet seen on this account, both regions tested topped out at 7-8); the "no lesson tickets" abort-immediately path (tickets hit exactly 0 mid-sweep during the real test, not at the start); a full 12-region sweep in one run (the test's 5 tickets ran out partway through region 3 of 12); a truly fresh "everything available, nothing done yet" run (this account had already done some lessons manually during calibration before the automated run started). Worth re-running `~/ba_dailies.sh lesson` after tickets next refill to exercise the untested tail of the region list.
## Prerequisites
### OCR
Not set up yet.
**Set up (Phase 10): `pytesseract` + the `tesseract-ocr` apt package.** `ba_auto/detector.py`'s `read_text()`/`read_int()` wrap it for occasional single-crop reads (a region number, a stage label) — no need for the reference's own socket/shared-memory PaddleOCR server, which exists there to make OCR fast across thousands of automation steps; this project's usage volume doesn't need that.
Needed for:
@ -314,8 +429,6 @@ Candidates:
- Tesseract
- PaddleOCR
Do not add OCR until a feature needs it.
### Auto-fight primitive
Needed for:
@ -355,11 +468,9 @@ This should become a reusable primitive, not arena-specific code.
### 1. Migration to Python-first
This is the highest priority.
**Status: Done.** See Phases 17 above for the detailed history, including the mailbox and cafe exit-game-dialog bug and its fix.
Do this before adding new features.
Goal:
Goal (all done):
- Bash launcher only
- Python CLI
@ -369,20 +480,14 @@ Goal:
- cafe migrated
- reference mapping started
Reference:
- Current local implementation
- Existing `ba_dailies.sh`
- Existing `scripts/detect_and_click.py`
### 2. Stamina/AP sweep
**Status: Partially done — see Phase 8.**
Claim:
- daily free AP purchase
- daily task-menu AP/pyroxene rewards
Likely mostly color/state detection and fixed clicks.
- ~~daily free AP purchase~~ — deferred; entry point is a real-money purchase menu, needs explicit confirmation before automating
- daily task-menu AP/pyroxene rewards — done, via the Mission panel's bulk claim button
Reference:
@ -393,7 +498,7 @@ Reference:
Local target: `ba_auto/tasks/stamina.py`
OCR: Probably not needed for first version.
OCR: Not needed — turned out to be a single bulk-claim button + Enter, no per-item detection required.
### 3. Club/Group AP claim
@ -411,6 +516,8 @@ OCR: Not expected.
### 4. Normal/Hard story AP sweep
**Status: Done — see Phase 10 (supersedes Phase 9's random-pick design).**
Sweep already-cleared main story stages to burn AP.
Reference:
@ -422,19 +529,13 @@ Reference:
Local target: `ba_auto/tasks/story_sweep.py`
OCR: Likely needed for current region/stage detection unless using fixed configured targets.
OCR: Used, for real, as of Phase 10 — region-number readout OCR + delta-click (porting `to_region`), and stage-row label OCR matching (a scoped-down `swipe_search_target_str`), replacing Phase 9's "next region arrow stops advancing, then a random stage" heuristic. See Phase 10's write-up for what was actually found live (a second, taller modal layout for regular numbered stages; an AP-usage-confirmation dialog the design hadn't accounted for; a genuine insufficient-AP dialog told apart from it only by button color).
Suggested first version:
Implemented version:
- user-configured fixed stage
- no region search
- no dynamic OCR
- sweep configured mission only
Later version:
- fuzzy stage/region selection
- OCR-assisted navigation
- exact configured `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS`), not a fixed single stage nor a random pick
- `count` is `"max"` or a specific int (the latter calibrated but not yet live-clicked — see Phase 10)
- opt-in only, not in the default daily flow
### 5. Bounty
@ -465,7 +566,7 @@ Reference:
Local target: `ba_auto/tasks/commission.py`
OCR: Probably avoidable for first version if configured fixed sweep target is used.
OCR: Port the reference's approach if it uses OCR here — do not default to a fixed-target workaround just to avoid OCR (see "OCR policy" in `CLAUDE.md` and the Phase 9 retrospective above).
### 7. Arena
@ -489,6 +590,8 @@ Arena is high-value but more risky than fixed claim tasks.
### 8. Common Shop + Tactical Shop
**Status: Done — see Phase 11.**
Auto-buy configured items.
Reference:
@ -504,18 +607,18 @@ Local targets:
```
ba_auto/tasks/shop_common.py
ba_auto/tasks/shop_tactical.py
ba_auto/tasks/shop_utils.py
```
Needs:
Implemented version:
- OCR for currency balances
- shop tab detection
- configured buy list
- safe purchase confirmation logic
- OCR for currency balances (top-bar credits, in-panel tactical coin) — done
- shop tab detection — done via a fixed click (tactical tab list fits on screen with no scroll needed on this account, confirmed live); no scroll/pagination logic yet since both current buy lists are fully visible without scrolling
- configured buy list — done, `config.COMMON_SHOP_TARGETS`/`config.TACTICAL_SHOP_TARGETS`, fixed `(row, col, name, expected_price)` per item (see Phase 11 for why identification is by grid position + price-OCR-verify rather than per-item OCR — the reference doesn't OCR item names here either)
- safe purchase confirmation logic — done, a single overlay-darkness probe covers both the confirm dialog and the reward-acquired banner
- no-refresh (the paid manual `更新` refresh button is deliberately not automated, same reasoning as Daily Free Power)
Start with a no-refresh, fixed configured buy list.
### 9. Lesson / Schedule
### 9. Lesson / Schedule — Done (Phase 12)
Affection farming via classes.
@ -525,22 +628,18 @@ Reference:
~/repo/baas-reference/module/lesson.py
```
Local target: `ba_auto/tasks/lesson.py`
Local target: `ba_auto/tasks/lesson.py` — implemented, live-tested with real tickets. See Phase 12 above for the full writeup.
Complexity: High
- region/area identification — done without OCR: this client's region list only settles at two fixed scroll positions, so navigation is deterministic index-based clicking, not the reference's OCR-a-name-then-page approach
- multi-page swipe search — not needed for the same reason
- student detection/portrait matching for specific students — deferred, per the original suggested scope below
- isometric grid location logic — done without porting the reference's geometry: per-cell status/affection reads via `detector.read_int_on_heart_badge` OCR + a checkmark color probe instead
Needs:
Suggested first version (as originally scoped, and what shipped):
- OCR for region/area names
- multi-page swipe search
- student detection/portrait matching if targeting specific students
- isometric grid location logic if following reference fully
Suggested first version:
- pick a fixed region
- select available/highest visible lesson
- avoid favorite-student targeting at first
- pick a fixed region — expanded to all 12, swept in order
- select available/highest visible lesson — affection-first, per explicit user direction
- avoid favorite-student targeting at first — still deferred
## Low priority backlog
@ -714,7 +813,7 @@ server = JP
game_window_name = BlueArchive
display = :0
asset_dir = ~/ba_assets
screenshot_dir = ./scratchpad
screenshot_dir = scratchpad/
cafe_max_clicks_per_room
story_sweep_target
shop_buy_list
@ -727,7 +826,7 @@ Keep config explicit. Do not bury user-specific settings deep inside task logic.
## Debugging conventions
Use `./scratchpad` for:
Use `scratchpad/` for:
- temporary screenshots
- cropped templates
@ -772,21 +871,23 @@ should run the default daily sequence.
## Near-term recommended task order
1. Rewrite `ba_dailies.sh` as a thin launcher.
2. Add `ba_daily.py`.
3. Add `ba_auto/driver.py`.
4. Add `ba_auto/detector.py`.
5. Add `ba_auto/navigation.py`.
6. Move mailbox logic to `ba_auto/tasks/mailbox.py`.
7. Move cafe logic to `ba_auto/tasks/cafe.py`.
8. Update `setup.sh`.
9. Add `ba_auto/reference_notes/mapping.md`.
10. Verify existing mailbox and cafe still work.
11. Implement stamina/AP.
12. Implement group/club AP.
13. Implement fixed-target sweep features.
14. Add OCR only when needed.
15. Attempt Arena/Shop/Lesson after the framework is stable.
1. Rewrite `ba_dailies.sh` as a thin launcher. — Done
2. Add `ba_daily.py`. — Done
3. Add `ba_auto/driver.py`. — Done
4. Add `ba_auto/detector.py`. — Done
5. Add `ba_auto/navigation.py`. — Done
6. Move mailbox logic to `ba_auto/tasks/mailbox.py`. — Done
7. Move cafe logic to `ba_auto/tasks/cafe.py`. — Done
8. Update `setup.sh`. — Done
9. Add `ba_auto/reference_notes/mapping.md`. — Done
10. Verify existing mailbox and cafe still work. — Done
11. Implement stamina/AP. — Done (Phase 8)
12. Implement Normal/Hard story AP sweep. — Done (Phase 9 built a random-pick heuristic; Phase 10 replaced it with the reference's actual OCR-based deterministic stage targeting, per Phase 9's own retrospective)
13. Implement group/club AP. — Not started
14. Set up OCR and port it for whichever remaining feature's reference implementation depends on it — not a blanket "only when needed" deferral; see `CLAUDE.md` → "OCR policy"
15. Implement Common Shop + Tactical Shop. — Done (Phase 11)
16. Implement Lesson/Schedule. — Done (Phase 12)
17. Attempt Arena once OCR is in place, since its reference implementation depends on it.
## Claude Code guidance summary
@ -796,5 +897,6 @@ When Claude Code works on this repo, it should follow this rule:
> Python first.
> Driver primitives before feature hacks.
> Bash launcher only.
> Port OCR when the reference uses it — don't invent non-OCR substitutes to avoid the setup cost.
Do not turn this project into a Bash recreation of Blue Archive Auto Script.

View File

@ -1,122 +0,0 @@
#!/bin/bash
# Legacy pre-migration click sequences; bridged from ba_auto/tasks/*.py until ported to Python (see CLAUDE.md).
export DISPLAY=:0
export XAUTHORITY=/run/user/1000/gdm/Xauthority
WIN_NAME="BlueArchive"
MAILBOX_ICON="1726 60"
CLAIM_ALL="1691 1128"
CAFE_ICON="165 1100"
CAFE_ROOM_SWITCH="190 160"
CAFE_INCOME="1780 1105"
VENV_PYTHON="$HOME/.venvs/ba-auto-daily/bin/python3"
DETECT_SPARKLE="$HOME/ba_scripts/detect_and_click.py"
CAFE_MAX_CLICKS_PER_ROOM=15
get_window() {
xdotool search --name "$WIN_NAME" | head -1
}
click() {
local coords="$1"
xdotool mousemove $coords click 1
sleep 0.5
}
press_enter() {
xdotool key Return
sleep 0.5
}
press_esc() {
xdotool key Escape
sleep 0.5
}
focus_game() {
local win
win=$(get_window)
if [ -z "$win" ]; then
echo "ERROR: Blue Archive window not found. Is the game running?"
exit 1
fi
xdotool windowactivate "$win"
sleep 0.5
}
do_mailbox() {
echo "[mailbox] Opening mailbox..."
click "$MAILBOX_ICON"
sleep 1.5
echo "[mailbox] Claiming all..."
click "$CLAIM_ALL"
sleep 1
echo "[mailbox] Confirming claim..."
press_enter
sleep 1
echo "[mailbox] Closing mailbox..."
press_esc
sleep 1
echo "[mailbox] Done."
}
do_cafe_room() {
local i
for ((i = 0; i < CAFE_MAX_CLICKS_PER_ROOM; i++)); do
if ! "$VENV_PYTHON" "$DETECT_SPARKLE" | grep -q "^MATCH"; then
break
fi
sleep 1
press_enter
done
}
do_cafe() {
echo "[cafe] Opening cafe..."
click "$CAFE_ICON"
sleep 3
press_enter
echo "[cafe] Room 1: farming affection..."
do_cafe_room
echo "[cafe] Switching to room 2..."
click "$CAFE_ROOM_SWITCH"
sleep 3
press_enter
echo "[cafe] Room 2: farming affection..."
do_cafe_room
echo "[cafe] Claiming cafe income..."
click "$CAFE_INCOME"
sleep 2
press_enter
sleep 2
press_enter
sleep 2
echo "[cafe] Exiting cafe..."
press_esc
sleep 1.5
press_esc
sleep 1.5
echo "[cafe] Done."
}
case "$1" in
mailbox)
focus_game
do_mailbox
;;
cafe)
focus_game
do_cafe
;;
"")
focus_game
do_mailbox
do_cafe
echo "All done."
;;
*)
echo "Unknown phase: $1"
exit 1
;;
esac

View File

@ -1,69 +0,0 @@
#!/usr/bin/env python3
"""Screenshot the game window, find a cafe affection sparkle, click it.
Runs entirely on nik-gpu (screenshot -> detect -> click all local) so the
whole cycle finishes in well under a second - roaming students move fast
enough that a multi-hop SSH/scp pipeline misses the click.
Prints "MATCH x y score" and exits 0 if a sparkle was found and clicked,
or prints "NO_MATCH" and exits 1 otherwise.
"""
import subprocess
import sys
import cv2
import numpy as np
TEMPLATE_PATH = "/home/nik/ba_assets/cafe_sparkle.png"
SHOT_PATH = "/tmp/ba_live.png"
OFFSET_X = 75
OFFSET_Y = 47
THRESHOLD = 0.97
ENV = {"DISPLAY": ":0", "XAUTHORITY": "/run/user/1000/gdm/Xauthority"}
def screenshot():
subprocess.run(
["scrot", "-a", "0,0,1920,1200", "-o", SHOT_PATH],
env=ENV, check=True,
)
def find_sparkles():
template = cv2.imread(TEMPLATE_PATH)
b, g, r = cv2.split(template.astype(np.int16))
yellow_white = ((r > 180) & (g > 140) & (r - b > 60)) | ((r > 200) & (g > 200) & (b > 200))
mask_plane = (yellow_white.astype(np.uint8)) * 255
mask = cv2.merge([mask_plane, mask_plane, mask_plane])
th, tw = template.shape[:2]
img = cv2.imread(SHOT_PATH)
result = cv2.matchTemplate(img, template, cv2.TM_CCORR_NORMED, mask=mask)
locs = np.where(result >= THRESHOLD)
points = sorted(zip(*locs[::-1]), key=lambda p: -result[p[1], p[0]])
merged = []
for x, y in points:
if all(abs(x - mx) > tw // 2 or abs(y - my) > th // 2 for mx, my, _ in merged):
merged.append((x, y, result[y, x]))
return [(x + tw // 2, y + th // 2, score) for x, y, score in merged]
def click(x, y):
subprocess.run(
["xdotool", "mousemove", str(x), str(y), "click", "1"],
env=ENV, check=True,
)
if __name__ == "__main__":
screenshot()
sparkles = find_sparkles()
if not sparkles:
print("NO_MATCH")
sys.exit(1)
x, y, score = sparkles[0]
click(x + OFFSET_X, y + OFFSET_Y)
print(f"MATCH {x} {y} {score:.4f}")
sys.exit(0)

View File

@ -5,13 +5,12 @@
# not from macOS, since it installs into nik-gpu-local paths.
#
# It does NOT touch the game or take any screenshots; it only installs the
# venv and copies files to the fixed paths ba_dailies.sh/ba_daily.py/
# detect_and_click.py expect (outside the repo, per the two-machine deploy
# convention in CLAUDE.md).
# venv and copies files to the fixed paths ba_dailies.sh/ba_daily.py/ba_auto
# expect (outside the repo, per the two-machine deploy convention in
# CLAUDE.md).
set -e
VENV_DIR="$HOME/.venvs/ba-auto-daily"
SCRIPTS_DIR="$HOME/ba_scripts"
ASSETS_DIR="$HOME/ba_assets"
echo "== Checking host tools =="
@ -28,19 +27,29 @@ if [ "$missing" = 1 ]; then
fi
echo "xdotool, scrot: OK"
if ! command -v tesseract >/dev/null 2>&1; then
echo "MISSING: tesseract (OCR engine binary -- e.g. sudo apt install tesseract-ocr)"
echo "story_sweep's region/stage-label OCR (see CLAUDE.md \"OCR policy\") needs this."
echo "This requires a password-interactive sudo, so it isn't installed for you here --"
echo "install it yourself, then re-run this script."
missing=1
fi
if [ "$missing" = 1 ]; then
exit 1
fi
echo "tesseract: OK"
echo "== Setting up Python venv at $VENV_DIR =="
if [ ! -x "$VENV_DIR/bin/python3" ]; then
python3 -m venv "$VENV_DIR"
fi
"$VENV_DIR/bin/pip" install --quiet --upgrade pip
"$VENV_DIR/bin/pip" install --quiet opencv-python-headless numpy
"$VENV_DIR/bin/pip" install --quiet opencv-python-headless numpy pytesseract
"$VENV_DIR/bin/python3" -c "import cv2, numpy; print('opencv', cv2.__version__, '/ numpy', numpy.__version__)"
"$VENV_DIR/bin/python3" -c "import pytesseract; print('pytesseract', pytesseract.get_tesseract_version())"
echo "== Deploying scripts + assets to fixed paths =="
mkdir -p "$SCRIPTS_DIR" "$ASSETS_DIR"
cp scripts/detect_and_click.py "$SCRIPTS_DIR/detect_and_click.py"
cp scripts/ba_dailies_legacy.sh "$SCRIPTS_DIR/ba_dailies_legacy.sh"
chmod +x "$SCRIPTS_DIR/ba_dailies_legacy.sh"
echo "== Deploying assets to fixed paths =="
mkdir -p "$ASSETS_DIR"
cp assets/cafe_sparkle.png "$ASSETS_DIR/cafe_sparkle.png"
echo "== Deploying Python-first entry point =="
@ -51,5 +60,5 @@ rm -rf "$HOME/ba_auto"
cp -r ba_auto "$HOME/ba_auto"
echo "== Done =="
echo "Run with: ~/ba_dailies.sh [mailbox|cafe]"
echo "Run with: ~/ba_dailies.sh [mailbox|cafe|stamina|story_sweep|shop_common|shop_tactical|lesson]"
echo "(requires the game already running, window titled 'BlueArchive')"