980 lines
87 KiB
Markdown
980 lines
87 KiB
Markdown
# ba-auto-daily implementation plan
|
||
|
||
Personal Blue Archive JP daily-automation project.
|
||
|
||
This project controls the PC/Steam/Proton Blue Archive client running on `nik-gpu` through local desktop automation:
|
||
|
||
- xdotool
|
||
- scrot
|
||
- Python
|
||
- OpenCV
|
||
- OCR, ported wherever the reference implementation uses it for a feature (see `CLAUDE.md` → "OCR policy" — this is no longer a "later, when needed" deferral)
|
||
|
||
Development happens on `nik-macbookair`.
|
||
|
||
The reference implementation lives at:
|
||
|
||
```
|
||
~/repo/baas-reference/
|
||
```
|
||
|
||
The reference project should be treated as the behavioral blueprint. This project should avoid recreating feature logic from scratch when the reference already implements it.
|
||
|
||
## Core project direction
|
||
|
||
This project is now Python-first.
|
||
|
||
The goal is not to grow a large Bash script.
|
||
|
||
The goal is to build a small local Python automation framework that adapts the reference project's Blue Archive logic to this user's unique PC/Steam/Proton environment.
|
||
|
||
`ba_dailies.sh` should only be a launcher.
|
||
|
||
Feature logic should live in Python.
|
||
|
||
## Target architecture
|
||
|
||
```
|
||
~/repo/ba-auto-daily/
|
||
├── ba_dailies.sh
|
||
├── ba_daily.py
|
||
├── ba_auto/
|
||
│ ├── __init__.py
|
||
│ ├── config.py
|
||
│ ├── driver.py
|
||
│ ├── detector.py
|
||
│ ├── navigation.py
|
||
│ ├── tasks/
|
||
│ │ ├── __init__.py
|
||
│ │ ├── mailbox.py
|
||
│ │ ├── cafe.py
|
||
│ │ ├── stamina.py
|
||
│ │ ├── group.py
|
||
│ │ ├── bounty.py
|
||
│ │ ├── commission.py
|
||
│ │ ├── arena.py
|
||
│ │ ├── shop_common.py
|
||
│ │ ├── shop_tactical.py
|
||
│ │ ├── lesson.py
|
||
│ │ └── ...
|
||
│ └── reference_notes/
|
||
│ └── mapping.md
|
||
├── assets/
|
||
├── screenshots/
|
||
├── scripts/
|
||
├── setup.sh
|
||
├── CLAUDE.md
|
||
└── plan.md
|
||
```
|
||
|
||
This is the intended direction. It does not have to be completed all at once.
|
||
|
||
## Project layout
|
||
|
||
| Path | What it is |
|
||
|---|---|
|
||
| `~/repo/ba-auto-daily/ba_dailies.sh` | Thin launcher only. It should call the Python entry point. Do not add new feature logic here. |
|
||
| `~/repo/ba-auto-daily/ba_daily.py` | Main Python CLI entry point. Dispatches tasks such as mailbox, cafe, stamina, group, etc. |
|
||
| `~/repo/ba-auto-daily/ba_auto/driver.py` | Local PC/Steam/Proton control backend. Wraps xdotool, scrot, waits, clicks, swipes, keypresses, screenshots, and window focus. |
|
||
| `~/repo/ba-auto-daily/ba_auto/detector.py` | OpenCV/template/color matching helpers. Currently has `find_cafe_sparkle()`, ported in-process from the retired `scripts/detect_and_click.py`. |
|
||
| `~/repo/ba-auto-daily/ba_auto/navigation.py` | Shared navigation/state-probe helpers: `is_on_subscreen`, `is_modal_open`, used by both `mailbox.py` and `cafe.py`. |
|
||
| `~/repo/ba-auto-daily/ba_auto/tasks/` | Feature implementations. Each task should adapt the relevant `baas-reference/module/...` logic where possible. |
|
||
| `~/repo/ba-auto-daily/ba_auto/reference_notes/mapping.md` | Reference mapping table: local feature → reference module → local implementation → driver gaps. |
|
||
| `~/repo/ba-auto-daily/assets/` | Locally captured template images, such as cafe sparkle. Do not blindly copy assets from the reference repo. |
|
||
| `~/repo/ba-auto-daily/screenshots/` | Human reference screenshots, mostly Moonlight/game captures, used for calibration and debugging. |
|
||
| `~/repo/ba-auto-daily/setup.sh` | Bootstrap/deploy helper for `nik-gpu`. Should install/check dependencies and copy runtime files. |
|
||
| `~/repo/baas-reference/` | Read-only GPL-3.0 reference clone. Study and adapt. Never edit. |
|
||
|
||
## Runtime paths on nik-gpu
|
||
|
||
Preferred runtime layout:
|
||
|
||
| Path | What it is |
|
||
|---|---|
|
||
| `nik-gpu:~/ba_dailies.sh` | Thin launcher. |
|
||
| `nik-gpu:~/ba_daily.py` | Python CLI entry point. |
|
||
| `nik-gpu:~/ba_auto/` | Python package copied from this repo. |
|
||
| `nik-gpu:~/ba_assets/` | Runtime assets/templates. |
|
||
| `nik-gpu:~/.venvs/ba-auto-daily/` | Python virtual environment. |
|
||
|
||
`nik-gpu:~/ba_scripts/` may still contain `detect_and_click.py` and `ba_dailies_legacy.sh` left over from before both mailbox and cafe were migrated off them. Neither is deployed or referenced by anything anymore (`setup.sh` stopped copying them once Phase 6 landed) — safe to delete manually on `nik-gpu`, just not automated here.
|
||
|
||
## Implementation strategy
|
||
|
||
For each feature:
|
||
|
||
1. Read the matching `~/repo/baas-reference/module/...` file.
|
||
2. Summarize the reference flow.
|
||
3. Identify reusable logic:
|
||
- state checks
|
||
- retry loops
|
||
- navigation sequence
|
||
- battle/sweep/shop rules
|
||
- detection method
|
||
- failure handling
|
||
4. Identify backend-specific calls that cannot be reused directly.
|
||
5. Implement missing generic primitives in `ba_auto/driver.py` or `ba_auto/detector.py`.
|
||
6. Implement the feature in `ba_auto/tasks/<feature>.py`.
|
||
7. Add CLI command dispatch in `ba_daily.py`.
|
||
8. Keep `ba_dailies.sh` unchanged unless launcher behavior changes.
|
||
9. Test syntax locally.
|
||
10. Deploy to `nik-gpu`.
|
||
11. Run against the live game.
|
||
12. Update this plan.
|
||
|
||
The intended result is not a Bash automation script.
|
||
|
||
The intended result is a Python automation framework using the reference repository as the behavioral blueprint.
|
||
|
||
## Reference mapping table
|
||
|
||
Maintain this table in `ba_auto/reference_notes/mapping.md`.
|
||
|
||
Initial seed:
|
||
|
||
| Local feature | Reference file | Reference functions/classes | Local file | Backend replacements | Status |
|
||
|---|---|---|---|---|---|
|
||
| Mailbox | Need to confirm in reference | Need to inspect | `ba_auto/tasks/mailbox.py` | tap/click via xdotool, screenshot via scrot | Existing Bash behavior; migrate to Python |
|
||
| Cafe | `module/cafe_reward.py` | `to_cafe`, `interaction_for_cafe_solve_method3`, `collect` | `ba_auto/tasks/cafe.py` | `picture.co_detect`/`color.rgb_in_range` → `driver.color_at` pixel-probe checks; sparkle template match ported in-process into `ba_auto/detector.py` | Migrated: real Python, state-verified via color probes, no legacy bridge |
|
||
| Stamina/AP | `module/collect_daily_free_power.py`, `module/collect_daily_task_power.py` | `to_tasks`/`implement` (task-power, ported); `to_purchase_pyroxenes_menu`/`detect_free_power_availability` (free-power, not ported — real-money purchase menu) | `ba_auto/tasks/stamina.py` | `color.rgb_in_range` → `driver.color_at`; reference's per-tab claim loop replaced by the live UI's single "一括受取" (claim-all) button, triggered via Enter | Partially migrated (Phase 8): Mission-panel task/weekly/achievement claim done. Daily Free Power deliberately not implemented. |
|
||
| Normal/Hard story AP sweep | `module/explore_tasks/sweep_task.py`, `module/explore_tasks/task_utils.py` | `to_region`/`to_normal_event` + OCR-driven per-stage claim loop (ported, Phase 10) | `ba_auto/tasks/story_sweep.py` | OCR-based region/stage-name matching, ported for real (Phase 10); reference's per-stage claim loop → this client's stage-info modal's self-contained 掃討 (sweep) sub-panel (MIN/-/+/MAX stepper + start button) | Done (Phase 10, supersedes Phase 9's random-pick design) |
|
||
| Group/Club AP | `module/group.py` | Need to inspect | `ba_auto/tasks/group.py` | fixed click + state check via local driver | Not started |
|
||
| Bounty | `module/rewarded_task.py` | Need to inspect | `ba_auto/tasks/bounty.py` | sweep/color/OCR adaptation | Not started |
|
||
| Commissions | `module/clear_special_task_power.py` | Need to inspect | `ba_auto/tasks/commission.py` | sweep/color adaptation | Not started |
|
||
| Arena | `module/arena.py` | Need to inspect | `ba_auto/tasks/arena.py` | auto-fight + OCR + local driver | Not started |
|
||
| Common Shop | `module/shop/common_shop.py`, `module/shop/shop_utils.py` | Need to inspect | `ba_auto/tasks/shop_common.py` | OCR + tab navigation + local clicks | Not started |
|
||
| Tactical Shop | `module/shop/tactical_challenge_shop.py`, `module/shop/shop_utils.py` | Need to inspect | `ba_auto/tasks/shop_tactical.py` | OCR + tab navigation + local clicks | Not started |
|
||
| Lesson/Schedule | `module/lesson.py` | Need to inspect | `ba_auto/tasks/lesson.py` | OCR + template/portrait search + local driver | Not started |
|
||
|
||
Do not implement a feature without filling at least the relevant row.
|
||
|
||
## Status snapshot
|
||
|
||
| Feature | Current status | Target status |
|
||
|---|---|---|
|
||
| Mailbox claim | Migrated: `ba_auto/tasks/mailbox.py` uses `driver.color_at` to verify the panel opened before acting (found live-testing bug: a marginal icon coordinate could miss and cascade into pressing Escape on the home screen, which triggers Blue Archive's own exit-game confirmation) | Done |
|
||
| Cafe pats + income | Migrated: `ba_auto/tasks/cafe.py` verifies each room/dialog opened via `driver.color_at` before acting; sparkle detection now runs in-process via `ba_auto/detector.py` instead of a per-click subprocess | Done |
|
||
| Stamina/AP (mission claim) | Migrated (partial, Phase 8): `ba_auto/tasks/stamina.py` claims the Mission panel's bulk "一括受取" button. Daily Free Power (real-money purchase menu) intentionally not implemented | Daily Free Power still not started |
|
||
| Normal/Hard story AP sweep | Done (Phase 10, supersedes Phase 9's random-pick design): `ba_auto/tasks/story_sweep.py` sweeps a config-driven list of exact `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS`), navigating to each via OCR (region-number read + delta-click, stage-label OCR match) instead of "latest region, random stage." Opt-in only (`story_sweep` command), not part of the default daily flow | Done |
|
||
| Common Shop / Tactical Shop | Done (Phase 11): `ba_auto/tasks/shop_common.py` / `shop_tactical.py` share a checkbox-grid-then-bulk-buy flow (`ba_auto/tasks/shop_utils.py`) against config-driven `(row, col, name, expected_price)` targets, price-OCR-verified before each click. Live-tested with real purchases in both shops. Opt-in only (`shop_common`/`shop_tactical` commands), not part of the default daily flow | Done |
|
||
| Lesson/Schedule | Done (Phase 12): `ba_auto/tasks/lesson.py` sweeps every unlocked region's schedule grid, picking the highest-affection available lesson each time via a dedicated heart-badge OCR read (`detector.read_int_on_heart_badge`) until tickets or lessons run out. Live-tested with real tickets spent; two real bugs (checkmark-doesn't-blank-the-number, badge OCR misreads) found and fixed. Opt-in only (`lesson` command), not part of the default daily flow | Done |
|
||
| Arena / Tactical Challenge | Done (Phase 13): `ba_auto/tasks/arena.py` fights exactly one ranked battle per invocation and collects both reward slots. Live-tested across all 5 of the account's daily tickets (2 WIN, 1 LOSE); the post-fight WIN/LOSE result modal proved undetectable by precise button-color search and was fixed via a bounded blind-Enter loop gated by a hard safety check. Opt-in only (`arena` command), not part of the default daily flow | Done |
|
||
| Event sweep | Implemented (Phase 14), calibrated live with zero real AP spent, not yet live-tested with a real sweep: `ba_auto/tasks/event_sweep.py` sweeps one config-rotated stage (9-12) of the currently-running event per invocation, reusing story_sweep's AP-confirm/result-screen config directly since the underlying dialog is pixel-identical. Opt-in only (`event_sweep` command), not part of the default daily flow | Needs a real live sweep test |
|
||
| Shared driver | `ba_auto/driver.py` built (`run_command`, `focus_game`, `click`, `move_mouse`, `scroll`, `keypress`, `screenshot`, `wait`, `color_at`); `click()` now splits `mousemove`/`click` into two xdotool calls (Phase 8 finding — fixes a real source of click flakiness); wired into `mailbox.py`, `cafe.py`, `stamina.py`, `story_sweep.py`, `shop_common.py`, `shop_tactical.py`, `lesson.py` | Extend with new primitives as future tasks need them |
|
||
| Python CLI | Built: `ba_daily.py` dispatches `mailbox`/`cafe`/`stamina`/`story_sweep`/`shop_common`/`shop_tactical`/`lesson`/default flow | Extend as new tasks are added |
|
||
| Reference mapping | Built: `ba_auto/reference_notes/mapping.md` | Fill in reference file/function columns per feature |
|
||
| Everything else | Not started | Implement reference-first in Python |
|
||
|
||
## Migration phase
|
||
|
||
Before adding new game features, migrate the existing working implementation.
|
||
|
||
### Phase 1: Python skeleton
|
||
|
||
**Status: Done.**
|
||
|
||
Create:
|
||
|
||
```
|
||
ba_daily.py
|
||
ba_auto/__init__.py
|
||
ba_auto/config.py
|
||
ba_auto/driver.py
|
||
ba_auto/detector.py
|
||
ba_auto/navigation.py
|
||
ba_auto/tasks/__init__.py
|
||
ba_auto/tasks/mailbox.py
|
||
ba_auto/tasks/cafe.py
|
||
ba_auto/reference_notes/mapping.md
|
||
```
|
||
|
||
### Phase 2: Launcher
|
||
|
||
**Status: Done.**
|
||
|
||
Change `ba_dailies.sh` into a thin launcher:
|
||
|
||
```bash
|
||
#!/usr/bin/env bash
|
||
set -euo pipefail
|
||
VENV_PYTHON="${VENV_PYTHON:-$HOME/.venvs/ba-auto-daily/bin/python3}"
|
||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||
exec "$VENV_PYTHON" "$SCRIPT_DIR/ba_daily.py" "$@"
|
||
```
|
||
|
||
Keep compatibility with:
|
||
|
||
```
|
||
./ba_dailies.sh
|
||
./ba_dailies.sh mailbox
|
||
./ba_dailies.sh cafe
|
||
```
|
||
|
||
### Phase 3: Driver extraction
|
||
|
||
**Status: Done.** Primitives (including `color_at`, added during the mailbox/cafe hardening work) are wired into both `ba_auto/tasks/mailbox.py` and `ba_auto/tasks/cafe.py`.
|
||
|
||
Move shell interactions into `ba_auto/driver.py`.
|
||
|
||
Driver primitives should include:
|
||
|
||
```
|
||
focus_game()
|
||
click(x, y)
|
||
double_click(x, y)
|
||
keypress(key)
|
||
screenshot(path=None)
|
||
swipe(...)
|
||
wait(seconds)
|
||
wait_until(...)
|
||
```
|
||
|
||
### Phase 4: Detector extraction
|
||
|
||
**Status: Done (scoped).** `scripts/detect_and_click.py`'s sparkle-matching logic (masked template match against `assets/cafe_sparkle.png`) was ported into `ba_auto/detector.py` as `find_cafe_sparkle()`, called in-process from `ba_auto/tasks/cafe.py` — this removed the old per-click Python cold start (a fresh `cv2`/`numpy` import per subprocess call) that CLAUDE.md's driver-layer guidance specifically warns against. `scripts/detect_and_click.py` had no remaining callers once this landed, so it was deleted rather than kept as a compatibility wrapper. The more generic primitives listed below (`load_template`, `match_template`, etc.) have not been built — only the one concrete sparkle-matching function needed so far exists; generalize when a second detector use case actually needs it.
|
||
|
||
Detector primitives, generalize later if needed:
|
||
|
||
```
|
||
load_template(...)
|
||
match_template(...)
|
||
find_best_match(...)
|
||
find_and_click_template(...)
|
||
color_mask(...)
|
||
debug_write_match(...)
|
||
```
|
||
|
||
### Phase 5: Mailbox migration
|
||
|
||
**Status: Done.** Live testing surfaced a real bug: the old `MAILBOX_ICON` coordinate `(1726, 60)` sat on the edge of the icon's hitbox and intermittently missed, and the fixed click sequence had no way to notice — it cascaded into pressing Escape on the bare home screen, which triggers Blue Archive's own "exit the game?" confirmation (dismissed safely with Cancel during testing; no game state was lost). The Python port in `ba_auto/tasks/mailbox.py` fixes the coordinate and, following `module/mail.py`'s `rgb_in_range` pattern, verifies the panel actually opened (and whether "claim all" is disabled) via `driver.color_at` before pressing any further keys, with a bounded retry and a safe abort if the panel never appears.
|
||
|
||
Move mailbox logic from Bash to:
|
||
|
||
```
|
||
ba_auto/tasks/mailbox.py
|
||
```
|
||
|
||
The CLI should call it through Python.
|
||
|
||
### Phase 6: Cafe migration
|
||
|
||
**Status: Done.** Same root cause as the mailbox bug (Phase 5), confirmed by step-by-step live replay with screenshots: `CAFE_ICON` clicks are flaky (missed on the first attempt, worked on retry at the identical coordinate — this is xdotool/Proton click-registration flakiness, not a coordinate-precision problem), and the old sequence had zero verification across its ~12 steps (open → dismiss notice → pat loop ×15 → switch room → dismiss notice → pat loop ×15 → claim income ×2 Enter → ×2 Escape). A missed click anywhere cascades into blind actions on whatever screen is actually showing, which — same as mailbox — very likely ends with an unverified Escape hitting the home screen and triggering Blue Archive's own exit-game confirmation.
|
||
|
||
`ba_auto/tasks/cafe.py` now verifies state at every transition using `driver.color_at`, following `module/cafe_reward.py`'s `picture.co_detect`/`rgb_in_range` pattern:
|
||
|
||
- opening the cafe icon and the room-switch button both retry (bounded) and confirm the panel actually opened via the same subscreen-header probe as mailbox, now shared in `ba_auto/navigation.is_on_subscreen`
|
||
- the "visited student list" notice that appears on every room entry is dismissed with Enter; this is harmless as a no-op if no popup is actually present (verified live), so no separate presence check was needed there
|
||
- the income dialog's own dimmed-overlay backdrop is checked (`navigation.is_modal_open`) before pressing Enter to claim, and the "receive" button's disabled-grey color is checked before attempting to claim at all (mirrors `collect()`'s `rgb_in_range` gate in the reference)
|
||
- the closing Escape(s) only fire when a subscreen/modal is confirmed still open, never blindly
|
||
|
||
Verified live (two full runs against the real game, plus a manual step-by-step replay of every transition):
|
||
|
||
- both rooms open and pat correctly
|
||
- sparkle detection still works and now runs in-process (see Phase 4) instead of shelling out per click
|
||
- cafe income claim works (confirmed gold +81,251 / AP +61 on an actual claim) and correctly no-ops when there's nothing to collect
|
||
- the reference's `zoom_out` step (camera zoom before sparkle detection — CLAUDE.md's "view centering/zoom" gap) was **not** ported: detection matched at 0.99 confidence without it in live testing, so it wasn't reproducibly broken here. Left as a documented open risk below rather than added speculatively.
|
||
|
||
Not verified / open risks:
|
||
|
||
- whether zoom/pan state could drift over a long unattended run and eventually break sparkle detection (see above — no evidence of this yet, but the reference project treats it as necessary)
|
||
|
||
#### Phase 6 follow-up: rank-up popups mid-pat-loop ("it will freeze a bit")
|
||
|
||
A user report during real usage: a pat that causes a bond-rank-up makes the loop "freeze a bit." Confirmed as a real, previously-unhandled gap, not a timing fluke — `find_cafe_sparkle()` was being asked to recognize a full-screen "絆ランクアップ!" cutscene (no cafe header, no chrome at all — see `screenshots/cafe/student/01`/`02`) as if it were the sparkle template, which it obviously never matches, so the loop just spun `driver.wait(1)` uselessly for the rest of the room's click budget. The reference's own `to_cafe()` navigation (`module/cafe_reward.py`) already treats `relationship_rank_up` as a recognized, reactively-dismissed popup checked after every pat round — this project's port had never carried that over.
|
||
|
||
Fix: `cafe.py`'s `_dismiss_rank_up_if_shown()`, called after every pat (click + Enter + move-mouse), reuses the *existing* `navigation.is_on_subscreen` header-brightness probe rather than adding a new one — directly confirmed against the user-provided screenshots: the header probe point reads `(183, 220, 240)` during the cutscene (r<200, fails the check) vs. `(248, 249, 250)` on the normal cafe screen (r>200, passes). Presses Enter (bounded, `config.CAFE_RANK_UP_DISMISS_RETRIES = 5`) until `is_on_subscreen` confirms the cafe room is back, rather than assuming one Enter is enough; if it never clears, the pat loop stops rather than continuing to click blindly.
|
||
|
||
Not yet live-confirmed against a real rank-up trigger — it's semi-random (tied to hitting an affection threshold) and didn't happen to occur during this session's testing. The fix is grounded in the user's own captured screenshots (a real observed state, precisely measured), not a guess, but a live run actually hitting this path and recovering cleanly is still open.
|
||
|
||
#### Phase 6 follow-up: "farming affection doesn't happen" report
|
||
|
||
A later report claimed pats weren't landing at all, with the original `ba_dailies.sh` `do_cafe_room`/`do_cafe` pasted as the expected-behavior reference. Re-reading that Bash carefully changed the diagnosis: the original `detect_and_click.py` did one screenshot → detect → click per invocation and the Bash loop only kept calling it back-to-back while hits kept landing, breaking immediately on the first miss (`grep -q "^MATCH" || break`) — i.e. give-up-on-first-miss was the *original design*, not a regression introduced by the Python port. Detection math (mask, threshold `0.97`, click offset `(75, 47)`) ported over byte-for-byte identical.
|
||
|
||
Changes made this round:
|
||
|
||
- `ba_auto/detector.py`: `find_cafe_sparkle()` now tries multiple template scales (`SPARKLE_SCALES`) instead of one fixed size, since the cafe camera's zoom isn't reset before farming and isn't guaranteed to match whatever zoom the template was captured at. Strictly more permissive than the original single-scale match — no observed downside — but not confirmed as the actual root cause of the report (no live zoom-mismatch case was reproduced/observed).
|
||
- `ba_auto/tasks/cafe.py`: `_pat_room` now polls for the full `CAFE_MAX_CLICKS_PER_ROOM` budget with a 1s wait between misses instead of breaking on the very first miss. This is a deliberate deviation from the original design (see above) — cheap (adds at most ~15s per room when nothing is available) and covers the case where a screenshot lands mid-animation right after the room transition.
|
||
- `driver.move_mouse` added and called after each pat to park the cursor away from the sparkle area, per `screenshots/cafe/sparkle/02_*_cursor_on_head.png` showing the cursor can occlude the icon.
|
||
|
||
What was directly verified live after these changes:
|
||
|
||
- the room-entry/no-modal state probes (`navigation.is_on_subscreen`, `is_modal_open`) read correctly on real captured frames from both rooms
|
||
- neither room had a visible sparkle on any student at the time of testing (confirmed by eye on the actual screenshots, not inferred from the "no sparkle found" log) — this is the most likely explanation for why a same-session automated run kept reporting no matches: this session's own manual+automated testing had already consumed the available per-student affection interactions, which regenerate on a real-world cooldown far longer than one room visit
|
||
|
||
**Not yet verified**: an actual end-to-end pat (detect → click → affection-up dialog dismissed) succeeding after this round's changes, because no interactable sparkle was available during testing to exercise it against. Re-run `~/ba_dailies.sh cafe` once interactions have had time to regenerate and confirm `[cafe] patted N sparkle(s)` appears with N > 0.
|
||
|
||
### Phase 7: setup.sh update
|
||
|
||
**Status: Done — `setup.sh` deploys `ba_daily.py` and `ba_auto/`.** `scripts/ba_dailies_legacy.sh` and `scripts/detect_and_click.py` were deleted once mailbox and cafe both migrated off them (Phases 5–6); `setup.sh` no longer references either.
|
||
|
||
Update `setup.sh` so it deploys:
|
||
|
||
```
|
||
ba_dailies.sh
|
||
ba_daily.py
|
||
ba_auto/
|
||
assets/
|
||
```
|
||
|
||
to the expected runtime paths on `nik-gpu`.
|
||
|
||
### Phase 8: Stamina/AP mission claim
|
||
|
||
**Status: Partially done.** Read `module/collect_daily_task_power.py` (the "Tasks" menu claim loop — `to_tasks` + `rgb_in_range` checks against two fixed pixel pairs, click, dismiss, repeat) and `module/collect_daily_free_power.py` (a `picture.co_detect` state-machine walk into the Pyroxene Purchase menu's Package tab to claim a genuinely free 10 AP item). Live reconnaissance on the home screen found the direct local equivalents: a `ミッション` (Mission) icon opening a panel with per-tab claim buttons *and* a single bulk "一括受取" (claim all) button whose keyboard shortcut is literally Enter — much simpler than porting the reference's per-tab color-probe loop. `ba_auto/tasks/stamina.py` opens the Mission panel, checks whether "一括受取" is enabled (bright yellow vs. flat grey background probe at `config.MISSION_CLAIM_PROBE`), and if so presses Enter to claim, Enter again to dismiss the reward-reveal card (same "harmless no-op if absent" assumption as cafe's room-entry dismiss), bounded to a few rounds in case multiple rewards queue up.
|
||
|
||
The 青輝石購入 (Pyroxene Purchase) icon — reference's Daily Free Power entry point — was opened once to confirm the free-claim flow's location, but turned out to be a real-money purchase menu (¥3,000–¥4,900 package buttons visible immediately) with the genuinely-free item buried in a further tab. Given `plan.md`'s own purchase-safety rules ("avoid unbounded spending", "avoid buying unknown items"), this was **deliberately not automated this round** — the dialog was closed without navigating further. Treat this as a separate, explicitly-confirmed piece of future work, not an oversight.
|
||
|
||
Two real bugs found and fixed during live calibration, both worth remembering for future coordinate-hunting:
|
||
|
||
- **Visual gridline coordinate estimates were wrong twice in a row.** Reading icon bounds off a scaled/annotated screenshot crop by eye put the Mission icon's center at `(146, 352)` — which is actually in the dead space between the Mission and Pyroxene-Purchase icons, close enough to the latter's edge that clicks there landed on Pyroxene Purchase instead. The fix was sampling actual pixel colors along a scanline (`img.getpixel`) to find each icon's true left/right edge against the background, rather than eyeballing gridlines — this put the real center at `(75, 350)`, squarely inside the icon graphic, confirmed live. Lesson: for icon coordinates, prefer a pixel-boundary scan over a visual grid-overlay estimate.
|
||
- **`driver.click()`'s combined `xdotool mousemove X Y click 1` invocation is unreliable; splitting it fixed a chunk of this project's long-documented click flakiness.** Repeated single-click tests at a *verified-correct* coordinate still missed intermittently until the mousemove and click were issued as two separate `xdotool` calls with a short (0.2s) pause between them — after that, every subsequent click registered. This plausibly explains some of the "icon click missed on the first attempt, worked on retry" flakiness documented in Phases 5–6 (mailbox/cafe icons). Applied to `driver.click()` itself (project-wide, since all tasks share it) rather than special-cased in `stamina.py`; regression-tested live against `mailbox` and `cafe` after the change — both still work.
|
||
|
||
Not yet done: Group/Club AP, and Daily Free Power (see above).
|
||
|
||
### Phase 9: Normal/Hard story AP sweep
|
||
|
||
**Status: Done.** Read `module/explore_tasks/sweep_task.py` and `module/explore_tasks/task_utils.py` — the reference flow reads the current region number and matches stage-name text via OCR (`swipe_search_target_str`) to navigate to a configured target stage, then runs a per-stage claim loop. This client exposes a much simpler path to the same goal (burn AP via already-3-starred stages) that avoids porting the OCR-based lookup entirely: each stage's own 任務情報 (task info) modal has a self-contained 掃討 (sweep) sub-panel with a MIN/-/+/MAX count stepper and a start button.
|
||
|
||
Per explicit user direction on target selection: rather than a fixed configured stage (the plan's original "suggested first version"), `ba_auto/tasks/story_sweep.py` gets the *latest unlocked* region by spamming the "next region" arrow until it stops advancing (a plain state-change check, no OCR — clicking past the last region is a harmless no-op, verified live), then picks one of that region's stages essentially at random (`_pick_random_stage_row`: scroll the stage list to one of its two extremes at random, then pick a random one of the 4 visible rows there — not perfectly uniform since middle stages are reachable from both extremes, but avoids OCR/generic scroll-enumeration). AP spend is bounded by the in-game MAX button per explicit user direction (no additional cap layered on top).
|
||
|
||
Three real bugs were found and fixed during live calibration, all specific to the fact that this task spends real AP (unlike every other task so far, which only claims free rewards):
|
||
|
||
- **`navigation.is_modal_open`'s default probe `(960, 200)` false-negatives on this modal.** The 任務情報 modal is wide enough that `(960, 200)` lands on the modal's own white card, not the dimmed backdrop. Fixed with a task-specific `config.STAGE_MODAL_PROBE = (1870, 600)` and a local `_is_stage_modal_open()` check. First live test aborted safely on this false negative (correctly spent 0 AP) before the fix.
|
||
- **The MAX button click was never verified, and silently under-delivered.** A live run completed without error but only spent ~10 AP (one sweep) instead of the ~190 AP a real MAX (19 sweeps) should cost — diagnosed by comparing the actual AP/gold delta against the AP preview text seen during manual calibration. Root cause: the same general click-flakiness documented in Phase 8, just unverified here because nothing checked it. Fixed with `_count_raised_above_one()`: the sweep count's "-" stepper button is flat grey at the default count of 1 and turns vivid orange once raised, so probing `config.SWEEP_MINUS_BUTTON_PROBE` after the MAX click cheaply confirms it landed, without needing OCR on the count itself. Wrapped in a bounded retry (`MAX_BUTTON_RETRIES = 3`), aborting with zero AP spent if it never confirms. A subsequent live test hit 0/3 on this retry (a real flakiness cluster, not a logic bug) and correctly aborted without spending; the very next live run succeeded on attempt 1 with a genuine MAX (count 1→19, AP 191→1 confirmed by screenshot), so the retry+verify mechanism does its job on both sides — safe abort on failure, correct spend on success.
|
||
- **Escape does not close this modal, and the fallback dismiss loop was a latent hazard.** After a real sweep, the post-sweep dismiss loop pressed Enter a fixed number of times to clear reward-summary popups; live testing showed that once those popups run out, the *same* underlying 任務情報 modal reappears — and its Enter hotkey is bound to the live "任務開始" (start manual mission) button, not a no-op. The original fixed round count (3) happened to land exactly on the modal's reappearance without going further, but a different reward-popup count on another run could just as easily have pressed one Enter too many and started a real manual battle attempt. Two fixes: `_dismiss_sweep_result` now checks `_is_stage_modal_open` before every Enter press and stops immediately once the modal reappears, instead of trusting a fixed count; and closing now happens via a new `_close_stage_modal()` that clicks the modal's own X button (`config.STAGE_MODAL_CLOSE_BUTTON`, pinned via pixel-scanline scan of the glyph, not visual estimate) with a bounded retry+verify, since two Escape presses were confirmed live to leave the modal open. If the X-click ever fails to confirm closed, the task logs a warning and stops rather than pressing any further keys blindly.
|
||
|
||
Verified live: work-hub → task-screen navigation, latest-region advance, random stage pick, stage-modal-open detection, MAX click+verify, a genuine MAX sweep (19 runs, AP 191→1, gold +9,144), and the modal-close-via-X-button fix (confirmed via direct scripted click that it reliably closes and returns to the stage list). Not yet re-verified end-to-end in one single run: the fixed dismiss-loop-then-X-close sequence together, since AP was down to 1/240 after the successful test and there wasn't a further real sweep available to test against before the fix was deployed — each half was verified independently instead. Re-run `~/ba_dailies.sh story_sweep` once AP has regenerated to confirm the full sequence end-to-end.
|
||
|
||
`story_sweep` is deliberately **not** in `ba_daily.py`'s `DEFAULT_ORDER` — it spends AP on a randomly-picked stage rather than reclaiming something free, which is a real resource decision the default unattended run shouldn't make blindly. It must be invoked explicitly (`~/ba_dailies.sh story_sweep`).
|
||
|
||
**Retrospective — OCR avoidance was a mistake here.** Three of this phase's four live bugs (wrong modal probe, unverified MAX click, Escape-doesn't-close-modal plus the latent accidental-battle-start hazard) trace back to one decision: avoiding the reference's OCR-driven, deterministic stage targeting in favor of a heuristic substitute (random-pick + pixel-probes). A deterministic "go to configured stage X" flow, ported from the reference the way `module/explore_tasks/sweep_task.py`/`task_utils.py` actually do it, would not have needed to guess whether a modal opened via an easily-mismatched color probe, nor would it have left ambiguity about what's under the cursor when dismissing reward popups. This project's policy is now to port the reference's OCR-driven logic when the reference uses OCR for a feature, rather than inventing a non-OCR substitute to avoid the setup cost (see `CLAUDE.md` → "OCR policy"). `story_sweep.py`'s random-stage-pick design is not being reverted retroactively without user direction, but any future rework of this task should prefer porting the reference's actual region/stage-name OCR matching over the current random-pick approach.
|
||
|
||
### Phase 10: story_sweep OCR/state-machine port (supersedes Phase 9's random-pick design)
|
||
|
||
**Status: Done.** Acted on Phase 9's retrospective: set up OCR for real (`pytesseract` + the `tesseract-ocr` apt package) and ported the reference's actual deterministic stage targeting, replacing the random-pick heuristic. See `Handoff.md`'s history (deleted once this phase landed) for the full brief; summary of what changed:
|
||
|
||
- **OCR primitive**: `ba_auto/detector.py`'s `read_text()`/`read_int()` crop a screenshot to a pixel rect, threshold it to pure black/white (this measurably fixed real digit misreads that survived every `psm` mode when left anti-aliased — see below), upscale 3x, and run `pytesseract`. `lang="eng"` is enough; the region-number and stage-label reads are pure digits/dashes, no Japanese trained data needed.
|
||
- **Config-driven targets**: `config.STORY_SWEEP_TARGETS = [(region, stage, count_or_"max"), ...]`, mirroring the reference's `unfinished_normal_tasks` shape. Ships with a placeholder `(1, 1, "max")` entry per explicit user direction — edit it to your own already-cleared stage(s) before running for real.
|
||
- **Deterministic region navigation** (`_go_to_region`, porting `task_utils.py::to_region`): OCR the region-number readout, click the exact left/right-arrow delta, re-check, bounded loop. Region-arrow presence (locked/last-region detection) is a `detector.region_contains_color` box scan, not a single fixed point — a centroid-derived single point landed in the concave notch of the "<"/">" chevron and read "absent" even while the arrow was clearly rendered a few pixels away.
|
||
- **Deterministic stage search** (`_find_stage_row`, a scoped-down `swipe_search_target_str`): OCR each of the 4 visible stage-row labels at both already-calibrated scroll extremes, matching by the label's suffix after the dash (e.g. "2" in "30-2") rather than the full string — the font's leading region digit reads unreliably even after threshold preprocessing (e.g. "3" as "2"), but the suffix read correctly on every row tested, and the region digit is redundant anyway since `_go_to_region` already confirmed it independently.
|
||
- **Scoped `co_detect` port**: `navigation.wait_for_state(driver, config, reactions, ends, max_iterations)` — checks named `ends` first each iteration (stop, return the name), then named `reactions` (run an action, keep polling), else waits and retries up to a bound. Generic and reusable beyond this task.
|
||
- **Named outcomes**: `_sweep_target` returns `"swept"`, `"inadequate_ap"`, `"region_unavailable"`, `"stage_not_found"`, or `"unrecognized_state"` instead of a single generic "Done".
|
||
|
||
Four real, live-discovered findings, on top of what Phase 9 already found:
|
||
|
||
- **Regular numbered stages (30-1..30-5) render a different, taller modal layout than the "-A" bonus stage Phase 9 exclusively calibrated against.** Regular stages add a "集中指揮"/"簡易攻略" tab row and a manual "任務開始" panel below the sweep sub-panel that "-A" doesn't have. Phase 9's `SWEEP_MAX_BUTTON`/`SWEEP_MINUS_BUTTON_PROBE`/`SWEEP_START_BUTTON` all missed by ~40-46px vertically against a real numbered stage (30-3) — caught live when the MAX-click retry correctly failed 3/3 and aborted without spending AP, rather than silently misfiring. Re-calibrated against the tabbed layout via pixel-scanning (not eyeballing); the "-A" layout's original Phase 9 coordinates are no longer what these constants hold, so a future sweep of a "-A" stage specifically would need its own re-check.
|
||
- **The modal's own X-close button also moves with the layout.** Not just the sweep sub-panel — the whole card is vertically positioned by its own content height rather than anchored at a fixed absolute position, so the X button sits at a different absolute Y (225 vs Phase 9's 271) in the taller tabbed layout. Caught live: `_close_stage_modal` correctly reported "not closed" (3/3 retries) against the stale coordinate, rather than silently believing it had closed.
|
||
- **Clicking 掃討開始 always raises an AP-usage-confirmation dialog the design had never accounted for at all.** ("APを`N`使用して、掃討を`M`回行いますか?", OK/Cancel.) A first live attempt at porting the "wait for outcome" step read this dialog's dimmed backdrop as a false "inadequate_ap" through an early, unverified placeholder probe — worth remembering: an uncalibrated placeholder check can be actively *wrong*, not just inert, if given a chance to run before it's confirmed. Fixed by explicitly clicking through this confirmation before watching for the real result.
|
||
- **Genuine insufficient-AP is a visually near-identical dialog at the exact same OK-button position, told apart only by color.** Deliberately triggered live (by emptying the sweep count via MAX/"+" at low AP) rather than guessed: a real "AP不足" case shows a dialog titled "AP購入" (spend real Pyroxene to buy more AP) whose OK button is gold/yellow, vs. the safe usage-confirm's cyan — same position, different color. `_is_ap_purchase_prompt`/`_is_sweep_usage_confirm` tell them apart by that color and only ever click the cyan one; the gold one is always cancelled, never clicked, matching this project's purchase-safety rules.
|
||
|
||
Also fixed in passing, found only because live testing exercised the actual home-screen click path repeatedly: `config.TASK_CARD`'s old coordinate `(1370, 450)` sat close enough to the 任務 card's bottom edge that one run missed and landed on the "総力戦" (Total War) card below it instead — confirmed via screenshot, safely backed out with zero AP spent, moved to `(1250, 380)` (squarely on the "任務" title text).
|
||
|
||
Verified live end-to-end at least once, real AP spent: a genuine 5x sweep of 30-3 (AP 53→~5, confirmed via the "掃討完了" results screen's reward totals), including clicking through the usage-confirm dialog, the SKIP animation-skip screen, and the final reward-totals OK, landing back on the bare stage-info modal afterward. The genuine insufficient-AP path was also verified live (correctly cancelled the real "AP購入" purchase prompt without spending Pyroxene). Not yet re-verified end-to-end with the final rewritten code specifically (the color-based dynamic button-finding in `_watch_sweep_result`) at a nonzero AP balance — the manual walkthrough that discovered the dialogs used direct scripted clicks before the code was rewritten to match; the rewritten code's color-matching logic was separately verified offline against the exact screenshots captured live (all four dialog states correctly classified), but a fresh live run once AP regenerates would close that last gap. Not verified: sweeping a "-A" bonus stage (needs its own layout re-check, see above), an integer (non-"max") configured count actually being clicked via `SWEEP_PLUS_BUTTON`, and Hard-mode tab stages.
|
||
|
||
**Phase 10 follow-up (user-reported):** the user ran `story_sweep` for real and it spent AP on region 1 stage 1 instead of region 30 (their actual current last region). Not a code bug — `config.STORY_SWEEP_TARGETS` still held the literal placeholder `(1, 1, "max")` shipped with this phase, and the user hadn't edited it yet. Rather than just filling in one static `(30, N, "max")` entry, the user asked for the stage within region 30 to rotate daily across all 6 of that region's stages instead of grinding one fixed stage every run. Added `config.STORY_SWEEP_ROTATION_REGION`/`STORY_SWEEP_ROTATION_STAGE_COUNT`/`STORY_SWEEP_ROTATION_COUNT` and `story_sweep._rotation_target()`, which computes `(region, stage, count)` from `datetime.date.today().toordinal() % stage_count` — a plain date-ordinal modulo rather than calendar day-of-year, so the 6-day cycle doesn't skip or repeat around a year boundary. This target is appended to (not a replacement for) whatever's in `STORY_SWEEP_TARGETS`, which is now empty by default. Verified the computed target offline (region 30, stage 6, on the date this was fixed) but not yet re-run against the live game since AP hadn't regenerated.
|
||
|
||
### Phase 11: Common Shop + Tactical Shop
|
||
|
||
**Status: Done.** Ported `module/shop/common_shop.py` / `module/shop/tactical_challenge_shop.py`'s `implement()` and the shared `module/shop/shop_utils.py` (`to_common_shop`, `get_item_position`/`ensure_choose`/`buy`) to `ba_auto/tasks/shop_common.py` / `shop_tactical.py`, sharing control flow through a new `ba_auto/tasks/shop_utils.py`.
|
||
|
||
Key design call, made after live-capturing both shop tabs before writing any code: the reference's own item identification inside the grid is **not** OCR-based. `get_item_position` scans fixed pixel columns and matches a purchasable-state color plus a currency-icon template, then maps that grid position to an item identity via `self.static_config.common_shop_price_list` — a table sourced from an external resource this repo doesn't contain (the reference dataclass just declares the field; the actual values are fetched elsewhere, at BAAS's own runtime). So porting this feature couldn't mean "OCR the item names" (the reference doesn't do that either) — it meant building our own local equivalent of that static table by live-capturing the real catalog and letting the user pick their buy list from it, same shape as `STORY_SWEEP_TARGETS`. `config.COMMON_SHOP_TARGETS` / `config.TACTICAL_SHOP_TARGETS` are `(row, col, item name (comment only), expected price)` tuples; identification is by fixed grid position, with price-digit OCR (something the reference doesn't even do per-item) layered on as an extra live-catalog-drift safety net, consistent with this project's verify-before-spend pattern elsewhere.
|
||
|
||
What was found live, captured before writing any code (see the session's live-capture screenshots, not kept in the repo):
|
||
|
||
- **Both shop tabs share one UI**: a 4-column checkbox grid per item, then a single bulk "購入" (Buy) button that appears once ≥1 item is checked, rather than the reference's per-item purchase flow. Checked state renders a distinct vivid yellow-green on the checkbox glyph, easily told apart from the plain white/grey unchecked state by `detector.region_contains_color` — no template matching needed.
|
||
- **The tactical shop's tab list (7 entries) fits on screen with no scrolling**, so `config.SHOP_TAB_TACTICAL` is a fixed click rather than a port of the reference's `goto_shop_by_name` OCR swipe-search — there's nothing to search for on this account, so a fixed click is the faithful choice here, not a shortcut around OCR (see `CLAUDE.md`'s OCR policy: only skip OCR where the reference's own need for it doesn't apply).
|
||
- **Price-digit OCR needed real calibration**, same as Phase 10's stage labels: a rect wide enough for the widest configured price (500,000) without also catching the neighboring column's card, and narrow enough on the left to exclude the currency icon (which OCR otherwise misreads as a spurious leading digit — confirmed live, e.g. the top-bar credit balance read "454920755" instead of "154920755" until the icon was excluded).
|
||
- **One shared corner-pixel probe (`config.SHOP_OVERLAY_PROBE`, a bottom-left point) detects both the purchase-confirm dialog and the post-purchase "報酬獲得!" (reward acquired) banner** — both dim it away from pure white; it stays pure white with nothing open, confirmed stable across tab switches and scrolling. `shop_utils.confirm_purchase` presses Enter in a bounded loop until it reads idle again, rather than tracking each dialog's own layout individually.
|
||
- **Live-tested with real purchases, not just offline-verified.** With the user's explicit buy lists confirmed first (8 Common Shop items: 初級/中級/上級/最上級レポート + 初級/中級/上級/最上級強化珠; 2 Tactical Shop items: 初級/中級栄養ドリンク), both shops were run for real: Common Shop total cost matched the pre-calculated 1,211,500 credits exactly (confirmed against the game's own confirmation-dialog total); Tactical Shop's AP gain (+90) and coin spend (-45) matched exactly. Both purchases resolved cleanly back to an idle screen via the overlay-probe loop.
|
||
- **Discovered live, not anticipated going in: these shop items have a per-refresh-cycle purchase cap that isn't shown as a visible counter** (unlike the Pyroxene-shop tab's explicit "あと1回購入可能" labels — Common Shop items just look normal until you've already bought them, then go quietly unresponsive). Found by re-running the actual `shop_common` CLI task shortly after the manual purchase above: every configured item's checkbox failed to register as checked, and a direct test of the individual per-item "購入" button and the page's own "全て選択" (select-all) control confirmed those specific items are genuinely non-interactive right now (select-all successfully picked up *other*, not-yet-purchased items further down the list), while credits stayed unchanged throughout. The task correctly reported "nothing selected, cancelling" and spent nothing, rather than misfiring — a real edge case the price-verify + checkbox-confirm design caught safely, not a bug in it. Net effect: a fully "fresh, everything-available" unattended run hasn't been re-verified end-to-end today, since this account's cycle allowance for these specific items was already spent via the same session's manual calibration. Next run after the shop's own refresh timer (~5h, shown in-game as "更新まで") should exercise that path for real.
|
||
- **Not verified**: non-fully-visible target rows requiring a scroll (both current buy lists happen to be fully visible without scrolling, so `shop_utils` has no scroll/pagination logic yet — would need porting `buy()`'s `last_checked_idx`/swipe-diff tracking if a future buy list needs it); the pre-purchase "insufficient assets" abort path (both currencies were comfortably sufficient this run); the paid manual shop-refresh flow (`更新` button) — deliberately not automated for v1, same reasoning as Daily Free Power (real-currency spend needs explicit human intent, not a default unattended path).
|
||
|
||
### Phase 12: Lesson/Schedule
|
||
|
||
**Status: Done.** Ported `module/lesson.py`'s control flow (`implement`, the `to_*` navigation state machine, `get_lesson_each_region_status`/`get_lesson_relationship_counts`, `choose_lesson`, `execute_lesson`) to `ba_auto/tasks/lesson.py`.
|
||
|
||
Scope, decided with the user before writing any code (see the two `AskUserQuestion` answers): affection-first selection (mirrors `lesson_relationship_first=True` — matches this feature's own "affection farming" framing, not raw reward-tier grinding), sweep every unlocked region in a fixed order until either lesson tickets or scoreable lessons run out (no per-region target list needed from the user, unlike shop's per-item buy list), no lesson-ticket purchasing and no favor-student targeting (both deferred, matching plan.md's original "Suggested first version").
|
||
|
||
Key finding before writing any code: **this client's UI is structurally the same two-level layout the reference describes (12 named regions, each with up to 9 individual lesson locations) but rendered completely differently** — a scrollable list of regions instead of the reference's paged single-region swipe view, and a clean "すべてのスケジュール" grid-card modal instead of the reference's raw isometric map + `Parallelogram`/`Triangle` pixel-scanned status grid. Two consequences:
|
||
|
||
- **No OCR is needed for region navigation at all.** The reference OCRs the current region name because its paged arrows leave position ambiguous. This client's region list only ever settles at two scroll positions (scrolled to top: regions 0-5; scrolled to bottom: regions 6-11, confirmed live — repeated scroll-down clicks don't keep scrolling past this), so navigation is just "scroll to the right state, click the row at a fixed Y" — deterministic, nothing to locate. The reference's own 12 JP region names (`core/config/default_config.py`'s `lesson_region_name.JP` — embedded directly there, not fetched externally like the shop price table) are kept locally purely for log readability, confirmed to match this account's list exactly, row for row.
|
||
- **No isometric geometry is needed for per-cell status either.** Each grid-modal card shows up to 3 student portraits with a heart-shaped affection-count badge — reading that number via OCR needs no geometry at all, replacing the reference's isometric pixel-scan outright.
|
||
|
||
Three real bugs were found and fixed via a live test run (5 real tickets spent across 3 regions, ticket accounting and cleanup navigation both confirmed correct throughout):
|
||
|
||
- **A checkmark does not blank the badge number — it renders alongside it.** The initial design assumed an "already done today" portrait would fail the digit-whitelisted OCR read (no number left to read), the same way a "no relationship yet" portrait does, and treated both as identically "not a candidate." Live testing showed this was wrong: a done portrait keeps its unchanged number and gets a small green checkmark added at top-right instead. Without separately checking for that checkmark, the same already-done cell could be re-picked immediately after being completed. Fixed by detecting the checkmark directly via its own fixed color/offset (`config.LESSON_GRID_CHECKMARK_*`, `lesson._is_slot_already_done`) rather than inferring done-ness from the OCR read.
|
||
- **The badge's own OCR read was unreliable, and not for the reason first suspected.** Reusing the project's existing generic `detector.read_int` (grayscale + hard threshold, tuned for this UI's normal dark-text-on-light-card look) on the pink/magenta heart badge produced wildly oversized results live — "13" read as "113", "19" as "119", "18" as "418". Root cause, confirmed by rendering the exact same threshold step locally: the heart's own darker outline stroke has a grayscale value that happens to land on the same side of the threshold as the digit glyph, surviving as stray black marks that tesseract sometimes fuses into extra leading digits. Fixed with a dedicated `detector.read_int_on_heart_badge`, which masks on the color relationship "R < G" (true for the navy digit glyph in every sample, false for every pink/magenta badge tone, light or dark) instead of raw brightness — this alone fixed most cases. A few slots still occasionally fuse a stray digit from the character's own portrait art bleeding into the crop's edge (this is character-art-dependent, not fixable by OCR config alone — confirmed reproducible across every psm mode tried). Rather than chase perfect crop geometry per-character, `config.LESSON_GRID_BADGE_MAX_PLAUSIBLE` discards any reading of 100+ as certain contamination (real affection values never reach that range in practice) — a second line of defense, not the primary fix.
|
||
- **A transient "2x schedule" campaign event (active on this account, ~6 days remaining) doubles how many intermediate screens appear after starting a lesson**, and a bond-rank-up cutscene can appear too (same full-screen "絆ランクアップ!" style already handled in `cafe.py`, confirmed live here for the first time against a real trigger). Rather than special-case either, `lesson._run_one_schedule` presses Enter in a bounded loop, checking two fixed markers each round: the grid modal's own title underline color (visible only when it's frontmost and idle — both the results modal and the cutscene cover it, confirmed live against screenshots of all three states) as the "done" signal, and the results modal's OK button color (shared with the info panel's Start button, confirmed by direct pixel sample) as the one intermediate state worth clicking precisely rather than folding into the blind-Enter fallback.
|
||
|
||
Also confirmed live: the "保有チケット N/M" ticket counter is directly visible on the region-list screen (no submenu needed, unlike the reference's `to_purchase_lesson_ticket`/OCR-a-modal approach) — `lesson._read_ticket_count` just OCRs it directly and re-reads it after every schedule to track spend, rather than assuming exactly 1 ticket per schedule (the 2x campaign event doesn't change the ticket cost, only the reward/cutscene count, but re-reading rather than assuming keeps this robust either way).
|
||
|
||
**Not verified**: a region with more than 9 currently-unlocked locations (would need scroll support inside the grid modal — not implemented, not yet seen on this account, both regions tested topped out at 7-8); the "no lesson tickets" abort-immediately path (tickets hit exactly 0 mid-sweep during the real test, not at the start); a full 12-region sweep in one run (the test's 5 tickets ran out partway through region 3 of 12); a truly fresh "everything available, nothing done yet" run (this account had already done some lessons manually during calibration before the automated run started). Worth re-running `~/ba_dailies.sh lesson` after tickets next refill to exercise the untested tail of the region list.
|
||
|
||
### Phase 12 follow-up: ticket-count OCR regression fix
|
||
|
||
Between sessions, `~/ba_dailies.sh lesson` started failing immediately every run with `[lesson] could not OCR ticket count, aborting without pressing further keys` — a full regression of a read that worked during Phase 12's own live test.
|
||
|
||
Root cause, confirmed by pulling a live screenshot back to `scratchpad/` and testing `detector.read_text` against the exact configured rect in isolation: `LESSON_TICKET_OCR_RECT`'s left edge (`x=295`) clipped in a stray few pixels of the "チケット" label's own trailing katakana glyph, immediately adjacent to the first digit. That fragment was enough to make tesseract drop the whole leading digit from its read — `"7/7"` came back as `"/7"`, confirmed reproducible across a fresh screenshot every single time (not a one-off render glitch, not a transient game-side layout change). Splitting `"/7"` on `/` gives an empty head, so `_read_ticket_count` returned `None`, exactly matching the failure.
|
||
|
||
Fixed by tightening the rect to `(315, 135, 375, 170)` — pixel-column analysis of the crop located the actual digit glyphs' bounding box and the new rect starts past the stray fragment. Re-confirmed stable across 6 consecutive fresh OCR reads before deploying, then validated with a full real run: 7 real lesson tickets correctly spent, ticket count correctly re-read after every single schedule (7→6→5→4→3→2→1→0), clean stop at 0, and a clean return to the home screen confirmed via a final screenshot.
|
||
|
||
This is the same class of fragility as the heart-badge OCR contamination fix from the original Phase 12 writeup above (a plausible-looking crop still picking up unrelated UI content at its edge) — worth keeping in mind for any other tightly-cropped OCR rect in this project if a similarly "worked before, fails now" regression shows up elsewhere.
|
||
|
||
### Phase 12 follow-up #2: schedule-icon navigation regression fix ("stuck on a region map")
|
||
|
||
Immediately after the ticket-OCR fix above, a full real run (`./ba_dailies.sh lesson`) still failed completely — this time past ticket reading (`starting tickets: 1` printed correctly), but `[lesson] schedule grid not detected for region index N (attempt 1-3/3)` for literally every region in sequence, 0 through 8, until the user Ctrl-C'd. The user's own suspicion was that `driver.click` was doing a click-and-hold instead of a single click; `driver.click`'s implementation was checked and is an ordinary `mousemove` + `click 1`, unchanged — that wasn't it.
|
||
|
||
Root cause, confirmed live by screenshotting the actual screen right after clicking `LESSON_ICON`: **the game does not always land on the Location Select list when the schedule icon is clicked.** It remembers the last-viewed region and reopens directly to that region's per-region isometric map instead — confirmed by reproducing the exact scenario (manually opening a region's map, then running the real task without returning to the list first) and seeing the identical failure signature. The previous session's run had been left stuck exactly like this by its own Ctrl-C interruption, mid-sweep, on some region's map — every subsequent `_open_region_grid` call for every region_index then fired its scroll/row-click sequence against that same stuck per-region map screen, which doesn't respond to any of it the way the list does, so nothing ever matched and every region failed identically.
|
||
|
||
This isn't a one-off leftover-state fluke either: since the game itself decides whether the schedule icon resumes on a region or resets to the list, any future interruption (or even manual browsing) before a run could reproduce it again.
|
||
|
||
Fixed with a new `lesson._ensure_location_select_list`, called right after `_open_schedule_screen` confirms a subscreen is open: it checks whether `LESSON_ALL_SCHEDULES_BUTTON`'s position already shows that button's known color *before any row here has been clicked* — that button only exists on the per-region map, never on the list (confirmed by direct pixel sample: same coordinate reads as plain dark background on the list) — and if so, presses `LESSON_BACK_BUTTON` (bounded, `OPEN_RETRIES` attempts) to return to the list before the sweep begins.
|
||
|
||
Validated live in two passes: first, a full real run from the recovered list (7→...→0 tickets from the earlier ticket-OCR fix test) confirmed the list-based flow itself was never broken. Second, the actual regression was reproduced on purpose — manually left the game on a region's per-region map, then ran the real task — and the new recovery step printed `schedule screen resumed on a specific region's map instead of the Location Select list -- returning`, correctly returned to the list, read the account's 1 remaining real ticket, spent it on a real schedule, and finished cleanly with a confirmed clean return to the home screen.
|
||
|
||
This is the same class of finding CLAUDE.md already documents from mailbox/cafe's fixed-coordinate click flakiness and story_sweep's OCR-based navigation: a screen's identity can't be safely assumed from "what button did we intend to click here," it has to be verified — this task's `_open_schedule_screen` had gotten away without that check only because the list had always been the landing screen in every session up to now.
|
||
|
||
### Phase 13: Arena / Tactical Challenge
|
||
|
||
**Status: Done.** Ported `module/arena.py`'s `implement` flow (`get_tickets`, `choose_enemy`, `check_skip_button`, `fight`, `collect_tactical_challenge_reward`) to `ba_auto/tasks/arena.py`. Live-tested for real across all 5 of the account's daily tickets — 3 real fights (2 WIN, 1 LOSE), plus real reward claims (credits, pyroxene, tactical coin all changed as expected).
|
||
|
||
**Scope decisions, made with the user before writing any code** (see the two `AskUserQuestion` answers, 2026-07-09):
|
||
|
||
- Fights exactly **one battle per invocation**, matching the reference's own per-call pacing, rather than looping to spend every available ticket in one run. The reference relies on its always-running background thread rescheduling itself 55 minutes later (`self.next_time = 55`) for the next ticket; this project's one-shot-per-invocation CLI has no equivalent, so repeated ticket spend means rerunning the task (e.g. via cron) rather than an internal loop.
|
||
- Full port including the actual auto-fight (not deferred to a later phase), since this is inherently a real-PvP-consequences feature and a ticket-only/no-fight v1 wouldn't be testable in a way that matters.
|
||
- Reward collection (`collect_tactical_challenge_reward`) runs unconditionally at the end of every invocation, unlike the reference's "only if this was the last ticket" rule — both reward slots are idempotent/harmless to check every time, and there's no scheduler here to guarantee a later invocation will do it.
|
||
|
||
**Real navigation differences found live, before writing the real logic:**
|
||
|
||
- **Tactical Challenge is a card inside the お仕事 (Work) hub** (`config.ARENA_WORK_HUB_CARD`), not a bottom-nav icon on the main page the way the reference's `to_tactical_challenge` assumes — the same "hub card, not a nav icon" pattern already found for story_sweep's task browser.
|
||
- **The reference's two separate screens — opponent-info, then a distinct formation-edit ("攻撃編成") screen — are merged into one modal here**, showing the matchup and the attack-formation button together with a live ticket-count preview (e.g. "5→4") that confirms it's the actual fight-commit step before ever clicking it for real.
|
||
- **`navigation.is_modal_open`'s shared darkness probe reads INVERTED on this specific screen**: the arena list's own background art at that probe point is darker than the modal's white card there, the opposite of every other screen using that probe. `arena.py` has its own `_is_modal_open` using `config.ARENA_MODAL_PROBE` with the opposite rule, rather than silently reusing the shared one wrong.
|
||
- **Self/opponent level OCR** (`choose_enemy`'s reroll logic) is read directly off the list screen — matching the reference's own `self_level_region`/`opponent_level_region`, both read there before any modal opens — not from inside the opponent-info modal.
|
||
|
||
**Real bugs found live** (across the session's 5 real tickets — 2 were spent purely debugging one design mistake, which is exactly why the third fix was validated offline against saved screenshots before risking the last ticket on it):
|
||
|
||
1. **A level-OCR crop that looked legible was still too small for tesseract.** The self/opponent-level "90" digit crop, tight-cropped to just the glyphs, OCR'd as `None` even against a correctly time-matched live screenshot. Debugged by dumping the exact thresholded image tesseract actually sees (`scratchpad/probe_arena_level_ocr*.py`): legible to the eye, but marginal enough at 3x upscale that tesseract succeeded on some rows/psm modes and not others. A few extra pixels of padding on each side fixed it outright across every row, using the same existing OCR pipeline — no upscale/threshold change needed, just a less tight crop.
|
||
2. **Self/opponent-level text is bright-on-dark, the opposite of this project's usual dark-on-light OCR assumption.** `detector.read_int` (built around a plain `THRESH_BINARY`) returned `None` against the profile card's white "Lv.90" text on its dark navy background. Added `detector.read_int_white_on_dark` (the same crop pipeline with `THRESH_BINARY_INV`) rather than reworking the shared path, since every other OCR read in this project genuinely is dark-on-light.
|
||
3. **The post-fight "対戦結果" WIN/LOSE result modal is not reliably detectable by precisely locating its own confirm button.** Three compounding problems, found in this order:
|
||
- A screenshot taken ~2s after Sortie showed the arena list already fully updated (rank, ticket count, a new "待機時間" cooldown) with no result modal visible at all — nearly mistaken for "no result screen needed here, this client resolves fights differently than the reference assumes." Wrong: the modal was still pending and only rendered after the next interaction (confirmed by clicking an opponent row afterward and getting the queued WIN screen instead of that row's own info). Must poll for it explicitly rather than trust a fixed delay, same as the reference's own `fight()` waiting on `arena_battle-win`/`arena_battle-lost`.
|
||
- WIN and LOSE modals are not the same height — WIN shows a reward showcase above its confirm button that LOSE doesn't, so LOSE's confirm button sits noticeably higher on screen. A fixed probe point calibrated only against WIN missed LOSE entirely, which *also* made reward collection silently read "not claimable" right after — it was actually checking the reward buttons while the still-undetected LOSE modal covered them, not a real reward-state problem.
|
||
- Widening the button search into a region spanning both known positions (the same fix already used for story_sweep's own two-position SKIP/OK button) caught WIN and LOSE correctly, but a live rerun then got stuck: the region also matched stray cyan-ish pixels in the opponent list's own portrait art (which changes every list refresh), and a false-positive centroid click there opened a *completely unrelated* opponent's info modal instead of confirming anything. A narrower, offline-revalidated region (checked against every saved WIN/LOSE/opponent-info/plain-list screenshot before risking the session's last ticket on it) hit the *identical* stuck state on the very next live run — the false-positive source is fundamentally unpredictable per-refresh portrait art, not something a fixed region can rule out.
|
||
|
||
Given the opponent-info modal's own attack-formation button is *also* Enter-bound and spends a real ticket, this was a real near-miss of exactly the class of hazard `CLAUDE.md` already documents from story_sweep ("a mistimed keypress could have started a real battle"). Fixed by abandoning per-button color detection entirely: `_wait_for_result` now presses Enter in a bounded blind-retry loop — the same pattern `lesson.py`'s `_run_one_schedule` already uses for its own "variable sequence of post-action screens" problem, and confirmed live to safely dismiss both the WIN modal, the LOSE modal, and an unrelated "リストの更新時間を超過しました" (list-refresh-expired) notice that can also appear if the season list's own countdown lapses mid-fight. A hard safety gate, checked before every single press, stops immediately without pressing Enter if the opponent-info modal is ever detected (via its own gold button at a fixed, reliable position, not a color-region search) — that modal should never legitimately be showing at this point in the flow, and the two prior tickets were spent confirming exactly how a stray click could get there.
|
||
|
||
**Bonus finding, not anticipated going in**: the "戦闘スキップ" (Battle Skip) toggle was already ON by default on this account, confirmed live via pixel-grid-scan. `check_skip_button`'s reroll-if-off logic is ported (`arena._ensure_skip_on`) but its "off" branch has never been exercised live, since no off state has been seen yet to calibrate against.
|
||
|
||
**Not verified**: the "no ticket" popup race the reference guards against (`get_tickets` going stale between the initial read and the attack-formation click — this account's tickets never hit exactly 0 mid-flow during testing); `choose_enemy`'s actual reroll click (`config.ARENA_REFRESH_LIST_BUTTON` is wired up per the reference's logic, but every opponent offered during testing was already an acceptable level, so a real reroll was never triggered); `ARENA_STOP_FIGHT_WHEN_RANK1`'s rank-1 stopping condition (default `False`, and this account's rank never got close enough to test the branch). All three are implemented per the reference's own logic, not guessed at, but none has been exercised against real game state yet.
|
||
|
||
### Phase 14: Event sweep
|
||
|
||
**Status: Implemented, calibrated live against zero real AP spend, NOT yet live-tested with a real sweep.** Ported `module/sweep_activity.py` -> `module/activities/activity_utils.py`'s `activity_sweep`/`start_sweep` to `ba_auto/tasks/event_sweep.py`, per explicit user request (2026-07-10): "the current event has up to 12 stages, randomly choose stage 9-12, same mod%4 date method as story sweep."
|
||
|
||
**Scope decision**: rather than porting the reference's config-string sweep-list parsing (arbitrary stage lists with per-stage float/fraction AP-fraction counts, e.g. `"9,10,11"` x `"0.5,3,1/3"`), this sweeps exactly one stage per run, chosen from a fixed 9-12 sub-range via the same date-ordinal-modulo rotation `story_sweep.py`'s `_rotation_target` already uses (a plain `date.today().toordinal() % span`, not day-of-year, so the cycle doesn't skip/repeat around a year boundary) — matching the user's own framing of the request.
|
||
|
||
**Live calibration, against the currently-running "鉄道爆走事件" (12 stages, all already 3-starred on this account), with zero real AP spent** (every AP-usage confirm dialog reached during calibration was cancelled via Escape, verified by the top-bar AP counter being unchanged before and after):
|
||
|
||
- **Entry point**: the home screen's top-right event countdown badge/thumbnail (`config.EVENT_BADGE_ICON`), not the bottom-left banner slot — that slot was found live to cycle between several unrelated banners (gacha pickups, other campaigns) across screenshots taken seconds apart, and clicking it landed on a *different, already-concluded* event once. The event list icon (イベント一覧) was also checked and ruled out: its "進行中" tab lists side campaigns (schedule-reward bonus, joint firepower drill, etc.), not the main story event.
|
||
- **The stage-info modal is structurally identical to story_sweep's**: same MIN/-/+/MAX count stepper, the same AP-usage confirm dialog (reused directly — `config.SWEEP_CONFIRM_BUTTON`/`SWEEP_CONFIRM_CANCEL_BUTTON`/`SWEEP_CONFIRM_CYAN`/`SWEEP_CONFIRM_GOLD` are pixel-identical live), and the same `SWEEP_RESULT_BUTTON_REGION` SKIP/OK result screen — expected, since the reference's own `activity_utils.py::start_sweep` is a near-duplicate of `sweep_task.py::start_sweep`, reusing the same underlying image names.
|
||
- **Simpler than story_sweep in three ways**, all confirmed live: no region concept (one flat stage list reached via the event's own Quest tab, not a Work-hub-card region browser); the stage list's bottom scroll extreme always reveals stages 9-12 regardless of starting scroll position, with **constant row height** regardless of 1-line vs 2-line title wrapping (no separate top/bottom row-position sets needed, unlike story_sweep's); and the modal has **one fixed layout** confirmed identical across two different stages tested (09 and 12) — no tabbed-vs-plain variant.
|
||
- **One behavior difference from story_sweep found live**: this modal *does* close on Escape (confirmed: AP counter and screen state both correct after Escape-cancelling the confirm dialog and closing the stage modal), whereas story_sweep's needs its own X button. `event_sweep.py` tries Escape first and falls back to the X button.
|
||
- **The "-" stepper's raised-count color signal is subtler here** than story_sweep's vivid-orange: a saturated coral `(251,173,152)` vs flat grey `(171,172,171)`, told apart by channel spread (`max-min > 40`) rather than story_sweep's simpler `r>200 and g<180` rule.
|
||
|
||
**Not yet exercised**: an actual real sweep/AP spend (calibration deliberately avoided this, per `CLAUDE.md`'s "validate offline before spending the resource" guidance — the confirm dialog, cyan/gold color distinction, and result-screen mechanics were all visually confirmed but never clicked through to completion); the reference's SSS-availability gate for a stage that has never been cleared (every stage 9-12 on this account was already 3-starred, so `check_sweep_availability`'s "not yet sweepable, needs a manual fight first" branch was never seen — `event_sweep.py` handles this the same defensive way story_sweep handles an unavailable target: if the MAX-button count-raise can't be verified, it aborts that target without spending AP rather than guessing what an unsweepable stage's modal looks like); behavior once this event ends 2026-07-22 and a future event's badge/stage-count/layout replaces it (the 9-12 rotation range and row-position calibration are specific to this 12-stage event, not proven durable across arbitrary future events).
|
||
|
||
#### Phase 14 follow-up: wrong/stale event page fix + shared `navigation.return_to_home`
|
||
|
||
**Reported by the user after a real run**: `event_sweep` opened an old, already-finished event's page instead of the current "鉄道爆走事件". Root cause: `config.EVENT_BADGE_ICON` (the home screen's top-right event badge) is itself a rotating carousel, not the stable single-event slot calibration happened to suggest — it cycles between the current event's own countdown and OTHER notices, including a finished event's leftover reward-claim-period reminder. Calibration's several screenshots all happened to catch it showing the right event, which masked this.
|
||
|
||
**Fix, in two parts:**
|
||
|
||
1. **New shared primitive**: `navigation.return_to_home(driver)` — a bounded "press Escape, re-check, fall back to clicking the shared back-arrow position, re-check" loop that returns once the home screen is confirmed reached (via `is_on_subscreen` reading `False`). Checks before every single press and never presses blind, matching the project's established "verify before acting" discipline — this specifically avoids the exit-game-confirmation hazard `CLAUDE.md` already documents (a stray Escape on the home screen itself raises Blue Archive's own "exit the game?" dialog). Generic across tasks: it only depends on `is_on_subscreen` and the shared back-arrow position (`navigation.BACK_BUTTON`, confirmed identical across `config.py`'s `LESSON_BACK_BUTTON`/`SHOP_BACK_BUTTON`/the now-removed `EVENT_BACK_BUTTON`), not on any event_sweep-specific state — built specifically so other tasks can reuse it later, per explicit user request ("make the back to home page a module so it can be used by other feature too").
|
||
2. **Wrong-page detection + retry in `event_sweep.py`**: `_find_stage_row` now also reports whether it OCR'd *any* valid stage number at all across the 5 checked rows, not just whether the target matched. Zero valid reads across all 5 is a strong "not actually on a stage list, likely the wrong/finished event" signal (a finished event's Quest tab shows plain "period ended" text instead of stage-row cards) — reported as a distinct `"wrong_page"` outcome, separate from `"stage_not_found"` (right page, this specific stage just isn't visible, which retrying the same badge click won't fix). `run()` now retries the whole entry sequence up to `WRONG_PAGE_RETRIES` (3) times on a `wrong_page` outcome, calling `navigation.return_to_home` and waiting a few seconds before each re-click of the badge, hoping the carousel has moved on by the next attempt. `run()` also now unconditionally calls `navigation.return_to_home` at the very end as a final safety net, replacing the old single unconditional `EVENT_BACK_BUTTON` click.
|
||
|
||
**Not yet live-tested**: this fix has only been syntax/compile-checked and deployed, not run against a real recurrence of the wrong-page bug — the retry timing (3 attempts, 3s apart) is a reasonable-but-unconfirmed guess at how fast the badge carousel actually cycles.
|
||
|
||
## Prerequisites
|
||
|
||
### OCR
|
||
|
||
**Set up (Phase 10): `pytesseract` + the `tesseract-ocr` apt package.** `ba_auto/detector.py`'s `read_text()`/`read_int()` wrap it for occasional single-crop reads (a region number, a stage label) — no need for the reference's own socket/shared-memory PaddleOCR server, which exists there to make OCR fast across thousands of automation steps; this project's usage volume doesn't need that.
|
||
|
||
Needed for:
|
||
|
||
- currency readouts
|
||
- ticket counts
|
||
- region/tab name matching
|
||
- some shop logic
|
||
- some lesson/schedule logic
|
||
- arena ticket/rank/level checks
|
||
- bounty coin balance if auto-refresh is implemented
|
||
|
||
Candidates:
|
||
|
||
- Tesseract
|
||
- PaddleOCR
|
||
|
||
### Auto-fight primitive
|
||
|
||
Needed for:
|
||
|
||
- Arena
|
||
- future main story push
|
||
- some battle automation
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/main_story.py
|
||
```
|
||
|
||
Look for:
|
||
|
||
```
|
||
auto_fight
|
||
enter_battle
|
||
```
|
||
|
||
Target local module may be:
|
||
|
||
```
|
||
ba_auto/tasks/battle.py
|
||
```
|
||
|
||
or:
|
||
|
||
```
|
||
ba_auto/battle.py
|
||
```
|
||
|
||
This should become a reusable primitive, not arena-specific code.
|
||
|
||
## High priority backlog
|
||
|
||
### 1. Migration to Python-first
|
||
|
||
**Status: Done.** See Phases 1–7 above for the detailed history, including the mailbox and cafe exit-game-dialog bug and its fix.
|
||
|
||
Goal (all done):
|
||
|
||
- Bash launcher only
|
||
- Python CLI
|
||
- Python driver
|
||
- Python detector
|
||
- mailbox migrated
|
||
- cafe migrated
|
||
- reference mapping started
|
||
|
||
### 2. Stamina/AP sweep
|
||
|
||
**Status: Partially done — see Phase 8.**
|
||
|
||
Claim:
|
||
|
||
- ~~daily free AP purchase~~ — deferred; entry point is a real-money purchase menu, needs explicit confirmation before automating
|
||
- daily task-menu AP/pyroxene rewards — done, via the Mission panel's bulk claim button
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/collect_daily_free_power.py
|
||
~/repo/baas-reference/module/collect_daily_task_power.py
|
||
```
|
||
|
||
Local target: `ba_auto/tasks/stamina.py`
|
||
|
||
OCR: Not needed — turned out to be a single bulk-claim button + Enter, no per-item detection required.
|
||
|
||
### 3. Club/Group AP claim
|
||
|
||
Claim AP from club/group.
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/group.py
|
||
```
|
||
|
||
Local target: `ba_auto/tasks/group.py`
|
||
|
||
OCR: Not expected.
|
||
|
||
### 4. Normal/Hard story AP sweep
|
||
|
||
**Status: Done — see Phase 10 (supersedes Phase 9's random-pick design).**
|
||
|
||
Sweep already-cleared main story stages to burn AP.
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/explore_tasks/sweep_task.py
|
||
~/repo/baas-reference/module/explore_tasks/task_utils.py
|
||
```
|
||
|
||
Local target: `ba_auto/tasks/story_sweep.py`
|
||
|
||
OCR: Used, for real, as of Phase 10 — region-number readout OCR + delta-click (porting `to_region`), and stage-row label OCR matching (a scoped-down `swipe_search_target_str`), replacing Phase 9's "next region arrow stops advancing, then a random stage" heuristic. See Phase 10's write-up for what was actually found live (a second, taller modal layout for regular numbered stages; an AP-usage-confirmation dialog the design hadn't accounted for; a genuine insufficient-AP dialog told apart from it only by button color).
|
||
|
||
Implemented version:
|
||
|
||
- exact configured `(region, stage, count)` targets (`config.STORY_SWEEP_TARGETS`), not a fixed single stage nor a random pick
|
||
- `count` is `"max"` or a specific int (the latter calibrated but not yet live-clicked — see Phase 10)
|
||
- opt-in only, not in the default daily flow
|
||
|
||
### 5. Bounty
|
||
|
||
Three sub-areas and sweep availability.
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/rewarded_task.py
|
||
```
|
||
|
||
Local target: `ba_auto/tasks/bounty.py`
|
||
|
||
OCR: Optional for coin balance/refresh logic. Can skip refresh for first version.
|
||
|
||
### 6. Commissions
|
||
|
||
Two sub-dungeons:
|
||
|
||
- Base Defense
|
||
- Item Retrieval
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/clear_special_task_power.py
|
||
```
|
||
|
||
Local target: `ba_auto/tasks/commission.py`
|
||
|
||
OCR: Port the reference's approach if it uses OCR here — do not default to a fixed-target workaround just to avoid OCR (see "OCR policy" in `CLAUDE.md` and the Phase 9 retrospective above).
|
||
|
||
### 7. Arena
|
||
|
||
**Status: Done — see Phase 13.**
|
||
|
||
Fights exactly one battle per invocation, not "until out of tickets" (deliberate — see Phase 13).
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/arena.py
|
||
```
|
||
|
||
Local target: `ba_auto/tasks/arena.py`
|
||
|
||
### 8. Common Shop + Tactical Shop
|
||
|
||
**Status: Done — see Phase 11.**
|
||
|
||
Auto-buy configured items.
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/shop/common_shop.py
|
||
~/repo/baas-reference/module/shop/tactical_challenge_shop.py
|
||
~/repo/baas-reference/module/shop/shop_utils.py
|
||
```
|
||
|
||
Local targets:
|
||
|
||
```
|
||
ba_auto/tasks/shop_common.py
|
||
ba_auto/tasks/shop_tactical.py
|
||
ba_auto/tasks/shop_utils.py
|
||
```
|
||
|
||
Implemented version:
|
||
|
||
- OCR for currency balances (top-bar credits, in-panel tactical coin) — done
|
||
- shop tab detection — done via a fixed click (tactical tab list fits on screen with no scroll needed on this account, confirmed live); no scroll/pagination logic yet since both current buy lists are fully visible without scrolling
|
||
- configured buy list — done, `config.COMMON_SHOP_TARGETS`/`config.TACTICAL_SHOP_TARGETS`, fixed `(row, col, name, expected_price)` per item (see Phase 11 for why identification is by grid position + price-OCR-verify rather than per-item OCR — the reference doesn't OCR item names here either)
|
||
- safe purchase confirmation logic — done, a single overlay-darkness probe covers both the confirm dialog and the reward-acquired banner
|
||
- no-refresh (the paid manual `更新` refresh button is deliberately not automated, same reasoning as Daily Free Power)
|
||
|
||
### 9. Lesson / Schedule — Done (Phase 12)
|
||
|
||
Affection farming via classes.
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/lesson.py
|
||
```
|
||
|
||
Local target: `ba_auto/tasks/lesson.py` — implemented, live-tested with real tickets. See Phase 12 above for the full writeup.
|
||
|
||
- region/area identification — done without OCR: this client's region list only settles at two fixed scroll positions, so navigation is deterministic index-based clicking, not the reference's OCR-a-name-then-page approach
|
||
- multi-page swipe search — not needed for the same reason
|
||
- student detection/portrait matching for specific students — deferred, per the original suggested scope below
|
||
- isometric grid location logic — done without porting the reference's geometry: per-cell status/affection reads via `detector.read_int_on_heart_badge` OCR + a checkmark color probe instead
|
||
|
||
Suggested first version (as originally scoped, and what shipped):
|
||
|
||
- pick a fixed region — expanded to all 12, swept in order
|
||
- select available/highest visible lesson — affection-first, per explicit user direction
|
||
- avoid favorite-student targeting at first — still deferred
|
||
|
||
## Low priority backlog
|
||
|
||
### Scrimmage
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/scrimmage.py
|
||
```
|
||
|
||
Local target: `ba_auto/tasks/scrimmage.py`
|
||
|
||
Similar shape to Bounty/Commissions.
|
||
|
||
### Crafting
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/create.py
|
||
```
|
||
|
||
Local target: `ba_auto/tasks/crafting.py`
|
||
|
||
Very complex. Contains:
|
||
|
||
- material selection
|
||
- priority lists
|
||
- rarity tiers
|
||
- stepper/quantity UI
|
||
- filtering/sorting
|
||
- OCR-like decision points
|
||
|
||
Do not start until the framework and OCR are mature.
|
||
|
||
### Battle Pass claim
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/collect_pass_reward.py
|
||
```
|
||
|
||
Local target: `ba_auto/tasks/battle_pass.py`
|
||
|
||
Should be simpler than crafting.
|
||
|
||
OCR only needed for optional stats.
|
||
|
||
### Momo Talk
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/momo_talk.py
|
||
```
|
||
|
||
Local target: `ba_auto/tasks/momo_talk.py`
|
||
|
||
Potentially useful because it runs on a different cadence from daily reset.
|
||
|
||
Likely no OCR. Mostly state scanning and click flow.
|
||
|
||
### Main story push
|
||
|
||
This means clearing new uncleared stages, not sweeping already-cleared stages.
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/main_story.py
|
||
~/repo/baas-reference/module/explore_tasks/explore_task.py
|
||
```
|
||
|
||
Low priority because full grid-mode support requires lots of per-stage scripting.
|
||
|
||
A simple auto-fight-only mode can be added later.
|
||
|
||
### Group Story / Mini Story
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/group_story.py
|
||
~/repo/baas-reference/module/mini_story.py
|
||
```
|
||
|
||
Convenience only.
|
||
|
||
### Event content
|
||
|
||
**Status: Event AP sweep done — see Phase 14.** The user asked for exactly the generic-sweep case anticipated below (reusing story_sweep's AP-confirm/result-screen logic, not a prebuilt event-specific script), scoped to the currently-running "鉄道爆走事件" event's 9-12 stage range via a rotation target.
|
||
|
||
Reference:
|
||
|
||
```
|
||
~/repo/baas-reference/module/activities/activity_utils.py
|
||
~/repo/baas-reference/module/sweep_activity.py
|
||
```
|
||
|
||
Still not built: anything beyond the sweep panel (`explore_activity_story`/`explore_activity_mission`/`explore_activity_challenge` — walking the event's map/fighting story stages manually, needed only for a stage that hasn't been SSS-cleared yet) and the reward-exchange shop (`exchange_reward`). Event-specific content expires; the 9-12 rotation range and row-position calibration are specific to this 12-stage event, not proven durable across future events with a different stage count or layout.
|
||
|
||
## Skip list
|
||
|
||
| Feature | Why skip |
|
||
|---|---|
|
||
| Total Assault / Raid | Low value and risky to automate. Reference support may be limited/stubbed. |
|
||
| Joint Firing Drill | Not worth prioritizing for JP if reference has server-specific limitations. |
|
||
| De-clothes localization toggle | CN-only / irrelevant. |
|
||
| Restart / refresh-uiautomator2 | Android/ADB backend maintenance, not applicable to PC/Steam/Proton. |
|
||
| Auto-unfriend | Risky, low value, destructive. |
|
||
| Daily minigame dispatcher | Event-specific and unstable. Handle ad hoc only. |
|
||
|
||
## Automation cadence notes
|
||
|
||
Some tasks decay on different schedules.
|
||
|
||
| Feature | Suggested cadence |
|
||
|---|---|
|
||
| Cafe income / affection | Every few hours |
|
||
| Momo Talk | Every few hours |
|
||
| Arena | Around reset windows / twice daily if implemented |
|
||
| Mailbox | Daily or with default run |
|
||
| AP/stamina/task rewards | Daily |
|
||
| Group AP | Daily |
|
||
| Bounty/Commissions/Scrimmage | Daily |
|
||
| Shops | Daily, after reset |
|
||
| Lesson/Schedule | Daily |
|
||
|
||
Scheduling should be handled outside the feature logic.
|
||
|
||
Feature code should perform one safe run and exit.
|
||
|
||
## Safety and robustness rules
|
||
|
||
Every task should have:
|
||
|
||
- maximum retry count
|
||
- timeout where appropriate
|
||
- safe failure mode
|
||
- clear stdout logging
|
||
- no infinite click loops
|
||
- no unbounded spending
|
||
- config guard for purchases
|
||
- dry-run or debug mode when useful
|
||
|
||
For purchases:
|
||
|
||
- default to conservative behavior
|
||
- avoid refresh loops until OCR/currency detection is reliable
|
||
- require explicit configured item list
|
||
- avoid buying unknown items
|
||
|
||
For battle features:
|
||
|
||
- require clear stop conditions
|
||
- avoid continuing blindly after unexpected state
|
||
- prefer returning failure over clicking randomly
|
||
|
||
## Configuration direction
|
||
|
||
Future config may live in `ba_auto/config.py` or `config.yaml`.
|
||
|
||
Possible config values:
|
||
|
||
```
|
||
server = JP
|
||
game_window_name = BlueArchive
|
||
display = :0
|
||
asset_dir = ~/ba_assets
|
||
screenshot_dir = scratchpad/
|
||
cafe_max_clicks_per_room
|
||
story_sweep_target
|
||
shop_buy_list
|
||
arena_stop_condition
|
||
ocr_enabled
|
||
debug_enabled
|
||
```
|
||
|
||
Keep config explicit. Do not bury user-specific settings deep inside task logic.
|
||
|
||
## Debugging conventions
|
||
|
||
Use `scratchpad/` for:
|
||
|
||
- temporary screenshots
|
||
- cropped templates
|
||
- annotated match images
|
||
- OCR debug output
|
||
- one-off notes
|
||
|
||
Do not use `/tmp` or `/private/tmp` unless unavoidable.
|
||
|
||
When detector behavior changes, save debug outputs with clear names, for example:
|
||
|
||
```
|
||
scratchpad/cafe_match_2026-07-05_001.png
|
||
scratchpad/shop_ocr_debug_001.png
|
||
```
|
||
|
||
## Local validation commands
|
||
|
||
On `nik-macbookair`:
|
||
|
||
```
|
||
bash -n ba_dailies.sh
|
||
python3 -m py_compile ba_daily.py
|
||
python3 -m py_compile ba_auto/*.py
|
||
python3 -m py_compile ba_auto/tasks/*.py
|
||
```
|
||
|
||
On `nik-gpu`:
|
||
|
||
```
|
||
~/ba_dailies.sh mailbox
|
||
~/ba_dailies.sh cafe
|
||
```
|
||
|
||
After migration:
|
||
|
||
```
|
||
~/ba_dailies.sh
|
||
```
|
||
|
||
should run the default daily sequence.
|
||
|
||
## Near-term recommended task order
|
||
|
||
1. Rewrite `ba_dailies.sh` as a thin launcher. — Done
|
||
2. Add `ba_daily.py`. — Done
|
||
3. Add `ba_auto/driver.py`. — Done
|
||
4. Add `ba_auto/detector.py`. — Done
|
||
5. Add `ba_auto/navigation.py`. — Done
|
||
6. Move mailbox logic to `ba_auto/tasks/mailbox.py`. — Done
|
||
7. Move cafe logic to `ba_auto/tasks/cafe.py`. — Done
|
||
8. Update `setup.sh`. — Done
|
||
9. Add `ba_auto/reference_notes/mapping.md`. — Done
|
||
10. Verify existing mailbox and cafe still work. — Done
|
||
11. Implement stamina/AP. — Done (Phase 8)
|
||
12. Implement Normal/Hard story AP sweep. — Done (Phase 9 built a random-pick heuristic; Phase 10 replaced it with the reference's actual OCR-based deterministic stage targeting, per Phase 9's own retrospective)
|
||
13. Implement group/club AP. — Not started
|
||
14. Set up OCR and port it for whichever remaining feature's reference implementation depends on it — not a blanket "only when needed" deferral; see `CLAUDE.md` → "OCR policy"
|
||
15. Implement Common Shop + Tactical Shop. — Done (Phase 11)
|
||
16. Implement Lesson/Schedule. — Done (Phase 12)
|
||
17. Implement Arena. — Done (Phase 13)
|
||
|
||
## Claude Code guidance summary
|
||
|
||
When Claude Code works on this repo, it should follow this rule:
|
||
|
||
> Reference first.
|
||
> Python first.
|
||
> Driver primitives before feature hacks.
|
||
> Bash launcher only.
|
||
> Port OCR when the reference uses it — don't invent non-OCR substitutes to avoid the setup cost.
|
||
|
||
Do not turn this project into a Bash recreation of Blue Archive Auto Script.
|