ba-auto-daily/plan.md

# ba-auto-daily implementation plan

Personal Blue Archive JP daily-automation project.

This project controls the PC/Steam/Proton Blue Archive client running on `nik-gpu` through local desktop automation:

- xdotool
- scrot
- Python
- OpenCV
- later OCR when needed

Development happens on `nik-macbookair`.

The reference implementation lives at:

```
~/repo/baas-reference/
```

The reference project should be treated as the behavioral blueprint. This project should avoid recreating feature logic from scratch when the reference already implements it.

## Core project direction

This project is now Python-first.

The goal is not to grow a large Bash script.

The goal is to build a small local Python automation framework that adapts the reference project's Blue Archive logic to this user's unique PC/Steam/Proton environment.

`ba_dailies.sh` should only be a launcher.

Feature logic should live in Python.

## Target architecture

```
~/repo/ba-auto-daily/
├── ba_dailies.sh
├── ba_daily.py
├── ba_auto/
│   ├── __init__.py
│   ├── config.py
│   ├── driver.py
│   ├── detector.py
│   ├── navigation.py
│   ├── tasks/
│   │   ├── __init__.py
│   │   ├── mailbox.py
│   │   ├── cafe.py
│   │   ├── stamina.py
│   │   ├── group.py
│   │   ├── bounty.py
│   │   ├── commission.py
│   │   ├── arena.py
│   │   ├── shop_common.py
│   │   ├── shop_tactical.py
│   │   ├── lesson.py
│   │   └── ...
│   └── reference_notes/
│       └── mapping.md
├── assets/
├── screenshots/
├── scripts/
├── setup.sh
├── CLAUDE.md
└── plan.md
```

This is the intended direction. It does not have to be completed all at once.

## Project layout

| Path | What it is |
|---|---|
| `~/repo/ba-auto-daily/ba_dailies.sh` | Thin launcher only. It should call the Python entry point. Do not add new feature logic here. |
| `~/repo/ba-auto-daily/ba_daily.py` | Main Python CLI entry point. Dispatches tasks such as mailbox, cafe, stamina, group, etc. |
| `~/repo/ba-auto-daily/ba_auto/driver.py` | Local PC/Steam/Proton control backend. Wraps xdotool, scrot, waits, clicks, swipes, keypresses, screenshots, and window focus. |
| `~/repo/ba-auto-daily/ba_auto/detector.py` | OpenCV/template/color matching helpers. Currently has `find_cafe_sparkle()`, ported in-process from the retired `scripts/detect_and_click.py`. |
| `~/repo/ba-auto-daily/ba_auto/navigation.py` | Shared navigation/state-probe helpers: `is_on_subscreen`, `is_modal_open`, used by both `mailbox.py` and `cafe.py`. |
| `~/repo/ba-auto-daily/ba_auto/tasks/` | Feature implementations. Each task should adapt the relevant `baas-reference/module/...` logic where possible. |
| `~/repo/ba-auto-daily/ba_auto/reference_notes/mapping.md` | Reference mapping table: local feature → reference module → local implementation → driver gaps. |
| `~/repo/ba-auto-daily/assets/` | Locally captured template images, such as cafe sparkle. Do not blindly copy assets from the reference repo. |
| `~/repo/ba-auto-daily/screenshots/` | Human reference screenshots, mostly Moonlight/game captures, used for calibration and debugging. |
| `~/repo/ba-auto-daily/setup.sh` | Bootstrap/deploy helper for `nik-gpu`. Should install/check dependencies and copy runtime files. |
| `~/repo/baas-reference/` | Read-only GPL-3.0 reference clone. Study and adapt. Never edit. |

## Runtime paths on nik-gpu

Preferred runtime layout:

| Path | What it is |
|---|---|
| `nik-gpu:~/ba_dailies.sh` | Thin launcher. |
| `nik-gpu:~/ba_daily.py` | Python CLI entry point. |
| `nik-gpu:~/ba_auto/` | Python package copied from this repo. |
| `nik-gpu:~/ba_assets/` | Runtime assets/templates. |
| `nik-gpu:~/.venvs/ba-auto-daily/` | Python virtual environment. |

`nik-gpu:~/ba_scripts/` may still contain `detect_and_click.py` and `ba_dailies_legacy.sh` left over from before both mailbox and cafe were migrated off them. Neither is deployed or referenced by anything anymore (`setup.sh` stopped copying them once Phase 6 landed) — safe to delete manually on `nik-gpu`, just not automated here.

## Implementation strategy

For each feature:

1. Read the matching `~/repo/baas-reference/module/...` file.
2. Summarize the reference flow.
3. Identify reusable logic:
   - state checks
   - retry loops
   - navigation sequence
   - battle/sweep/shop rules
   - detection method
   - failure handling
4. Identify backend-specific calls that cannot be reused directly.
5. Implement missing generic primitives in `ba_auto/driver.py` or `ba_auto/detector.py`.
6. Implement the feature in `ba_auto/tasks/<feature>.py`.
7. Add CLI command dispatch in `ba_daily.py`.
8. Keep `ba_dailies.sh` unchanged unless launcher behavior changes.
9. Test syntax locally.
10. Deploy to `nik-gpu`.
11. Run against the live game.
12. Update this plan.

The intended result is not a Bash automation script.

The intended result is a Python automation framework using the reference repository as the behavioral blueprint.

## Reference mapping table

Maintain this table in `ba_auto/reference_notes/mapping.md`.

Initial seed:

| Local feature | Reference file | Reference functions/classes | Local file | Backend replacements | Status |
|---|---|---|---|---|---|
| Mailbox | Need to confirm in reference | Need to inspect | `ba_auto/tasks/mailbox.py` | tap/click via xdotool, screenshot via scrot | Existing Bash behavior; migrate to Python |
| Cafe | `module/cafe_reward.py` | `to_cafe`, `interaction_for_cafe_solve_method3`, `collect` | `ba_auto/tasks/cafe.py` | `picture.co_detect`/`color.rgb_in_range` → `driver.color_at` pixel-probe checks; sparkle template match ported in-process into `ba_auto/detector.py` | Migrated: real Python, state-verified via color probes, no legacy bridge |
| Stamina/AP | `module/collect_daily_free_power.py`, `module/collect_daily_task_power.py` | Need to inspect | `ba_auto/tasks/stamina.py` | color checks/clicks via local driver | Not started |
| Group/Club AP | `module/group.py` | Need to inspect | `ba_auto/tasks/group.py` | fixed click + state check via local driver | Not started |
| Bounty | `module/rewarded_task.py` | Need to inspect | `ba_auto/tasks/bounty.py` | sweep/color/OCR adaptation | Not started |
| Commissions | `module/clear_special_task_power.py` | Need to inspect | `ba_auto/tasks/commission.py` | sweep/color adaptation | Not started |
| Arena | `module/arena.py` | Need to inspect | `ba_auto/tasks/arena.py` | auto-fight + OCR + local driver | Not started |
| Common Shop | `module/shop/common_shop.py`, `module/shop/shop_utils.py` | Need to inspect | `ba_auto/tasks/shop_common.py` | OCR + tab navigation + local clicks | Not started |
| Tactical Shop | `module/shop/tactical_challenge_shop.py`, `module/shop/shop_utils.py` | Need to inspect | `ba_auto/tasks/shop_tactical.py` | OCR + tab navigation + local clicks | Not started |
| Lesson/Schedule | `module/lesson.py` | Need to inspect | `ba_auto/tasks/lesson.py` | OCR + template/portrait search + local driver | Not started |

Do not implement a feature without filling at least the relevant row.

## Status snapshot

| Feature | Current status | Target status |
|---|---|---|
| Mailbox claim | Migrated: `ba_auto/tasks/mailbox.py` uses `driver.color_at` to verify the panel opened before acting (found live-testing bug: a marginal icon coordinate could miss and cascade into pressing Escape on the home screen, which triggers Blue Archive's own exit-game confirmation) | Done |
| Cafe pats + income | Migrated: `ba_auto/tasks/cafe.py` verifies each room/dialog opened via `driver.color_at` before acting; sparkle detection now runs in-process via `ba_auto/detector.py` instead of a per-click subprocess | Done |
| Shared driver | `ba_auto/driver.py` built (`run_command`, `focus_game`, `click`, `keypress`, `screenshot`, `wait`, `color_at`); wired into both `mailbox.py` and `cafe.py` | Extend with new primitives as future tasks need them |
| Python CLI | Built: `ba_daily.py` dispatches `mailbox`/`cafe`/default flow | Extend as new tasks are added |
| Reference mapping | Built: `ba_auto/reference_notes/mapping.md` | Fill in reference file/function columns per feature |
| Everything else | Not started | Implement reference-first in Python |

## Migration phase

Before adding new game features, migrate the existing working implementation.

### Phase 1: Python skeleton

**Status: Done.**

Create:

```
ba_daily.py
ba_auto/__init__.py
ba_auto/config.py
ba_auto/driver.py
ba_auto/detector.py
ba_auto/navigation.py
ba_auto/tasks/__init__.py
ba_auto/tasks/mailbox.py
ba_auto/tasks/cafe.py
ba_auto/reference_notes/mapping.md
```

### Phase 2: Launcher

**Status: Done.**

Change `ba_dailies.sh` into a thin launcher:

```bash
#!/usr/bin/env bash
set -euo pipefail
VENV_PYTHON="${VENV_PYTHON:-$HOME/.venvs/ba-auto-daily/bin/python3}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
exec "$VENV_PYTHON" "$SCRIPT_DIR/ba_daily.py" "$@"
```

Keep compatibility with:

```
./ba_dailies.sh
./ba_dailies.sh mailbox
./ba_dailies.sh cafe
```

### Phase 3: Driver extraction

**Status: Done.** Primitives (including `color_at`, added during the mailbox/cafe hardening work) are wired into both `ba_auto/tasks/mailbox.py` and `ba_auto/tasks/cafe.py`.

Move shell interactions into `ba_auto/driver.py`.

Driver primitives should include:

```
focus_game()
click(x, y)
double_click(x, y)
keypress(key)
screenshot(path=None)
swipe(...)
wait(seconds)
wait_until(...)
```

### Phase 4: Detector extraction

**Status: Done (scoped).** `scripts/detect_and_click.py`'s sparkle-matching logic (masked template match against `assets/cafe_sparkle.png`) was ported into `ba_auto/detector.py` as `find_cafe_sparkle()`, called in-process from `ba_auto/tasks/cafe.py` — this removed the old per-click Python cold start (a fresh `cv2`/`numpy` import per subprocess call) that CLAUDE.md's driver-layer guidance specifically warns against. `scripts/detect_and_click.py` had no remaining callers once this landed, so it was deleted rather than kept as a compatibility wrapper. The more generic primitives listed below (`load_template`, `match_template`, etc.) have not been built — only the one concrete sparkle-matching function needed so far exists; generalize when a second detector use case actually needs it.

Detector primitives, generalize later if needed:

```
load_template(...)
match_template(...)
find_best_match(...)
find_and_click_template(...)
color_mask(...)
debug_write_match(...)
```

### Phase 5: Mailbox migration

**Status: Done.** Live testing surfaced a real bug: the old `MAILBOX_ICON` coordinate `(1726, 60)` sat on the edge of the icon's hitbox and intermittently missed, and the fixed click sequence had no way to notice — it cascaded into pressing Escape on the bare home screen, which triggers Blue Archive's own "exit the game?" confirmation (dismissed safely with Cancel during testing; no game state was lost). The Python port in `ba_auto/tasks/mailbox.py` fixes the coordinate and, following `module/mail.py`'s `rgb_in_range` pattern, verifies the panel actually opened (and whether "claim all" is disabled) via `driver.color_at` before pressing any further keys, with a bounded retry and a safe abort if the panel never appears.

Move mailbox logic from Bash to:

```
ba_auto/tasks/mailbox.py
```

The CLI should call it through Python.

### Phase 6: Cafe migration

**Status: Done.** Same root cause as the mailbox bug (Phase 5), confirmed by step-by-step live replay with screenshots: `CAFE_ICON` clicks are flaky (missed on the first attempt, worked on retry at the identical coordinate — this is xdotool/Proton click-registration flakiness, not a coordinate-precision problem), and the old sequence had zero verification across its ~12 steps (open → dismiss notice → pat loop ×15 → switch room → dismiss notice → pat loop ×15 → claim income ×2 Enter → ×2 Escape). A missed click anywhere cascades into blind actions on whatever screen is actually showing, which — same as mailbox — very likely ends with an unverified Escape hitting the home screen and triggering Blue Archive's own exit-game confirmation.

`ba_auto/tasks/cafe.py` now verifies state at every transition using `driver.color_at`, following `module/cafe_reward.py`'s `picture.co_detect`/`rgb_in_range` pattern:

- opening the cafe icon and the room-switch button both retry (bounded) and confirm the panel actually opened via the same subscreen-header probe as mailbox, now shared in `ba_auto/navigation.is_on_subscreen`
- the "visited student list" notice that appears on every room entry is dismissed with Enter; this is harmless as a no-op if no popup is actually present (verified live), so no separate presence check was needed there
- the income dialog's own dimmed-overlay backdrop is checked (`navigation.is_modal_open`) before pressing Enter to claim, and the "receive" button's disabled-grey color is checked before attempting to claim at all (mirrors `collect()`'s `rgb_in_range` gate in the reference)
- the closing Escape(s) only fire when a subscreen/modal is confirmed still open, never blindly

Verified live (two full runs against the real game, plus a manual step-by-step replay of every transition):

- both rooms open and pat correctly
- sparkle detection still works and now runs in-process (see Phase 4) instead of shelling out per click
- cafe income claim works (confirmed gold +81,251 / AP +61 on an actual claim) and correctly no-ops when there's nothing to collect
- the reference's `zoom_out` step (camera zoom before sparkle detection — CLAUDE.md's "view centering/zoom" gap) was **not** ported: detection matched at 0.99 confidence without it in live testing, so it wasn't reproducibly broken here. Left as a documented open risk below rather than added speculatively.

Not verified / open risks:

- rank-up popups: not observed during testing (no student ranked up while testing), still unhandled if one appears mid-loop
- whether zoom/pan state could drift over a long unattended run and eventually break sparkle detection (see above — no evidence of this yet, but the reference project treats it as necessary)

### Phase 7: setup.sh update

**Status: Done — `setup.sh` deploys `ba_daily.py` and `ba_auto/`.** `scripts/ba_dailies_legacy.sh` and `scripts/detect_and_click.py` were deleted once mailbox and cafe both migrated off them (Phases 5–6); `setup.sh` no longer references either.

Update `setup.sh` so it deploys:

```
ba_dailies.sh
ba_daily.py
ba_auto/
assets/
```

to the expected runtime paths on `nik-gpu`.

## Prerequisites

### OCR

Not set up yet.

Needed for:

- currency readouts
- ticket counts
- region/tab name matching
- some shop logic
- some lesson/schedule logic
- arena ticket/rank/level checks
- bounty coin balance if auto-refresh is implemented

Candidates:

- Tesseract
- PaddleOCR

Do not add OCR until a feature needs it.

### Auto-fight primitive

Needed for:

- Arena
- future main story push
- some battle automation

Reference:

```
~/repo/baas-reference/module/main_story.py
```

Look for:

```
auto_fight
enter_battle
```

Target local module may be:

```
ba_auto/tasks/battle.py
```

or:

```
ba_auto/battle.py
```

This should become a reusable primitive, not arena-specific code.

## High priority backlog

### 1. Migration to Python-first

**Status: Done.** See Phases 1–7 above for the detailed history, including the mailbox and cafe exit-game-dialog bug and its fix.

Goal (all done):

- Bash launcher only
- Python CLI
- Python driver
- Python detector
- mailbox migrated
- cafe migrated
- reference mapping started

### 2. Stamina/AP sweep

Claim:

- daily free AP purchase
- daily task-menu AP/pyroxene rewards

Likely mostly color/state detection and fixed clicks.

Reference:

```
~/repo/baas-reference/module/collect_daily_free_power.py
~/repo/baas-reference/module/collect_daily_task_power.py
```

Local target: `ba_auto/tasks/stamina.py`

OCR: Probably not needed for first version.

### 3. Club/Group AP claim

Claim AP from club/group.

Reference:

```
~/repo/baas-reference/module/group.py
```

Local target: `ba_auto/tasks/group.py`

OCR: Not expected.

### 4. Normal/Hard story AP sweep

Sweep already-cleared main story stages to burn AP.

Reference:

```
~/repo/baas-reference/module/explore_tasks/sweep_task.py
~/repo/baas-reference/module/explore_tasks/task_utils.py
```

Local target: `ba_auto/tasks/story_sweep.py`

OCR: Likely needed for current region/stage detection unless using fixed configured targets.

Suggested first version:

- user-configured fixed stage
- no region search
- no dynamic OCR
- sweep configured mission only

Later version:

- fuzzy stage/region selection
- OCR-assisted navigation

### 5. Bounty

Three sub-areas and sweep availability.

Reference:

```
~/repo/baas-reference/module/rewarded_task.py
```

Local target: `ba_auto/tasks/bounty.py`

OCR: Optional for coin balance/refresh logic. Can skip refresh for first version.

### 6. Commissions

Two sub-dungeons:

- Base Defense
- Item Retrieval

Reference:

```
~/repo/baas-reference/module/clear_special_task_power.py
```

Local target: `ba_auto/tasks/commission.py`

OCR: Probably avoidable for first version if configured fixed sweep target is used.

### 7. Arena

Fight until out of tickets or until configured stopping condition.

Reference:

```
~/repo/baas-reference/module/arena.py
```

Local target: `ba_auto/tasks/arena.py`

Needs:

- auto-fight primitive
- OCR or visual detection for ticket/rank state
- careful safety limits

Arena is high-value but more risky than fixed claim tasks.

### 8. Common Shop + Tactical Shop

Auto-buy configured items.

Reference:

```
~/repo/baas-reference/module/shop/common_shop.py
~/repo/baas-reference/module/shop/tactical_challenge_shop.py
~/repo/baas-reference/module/shop/shop_utils.py
```

Local targets:

```
ba_auto/tasks/shop_common.py
ba_auto/tasks/shop_tactical.py
```

Needs:

- OCR for currency balances
- shop tab detection
- configured buy list
- safe purchase confirmation logic

Start with a no-refresh, fixed configured buy list.

### 9. Lesson / Schedule

Affection farming via classes.

Reference:

```
~/repo/baas-reference/module/lesson.py
```

Local target: `ba_auto/tasks/lesson.py`

Complexity: High

Needs:

- OCR for region/area names
- multi-page swipe search
- student detection/portrait matching if targeting specific students
- isometric grid location logic if following reference fully

Suggested first version:

- pick a fixed region
- select available/highest visible lesson
- avoid favorite-student targeting at first

## Low priority backlog

### Scrimmage

Reference:

```
~/repo/baas-reference/module/scrimmage.py
```

Local target: `ba_auto/tasks/scrimmage.py`

Similar shape to Bounty/Commissions.

### Crafting

Reference:

```
~/repo/baas-reference/module/create.py
```

Local target: `ba_auto/tasks/crafting.py`

Very complex. Contains:

- material selection
- priority lists
- rarity tiers
- stepper/quantity UI
- filtering/sorting
- OCR-like decision points

Do not start until the framework and OCR are mature.

### Battle Pass claim

Reference:

```
~/repo/baas-reference/module/collect_pass_reward.py
```

Local target: `ba_auto/tasks/battle_pass.py`

Should be simpler than crafting.

OCR only needed for optional stats.

### Momo Talk

Reference:

```
~/repo/baas-reference/module/momo_talk.py
```

Local target: `ba_auto/tasks/momo_talk.py`

Potentially useful because it runs on a different cadence from daily reset.

Likely no OCR. Mostly state scanning and click flow.

### Main story push

This means clearing new uncleared stages, not sweeping already-cleared stages.

Reference:

```
~/repo/baas-reference/module/main_story.py
~/repo/baas-reference/module/explore_tasks/explore_task.py
```

Low priority because full grid-mode support requires lots of per-stage scripting.

A simple auto-fight-only mode can be added later.

### Group Story / Mini Story

Reference:

```
~/repo/baas-reference/module/group_story.py
~/repo/baas-reference/module/mini_story.py
```

Convenience only.

### Event content

Reference:

```
~/repo/baas-reference/module/activities/activity_utils.py
~/repo/baas-reference/module/sweep_activity.py
```

Do not prebuild specific event scripts.

Build only generic event AP sweep if it can reuse story sweep logic.

Event-specific content expires and is not worth hardcoding unless the user asks for a specific live event.

## Skip list

| Feature | Why skip |
|---|---|
| Total Assault / Raid | Low value and risky to automate. Reference support may be limited/stubbed. |
| Joint Firing Drill | Not worth prioritizing for JP if reference has server-specific limitations. |
| De-clothes localization toggle | CN-only / irrelevant. |
| Restart / refresh-uiautomator2 | Android/ADB backend maintenance, not applicable to PC/Steam/Proton. |
| Auto-unfriend | Risky, low value, destructive. |
| Daily minigame dispatcher | Event-specific and unstable. Handle ad hoc only. |

## Automation cadence notes

Some tasks decay on different schedules.

| Feature | Suggested cadence |
|---|---|
| Cafe income / affection | Every few hours |
| Momo Talk | Every few hours |
| Arena | Around reset windows / twice daily if implemented |
| Mailbox | Daily or with default run |
| AP/stamina/task rewards | Daily |
| Group AP | Daily |
| Bounty/Commissions/Scrimmage | Daily |
| Shops | Daily, after reset |
| Lesson/Schedule | Daily |

Scheduling should be handled outside the feature logic.

Feature code should perform one safe run and exit.

## Safety and robustness rules

Every task should have:

- maximum retry count
- timeout where appropriate
- safe failure mode
- clear stdout logging
- no infinite click loops
- no unbounded spending
- config guard for purchases
- dry-run or debug mode when useful

For purchases:

- default to conservative behavior
- avoid refresh loops until OCR/currency detection is reliable
- require explicit configured item list
- avoid buying unknown items

For battle features:

- require clear stop conditions
- avoid continuing blindly after unexpected state
- prefer returning failure over clicking randomly

## Configuration direction

Future config may live in `ba_auto/config.py` or `config.yaml`.

Possible config values:

```
server = JP
game_window_name = BlueArchive
display = :0
asset_dir = ~/ba_assets
screenshot_dir = ./scratchpad
cafe_max_clicks_per_room
story_sweep_target
shop_buy_list
arena_stop_condition
ocr_enabled
debug_enabled
```

Keep config explicit. Do not bury user-specific settings deep inside task logic.

## Debugging conventions

Use `./scratchpad` for:

- temporary screenshots
- cropped templates
- annotated match images
- OCR debug output
- one-off notes

Do not use `/tmp` or `/private/tmp` unless unavoidable.

When detector behavior changes, save debug outputs with clear names, for example:

```
scratchpad/cafe_match_2026-07-05_001.png
scratchpad/shop_ocr_debug_001.png
```

## Local validation commands

On `nik-macbookair`:

```
bash -n ba_dailies.sh
python3 -m py_compile ba_daily.py
python3 -m py_compile ba_auto/*.py
python3 -m py_compile ba_auto/tasks/*.py
```

On `nik-gpu`:

```
~/ba_dailies.sh mailbox
~/ba_dailies.sh cafe
```

After migration:

```
~/ba_dailies.sh
```

should run the default daily sequence.

## Near-term recommended task order

1. Rewrite `ba_dailies.sh` as a thin launcher.
2. Add `ba_daily.py`.
3. Add `ba_auto/driver.py`.
4. Add `ba_auto/detector.py`.
5. Add `ba_auto/navigation.py`.
6. Move mailbox logic to `ba_auto/tasks/mailbox.py`.
7. Move cafe logic to `ba_auto/tasks/cafe.py`.
8. Update `setup.sh`.
9. Add `ba_auto/reference_notes/mapping.md`.
10. Verify existing mailbox and cafe still work.
11. Implement stamina/AP.
12. Implement group/club AP.
13. Implement fixed-target sweep features.
14. Add OCR only when needed.
15. Attempt Arena/Shop/Lesson after the framework is stable.

## Claude Code guidance summary

When Claude Code works on this repo, it should follow this rule:

> Reference first.
> Python first.
> Driver primitives before feature hacks.
> Bash launcher only.

Do not turn this project into a Bash recreation of Blue Archive Auto Script.