ba-auto-daily/plan.md

# ba-auto-daily implementation plan

Personal Blue Archive JP daily-automation project.

This project controls the PC/Steam/Proton Blue Archive client running on `nik-gpu` through local desktop automation:

- xdotool
- scrot
- Python
- OpenCV
- later OCR when needed

Development happens on `nik-macbookair`.

The reference implementation lives at:

```
~/repo/baas-reference/
```

The reference project should be treated as the behavioral blueprint. This project should avoid recreating feature logic from scratch when the reference already implements it.

## Core project direction

This project is now Python-first.

The goal is not to grow a large Bash script.

The goal is to build a small local Python automation framework that adapts the reference project's Blue Archive logic to this user's unique PC/Steam/Proton environment.

`ba_dailies.sh` should only be a launcher.

Feature logic should live in Python.

## Target architecture

```
~/repo/ba-auto-daily/
├── ba_dailies.sh
├── ba_daily.py
├── ba_auto/
│   ├── __init__.py
│   ├── config.py
│   ├── driver.py
│   ├── detector.py
│   ├── navigation.py
│   ├── tasks/
│   │   ├── __init__.py
│   │   ├── mailbox.py
│   │   ├── cafe.py
│   │   ├── stamina.py
│   │   ├── group.py
│   │   ├── bounty.py
│   │   ├── commission.py
│   │   ├── arena.py
│   │   ├── shop_common.py
│   │   ├── shop_tactical.py
│   │   ├── lesson.py
│   │   └── ...
│   └── reference_notes/
│       └── mapping.md
├── assets/
├── screenshots/
├── scripts/
├── setup.sh
├── CLAUDE.md
└── plan.md
```

This is the intended direction. It does not have to be completed all at once.

## Project layout

| Path | What it is |
|---|---|
| `~/repo/ba-auto-daily/ba_dailies.sh` | Thin launcher only. It should call the Python entry point. Do not add new feature logic here. |
| `~/repo/ba-auto-daily/ba_daily.py` | Main Python CLI entry point. Dispatches tasks such as mailbox, cafe, stamina, group, etc. |
| `~/repo/ba-auto-daily/ba_auto/driver.py` | Local PC/Steam/Proton control backend. Wraps xdotool, scrot, waits, clicks, swipes, keypresses, screenshots, and window focus. |
| `~/repo/ba-auto-daily/ba_auto/detector.py` | OpenCV/template/color matching helpers. Existing `scripts/detect_and_click.py` logic should be migrated here. |
| `~/repo/ba-auto-daily/ba_auto/navigation.py` | Shared navigation helpers: home, menu, close popup, back, open feature screens. |
| `~/repo/ba-auto-daily/ba_auto/tasks/` | Feature implementations. Each task should adapt the relevant `baas-reference/module/...` logic where possible. |
| `~/repo/ba-auto-daily/ba_auto/reference_notes/mapping.md` | Reference mapping table: local feature → reference module → local implementation → driver gaps. |
| `~/repo/ba-auto-daily/assets/` | Locally captured template images, such as cafe sparkle. Do not blindly copy assets from the reference repo. |
| `~/repo/ba-auto-daily/screenshots/` | Human reference screenshots, mostly Moonlight/game captures, used for calibration and debugging. |
| `~/repo/ba-auto-daily/scripts/` | Transitional scripts. Long-term reusable Python logic should move into `ba_auto/`. |
| `~/repo/ba-auto-daily/setup.sh` | Bootstrap/deploy helper for `nik-gpu`. Should install/check dependencies and copy runtime files. |
| `~/repo/baas-reference/` | Read-only GPL-3.0 reference clone. Study and adapt. Never edit. |

## Runtime paths on nik-gpu

Preferred runtime layout:

| Path | What it is |
|---|---|
| `nik-gpu:~/ba_dailies.sh` | Thin launcher. |
| `nik-gpu:~/ba_daily.py` | Python CLI entry point. |
| `nik-gpu:~/ba_auto/` | Python package copied from this repo. |
| `nik-gpu:~/ba_assets/` | Runtime assets/templates. |
| `nik-gpu:~/.venvs/ba-auto-daily/` | Python virtual environment. |

Current older layout may include:

| Path | What it is |
|---|---|
| `nik-gpu:~/ba_scripts/detect_and_click.py` | Old standalone detector helper. Should eventually be replaced by `ba_auto/detector.py`. |

## Implementation strategy

For each feature:

1. Read the matching `~/repo/baas-reference/module/...` file.
2. Summarize the reference flow.
3. Identify reusable logic:
   - state checks
   - retry loops
   - navigation sequence
   - battle/sweep/shop rules
   - detection method
   - failure handling
4. Identify backend-specific calls that cannot be reused directly.
5. Implement missing generic primitives in `ba_auto/driver.py` or `ba_auto/detector.py`.
6. Implement the feature in `ba_auto/tasks/<feature>.py`.
7. Add CLI command dispatch in `ba_daily.py`.
8. Keep `ba_dailies.sh` unchanged unless launcher behavior changes.
9. Test syntax locally.
10. Deploy to `nik-gpu`.
11. Run against the live game.
12. Update this plan.

The intended result is not a Bash automation script.

The intended result is a Python automation framework using the reference repository as the behavioral blueprint.

## Reference mapping table

Maintain this table in `ba_auto/reference_notes/mapping.md`.

Initial seed:

| Local feature | Reference file | Reference functions/classes | Local file | Backend replacements | Status |
|---|---|---|---|---|---|
| Mailbox | Need to confirm in reference | Need to inspect | `ba_auto/tasks/mailbox.py` | tap/click via xdotool, screenshot via scrot | Existing Bash behavior; migrate to Python |
| Cafe | Need to confirm in reference | Need to inspect | `ba_auto/tasks/cafe.py` | template matching via OpenCV, click via xdotool | Existing Bash + helper behavior; migrate to Python |
| Stamina/AP | `module/collect_daily_free_power.py`, `module/collect_daily_task_power.py` | Need to inspect | `ba_auto/tasks/stamina.py` | color checks/clicks via local driver | Not started |
| Group/Club AP | `module/group.py` | Need to inspect | `ba_auto/tasks/group.py` | fixed click + state check via local driver | Not started |
| Bounty | `module/rewarded_task.py` | Need to inspect | `ba_auto/tasks/bounty.py` | sweep/color/OCR adaptation | Not started |
| Commissions | `module/clear_special_task_power.py` | Need to inspect | `ba_auto/tasks/commission.py` | sweep/color adaptation | Not started |
| Arena | `module/arena.py` | Need to inspect | `ba_auto/tasks/arena.py` | auto-fight + OCR + local driver | Not started |
| Common Shop | `module/shop/common_shop.py`, `module/shop/shop_utils.py` | Need to inspect | `ba_auto/tasks/shop_common.py` | OCR + tab navigation + local clicks | Not started |
| Tactical Shop | `module/shop/tactical_challenge_shop.py`, `module/shop/shop_utils.py` | Need to inspect | `ba_auto/tasks/shop_tactical.py` | OCR + tab navigation + local clicks | Not started |
| Lesson/Schedule | `module/lesson.py` | Need to inspect | `ba_auto/tasks/lesson.py` | OCR + template/portrait search + local driver | Not started |

Do not implement a feature without filling at least the relevant row.

## Status snapshot

| Feature | Current status | Target status |
|---|---|---|
| Mailbox claim | Migrated: `ba_auto/tasks/mailbox.py` uses `driver.color_at` to verify the panel opened before acting (found live-testing bug: a marginal icon coordinate could miss and cascade into pressing Escape on the home screen, which triggers Blue Archive's own exit-game confirmation) | Done |
| Cafe pats + income | Bridged: `ba_auto/tasks/cafe.py` calls `scripts/ba_dailies_legacy.sh cafe` (which still shells out to `scripts/detect_and_click.py`) | Migrate to `ba_auto/tasks/cafe.py` and `ba_auto/detector.py` (Phase 6) |
| Shared driver | `ba_auto/driver.py` built (`run_command`, `focus_game`, `click`, `keypress`, `screenshot`, `wait`); not yet wired into tasks | Wire into mailbox/cafe as they migrate off the Bash bridge |
| Python CLI | Built: `ba_daily.py` dispatches `mailbox`/`cafe`/default flow | Extend as new tasks are added |
| Reference mapping | Built: `ba_auto/reference_notes/mapping.md` | Fill in reference file/function columns per feature |
| Everything else | Not started | Implement reference-first in Python |

## Migration phase

Before adding new game features, migrate the existing working implementation.

### Phase 1: Python skeleton

**Status: Done.**

Create:

```
ba_daily.py
ba_auto/__init__.py
ba_auto/config.py
ba_auto/driver.py
ba_auto/detector.py
ba_auto/navigation.py
ba_auto/tasks/__init__.py
ba_auto/tasks/mailbox.py
ba_auto/tasks/cafe.py
ba_auto/reference_notes/mapping.md
```

### Phase 2: Launcher

**Status: Done.**

Change `ba_dailies.sh` into a thin launcher:

```bash
#!/usr/bin/env bash
set -euo pipefail
VENV_PYTHON="${VENV_PYTHON:-$HOME/.venvs/ba-auto-daily/bin/python3}"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
exec "$VENV_PYTHON" "$SCRIPT_DIR/ba_daily.py" "$@"
```

Keep compatibility with:

```
./ba_dailies.sh
./ba_dailies.sh mailbox
./ba_dailies.sh cafe
```

### Phase 3: Driver extraction

**Status: Primitives added to `ba_auto/driver.py`, not yet wired into task modules.**

Move shell interactions into `ba_auto/driver.py`.

Driver primitives should include:

```
focus_game()
click(x, y)
double_click(x, y)
keypress(key)
screenshot(path=None)
swipe(...)
wait(seconds)
wait_until(...)
```

### Phase 4: Detector extraction

**Status: Not started — `ba_auto/detector.py` is currently a placeholder.**

Move `scripts/detect_and_click.py` logic into `ba_auto/detector.py`.

Detector primitives should include:

```
load_template(...)
match_template(...)
find_best_match(...)
find_and_click_template(...)
color_mask(...)
debug_write_match(...)
```

The old script may remain as a compatibility wrapper temporarily, but the reusable logic should live under `ba_auto/`.

### Phase 5: Mailbox migration

**Status: Done.** Live testing surfaced a real bug: the old `MAILBOX_ICON` coordinate `(1726, 60)` sat on the edge of the icon's hitbox and intermittently missed, and the fixed click sequence had no way to notice — it cascaded into pressing Escape on the bare home screen, which triggers Blue Archive's own "exit the game?" confirmation (dismissed safely with Cancel during testing; no game state was lost). The Python port in `ba_auto/tasks/mailbox.py` fixes the coordinate and, following `module/mail.py`'s `rgb_in_range` pattern, verifies the panel actually opened (and whether "claim all" is disabled) via `driver.color_at` before pressing any further keys, with a bounded retry and a safe abort if the panel never appears.

Move mailbox logic from Bash to:

```
ba_auto/tasks/mailbox.py
```

The CLI should call it through Python.

### Phase 6: Cafe migration

**Status: Not started — `ba_auto/tasks/cafe.py` currently bridges to `scripts/ba_dailies_legacy.sh`.**

Move cafe logic from Bash to:

```
ba_auto/tasks/cafe.py
```

During migration, verify:

- both rooms still work
- sparkle detection still works
- cafe income claim still works
- rank-up popups are handled or explicitly documented as not handled
- student rotation popups are handled or explicitly documented as not handled
- view centering/zoom state is robust
- repeated detection does not suffer from Python cold-start delay

### Phase 7: setup.sh update

**Status: Done — `setup.sh` now deploys `ba_daily.py`, `ba_auto/`, and `scripts/ba_dailies_legacy.sh`.**

Update `setup.sh` so it deploys:

```
ba_dailies.sh
ba_daily.py
ba_auto/
assets/
```

to the expected runtime paths on `nik-gpu`.

## Prerequisites

### OCR

Not set up yet.

Needed for:

- currency readouts
- ticket counts
- region/tab name matching
- some shop logic
- some lesson/schedule logic
- arena ticket/rank/level checks
- bounty coin balance if auto-refresh is implemented

Candidates:

- Tesseract
- PaddleOCR

Do not add OCR until a feature needs it.

### Auto-fight primitive

Needed for:

- Arena
- future main story push
- some battle automation

Reference:

```
~/repo/baas-reference/module/main_story.py
```

Look for:

```
auto_fight
enter_battle
```

Target local module may be:

```
ba_auto/tasks/battle.py
```

or:

```
ba_auto/battle.py
```

This should become a reusable primitive, not arena-specific code.

## High priority backlog

### 1. Migration to Python-first

This is the highest priority.

Do this before adding new features.

Goal:

- Bash launcher only
- Python CLI
- Python driver
- Python detector
- mailbox migrated
- cafe migrated
- reference mapping started

Reference:

- Current local implementation
- Existing `ba_dailies.sh`
- Existing `scripts/detect_and_click.py`

### 2. Stamina/AP sweep

Claim:

- daily free AP purchase
- daily task-menu AP/pyroxene rewards

Likely mostly color/state detection and fixed clicks.

Reference:

```
~/repo/baas-reference/module/collect_daily_free_power.py
~/repo/baas-reference/module/collect_daily_task_power.py
```

Local target: `ba_auto/tasks/stamina.py`

OCR: Probably not needed for first version.

### 3. Club/Group AP claim

Claim AP from club/group.

Reference:

```
~/repo/baas-reference/module/group.py
```

Local target: `ba_auto/tasks/group.py`

OCR: Not expected.

### 4. Normal/Hard story AP sweep

Sweep already-cleared main story stages to burn AP.

Reference:

```
~/repo/baas-reference/module/explore_tasks/sweep_task.py
~/repo/baas-reference/module/explore_tasks/task_utils.py
```

Local target: `ba_auto/tasks/story_sweep.py`

OCR: Likely needed for current region/stage detection unless using fixed configured targets.

Suggested first version:

- user-configured fixed stage
- no region search
- no dynamic OCR
- sweep configured mission only

Later version:

- fuzzy stage/region selection
- OCR-assisted navigation

### 5. Bounty

Three sub-areas and sweep availability.

Reference:

```
~/repo/baas-reference/module/rewarded_task.py
```

Local target: `ba_auto/tasks/bounty.py`

OCR: Optional for coin balance/refresh logic. Can skip refresh for first version.

### 6. Commissions

Two sub-dungeons:

- Base Defense
- Item Retrieval

Reference:

```
~/repo/baas-reference/module/clear_special_task_power.py
```

Local target: `ba_auto/tasks/commission.py`

OCR: Probably avoidable for first version if configured fixed sweep target is used.

### 7. Arena

Fight until out of tickets or until configured stopping condition.

Reference:

```
~/repo/baas-reference/module/arena.py
```

Local target: `ba_auto/tasks/arena.py`

Needs:

- auto-fight primitive
- OCR or visual detection for ticket/rank state
- careful safety limits

Arena is high-value but more risky than fixed claim tasks.

### 8. Common Shop + Tactical Shop

Auto-buy configured items.

Reference:

```
~/repo/baas-reference/module/shop/common_shop.py
~/repo/baas-reference/module/shop/tactical_challenge_shop.py
~/repo/baas-reference/module/shop/shop_utils.py
```

Local targets:

```
ba_auto/tasks/shop_common.py
ba_auto/tasks/shop_tactical.py
```

Needs:

- OCR for currency balances
- shop tab detection
- configured buy list
- safe purchase confirmation logic

Start with a no-refresh, fixed configured buy list.

### 9. Lesson / Schedule

Affection farming via classes.

Reference:

```
~/repo/baas-reference/module/lesson.py
```

Local target: `ba_auto/tasks/lesson.py`

Complexity: High

Needs:

- OCR for region/area names
- multi-page swipe search
- student detection/portrait matching if targeting specific students
- isometric grid location logic if following reference fully

Suggested first version:

- pick a fixed region
- select available/highest visible lesson
- avoid favorite-student targeting at first

## Low priority backlog

### Scrimmage

Reference:

```
~/repo/baas-reference/module/scrimmage.py
```

Local target: `ba_auto/tasks/scrimmage.py`

Similar shape to Bounty/Commissions.

### Crafting

Reference:

```
~/repo/baas-reference/module/create.py
```

Local target: `ba_auto/tasks/crafting.py`

Very complex. Contains:

- material selection
- priority lists
- rarity tiers
- stepper/quantity UI
- filtering/sorting
- OCR-like decision points

Do not start until the framework and OCR are mature.

### Battle Pass claim

Reference:

```
~/repo/baas-reference/module/collect_pass_reward.py
```

Local target: `ba_auto/tasks/battle_pass.py`

Should be simpler than crafting.

OCR only needed for optional stats.

### Momo Talk

Reference:

```
~/repo/baas-reference/module/momo_talk.py
```

Local target: `ba_auto/tasks/momo_talk.py`

Potentially useful because it runs on a different cadence from daily reset.

Likely no OCR. Mostly state scanning and click flow.

### Main story push

This means clearing new uncleared stages, not sweeping already-cleared stages.

Reference:

```
~/repo/baas-reference/module/main_story.py
~/repo/baas-reference/module/explore_tasks/explore_task.py
```

Low priority because full grid-mode support requires lots of per-stage scripting.

A simple auto-fight-only mode can be added later.

### Group Story / Mini Story

Reference:

```
~/repo/baas-reference/module/group_story.py
~/repo/baas-reference/module/mini_story.py
```

Convenience only.

### Event content

Reference:

```
~/repo/baas-reference/module/activities/activity_utils.py
~/repo/baas-reference/module/sweep_activity.py
```

Do not prebuild specific event scripts.

Build only generic event AP sweep if it can reuse story sweep logic.

Event-specific content expires and is not worth hardcoding unless the user asks for a specific live event.

## Skip list

| Feature | Why skip |
|---|---|
| Total Assault / Raid | Low value and risky to automate. Reference support may be limited/stubbed. |
| Joint Firing Drill | Not worth prioritizing for JP if reference has server-specific limitations. |
| De-clothes localization toggle | CN-only / irrelevant. |
| Restart / refresh-uiautomator2 | Android/ADB backend maintenance, not applicable to PC/Steam/Proton. |
| Auto-unfriend | Risky, low value, destructive. |
| Daily minigame dispatcher | Event-specific and unstable. Handle ad hoc only. |

## Automation cadence notes

Some tasks decay on different schedules.

| Feature | Suggested cadence |
|---|---|
| Cafe income / affection | Every few hours |
| Momo Talk | Every few hours |
| Arena | Around reset windows / twice daily if implemented |
| Mailbox | Daily or with default run |
| AP/stamina/task rewards | Daily |
| Group AP | Daily |
| Bounty/Commissions/Scrimmage | Daily |
| Shops | Daily, after reset |
| Lesson/Schedule | Daily |

Scheduling should be handled outside the feature logic.

Feature code should perform one safe run and exit.

## Safety and robustness rules

Every task should have:

- maximum retry count
- timeout where appropriate
- safe failure mode
- clear stdout logging
- no infinite click loops
- no unbounded spending
- config guard for purchases
- dry-run or debug mode when useful

For purchases:

- default to conservative behavior
- avoid refresh loops until OCR/currency detection is reliable
- require explicit configured item list
- avoid buying unknown items

For battle features:

- require clear stop conditions
- avoid continuing blindly after unexpected state
- prefer returning failure over clicking randomly

## Configuration direction

Future config may live in `ba_auto/config.py` or `config.yaml`.

Possible config values:

```
server = JP
game_window_name = BlueArchive
display = :0
asset_dir = ~/ba_assets
screenshot_dir = ./scratchpad
cafe_max_clicks_per_room
story_sweep_target
shop_buy_list
arena_stop_condition
ocr_enabled
debug_enabled
```

Keep config explicit. Do not bury user-specific settings deep inside task logic.

## Debugging conventions

Use `./scratchpad` for:

- temporary screenshots
- cropped templates
- annotated match images
- OCR debug output
- one-off notes

Do not use `/tmp` or `/private/tmp` unless unavoidable.

When detector behavior changes, save debug outputs with clear names, for example:

```
scratchpad/cafe_match_2026-07-05_001.png
scratchpad/shop_ocr_debug_001.png
```

## Local validation commands

On `nik-macbookair`:

```
bash -n ba_dailies.sh
python3 -m py_compile ba_daily.py
python3 -m py_compile ba_auto/*.py
python3 -m py_compile ba_auto/tasks/*.py
```

On `nik-gpu`:

```
~/ba_dailies.sh mailbox
~/ba_dailies.sh cafe
```

After migration:

```
~/ba_dailies.sh
```

should run the default daily sequence.

## Near-term recommended task order

1. Rewrite `ba_dailies.sh` as a thin launcher.
2. Add `ba_daily.py`.
3. Add `ba_auto/driver.py`.
4. Add `ba_auto/detector.py`.
5. Add `ba_auto/navigation.py`.
6. Move mailbox logic to `ba_auto/tasks/mailbox.py`.
7. Move cafe logic to `ba_auto/tasks/cafe.py`.
8. Update `setup.sh`.
9. Add `ba_auto/reference_notes/mapping.md`.
10. Verify existing mailbox and cafe still work.
11. Implement stamina/AP.
12. Implement group/club AP.
13. Implement fixed-target sweep features.
14. Add OCR only when needed.
15. Attempt Arena/Shop/Lesson after the framework is stable.

## Claude Code guidance summary

When Claude Code works on this repo, it should follow this rule:

> Reference first.
> Python first.
> Driver primitives before feature hacks.
> Bash launcher only.

Do not turn this project into a Bash recreation of Blue Archive Auto Script.