Implement dynamic OCR for gem shop sidebar navigation and free card status
- Added `read_lines` function to `detector.py` for OCR of multiple text lines in a specified region, improving the ability to dynamically locate text in the UI. - Rewrote the gem shop task in `gem_shop.py` to utilize OCR for sidebar navigation and free card status checks, replacing fixed coordinates with dynamic text detection. - Introduced helper functions `_find_sidebar_label`, `_on_general_products_tab`, `_ensure_general_products_tab`, `_read_card_lines`, `_read_free_card_status`, and `_find_buy_button` to streamline the OCR process and improve reliability against UI changes. - Removed outdated color probe methods for checking the free card status, enhancing the robustness of the gem shop interaction flow. - Live-calibrated the new OCR functionality to ensure accurate detection of UI elements and their states.
This commit is contained in:
parent
b10824d978
commit
0ff3dca0b9
@ -1435,37 +1435,72 @@ BOUNTY_SWEEP_CONFIRM_TEXT_RECT = (605, 505, 1320, 615)
|
||||
# for the "available -> claim" path in general. Re-confirm the exact
|
||||
# first time gem_shop actually claims a package post-redesign.
|
||||
GEM_SHOP_ICON = (204, 345)
|
||||
# 一般商品 (general products), 3rd item in the left sidebar
|
||||
# (おすすめの商品/限定商品/一般商品/定額商品/青輝石/特別支援). Always
|
||||
# clicked explicitly rather than relying on it already being selected --
|
||||
# even though it was the landing tab in this recalibration, per this
|
||||
# project's own established discipline of never trusting a remembered/
|
||||
# default UI selection before a real click (see memory: BA count/quantity
|
||||
# steppers remember last-used values; the same caution applies to any
|
||||
# selected-tab state here).
|
||||
GEM_SHOP_GENERAL_PRODUCTS_TAB = (196, 376)
|
||||
# デイリー (daily), 1st of 3 sub-tabs (デイリー/ウィークリー/マンスリー)
|
||||
# shown top-right once 一般商品 is open. Also always clicked explicitly for
|
||||
# the same reason as GEM_SHOP_GENERAL_PRODUCTS_TAB above.
|
||||
GEM_SHOP_DAILY_SUBTAB = (1330, 148)
|
||||
# 毎日無料パッケージ card's own 購入 (purchase) button -- the only card
|
||||
# shown under 一般商品 -> デイリー (fixed position, no scrolling/search
|
||||
# needed).
|
||||
GEM_SHOP_FREE_CARD_BUY_BUTTON = (576, 630)
|
||||
|
||||
# The free card's own status bar (just below its artwork, above the price
|
||||
# button) reads a flat, highly distinct color depending on claim state --
|
||||
# see the RECALIBRATED note above for how each value was obtained:
|
||||
# available ("一日にN回まで購入可能"): flat light blue ~(226, 236, 246)
|
||||
# claimed ("一日に0回まで購入可能"): flat dark red ~(190, 56, 66)
|
||||
# The two are far enough apart (light vs dark, blue-family vs red-family)
|
||||
# that no OCR of the "0"/"N" count text is needed -- the reference itself
|
||||
# doesn't OCR this either, it template-matches two whole separate
|
||||
# "purchasable"/"non-purchasable" card images.
|
||||
GEM_SHOP_FREE_CARD_STATUS_PROBE = (410, 485)
|
||||
GEM_SHOP_STATUS_AVAILABLE_RGB = (226, 236, 246)
|
||||
GEM_SHOP_STATUS_CLAIMED_RGB = (190, 56, 66)
|
||||
GEM_SHOP_STATUS_TOLERANCE = 20
|
||||
# RECALIBRATED AGAIN 2026-09-17 after a real live failure ("free card status
|
||||
# not recognized after click", 3/3 attempts, reported live with the run's
|
||||
# own log). Root-caused by driving the real screen step by step on nik-gpu
|
||||
# (scratchpad/probe_gem_shop_careful_nav.py/probe_gem_shop_general_tab_fixed.py):
|
||||
# the sidebar gained a new 限定商品 (NEW) entry above 一般商品 sometime after
|
||||
# the 2026-07-29 recalibration above, which is now
|
||||
# おすすめの商品/限定商品/定額商品/一般商品/青輝石/特別支援/ステップアップ商品
|
||||
# (also note 定額商品 and 一般商品 swapped order from the 2026-07-29 comment,
|
||||
# and a new ステップアップ商品 row was appended -- see memory:
|
||||
# project_gem_shop_stepup_not_automated). The old fixed
|
||||
# GEM_SHOP_GENERAL_PRODUCTS_TAB=(196, 376) coordinate, aimed at the 3rd slot,
|
||||
# now lands on 定額商品's 3rd slot instead of 一般商品's real 4th slot --
|
||||
# confirmed live by screenshot, the run was silently landing on and reading
|
||||
# an entirely unrelated tab's card every attempt, never able to match either
|
||||
# status color there. Retrying the exact same wrong click three times could
|
||||
# never recover, matching the reported symptom exactly.
|
||||
#
|
||||
# Per explicit user request ("make the button press dynamic. Read word then
|
||||
# click the correct position of the word"), the sidebar click, the status
|
||||
# read, and the buy-button click are no longer fixed coordinates/colors at
|
||||
# all -- gem_shop.py now OCRs each of these dynamically every run via
|
||||
# detector.read_lines() and clicks/parses whatever position the target text
|
||||
# actually rendered at, so a future sidebar reorder (or another new entry
|
||||
# above 一般商品) can't silently break this again the same way. See
|
||||
# gem_shop.py's module docstring for the live OCR calibration writeup
|
||||
# (scratchpad/probe_gem_shop_ocr_regions.py) -- both the sidebar
|
||||
# (vertically-stacked labels) and the free card's own status/button text
|
||||
# read cleanly via one detector.read_lines() call per region, in their
|
||||
# normal (unselected, dark-text-on-light) state.
|
||||
#
|
||||
# The one exception is the デイリー/ウィークリー/マンスリー subtab row,
|
||||
# which stays a fixed click below: it is a horizontal (side-by-side) layout,
|
||||
# confirmed live NOT to segment reliably as one OCR region (tesseract
|
||||
# fragments multi-column text far worse than the vertically-stacked sidebar
|
||||
# or card regions -- see the module docstring), and unlike
|
||||
# GEM_SHOP_GENERAL_PRODUCTS_TAB above, this exact coordinate was NOT the
|
||||
# demonstrated root cause (it landed correctly every time in live testing).
|
||||
# GEM_SHOP_WEEKLY_SUBTAB_LABEL_REGION below reads ウィークリー as a
|
||||
# read-only "did we actually land on 一般商品" confirmation instead --
|
||||
# ウィークリー is never the tab gem_shop.py itself selects, so it stays in
|
||||
# its normal unselected (OCR-readable) state every run.
|
||||
#
|
||||
# 一般商品 (general products) sidebar item -- OCR target, not a coordinate.
|
||||
# Region covers the whole scrollable sidebar column so it keeps working
|
||||
# regardless of how many entries render above 一般商品.
|
||||
GEM_SHOP_SIDEBAR_REGION = (40, 130, 360, 900)
|
||||
# ウィークリー subtab label -- OCR target used only to confirm we're on the
|
||||
# 一般商品 tab (this subtab row only renders there); not clicked.
|
||||
GEM_SHOP_WEEKLY_SUBTAB_LABEL_REGION = (1447, 120, 1660, 180)
|
||||
# デイリー (daily), 1st of 3 sub-tabs (デイリー/ウィークリー/マンスリー)
|
||||
# shown top-right once 一般商品 is open. Always clicked explicitly rather
|
||||
# than relying on it already being selected, per this project's own
|
||||
# established discipline of never trusting a remembered/default UI
|
||||
# selection before a real click (see memory: BA count/quantity steppers
|
||||
# remember last-used values; the same caution applies to any selected-tab
|
||||
# state here). See the note above for why this one stays a fixed
|
||||
# coordinate while the sidebar click does not.
|
||||
GEM_SHOP_DAILY_SUBTAB = (1330, 148)
|
||||
# 毎日無料パッケージ card's content area (title, status line, and 無料 buy
|
||||
# button) -- the only card shown under 一般商品 -> デイリー (no
|
||||
# scrolling/search needed). OCR target region, not a fixed button
|
||||
# coordinate: gem_shop.py reads every text line in here and finds "無料"
|
||||
# (the buy button) and the "◯回まで購入可能" status line dynamically,
|
||||
# rather than assuming either one's exact position or color.
|
||||
GEM_SHOP_FREE_CARD_REGION = (397, 220, 760, 680)
|
||||
|
||||
GEM_SHOP_ICON_RETRIES = 3
|
||||
# Bounds the "press Enter, re-check the free card's status" loop that
|
||||
|
||||
@ -286,6 +286,68 @@ def read_int_bordered(region, psm=7, border=20):
|
||||
return int(digits) if digits else None
|
||||
|
||||
|
||||
def read_lines(region, lang="jpn", psm=6, image=None):
|
||||
"""OCR `region` and return `(x, y, text)` for every distinct text line
|
||||
found in it -- `(x, y)` is that line's on-screen center (for clicking
|
||||
it) and `text` is the line's full content, re-joined left-to-right from
|
||||
tesseract's own per-token boxes (CJK text gets segmented into several
|
||||
word/glyph tokens even on one visual line).
|
||||
|
||||
Unlike read_text/read_int (a single known-fixed crop, content only, no
|
||||
position), this scans a region that may contain several stacked lines
|
||||
at once and reports WHERE each one actually rendered -- for a caller
|
||||
that needs to dynamically locate and click a label/button whose exact
|
||||
position isn't a reliable fixed coordinate, e.g. a sidebar list that
|
||||
reflows when the game adds a new entry above an existing one. Built for
|
||||
gem_shop.py after a real live failure where a fixed sidebar coordinate,
|
||||
aimed at 一般商品 (general products), silently started landing on
|
||||
定額商品 instead once the game added a new entry above it -- see
|
||||
gem_shop.py's module docstring and config.py's "RECALIBRATED AGAIN
|
||||
2026-09-17" note for the full incident.
|
||||
|
||||
Only confirmed reliable for vertically-stacked lines rendered in this
|
||||
UI's usual dark-text-on-light styling (same assumption `_ocr_crop`
|
||||
already makes elsewhere) -- confirmed live NOT to segment a horizontal
|
||||
row of side-by-side labels (e.g. gem shop's own
|
||||
デイリー/ウィークリー/マンスリー sub-tabs) reliably in one call; OCR
|
||||
each column as its own small region instead for that shape. Also not
|
||||
reliable against a selected/highlighted item's own inverted (light-
|
||||
text-on-dark-pill) styling -- confirmed live to misread rather than
|
||||
just fail quietly, so prefer checking for a different, still-unselected
|
||||
marker over re-reading a label the caller itself just selected.
|
||||
"""
|
||||
x1, y1, x2, y2 = region
|
||||
img = image if image is not None else driver.read_screenshot(OCR_SHOT_PATH)
|
||||
crop = img[y1:y2, x1:x2]
|
||||
gray = cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY)
|
||||
_, thresh = cv2.threshold(gray, 150, 255, cv2.THRESH_BINARY)
|
||||
upscaled = cv2.resize(thresh, None, fx=OCR_UPSCALE, fy=OCR_UPSCALE, interpolation=cv2.INTER_CUBIC)
|
||||
|
||||
tess_config = f"--psm {psm}"
|
||||
data = pytesseract.image_to_data(upscaled, lang=lang, config=tess_config, output_type=pytesseract.Output.DICT)
|
||||
|
||||
lines = {}
|
||||
for i, word in enumerate(data["text"]):
|
||||
word = word.strip()
|
||||
if not word:
|
||||
continue
|
||||
key = (data["block_num"][i], data["par_num"][i], data["line_num"][i])
|
||||
lines.setdefault(key, []).append((data["left"][i], data["top"][i], data["width"][i], data["height"][i], word))
|
||||
|
||||
results = []
|
||||
for tokens in lines.values():
|
||||
tokens.sort(key=lambda t: t[0])
|
||||
joined = "".join(t[4] for t in tokens)
|
||||
lefts = [t[0] for t in tokens]
|
||||
rights = [t[0] + t[2] for t in tokens]
|
||||
tops = [t[1] for t in tokens]
|
||||
bottoms = [t[1] + t[3] for t in tokens]
|
||||
cx = (min(lefts) + max(rights)) / 2 / OCR_UPSCALE
|
||||
cy = (min(tops) + max(bottoms)) / 2 / OCR_UPSCALE
|
||||
results.append((x1 + round(cx), y1 + round(cy), joined))
|
||||
return results
|
||||
|
||||
|
||||
def read_int_on_heart_badge(region, psm=7, image=None):
|
||||
"""OCR a small dark-navy digit rendered on lesson.py's pink/magenta
|
||||
heart-shaped affection badge.
|
||||
|
||||
File diff suppressed because one or more lines are too long
@ -9,12 +9,36 @@ wait for the reward) else log "already collected" -> return_to_main_page
|
||||
equivalent server concept in this project, so not ported.
|
||||
|
||||
The reference detects every step via image template matching (no OCR
|
||||
anywhere in this flow) -- this port uses plain color probes for the same
|
||||
states instead (see config.py's "Gem shop daily free package" section for
|
||||
the full live-calibration writeup and exact pixel values), matching this
|
||||
project's own established equivalent for a simple, high-contrast
|
||||
state-A/state-B visual difference, same as cafe.py's CLAIM_DISABLED_RGB or
|
||||
stamina.py's MISSION_CLAIM_PROBE.
|
||||
anywhere in this flow). This port originally used plain color probes for
|
||||
the same states instead (see config.py's "Gem shop daily free package"
|
||||
section for the original 2026-07-15/07-29 live-calibration writeup),
|
||||
matching this project's own established equivalent for a simple,
|
||||
high-contrast state-A/state-B visual difference, same as cafe.py's
|
||||
CLAIM_DISABLED_RGB or stamina.py's MISSION_CLAIM_PROBE.
|
||||
|
||||
REWRITTEN 2026-09-17 after a real live failure: the sidebar gained a new
|
||||
限定商品 entry above 一般商品 sometime after the 2026-07-29 recalibration,
|
||||
silently shifting 一般商品 down one slot and making the old fixed
|
||||
GEM_SHOP_GENERAL_PRODUCTS_TAB coordinate land on 定額商品 instead --
|
||||
confirmed live via scratchpad/probe_gem_shop_careful_nav.py and
|
||||
probe_gem_shop_general_tab_fixed.py, which walked the real screen step by
|
||||
step and captured the wrong-tab landing directly. Every retry just
|
||||
re-clicked the same wrong position, matching the reported symptom exactly
|
||||
("free card status not recognized after click", 3/3 attempts).
|
||||
|
||||
Per explicit user request ("make the button press dynamic. Read word then
|
||||
click the correct position of the word"), the sidebar navigation, the
|
||||
status read, and the buy-button click are now all OCR-driven via
|
||||
detector.read_lines() rather than fixed coordinates/colors -- each one
|
||||
finds its target text wherever it actually rendered this run and
|
||||
reads/clicks that position, so a future sidebar reorder can't silently
|
||||
break this the same way again. See config.py's "RECALIBRATED AGAIN
|
||||
2026-09-17" note and scratchpad/probe_gem_shop_ocr_regions.py for the live
|
||||
OCR calibration this was based on. The デイリー/ウィークリー/マンスリー
|
||||
subtab row is the one piece that stays a fixed click -- it's a horizontal
|
||||
layout that doesn't OCR-segment reliably as one region (see
|
||||
detector.read_lines' own docstring), and it was not the coordinate that
|
||||
actually broke.
|
||||
|
||||
Live-calibrated 2026-07-15 by manually driving the real dialog end-to-end
|
||||
on nik-gpu, which genuinely claimed the account's real free package for the
|
||||
@ -26,15 +50,14 @@ than this actual module.
|
||||
RECALIBRATED 2026-07-29 after a real Blue Archive client UI update replaced
|
||||
the old 3-top-tab overlay dialog (期間限定/青輝石/パッケージ) with a real
|
||||
full subscreen (left sidebar + デイリー/ウィークリー/マンスリー sub-tabs
|
||||
under 一般商品). See config.py's "Gem shop daily free package" section for
|
||||
the full writeup of what was re-verified live and what's still an inferred
|
||||
value. The old `_dialog_open`/`_close_gem_shop` custom probes existed only
|
||||
because the pre-redesign overlay dialog was invisible to
|
||||
under 一般商品). See config.py for the full writeup of what was
|
||||
re-verified live and what's still an inferred value. The old
|
||||
`_dialog_open`/`_close_gem_shop` custom probes existed only because the
|
||||
pre-redesign overlay dialog was invisible to
|
||||
navigation.is_on_subscreen/is_modal_open; the redesigned screen is a real
|
||||
subscreen, confirmed live, so this module now shares the same
|
||||
subscreen, confirmed live, so this module shares the same
|
||||
is_on_subscreen/return_to_home machinery every other subscreen-based task
|
||||
(mailbox/cafe/shop/lesson) already uses, and those two custom probes are
|
||||
gone.
|
||||
(mailbox/cafe/shop/lesson) already uses.
|
||||
|
||||
A `reference-parity-reviewer` pass on the original version caught a real
|
||||
bug: the final dialog-close step verified success via
|
||||
@ -46,7 +69,9 @@ screen from home, confirmed live), which is what made switching to the
|
||||
shared navigation.return_to_home safe here.
|
||||
"""
|
||||
|
||||
from ba_auto import navigation
|
||||
import re
|
||||
|
||||
from ba_auto import detector, navigation
|
||||
|
||||
|
||||
def _open_gem_shop(driver, config):
|
||||
@ -59,10 +84,74 @@ def _open_gem_shop(driver, config):
|
||||
return False
|
||||
|
||||
|
||||
def _open_daily_general_tab(driver, config):
|
||||
def _find_sidebar_label(driver, config, label):
|
||||
"""OCR the gem shop's own left sidebar for `label` and return its
|
||||
actual on-screen position, or None if not found in its normal
|
||||
(unselected) styling."""
|
||||
for x, y, text in detector.read_lines(config.GEM_SHOP_SIDEBAR_REGION):
|
||||
if label in text:
|
||||
return (x, y)
|
||||
return None
|
||||
|
||||
|
||||
def _on_general_products_tab(driver, config):
|
||||
"""The デイリー/ウィークリー/マンスリー subtab row only renders under
|
||||
一般商品 -- checking for ウィークリー specifically (rather than
|
||||
whichever subtab gem_shop.py itself just selected) avoids ever needing
|
||||
to OCR a selected/highlighted pill, which reads unreliably (see
|
||||
detector.read_lines' docstring)."""
|
||||
return "ウィークリー" in detector.read_text(config.GEM_SHOP_WEEKLY_SUBTAB_LABEL_REGION, lang="jpn", psm=7)
|
||||
|
||||
|
||||
def _ensure_general_products_tab(driver, config):
|
||||
if _on_general_products_tab(driver, config):
|
||||
return True
|
||||
for attempt in range(1, config.GEM_SHOP_ICON_RETRIES + 1):
|
||||
driver.click(*config.GEM_SHOP_GENERAL_PRODUCTS_TAB)
|
||||
pos = _find_sidebar_label(driver, config, "一般商品")
|
||||
if pos is None:
|
||||
print(f"[gem_shop] could not OCR '一般商品' in the sidebar (attempt {attempt}/{config.GEM_SHOP_ICON_RETRIES})")
|
||||
driver.wait(1)
|
||||
continue
|
||||
driver.click(*pos)
|
||||
driver.wait(1.5)
|
||||
if _on_general_products_tab(driver, config):
|
||||
return True
|
||||
print(f"[gem_shop] general products tab not confirmed after click (attempt {attempt}/{config.GEM_SHOP_ICON_RETRIES})")
|
||||
return False
|
||||
|
||||
|
||||
def _read_card_lines(driver, config):
|
||||
return detector.read_lines(config.GEM_SHOP_FREE_CARD_REGION)
|
||||
|
||||
|
||||
def _read_free_card_status(driver, config):
|
||||
for _, _, text in _read_card_lines(driver, config):
|
||||
if "購入可能" not in text:
|
||||
continue
|
||||
counts = re.findall(r"(\d+)回", text)
|
||||
if counts:
|
||||
return "claimed" if counts[0] == "0" else "available"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _find_buy_button(driver, config):
|
||||
"""The free card's own 無料 (free) buy button, located by OCR rather
|
||||
than a fixed coordinate. "無料" can also appear as a substring of the
|
||||
card's title (毎日無料パッケージ) in the same region -- picking the
|
||||
shortest matching line prefers the standalone button label over that
|
||||
longer title line."""
|
||||
candidates = [(x, y, text) for x, y, text in _read_card_lines(driver, config) if "無料" in text]
|
||||
if not candidates:
|
||||
return None
|
||||
x, y, _ = min(candidates, key=lambda c: len(c[2]))
|
||||
return (x, y)
|
||||
|
||||
|
||||
def _open_daily_general_tab(driver, config):
|
||||
if not _ensure_general_products_tab(driver, config):
|
||||
print("[gem_shop] could not confirm the general products tab opened")
|
||||
return "unknown"
|
||||
for attempt in range(1, config.GEM_SHOP_ICON_RETRIES + 1):
|
||||
driver.click(*config.GEM_SHOP_DAILY_SUBTAB)
|
||||
driver.wait(1.5)
|
||||
status = _read_free_card_status(driver, config)
|
||||
@ -72,21 +161,12 @@ def _open_daily_general_tab(driver, config):
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _color_matches(rgb, target, tolerance):
|
||||
return all(abs(c - t) <= tolerance for c, t in zip(rgb, target))
|
||||
|
||||
|
||||
def _read_free_card_status(driver, config):
|
||||
rgb = driver.color_at(*config.GEM_SHOP_FREE_CARD_STATUS_PROBE)
|
||||
if _color_matches(rgb, config.GEM_SHOP_STATUS_CLAIMED_RGB, config.GEM_SHOP_STATUS_TOLERANCE):
|
||||
return "claimed"
|
||||
if _color_matches(rgb, config.GEM_SHOP_STATUS_AVAILABLE_RGB, config.GEM_SHOP_STATUS_TOLERANCE):
|
||||
return "available"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _claim_free_package(driver, config):
|
||||
driver.click(*config.GEM_SHOP_FREE_CARD_BUY_BUTTON)
|
||||
pos = _find_buy_button(driver, config)
|
||||
if pos is None:
|
||||
print("[gem_shop] could not OCR the free package's buy button")
|
||||
return False
|
||||
driver.click(*pos)
|
||||
driver.wait(2)
|
||||
for _ in range(config.GEM_SHOP_CLAIM_MAX_ATTEMPTS):
|
||||
if _read_free_card_status(driver, config) == "claimed":
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user