Implement dynamic OCR for gem shop sidebar navigation and free card status

- Added `read_lines` function to `detector.py` for OCR of multiple text lines in a specified region, improving the ability to dynamically locate text in the UI.
- Rewrote the gem shop task in `gem_shop.py` to utilize OCR for sidebar navigation and free card status checks, replacing fixed coordinates with dynamic text detection.
- Introduced helper functions `_find_sidebar_label`, `_on_general_products_tab`, `_ensure_general_products_tab`, `_read_card_lines`, `_read_free_card_status`, and `_find_buy_button` to streamline the OCR process and improve reliability against UI changes.
- Removed outdated color probe methods for checking the free card status, enhancing the robustness of the gem shop interaction flow.
- Live-calibrated the new OCR functionality to ensure accurate detection of UI elements and their states.

ba_auto/config.py

@@ -1435,37 +1435,72 @@ BOUNTY_SWEEP_CONFIRM_TEXT_RECT = (605, 505, 1320, 615)
 #     for the "available -> claim" path in general. Re-confirm the exact
 #     first time gem_shop actually claims a package post-redesign.
 GEM_SHOP_ICON = (204, 345)
-# 一般商品 (general products), 3rd item in the left sidebar
-# (おすすめの商品/限定商品/一般商品/定額商品/青輝石/特別支援). Always
-# clicked explicitly rather than relying on it already being selected --
-# even though it was the landing tab in this recalibration, per this
-# project's own established discipline of never trusting a remembered/
-# default UI selection before a real click (see memory: BA count/quantity
-# steppers remember last-used values; the same caution applies to any
-# selected-tab state here).
-GEM_SHOP_GENERAL_PRODUCTS_TAB = (196, 376)
-# デイリー (daily), 1st of 3 sub-tabs (デイリー/ウィークリー/マンスリー)
-# shown top-right once 一般商品 is open. Also always clicked explicitly for
-# the same reason as GEM_SHOP_GENERAL_PRODUCTS_TAB above.
-GEM_SHOP_DAILY_SUBTAB = (1330, 148)
-# 毎日無料パッケージ card's own 購入 (purchase) button -- the only card
-# shown under 一般商品 -> デイリー (fixed position, no scrolling/search
-# needed).
-GEM_SHOP_FREE_CARD_BUY_BUTTON = (576, 630)
 
-# The free card's own status bar (just below its artwork, above the price
+# RECALIBRATED AGAIN 2026-09-17 after a real live failure ("free card status
-# button) reads a flat, highly distinct color depending on claim state --
+# not recognized after click", 3/3 attempts, reported live with the run's
-# see the RECALIBRATED note above for how each value was obtained:
+# own log). Root-caused by driving the real screen step by step on nik-gpu
-#   available ("一日にN回まで購入可能"): flat light blue ~(226, 236, 246)
+# (scratchpad/probe_gem_shop_careful_nav.py/probe_gem_shop_general_tab_fixed.py):
-#   claimed   ("一日に0回まで購入可能"):  flat dark red   ~(190, 56, 66)
+# the sidebar gained a new 限定商品 (NEW) entry above 一般商品 sometime after
-# The two are far enough apart (light vs dark, blue-family vs red-family)
+# the 2026-07-29 recalibration above, which is now
-# that no OCR of the "0"/"N" count text is needed -- the reference itself
+# おすすめの商品/限定商品/定額商品/一般商品/青輝石/特別支援/ステップアップ商品
-# doesn't OCR this either, it template-matches two whole separate
+# (also note 定額商品 and 一般商品 swapped order from the 2026-07-29 comment,
-# "purchasable"/"non-purchasable" card images.
+# and a new ステップアップ商品 row was appended -- see memory:
-GEM_SHOP_FREE_CARD_STATUS_PROBE = (410, 485)
+# project_gem_shop_stepup_not_automated). The old fixed
-GEM_SHOP_STATUS_AVAILABLE_RGB = (226, 236, 246)
+# GEM_SHOP_GENERAL_PRODUCTS_TAB=(196, 376) coordinate, aimed at the 3rd slot,
-GEM_SHOP_STATUS_CLAIMED_RGB = (190, 56, 66)
+# now lands on 定額商品's 3rd slot instead of 一般商品's real 4th slot --
-GEM_SHOP_STATUS_TOLERANCE = 20
+# confirmed live by screenshot, the run was silently landing on and reading
+# an entirely unrelated tab's card every attempt, never able to match either
+# status color there. Retrying the exact same wrong click three times could
+# never recover, matching the reported symptom exactly.
+#
+# Per explicit user request ("make the button press dynamic. Read word then
+# click the correct position of the word"), the sidebar click, the status
+# read, and the buy-button click are no longer fixed coordinates/colors at
+# all -- gem_shop.py now OCRs each of these dynamically every run via
+# detector.read_lines() and clicks/parses whatever position the target text
+# actually rendered at, so a future sidebar reorder (or another new entry
+# above 一般商品) can't silently break this again the same way. See
+# gem_shop.py's module docstring for the live OCR calibration writeup
+# (scratchpad/probe_gem_shop_ocr_regions.py) -- both the sidebar
+# (vertically-stacked labels) and the free card's own status/button text
+# read cleanly via one detector.read_lines() call per region, in their
+# normal (unselected, dark-text-on-light) state.
+#
+# The one exception is the デイリー/ウィークリー/マンスリー subtab row,
+# which stays a fixed click below: it is a horizontal (side-by-side) layout,
+# confirmed live NOT to segment reliably as one OCR region (tesseract
+# fragments multi-column text far worse than the vertically-stacked sidebar
+# or card regions -- see the module docstring), and unlike
+# GEM_SHOP_GENERAL_PRODUCTS_TAB above, this exact coordinate was NOT the
+# demonstrated root cause (it landed correctly every time in live testing).
+# GEM_SHOP_WEEKLY_SUBTAB_LABEL_REGION below reads ウィークリー as a
+# read-only "did we actually land on 一般商品" confirmation instead --
+# ウィークリー is never the tab gem_shop.py itself selects, so it stays in
+# its normal unselected (OCR-readable) state every run.
+#
+# 一般商品 (general products) sidebar item -- OCR target, not a coordinate.
+# Region covers the whole scrollable sidebar column so it keeps working
+# regardless of how many entries render above 一般商品.
+GEM_SHOP_SIDEBAR_REGION = (40, 130, 360, 900)
+# ウィークリー subtab label -- OCR target used only to confirm we're on the
+# 一般商品 tab (this subtab row only renders there); not clicked.
+GEM_SHOP_WEEKLY_SUBTAB_LABEL_REGION = (1447, 120, 1660, 180)
+# デイリー (daily), 1st of 3 sub-tabs (デイリー/ウィークリー/マンスリー)
+# shown top-right once 一般商品 is open. Always clicked explicitly rather
+# than relying on it already being selected, per this project's own
+# established discipline of never trusting a remembered/default UI
+# selection before a real click (see memory: BA count/quantity steppers
+# remember last-used values; the same caution applies to any selected-tab
+# state here). See the note above for why this one stays a fixed
+# coordinate while the sidebar click does not.
+GEM_SHOP_DAILY_SUBTAB = (1330, 148)
+# 毎日無料パッケージ card's content area (title, status line, and 無料 buy
+# button) -- the only card shown under 一般商品 -> デイリー (no
+# scrolling/search needed). OCR target region, not a fixed button
+# coordinate: gem_shop.py reads every text line in here and finds "無料"
+# (the buy button) and the "◯回まで購入可能" status line dynamically,
+# rather than assuming either one's exact position or color.
+GEM_SHOP_FREE_CARD_REGION = (397, 220, 760, 680)
 
 GEM_SHOP_ICON_RETRIES = 3
 # Bounds the "press Enter, re-check the free card's status" loop that

ba_auto/detector.py

@@ -286,6 +286,68 @@ def read_int_bordered(region, psm=7, border=20):
     return int(digits) if digits else None
 
 
+def read_lines(region, lang="jpn", psm=6, image=None):
+    """OCR `region` and return `(x, y, text)` for every distinct text line
+    found in it -- `(x, y)` is that line's on-screen center (for clicking
+    it) and `text` is the line's full content, re-joined left-to-right from
+    tesseract's own per-token boxes (CJK text gets segmented into several
+    word/glyph tokens even on one visual line).
+
+    Unlike read_text/read_int (a single known-fixed crop, content only, no
+    position), this scans a region that may contain several stacked lines
+    at once and reports WHERE each one actually rendered -- for a caller
+    that needs to dynamically locate and click a label/button whose exact
+    position isn't a reliable fixed coordinate, e.g. a sidebar list that
+    reflows when the game adds a new entry above an existing one. Built for
+    gem_shop.py after a real live failure where a fixed sidebar coordinate,
+    aimed at 一般商品 (general products), silently started landing on
+    定額商品 instead once the game added a new entry above it -- see
+    gem_shop.py's module docstring and config.py's "RECALIBRATED AGAIN
+    2026-09-17" note for the full incident.
+
+    Only confirmed reliable for vertically-stacked lines rendered in this
+    UI's usual dark-text-on-light styling (same assumption `_ocr_crop`
+    already makes elsewhere) -- confirmed live NOT to segment a horizontal
+    row of side-by-side labels (e.g. gem shop's own
+    デイリー/ウィークリー/マンスリー sub-tabs) reliably in one call; OCR
+    each column as its own small region instead for that shape. Also not
+    reliable against a selected/highlighted item's own inverted (light-
+    text-on-dark-pill) styling -- confirmed live to misread rather than
+    just fail quietly, so prefer checking for a different, still-unselected
+    marker over re-reading a label the caller itself just selected.
+    """
+    x1, y1, x2, y2 = region
+    img = image if image is not None else driver.read_screenshot(OCR_SHOT_PATH)
+    crop = img[y1:y2, x1:x2]
+    gray = cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY)
+    _, thresh = cv2.threshold(gray, 150, 255, cv2.THRESH_BINARY)
+    upscaled = cv2.resize(thresh, None, fx=OCR_UPSCALE, fy=OCR_UPSCALE, interpolation=cv2.INTER_CUBIC)
+
+    tess_config = f"--psm {psm}"
+    data = pytesseract.image_to_data(upscaled, lang=lang, config=tess_config, output_type=pytesseract.Output.DICT)
+
+    lines = {}
+    for i, word in enumerate(data["text"]):
+        word = word.strip()
+        if not word:
+            continue
+        key = (data["block_num"][i], data["par_num"][i], data["line_num"][i])
+        lines.setdefault(key, []).append((data["left"][i], data["top"][i], data["width"][i], data["height"][i], word))
+
+    results = []
+    for tokens in lines.values():
+        tokens.sort(key=lambda t: t[0])
+        joined = "".join(t[4] for t in tokens)
+        lefts = [t[0] for t in tokens]
+        rights = [t[0] + t[2] for t in tokens]
+        tops = [t[1] for t in tokens]
+        bottoms = [t[1] + t[3] for t in tokens]
+        cx = (min(lefts) + max(rights)) / 2 / OCR_UPSCALE
+        cy = (min(tops) + max(bottoms)) / 2 / OCR_UPSCALE
+        results.append((x1 + round(cx), y1 + round(cy), joined))
+    return results
+
+
 def read_int_on_heart_badge(region, psm=7, image=None):
     """OCR a small dark-navy digit rendered on lesson.py's pink/magenta
     heart-shaped affection badge.

ba_auto/tasks/gem_shop.py

@@ -9,12 +9,36 @@ wait for the reward) else log "already collected" -> return_to_main_page
 equivalent server concept in this project, so not ported.
 
 The reference detects every step via image template matching (no OCR
-anywhere in this flow) -- this port uses plain color probes for the same
+anywhere in this flow). This port originally used plain color probes for
-states instead (see config.py's "Gem shop daily free package" section for
+the same states instead (see config.py's "Gem shop daily free package"
-the full live-calibration writeup and exact pixel values), matching this
+section for the original 2026-07-15/07-29 live-calibration writeup),
-project's own established equivalent for a simple, high-contrast
+matching this project's own established equivalent for a simple,
-state-A/state-B visual difference, same as cafe.py's CLAIM_DISABLED_RGB or
+high-contrast state-A/state-B visual difference, same as cafe.py's
-stamina.py's MISSION_CLAIM_PROBE.
+CLAIM_DISABLED_RGB or stamina.py's MISSION_CLAIM_PROBE.
+
+REWRITTEN 2026-09-17 after a real live failure: the sidebar gained a new
+限定商品 entry above 一般商品 sometime after the 2026-07-29 recalibration,
+silently shifting 一般商品 down one slot and making the old fixed
+GEM_SHOP_GENERAL_PRODUCTS_TAB coordinate land on 定額商品 instead --
+confirmed live via scratchpad/probe_gem_shop_careful_nav.py and
+probe_gem_shop_general_tab_fixed.py, which walked the real screen step by
+step and captured the wrong-tab landing directly. Every retry just
+re-clicked the same wrong position, matching the reported symptom exactly
+("free card status not recognized after click", 3/3 attempts).
+
+Per explicit user request ("make the button press dynamic. Read word then
+click the correct position of the word"), the sidebar navigation, the
+status read, and the buy-button click are now all OCR-driven via
+detector.read_lines() rather than fixed coordinates/colors -- each one
+finds its target text wherever it actually rendered this run and
+reads/clicks that position, so a future sidebar reorder can't silently
+break this the same way again. See config.py's "RECALIBRATED AGAIN
+2026-09-17" note and scratchpad/probe_gem_shop_ocr_regions.py for the live
+OCR calibration this was based on. The デイリー/ウィークリー/マンスリー
+subtab row is the one piece that stays a fixed click -- it's a horizontal
+layout that doesn't OCR-segment reliably as one region (see
+detector.read_lines' own docstring), and it was not the coordinate that
+actually broke.
 
 Live-calibrated 2026-07-15 by manually driving the real dialog end-to-end
 on nik-gpu, which genuinely claimed the account's real free package for the
@@ -26,15 +50,14 @@ than this actual module.
 RECALIBRATED 2026-07-29 after a real Blue Archive client UI update replaced
 the old 3-top-tab overlay dialog (期間限定/青輝石/パッケージ) with a real
 full subscreen (left sidebar + デイリー/ウィークリー/マンスリー sub-tabs
-under 一般商品). See config.py's "Gem shop daily free package" section for
+under 一般商品). See config.py for the full writeup of what was
-the full writeup of what was re-verified live and what's still an inferred
+re-verified live and what's still an inferred value. The old
-value. The old `_dialog_open`/`_close_gem_shop` custom probes existed only
+`_dialog_open`/`_close_gem_shop` custom probes existed only because the
-because the pre-redesign overlay dialog was invisible to
+pre-redesign overlay dialog was invisible to
 navigation.is_on_subscreen/is_modal_open; the redesigned screen is a real
-subscreen, confirmed live, so this module now shares the same
+subscreen, confirmed live, so this module shares the same
 is_on_subscreen/return_to_home machinery every other subscreen-based task
-(mailbox/cafe/shop/lesson) already uses, and those two custom probes are
+(mailbox/cafe/shop/lesson) already uses.
-gone.
 
 A `reference-parity-reviewer` pass on the original version caught a real
 bug: the final dialog-close step verified success via
@@ -46,7 +69,9 @@ screen from home, confirmed live), which is what made switching to the
 shared navigation.return_to_home safe here.
 """
 
-from ba_auto import navigation
+import re
+
+from ba_auto import detector, navigation
 
 
 def _open_gem_shop(driver, config):
@@ -59,10 +84,74 @@ def _open_gem_shop(driver, config):
     return False
 
 
-def _open_daily_general_tab(driver, config):
+def _find_sidebar_label(driver, config, label):
+    """OCR the gem shop's own left sidebar for `label` and return its
+    actual on-screen position, or None if not found in its normal
+    (unselected) styling."""
+    for x, y, text in detector.read_lines(config.GEM_SHOP_SIDEBAR_REGION):
+        if label in text:
+            return (x, y)
+    return None
+
+
+def _on_general_products_tab(driver, config):
+    """The デイリー/ウィークリー/マンスリー subtab row only renders under
+    一般商品 -- checking for ウィークリー specifically (rather than
+    whichever subtab gem_shop.py itself just selected) avoids ever needing
+    to OCR a selected/highlighted pill, which reads unreliably (see
+    detector.read_lines' docstring)."""
+    return "ウィークリー" in detector.read_text(config.GEM_SHOP_WEEKLY_SUBTAB_LABEL_REGION, lang="jpn", psm=7)
+
+
+def _ensure_general_products_tab(driver, config):
+    if _on_general_products_tab(driver, config):
+        return True
+    for attempt in range(1, config.GEM_SHOP_ICON_RETRIES + 1):
+        pos = _find_sidebar_label(driver, config, "一般商品")
+        if pos is None:
+            print(f"[gem_shop] could not OCR '一般商品' in the sidebar (attempt {attempt}/{config.GEM_SHOP_ICON_RETRIES})")
+            driver.wait(1)
+            continue
+        driver.click(*pos)
+        driver.wait(1.5)
+        if _on_general_products_tab(driver, config):
+            return True
+        print(f"[gem_shop] general products tab not confirmed after click (attempt {attempt}/{config.GEM_SHOP_ICON_RETRIES})")
+    return False
+
+
+def _read_card_lines(driver, config):
+    return detector.read_lines(config.GEM_SHOP_FREE_CARD_REGION)
+
+
+def _read_free_card_status(driver, config):
+    for _, _, text in _read_card_lines(driver, config):
+        if "購入可能" not in text:
+            continue
+        counts = re.findall(r"(\d+)回", text)
+        if counts:
+            return "claimed" if counts[0] == "0" else "available"
+    return "unknown"
+
+
+def _find_buy_button(driver, config):
+    """The free card's own 無料 (free) buy button, located by OCR rather
+    than a fixed coordinate. "無料" can also appear as a substring of the
+    card's title (毎日無料パッケージ) in the same region -- picking the
+    shortest matching line prefers the standalone button label over that
+    longer title line."""
+    candidates = [(x, y, text) for x, y, text in _read_card_lines(driver, config) if "無料" in text]
+    if not candidates:
+        return None
+    x, y, _ = min(candidates, key=lambda c: len(c[2]))
+    return (x, y)
+
+
+def _open_daily_general_tab(driver, config):
+    if not _ensure_general_products_tab(driver, config):
+        print("[gem_shop] could not confirm the general products tab opened")
+        return "unknown"
     for attempt in range(1, config.GEM_SHOP_ICON_RETRIES + 1):
-        driver.click(*config.GEM_SHOP_GENERAL_PRODUCTS_TAB)
-        driver.wait(1)
         driver.click(*config.GEM_SHOP_DAILY_SUBTAB)
         driver.wait(1.5)
         status = _read_free_card_status(driver, config)
@@ -72,21 +161,12 @@ def _open_daily_general_tab(driver, config):
     return "unknown"
 
 
-def _color_matches(rgb, target, tolerance):
-    return all(abs(c - t) <= tolerance for c, t in zip(rgb, target))
-
-
-def _read_free_card_status(driver, config):
-    rgb = driver.color_at(*config.GEM_SHOP_FREE_CARD_STATUS_PROBE)
-    if _color_matches(rgb, config.GEM_SHOP_STATUS_CLAIMED_RGB, config.GEM_SHOP_STATUS_TOLERANCE):
-        return "claimed"
-    if _color_matches(rgb, config.GEM_SHOP_STATUS_AVAILABLE_RGB, config.GEM_SHOP_STATUS_TOLERANCE):
-        return "available"
-    return "unknown"
-
-
 def _claim_free_package(driver, config):
-    driver.click(*config.GEM_SHOP_FREE_CARD_BUY_BUTTON)
+    pos = _find_buy_button(driver, config)
+    if pos is None:
+        print("[gem_shop] could not OCR the free package's buy button")
+        return False
+    driver.click(*pos)
     driver.wait(2)
     for _ in range(config.GEM_SHOP_CLAIM_MAX_ATTEMPTS):
         if _read_free_card_status(driver, config) == "claimed":