Enhance OCR name matching and purchase handling in shop utilities

- Introduced a fallback mechanism using difflib for improved name matching when OCR results contain internal character corruption.
- Implemented a two-stage fallback for name matching: stripping whitespace and using similarity scoring if no exact or substring matches are found.
- Modified `select_targets_by_name` to handle incremental purchases, committing batches before reopening the shop tab to avoid under-purchases due to UI state resets.
- Added retry logic for handling OCR instability, allowing multiple attempts to read item names before resorting to reopening the shop tab.
- Updated the return values of `select_targets_by_name` to reflect purchased items and skipped targets, improving clarity on transaction outcomes.
This commit is contained in:
Nik Afiq 2026-09-24 13:25:56 +09:00
parent 3d8af066fa
commit 5a6d964fe8
4 changed files with 320 additions and 23 deletions

View File

@ -823,7 +823,32 @@ SHOP_PRICE_OCR_RETRIES = 3
# Common Shop target name (single-line names only -- two-line-wrapping
# names like the bundle/material items sit differently and aren't covered,
# but none of the configured targets below wrap).
SHOP_NAME_OCR_OFFSET = (-75, -105, 160, -45)
#
# Left edge moved from -75 to -30 (2026-09-24, live incident): a promotional
# ribbon/badge decoration -- confirmed live on '初級強化珠' (Novice tier) and
# '最上級強化珠' (Highest tier) specifically, not on the two middle tiers --
# overlaps the left ~40px of the old crop, corrupting the OCR'd name badly
# enough that neither the exact, substring, nor fuzzy-similarity match stage
# in _match_name could resolve it (confirmed live: read as total garbage like
# '入弧比味' for '初級強化珠'). Pixel-column analysis of real live crops
# (scratchpad captures, both ribbon colors) found the decoration itself never
# extends past x=42 within the old crop, with the actual name text not
# starting until x=68-82 -- so shifting the left edge by +45px clears every
# observed ribbon with margin to spare, while the right edge is untouched
# (still wide enough for the longest configured name, e.g. 最上級強化珠).
#
# Top edge moved from -105 to -95 (2026-09-24, same incident, found after
# the left-edge fix above still left '最上級強化珠' unmatched across 12
# straight reopen attempts): even with a raw crop that looked completely
# clean to the eye, the actual thresholded/upscaled image tesseract sees
# had two small stray dark blobs in its first ~5px, above the real text --
# confirmed by row-by-row brightness analysis of a live crop, blob at
# y=0-4, then clean background to y=26, then real text starting at y=28.
# These are faint enough to be easy to miss by eye but enough to confuse
# psm 7's single-text-line assumption. Shifting the top edge down by 10px
# clears the blob with comfortable margin before the real text on every
# row, ribbon-affected or not.
SHOP_NAME_OCR_OFFSET = (-30, -95, 160, -45)
# A point inside each cell's own buy-button bar. First calibrated against
# two downloaded screenshots as a plain brightness check (purchasable
# bright ~(235,241,241) vs. sold-out/greyed ~172-174 flat) -- but a live
@ -870,6 +895,46 @@ SHOP_SCROLL_REALIGN_MAX_NUDGES = 5
# the live catalog was ~6 rows deep (24 slots) when this was built, so this
# leaves headroom without scrolling forever if that detection ever misses.
SHOP_NAME_SCAN_MAX_STEPS = 12
# How many times select_targets_by_name retries a full row0/row1 pass that
# matched nothing before concluding the current (unscrolled) viewport
# genuinely has nothing left and scrolling away from it. Added 2026-09-24:
# real live testing (five consecutive, purchase-free selection-only runs)
# found OCR reads of the exact same still-visible, still-purchasable cell
# flip between clean and corrupted from one fresh screenshot to the next --
# a transient capture/render issue, not a fixed per-item bug (which target
# failed varied run to run). A single bad pass used to be treated as proof
# nothing was left, immediately triggering a scroll and compounding the
# problem with that operation's own separate fragility -- retrying first
# gives a transient bad frame a chance to clear before paying that cost.
#
# **Lowered from 10 to 2 (2026-09-24)**: a live stability test proved this
# retry, as originally designed, can never help -- reading the exact same
# still-open cell 5 times in a row gave the identical wrong text every
# single time, so the corruption is stable for as long as the tab stays
# open, not a per-screenshot flake that a wait-and-reread can catch. Kept
# small (covers a genuine one-off, e.g. a mid-transition frame right after
# a click) rather than removed outright, but the real fix is
# SHOP_REOPEN_MAX_ATTEMPTS below -- a full tab reopen actually gets a fresh
# render, confirmed live to differ on 4 of 5 reopens for the same cell.
SHOP_NO_MATCH_RETRIES = 2
# How many times select_targets_by_name fully leaves and re-enters the shop
# tab, once SHOP_NO_MATCH_RETRIES's in-place rescans are exhausted, before
# falling back to scrolling. Added 2026-09-24 alongside the retry count
# above -- see shop_utils.select_targets_by_name's own docstring for the
# live evidence this was built from. **Raised from 5 to 12** the same day:
# 5 was enough for most stuck targets (a single-character OCR substitution
# usually clears _match_name's fuzzy-similarity threshold within a couple
# of reopens), but one real live target's corruption pattern happened to
# score higher against a WRONG neighbor (0.727) than its own true target
# (0.667) -- both below/at the fuzzy threshold either way -- so it needs a
# reopen that renders noticeably more cleanly than usual, not just
# differently, to ever match; exhausted all 5 attempts in one real run.
# Giving more attempts (each one just a normal navigation cycle, not a
# blind guess) is the correct lever here, not lowering the similarity
# threshold -- that was tried and tested worse, since a looser threshold
# let OTHER, previously-correct fuzzy matches resolve to the wrong
# same-category neighbor instead.
SHOP_REOPEN_MAX_ATTEMPTS = 12
# Extra wait after a confirmed checkbox click, before select_targets_by_name
# rescans the grid for its next target -- on top of the 0.3s already spent
# confirming the checkbox itself registered. Added 2026-09-17: a real live

View File

@ -319,8 +319,26 @@ def read_text_white_on_dark(region, whitelist=None, psm=7, lang="eng", threshold
def read_int(region, psm=7):
"""Local equivalent of the reference's recognize_int."""
digits = "".join(ch for ch in read_text(region, whitelist="0123456789", psm=psm) if ch.isdigit())
"""Local equivalent of the reference's recognize_int.
Whitelist includes ',' even though the return value is comma-stripped
digits only -- a real live misread (2026-09-24) found a digit-only
whitelist forces tesseract to misclassify a genuine thousands-separator
comma as a phantom digit instead of punctuation it's simply not allowed
to output, corrupting the whole read (confirmed live: Common Shop's
'中級レポート' at 125,000 read as 1250060/195900/4925000 depending on
the exact frame, never the correct value, across a digit-only whitelist
and multiple psm modes) -- allowing ',' through lets tesseract correctly
segment it as punctuation, and the existing isdigit() filter here already
strips it either way, so this is a strict improvement with no behavior
change for any comma-free read (region numbers, ticket counts, etc.).
This same whitelist also covers config.CREDIT_BALANCE_OCR_RECT reads
(routinely multi-comma, e.g. "412,271,037"), used by
shop_utils.confirm_purchase's balance-delta verification -- a likely
contributor to that function's own intermittent "purchase confirmation
did not resolve" false negatives, though not independently confirmed.
"""
digits = "".join(ch for ch in read_text(region, whitelist="0123456789,", psm=psm) if ch.isdigit())
return int(digits) if digits else None

File diff suppressed because one or more lines are too long

View File

@ -28,8 +28,16 @@ already gives. Tactical Shop (`select_targets`, unchanged) has never shown
this symptom -- one row, two items, no observed reordering -- so it stays
on the simpler position-based path rather than migrating speculatively.
"""
import difflib
from ba_auto import detector, navigation
# _match_name's last-resort fallback for OCR text with an internal inserted/
# dropped character that breaks plain substring containment (see that
# function's own docstring for the live incident and real similarity scores
# this was calibrated against).
SIMILARITY_THRESHOLD = 0.75
def _checkbox_center(config, row, col):
return config.SHOP_ITEM_COL_X[col], config.SHOP_ITEM_ROW_Y[row]
@ -129,17 +137,58 @@ def _match_name(name_text, remaining):
An exact match is checked first and always wins outright. The
substring fallback (for a genuinely imperfect OCR read missing/adding
a stray character) still exists, but picks the LONGEST candidate --
the more specific name -- to minimize the same class of collision.
a stray character at the START or END) still exists, but picks the
LONGEST candidate -- the more specific name -- to minimize the same
class of collision.
**Real live under-match, 2026-09-24**: even with a clean, purchasable,
correctly-priced card on screen, several targets still went unmatched
every single pass, forcing an unnecessary scroll (and from there into
the separate scroll-misalignment failure -- see _grid_looks_aligned).
Root-caused with a per-cell verbose trace against the live game: the
OCR noise wasn't just a prefix/suffix character any more -- it was an
INTERNAL corruption breaking contiguity, e.g. '初級レポート' (12,500)
read as '初級レポボート' (a phantom character inserted mid-string) and
'中級強化珠' (40,000) read as '中級化珠' (a real character, '強',
dropped from the middle) -- both stable/repeatable across several
consecutive fresh screenshots of the same still-unclicked cell, not a
one-off flake. Neither direction of plain substring containment can
ever match these: the target string literally does not appear
contiguously inside the corrupted OCR text, or vice versa. A separate
case, '中級レポート' read as '。 日 中級レポー ト', had a plain space
inserted before the final character -- also breaks substring
containment, but is fixed for free by stripping whitespace before any
comparison (real target names never contain spaces).
Fixed with a two-stage fallback: (1) strip whitespace before every
comparison stage above; (2) if plain substring containment still finds
nothing, fall back to whole-string similarity (difflib.SequenceMatcher
ratio) against every remaining candidate, taking the best match if it
clears SIMILARITY_THRESHOLD. Confirmed against this incident's own
real corrupted readings: '初級レポボート' vs '初級レポート' scores
~0.92, '中級化珠' vs '中級強化珠' scores ~0.89 -- both comfortably
above threshold, while unrelated/heavily-garbled reads (e.g. a
genuinely different item's name, or a cell read as pure noise) score
well below it in the same live samples. This is a fallback of last
resort, not a replacement for the stages above: an exact or clean
substring match is always preferred, and this project's usual
price-match safety net still guards the actual purchase regardless of
which stage resolved the name.
"""
if not name_text:
return None
if name_text in remaining:
return name_text
candidates = [n for n in remaining if n in name_text or name_text in n]
if not candidates:
return None
normalized = "".join(name_text.split())
if normalized in remaining:
return normalized
candidates = [n for n in remaining if n in normalized or normalized in n]
if candidates:
return max(candidates, key=len)
best_name, best_ratio = None, 0.0
for n in remaining:
ratio = difflib.SequenceMatcher(None, normalized, n).ratio()
if ratio > best_ratio:
best_name, best_ratio = n, ratio
return best_name if best_ratio >= SIMILARITY_THRESHOLD else None
def _scan_current_view(driver, config, remaining):
@ -240,7 +289,7 @@ def _ensure_grid_aligned(driver, config):
return False
def select_targets_by_name(driver, config, targets):
def select_targets_by_name(driver, config, targets, *, balance_rect):
"""Name-OCR-based sibling of select_targets, for grids where a bought
(or sold-out) item sinks to the bottom and shifts everything else's
position (see this module's own docstring and config.SHOP_PRICE_OCR_OFFSET's
@ -311,12 +360,114 @@ def select_targets_by_name(driver, config, targets):
available", so a run cut short this way is never confused with the
genuinely-benign sold-out/escalated-price case in the log or by anyone
reading it later.
**Real live OCR instability, 2026-09-24**: even with the fixes above,
the exact same still-visible, still-purchasable, still-unscrolled cell
would sometimes read cleanly and sometimes come back corrupted --
confirmed live across five consecutive selection-only test runs (never
clicking Buy, so free to repeat) that which of the 8 targets failed to
match varied run to run, not a fixed set tied to one item's name. A
single full-pass "nothing matched" was previously treated as proof the
remaining targets weren't in the current view at all, immediately
triggering a scroll -- compounding the problem, since scrolling has its
own separate fragility (see _grid_looks_aligned). First fix attempt:
retry a "nothing matched" pass up to `config.SHOP_NO_MATCH_RETRIES`
times, waiting between attempts, before giving up and scrolling.
**That in-place retry turned out not to work, confirmed live the same
session**: reading the exact same still-unclicked cell 5 times in a row
without leaving the shop tab gave the identical wrong text every time
(e.g. '中級化珠' for '中級強化珠', all 5 reads) -- the corruption is
stable for as long as the tab stays open, not a per-frame flake, so
re-reading in place can never see anything different. Fully leaving and
re-entering the shop tab, by contrast, gave a DIFFERENT reading on
4 of 5 attempts in the same live test (one of them exactly correct) --
proving the render itself resets on each fresh tab-open, even though it
doesn't change while sitting on it. Fixed by escalating to a full
return-to-home-and-reopen cycle (bounded by
`config.SHOP_REOPEN_MAX_ATTEMPTS`) once the in-place retries are
exhausted, before ever falling back to scrolling -- most of that live
test's reopen variants (e.g. '中級衝化珠', '中級測化珠', both a single
substituted character) already clear `_match_name`'s fuzzy-similarity
threshold on their own, so a reopen or two is normally enough. A
successful match resets `reopen_attempts` back to 0, so a later,
unrelated stuck target still gets its own full budget.
**Real live under-purchase, 2026-09-24, same day**: leaving the shop
tab does not just reset the render -- it also unchecks every checkbox
already ticked this call, since checked state lives in the tab's own
UI, not the account, until Buy is actually clicked. The first version
of the reopen fix above didn't account for this: a real cron-triggered
run logged all 8 targets as matched (zero skips, `selected` summing to
the full 1,211,500) after one reopen mid-scan, yet the real purchase
only deducted 125,000 -- exactly one target's price, the one found
right after that reopen. Every target confirmed-checked *before* the
reopen had silently gone back to unchecked in the live game, but
`selected` still listed them as done, so confirm_purchase clicked Buy
against a cart that was actually down to one item. Confirmed via the
account's own real credit balance delta.
**First fix (moving already-selected entries back into `remaining` on
every reopen) was correct but pathological**: it eliminated the
under-purchase, but since a reopen now had to re-earn EVERY target
checked so far, not just the one that got stuck, one flaky match
anywhere could force the whole session to redo several already-solved
targets -- confirmed live: a real run cycled through 20+ consecutive
reopens (each one individually reasonable) chasing a shrinking-then-
growing batch, and burned enough wall-clock time doing so that it
eventually collided with a real scheduled `q4h` cron fire on the same
game window, corrupting screenshots for both processes and crashing.
**Real fix: buy each batch before it can be invalidated**, rather than
holding the whole cart in memory across a reopen. `current_batch`
(this session's checked-but-not-yet-bought items) is committed via a
real, balance-verified `confirm_purchase` call -- exactly like a normal
end-of-run purchase, just potentially several times per call instead of
once -- immediately before any reopen, and once more at the very end.
A committed batch moves into `purchased` and can never be undone by a
later reopen, since the real account balance already reflects it. This
function now owns its own purchase confirmation end-to-end (hence the
required `balance_rect` and the `HANDLES_OWN_PURCHASE` flag below) --
`run_shop_tab` no longer buys on this path's behalf. If a batch
purchase itself fails to verify, scanning stops immediately (further
spending on top of an already-unclear balance state would only make a
manual check harder), and every still-remaining target is reported
with a distinct 'unrecognized_state' reason rather than the
genuinely-benign "not found available".
"""
remaining = {name: price for name, price in targets}
selected = []
current_batch = []
purchased = []
prev_shot = None
blocked_by_modal = False
for _ in range(config.SHOP_NAME_SCAN_MAX_STEPS):
purchase_failed = False
scroll_steps = 0
reopen_attempts = 0
def commit_current_batch():
"""Buy whatever's checked right now, before anything (a reopen) can
invalidate it. Returns False if the purchase itself doesn't verify
(see confirm_purchase) -- a real, distinct failure from "nothing to
buy", which the caller must stop on rather than paper over."""
nonlocal current_batch, purchase_failed
if not current_batch:
return True
balance_before = detector.read_int(balance_rect)
if balance_before is None:
print("[shop] could not read balance before committing this batch -- stopping without buying")
purchase_failed = True
return False
spent = sum(price for _, price in current_batch)
if not confirm_purchase(driver, config, balance_rect=balance_rect, balance_before=balance_before, expected_spend=spent):
print(f"[shop] purchase confirmation for this batch ({len(current_batch)} item(s), {spent}) did not resolve to a verified balance change -- stopping, check the game manually")
purchase_failed = True
return False
print(f"[shop] bought {len(current_batch)} item(s) for {spent} this batch: {[name for name, _ in current_batch]}")
purchased.extend(current_batch)
current_batch = []
return True
while scroll_steps < config.SHOP_NAME_SCAN_MAX_STEPS:
if not remaining:
break
if navigation.is_modal_open(driver):
@ -329,27 +480,62 @@ def select_targets_by_name(driver, config, targets):
break
prev_shot = shot
no_match_streak = 0
stuck = False
while remaining:
match = _scan_current_view(driver, config, remaining)
if match is None:
no_match_streak += 1
if no_match_streak < config.SHOP_NO_MATCH_RETRIES:
driver.wait(0.5)
continue
stuck = True
break
selected.append((match, remaining.pop(match)))
no_match_streak = 0
reopen_attempts = 0
current_batch.append((match, remaining.pop(match)))
if not remaining:
break
if stuck and reopen_attempts < config.SHOP_REOPEN_MAX_ATTEMPTS:
reopen_attempts += 1
print(f"[shop] no match after {config.SHOP_NO_MATCH_RETRIES} in-place rescans -- reopening the shop tab for a fresh render (attempt {reopen_attempts}/{config.SHOP_REOPEN_MAX_ATTEMPTS})")
if not commit_current_batch():
break
navigation.return_to_home(driver)
driver.click(*config.SHOP_ICON)
driver.wait(2)
driver.click(*config.SHOP_TAB_COMMON)
driver.wait(1.5)
prev_shot = None
continue
if remaining:
driver.scroll(*config.SHOP_SCROLL_POINT, "down", clicks=config.SHOP_SCROLL_STEP_CLICKS)
driver.wait(0.3)
scroll_steps += 1
if not _ensure_grid_aligned(driver, config):
print("[shop] grid scroll landed on a misaligned frame and could not realign -- stopping scan early (remaining targets may still be genuinely available)")
break
if not purchase_failed:
commit_current_batch()
skipped = []
for name in remaining:
if blocked_by_modal:
if purchase_failed:
print(f"[shop] '{name}' scan stopped after a batch purchase failed to verify -- NOT confirmed unavailable, check the game manually")
skipped.append((name, "unrecognized_state"))
elif blocked_by_modal:
print(f"[shop] '{name}' scan aborted by a blocking dialog -- NOT confirmed unavailable, check the game manually")
skipped.append((name, "blocked_by_modal"))
else:
print(f"[shop] '{name}' not found available at its expected price {remaining[name]} -- skipping (sold out this cycle, or catalog changed)")
skipped.append((name, "not_available"))
return selected, skipped
return purchased, skipped
select_targets_by_name.HANDLES_OWN_PURCHASE = True
def confirm_purchase(driver, config, *, balance_rect, balance_before, expected_spend):
@ -419,6 +605,13 @@ def run_shop_tab(driver, config, *, tab_button, targets, balance_rect, currency_
select_targets_by_name instead, for its (name, price) targets -- see
that function's docstring for why Common Shop needs name-based
identification specifically.
`select_fn` normally only selects (checkbox state); this function buys
once at the end. select_targets_by_name is the one exception, flagged
via its own `HANDLES_OWN_PURCHASE` attribute -- it needs to buy
incrementally in its own batches (see its docstring's 2026-09-24
entries for why one end-of-scan purchase isn't safe for it), so this
function defers to whatever it already bought instead of buying again.
"""
if not targets:
print(f"[shop] no targets configured for {currency_label}, nothing to do")
@ -440,6 +633,27 @@ def run_shop_tab(driver, config, *, tab_button, targets, balance_rect, currency_
print(f"[shop] insufficient {currency_label}: have {balance}, need {total_cost} -- stopping, nothing bought")
return "inadequate_assets"
if getattr(select_fn, "HANDLES_OWN_PURCHASE", False):
# select_targets_by_name buys as it goes (possibly in several
# batches, each independently balance-verified) rather than
# returning an unbought selection for this function to purchase in
# one shot -- see that function's own docstring for why a single
# end-of-scan purchase isn't safe for it. `purchased` below is
# already real, confirmed spending, not a pending cart.
purchased, skipped = select_fn(driver, config, targets, balance_rect=balance_rect)
if not purchased:
if any(reason == "unrecognized_state" for _, reason in skipped):
print("[shop] a batch purchase failed to verify partway through -- stopping, check the game manually")
return "unrecognized_state"
print("[shop] nothing purchased (all targets skipped)")
return "nothing_selected"
spent = sum(price for _, price in purchased)
msg = f"[shop] bought {len(purchased)} item(s) for {spent} {currency_label}"
if skipped:
msg += f", skipped {len(skipped)}"
print(msg)
return "purchased"
selected, skipped = select_fn(driver, config, targets)
if not selected:
print("[shop] nothing selected (all targets skipped), cancelling")