Skip to content

Sprint 1B: Hero catalog, slot extraction and recognition - #3

Merged
xeriomy merged 33 commits into
mainfrom
sprint-1b-hero-recognition
Nov 16, 2026
Merged

xeriomy merged 33 commits into
mainfrom
sprint-1b-hero-recognition

Conversation

@xeriomy

@xeriomy xeriomy commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

First real, on-device hero-identification pipeline for the Vision Lab. Draft.

This branch also contains the review-fix pass for the PR #3 review findings (freeze the game frame from the notification; rotation correctness; blank-image rejection; manual correction everywhere; cache freshness; atomic persistence; reference-loader ownership). Sprint 1A interaction behavior is preserved unchanged.

  • Freeze from the notification — the ongoing capture notification exposes Freeze current frame (and Resume live capture once frozen). The action is an explicit, immutable PendingIntent.getService back to the running capture service; it never launches an Activity, never re-promotes the service, never re-requests consent and never creates a second VirtualDisplay. While frozen, images keep being acquired and closed (the reader stays healthy) but no conversion or publication happens; the Vision Lab shows the pinned game frame instead of the DraftLens screen; the in-app Resume and the notification Resume live capture both reveal live frames again. Full-device capture guidance ("Entire screen / full device display") is shown where capture is started.
  • Hero catalog — provider-independent identity (name-derived slug), Rone Arena primary source (GET /api/heroes, paginated, HTTPS only), MLBBDex completeness cross-check and roles/lanes by slug. Temp-sibling + flush/sync + ATOMIC_MOVE snapshot; private image cache bound to URL + SHA-256 with serve-time validation (hash, magic bytes, container structure; actual decode on Android); offline browsing after sync; manual sync only.
  • Slot extraction — upright crop for every rotation following the FrameTransform.sourceToOriented convention (pixel-exact on JVM and Android paths); no mutation of the source bitmap.
  • Recognition v0 — dHash perceptual hash + normalized 3×3 mean-RGB descriptor, an explicitly uncalibrated heuristic similarity, not a probability. Blank/near-uniform/detail-free crops are rejected up front as InsufficientVisualInformation and never produce candidates. Below threshold the result is an honest no match; the user always confirms or corrects manually, from every recognition state. No cloud API, no ML model, no OpenCV.
  • UI — explicit Identify action; per-slot states (idle / processing / candidates / no-match / insufficient information / confirmed / failed); top-3 cards; manual confirm and manual correction available everywhere by name search; confirmed heroes persist per slot.
  • Privacy — captured frames and slot crops stay on-device and in memory. Only public static hero catalog data and reference images are downloaded. No game-memory reading, no injection, no automated touches, no upload.

Critical fix — landscape capture (device-reported, MLBB)

Reported by the maintainer on a physical phone: a normal screenshot captures MLBB correctly, but the projection preview/capture shows only part of the game with the rest black; portrait apps capture fine. Root cause: the capture size was fixed at session start and never re-derived, late frames could cross resize generations, and image.cropRect was ignored by the plane conversion.

Fix (this PR's final head):

  • onCapturedContentResize and a DisplayManager fallback to current window metrics now feed one authoritative, idempotent resize path that re-creates the ImageReader/Surface generation at the reported size (all API levels ≥ 26).
  • Every reader/surface generation is numbered; stale callbacks are dropped at the listener and again at publish time, so a late frame from a released reader can never reach the UI or the pinned freeze snapshot.
  • The plane conversion was rewritten as a pure, JVM-tested RgbaPlaneCopy: honors padded rows (rowStride > width * pixelStride), image.cropRect (letterboxed/offset content yields a tightly packed image of the content), arbitrary pixel strides and both byte orders; it never stretches, pads or invents pixels; malformed planes drop the frame.
  • Structured key=value diagnostics in CaptureSessionService / FramePipeline / VisionPreprocessor (dimensions, resize events, plane layout, cropRect, rowStride, pixelStride, generations, preprocessor in/out) so acquisition-side black vs conversion-side black is distinguishable from logcat. Image contents are never logged, written or uploaded.
  • Freeze keeps working: the pinned frame is never overwritten by stale frames; a resize during freeze pins the newest valid frame; Resume continues with the current generation; stale frames never reach the pinned slot.

On-device verification is pending. The fix is not claimed to work on a phone until rows 23–32 of docs/TESTING_PHYSICAL_DEVICE.md are run (rotation matrix, resize storm, freeze-during-rotate, letterbox transient, API-34+ single-app resize). CI proves compilation, logic and the pure conversion fixtures; it does not prove MediaProjection behavior on hardware.

Review-fix pass in this revision (A–K)

A. Notification freeze/resume through the capture architecture (CaptureFreeze, ACTION_FREEZE/ACTION_RESUME_LIVE), independently owned snapshot (FrameSnapshots), preserved projection on stop, one pinned frame, LIVE/FROZEN notification, stale-command handling, full-device capture guidance, POST_NOTIFICATIONS handling.
B. Pixel rotation rewritten to the FrameTransform.sourceToOriented convention; tests use asymmetric non-square images with hand-computed expectations.
C. Android bitmap extractor validates Bitmap dimensions vs OriginalFrame.sourceWidth/sourceHeight, rotates pixel-exactly, keeps the source bitmap valid, recycles every temporary exactly once; instrumentation tests added.
D. New pure-Kotlin VisualInformationAnalyzer (variance, edge density, meaningful differences, minimum dimensions) gates matching; new distinct InsufficientVisualInformation outcome and INSUFFICIENT_INFO UI state.
E. Manual correction reachable from CANDIDATES, NO_MATCH, FAILED, INSUFFICIENT_INFO and CONFIRMED; Change… opens correction instead of re-running Identify; failed persistence is reported, never shown as success.
F. Cache freshness — per-asset *.meta sidecar (URL, SHA-256, size, content type), serve-time revalidation, changed URLs force re-download, failed refresh keeps the previous asset and never exposes it under a changed URL, HTTPS-only redirect policy for every hop.
G. writeAtomically() is temp-sibling + flush/sync + ATOMIC_MOVE only; failure preserves the previous snapshot; no direct-write fallback; temp files swept.
H. Reference Bitmap dimensions read before recycling; pixel reads complete before release; exceptional paths recycle.
I. Matcher kept clearly labelled a prototype ranking baseline; honest evaluation section added to docs/sprints/SPRINT_1B_HERO_RECOGNITION.md.
J. Sprint 1A behavior preserved; no new feature claims.
K. Verification: see below. Instrumentation tests are compiled, not executed.

Not yet claimed

  • Accuracy on real MLBB screens is not measured. Top-1/Top-3 and rejection behavior are evaluated on synthetic fixtures only; the no-match threshold (0.55) and the matcher weights are uncalibrated starting defaults, not calibrated thresholds; the visual-information heuristics are conservative starting points, not MLBB-calibrated.
  • Physical-device smoke test of the freeze workflow: pending (rows 14–22 of docs/TESTING_PHYSICAL_DEVICE.md). The MediaProjection lifecycle matrix remains NOT RUN.
  • Instrumentation tests compiled, not executed — the CI runner has no KVM access: the default API-30 image does not boot within the timeout, and the aosp_atd image boots but is reported as "Unknown API Level" and skipped by Gradle. Execute them on a physical device or a KVM-enabled emulator before marking verified.
  • Catalog entries actually loaded are measured at device sync time, not in CI; the 1B transport is exercised only through fake responses — no unit test performs a network call.
  • No screenshots, frames or crops are committed or uploaded anywhere.

Status as of the final head

9ca1c362d5615f371381a5105370201805f6aeb6 — latest successful run: https://github.com/xeriomy/DraftLens/actions/runs/43773953944

  • 840 unit tests (debug + release), 0 failures, 0 errors, 0 skipped
  • strict lint (abortOnError = true), assembleDebug — green
  • assembleAndroidTest — compiles, not executed (no KVM on the runner)
  • Artifact draftlens-debug, 10 369 783 bytes, digest sha256:fbb5c317a7f8abfbfa2c535ea7b6d7b184bfee7793bc0244369e3dee7fdb3c26
  • Test-only commits kept the 1A interaction behavior unchanged

Catalog entries actually loaded will be whatever Rone Arena returns at the device sync; the value was not measured here (no device). The 1B transport is exercised only through fake responses — no unit test performs a live network call. Accuracy on real MLBB screens is not measured; the no-match threshold is a starting default. Physical-device smoke test: pending.

Base

main after Sprint 1A merged (#2). Sprint 1A interaction behavior is preserved as-is (original compact X delete control, long-press create without auto-opening the editor). PR remains Draft and unmerged.

Rone Arena is the primary catalog (GET /api/heroes, paginated, HTTPS only);
MLBBDex is used for the completeness cross-check and roles/lanes by slug.
Provider-independent hero identity is a name-derived slug. A file-based
snapshot plus a private image cache serve offline browsing; the catalog and
its images are downloaded only on the user's explicit Sync.

Slot extraction crops the source rectangle out of the frame upright and is
covered by rotation, edge, minimum-size and payload-mismatch tests. Matching
is a dHash perceptual hash plus a normalized 3x3 mean-RGB descriptor, combined
into an explicit non-probability similarity; below threshold the result is
no-match, and the user confirms or corrects manually.
… (parts A-H)

- Part A: capture-side freeze command (ACTION_FREEZE / ACTION_RESUME_LIVE)
  routed to the running service; FramePipeline freezes publication while
  images keep being acquired and closed; the frozen frame is an
  independently owned copy (FrameSnapshots); holder survives projection
  stops; notification re-posts LIVE/FROZEN with Freeze/Resume actions;
  Vision Lab resumes from the UI and from the notification; full-device
  capture guidance and notification-permission handling added.
- Part B: pixel extractor rotation rewritten to the FrameTransform
  sourceToOriented convention; tests rewritten with asymmetric fixtures,
  edge crops and hand-computed expectations.
- Part C: Android bitmap extractor validates payload dimensions,
  rotates pixel-exactly, keeps source bitmap valid, clear ownership and
  try/catch release of temporaries; instrumentation tests for all
  rotations, edges, mismatches and bitmap snapshot ownership.
- Part D: VisualInformationAnalyzer (pure Kotlin) rejects blank,
  near-uniform and detail-free crops before matching; new distinct
  InsufficientVisualInformation outcome and INSUFFICIENT_INFO phase.
- Part E: manual correction reachable from CANDIDATES, NO_MATCH, FAILED,
  INSUFFICIENT_INFO and CONFIRMED; Change… opens correction instead of
  re-running Identify; failed clears no longer report success.
- Part F: asset cache freshness via per-asset metadata sidecars binding
  URL/SHA-256/size/content-type; full serve-time validation incl.
  structural decode check (Android: real decode); changed URLs force
  re-download; corrupt cache re-downloads; failed refresh keeps the
  previous valid asset; HTTP redirects refused everywhere
  (HeroRedirectPolicy).
- Part G: writeAtomically is temp-sibling + sync + atomic move only;
  no direct-write fallback; failure preserves the previous target and
  cleans temp files; stale *.tmp sweep.
- Part H: reference Bitmap dimensions read before recycling; pixel
  reads complete before release; exceptions still recycle.
The instrumentation tests are deterministic (bitmap extraction, image
validation, manifest checks) and need no MediaProjection session. The
physical capture path stays in the manual checklist.
Duplicate capture_notification_text string removed; README, Vision Lab,
screen capture, Vision architecture, Sprint 1B review-fix doc, plan and
todo updated to the new behavior.
imageValidator is a SAM; as the last parameter it stole the { 1234L }
clock lambdas. Also assert the frozen delivery non-null in the adapter
test.
- a uniform crop is now InsufficientVisualInformation; the old black-crop
  case belongs to the new gate, so the no-match test uses an inverse
  gradient; a uniform-crop test pins the distinct outcome
- sabotage the confirmations store with a file at its directory path
  (the DataStore recreates missing dirs, plain deletion was not enough)
- the temp-sweep test needs a wall-clock, not the fixed 1234L clock
The runner has no KVM: the default API-30 image does not boot in time
and aosp_atd is reported as an unknown API level. Instrumentation tests
compile but do not run; physical-device verification is pending.
The action now stops the service whenever no pipeline is attached,
regardless of whether a frozen frame was preserved from a previous
session. freezeCurrentFrame keeps the holder-first shortcut so the
AlreadyFrozen path stays testable on the JVM.
Root causes addressed:
- The VirtualDisplay kept the session-start dimensions forever: rotation
  relied on onCapturedContentResize (API 34+) and, failing that, on no
  signal at all; nothing re-derived the capture size from the display.
  Now onCapturedContentResize AND a DisplayManager fallback both funnel
  into one idempotent resize path re-creating the ImageReader/Surface
  at the reported size.
- Late frames from a released reader could publish into the replacement
  reader's stream; every generation is now numbered and stale callbacks
  are dropped at the listener and at publish time.
- image.cropRect was ignored; letterboxed/partial content produced
  shifted garbage. The conversion now honors cropRect (offset crops
  become tightly packed images of the content region), padded rows and
  arbitrary pixel strides, via a pure, JVM-tested RgbaPlaneCopy.
- No diagnostics existed to tell acquisition-side black from conversion-
  side black. Structured key=value logs now cover dimensions, resize
  events, plane layout, cropRect, rowStride, frame generation and the
  preprocessor in/out dimensions; contents are never logged.

Freeze keeps working: the pinned frame is never overwritten, resize
preserves the frozen snapshot, Resume continues the current generation.
@xeriomy
xeriomy marked this pull request as ready for review November 16, 2026 19:19
@xeriomy
xeriomy merged commit 5c4ac89 into main Nov 16, 2026
1 check passed
@xeriomy
xeriomy deleted the sprint-1b-hero-recognition branch November 16, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant