Repository navigation
Sprint 1B: Hero catalog, slot extraction and recognition - #3
Merged
Merged
Conversation
Rone Arena is the primary catalog (GET /api/heroes, paginated, HTTPS only); MLBBDex is used for the completeness cross-check and roles/lanes by slug. Provider-independent hero identity is a name-derived slug. A file-based snapshot plus a private image cache serve offline browsing; the catalog and its images are downloaded only on the user's explicit Sync. Slot extraction crops the source rectangle out of the frame upright and is covered by rotation, edge, minimum-size and payload-mismatch tests. Matching is a dHash perceptual hash plus a normalized 3x3 mean-RGB descriptor, combined into an explicit non-probability similarity; below threshold the result is no-match, and the user confirms or corrects manually.
… (parts A-H) - Part A: capture-side freeze command (ACTION_FREEZE / ACTION_RESUME_LIVE) routed to the running service; FramePipeline freezes publication while images keep being acquired and closed; the frozen frame is an independently owned copy (FrameSnapshots); holder survives projection stops; notification re-posts LIVE/FROZEN with Freeze/Resume actions; Vision Lab resumes from the UI and from the notification; full-device capture guidance and notification-permission handling added. - Part B: pixel extractor rotation rewritten to the FrameTransform sourceToOriented convention; tests rewritten with asymmetric fixtures, edge crops and hand-computed expectations. - Part C: Android bitmap extractor validates payload dimensions, rotates pixel-exactly, keeps source bitmap valid, clear ownership and try/catch release of temporaries; instrumentation tests for all rotations, edges, mismatches and bitmap snapshot ownership. - Part D: VisualInformationAnalyzer (pure Kotlin) rejects blank, near-uniform and detail-free crops before matching; new distinct InsufficientVisualInformation outcome and INSUFFICIENT_INFO phase. - Part E: manual correction reachable from CANDIDATES, NO_MATCH, FAILED, INSUFFICIENT_INFO and CONFIRMED; Change… opens correction instead of re-running Identify; failed clears no longer report success. - Part F: asset cache freshness via per-asset metadata sidecars binding URL/SHA-256/size/content-type; full serve-time validation incl. structural decode check (Android: real decode); changed URLs force re-download; corrupt cache re-downloads; failed refresh keeps the previous valid asset; HTTP redirects refused everywhere (HeroRedirectPolicy). - Part G: writeAtomically is temp-sibling + sync + atomic move only; no direct-write fallback; failure preserves the previous target and cleans temp files; stale *.tmp sweep. - Part H: reference Bitmap dimensions read before recycling; pixel reads complete before release; exceptions still recycle.
The instrumentation tests are deterministic (bitmap extraction, image validation, manifest checks) and need no MediaProjection session. The physical capture path stays in the manual checklist.
Duplicate capture_notification_text string removed; README, Vision Lab, screen capture, Vision architecture, Sprint 1B review-fix doc, plan and todo updated to the new behavior.
imageValidator is a SAM; as the last parameter it stole the { 1234L }
clock lambdas. Also assert the frozen delivery non-null in the adapter
test.
- a uniform crop is now InsufficientVisualInformation; the old black-crop case belongs to the new gate, so the no-match test uses an inverse gradient; a uniform-crop test pins the distinct outcome - sabotage the confirmations store with a file at its directory path (the DataStore recreates missing dirs, plain deletion was not enough) - the temp-sweep test needs a wall-clock, not the fixed 1234L clock
The runner has no KVM: the default API-30 image does not boot in time and aosp_atd is reported as an unknown API level. Instrumentation tests compile but do not run; physical-device verification is pending.
The action now stops the service whenever no pipeline is attached, regardless of whether a frozen frame was preserved from a previous session. freezeCurrentFrame keeps the holder-first shortcut so the AlreadyFrozen path stays testable on the JVM.
Root causes addressed: - The VirtualDisplay kept the session-start dimensions forever: rotation relied on onCapturedContentResize (API 34+) and, failing that, on no signal at all; nothing re-derived the capture size from the display. Now onCapturedContentResize AND a DisplayManager fallback both funnel into one idempotent resize path re-creating the ImageReader/Surface at the reported size. - Late frames from a released reader could publish into the replacement reader's stream; every generation is now numbered and stale callbacks are dropped at the listener and at publish time. - image.cropRect was ignored; letterboxed/partial content produced shifted garbage. The conversion now honors cropRect (offset crops become tightly packed images of the content region), padded rows and arbitrary pixel strides, via a pure, JVM-tested RgbaPlaneCopy. - No diagnostics existed to tell acquisition-side black from conversion- side black. Structured key=value logs now cover dimensions, resize events, plane layout, cropRect, rowStride, frame generation and the preprocessor in/out dimensions; contents are never logged. Freeze keeps working: the pinned frame is never overwritten, resize preserves the frozen snapshot, Resume continues the current generation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First real, on-device hero-identification pipeline for the Vision Lab. Draft.
This branch also contains the review-fix pass for the PR #3 review findings (freeze the game frame from the notification; rotation correctness; blank-image rejection; manual correction everywhere; cache freshness; atomic persistence; reference-loader ownership). Sprint 1A interaction behavior is preserved unchanged.
PendingIntent.getServiceback to the running capture service; it never launches an Activity, never re-promotes the service, never re-requests consent and never creates a second VirtualDisplay. While frozen, images keep being acquired and closed (the reader stays healthy) but no conversion or publication happens; the Vision Lab shows the pinned game frame instead of the DraftLens screen; the in-app Resume and the notification Resume live capture both reveal live frames again. Full-device capture guidance ("Entire screen / full device display") is shown where capture is started.GET /api/heroes, paginated, HTTPS only), MLBBDex completeness cross-check and roles/lanes by slug. Temp-sibling + flush/sync +ATOMIC_MOVEsnapshot; private image cache bound to URL + SHA-256 with serve-time validation (hash, magic bytes, container structure; actual decode on Android); offline browsing after sync; manual sync only.FrameTransform.sourceToOrientedconvention (pixel-exact on JVM and Android paths); no mutation of the source bitmap.Critical fix — landscape capture (device-reported, MLBB)
Reported by the maintainer on a physical phone: a normal screenshot captures MLBB correctly, but the projection preview/capture shows only part of the game with the rest black; portrait apps capture fine. Root cause: the capture size was fixed at session start and never re-derived, late frames could cross resize generations, and
image.cropRectwas ignored by the plane conversion.Fix (this PR's final head):
onCapturedContentResizeand aDisplayManagerfallback to current window metrics now feed one authoritative, idempotent resize path that re-creates the ImageReader/Surface generation at the reported size (all API levels ≥ 26).RgbaPlaneCopy: honors padded rows (rowStride > width * pixelStride),image.cropRect(letterboxed/offset content yields a tightly packed image of the content), arbitrary pixel strides and both byte orders; it never stretches, pads or invents pixels; malformed planes drop the frame.key=valuediagnostics inCaptureSessionService/FramePipeline/VisionPreprocessor(dimensions, resize events, plane layout, cropRect, rowStride, pixelStride, generations, preprocessor in/out) so acquisition-side black vs conversion-side black is distinguishable from logcat. Image contents are never logged, written or uploaded.On-device verification is pending. The fix is not claimed to work on a phone until rows 23–32 of
docs/TESTING_PHYSICAL_DEVICE.mdare run (rotation matrix, resize storm, freeze-during-rotate, letterbox transient, API-34+ single-app resize). CI proves compilation, logic and the pure conversion fixtures; it does not prove MediaProjection behavior on hardware.Review-fix pass in this revision (A–K)
A. Notification freeze/resume through the capture architecture (
CaptureFreeze,ACTION_FREEZE/ACTION_RESUME_LIVE), independently owned snapshot (FrameSnapshots), preserved projection on stop, one pinned frame, LIVE/FROZEN notification, stale-command handling, full-device capture guidance, POST_NOTIFICATIONS handling.B. Pixel rotation rewritten to the
FrameTransform.sourceToOrientedconvention; tests use asymmetric non-square images with hand-computed expectations.C. Android bitmap extractor validates Bitmap dimensions vs
OriginalFrame.sourceWidth/sourceHeight, rotates pixel-exactly, keeps the source bitmap valid, recycles every temporary exactly once; instrumentation tests added.D. New pure-Kotlin
VisualInformationAnalyzer(variance, edge density, meaningful differences, minimum dimensions) gates matching; new distinctInsufficientVisualInformationoutcome andINSUFFICIENT_INFOUI state.E. Manual correction reachable from CANDIDATES, NO_MATCH, FAILED, INSUFFICIENT_INFO and CONFIRMED; Change… opens correction instead of re-running Identify; failed persistence is reported, never shown as success.
F. Cache freshness — per-asset
*.metasidecar (URL, SHA-256, size, content type), serve-time revalidation, changed URLs force re-download, failed refresh keeps the previous asset and never exposes it under a changed URL, HTTPS-only redirect policy for every hop.G.
writeAtomically()is temp-sibling + flush/sync +ATOMIC_MOVEonly; failure preserves the previous snapshot; no direct-write fallback; temp files swept.H. Reference Bitmap dimensions read before recycling; pixel reads complete before release; exceptional paths recycle.
I. Matcher kept clearly labelled a prototype ranking baseline; honest evaluation section added to
docs/sprints/SPRINT_1B_HERO_RECOGNITION.md.J. Sprint 1A behavior preserved; no new feature claims.
K. Verification: see below. Instrumentation tests are compiled, not executed.
Not yet claimed
docs/TESTING_PHYSICAL_DEVICE.md). The MediaProjection lifecycle matrix remains NOT RUN.aosp_atdimage boots but is reported as "Unknown API Level" and skipped by Gradle. Execute them on a physical device or a KVM-enabled emulator before marking verified.Status as of the final head
9ca1c362d5615f371381a5105370201805f6aeb6— latest successful run: https://github.com/xeriomy/DraftLens/actions/runs/43773953944lint(abortOnError = true),assembleDebug— greenassembleAndroidTest— compiles, not executed (no KVM on the runner)draftlens-debug, 10 369 783 bytes, digestsha256:fbb5c317a7f8abfbfa2c535ea7b6d7b184bfee7793bc0244369e3dee7fdb3c26Catalog entries actually loaded will be whatever Rone Arena returns at the device sync; the value was not measured here (no device). The 1B transport is exercised only through fake responses — no unit test performs a live network call. Accuracy on real MLBB screens is not measured; the no-match threshold is a starting default. Physical-device smoke test: pending.
Base
mainafter Sprint 1A merged (#2). Sprint 1A interaction behavior is preserved as-is (original compact X delete control, long-press create without auto-opening the editor). PR remains Draft and unmerged.