Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
f91bc9d
feat: hero catalog sync, slot extraction and recognition v0
xeriomy Nov 15, 2026
2ba69e2
fix: keep the cache-read error state instead of overwriting it
xeriomy Nov 15, 2026
730b44a
fix: correct package paths and parameter order for the recognition wi…
xeriomy Nov 15, 2026
8d960b8
test: pass the hero collaborators as named arguments
xeriomy Nov 15, 2026
9055826
test: fix property name in the recognition orchestration test
xeriomy Nov 15, 2026
9573b52
test: fix float-precision expectations and real-time waits in the 1B …
xeriomy Nov 15, 2026
7d0c903
test: fix hamming vector and sub-pixel rects on the 4x2 synthetic frame
xeriomy Nov 15, 2026
3398891
docs: record the Sprint 1B green run and acceptance state
xeriomy Nov 15, 2026
b4eac9d
test: reset Main defensively in setUp to survive order-dependent leaks
xeriomy Nov 15, 2026
a8bbead
docs: record both Sprint 1B green runs
xeriomy Nov 15, 2026
e58611c
test: class-scoped Main dispatcher to kill the order-dependent race
xeriomy Nov 15, 2026
84f3d01
docs: point the Sprint 1B verification at the latest green head
xeriomy Nov 15, 2026
2e115b0
feat: freeze game frames from the capture notification + review fixes…
xeriomy Nov 15, 2026
044ddd8
ci: run the instrumented suite on an emulator
xeriomy Nov 15, 2026
24fe770
docs: freeze workflow, review fixes and reporting skeleton
xeriomy Nov 15, 2026
0e95515
fix: exhaustive when in Android bitmap extractor rotation
xeriomy Nov 15, 2026
7ce688b
fix: keep the repository clock last so trailing lambdas bind to it
xeriomy Nov 15, 2026
860c1c4
fix: test failures from the review-fix pass
xeriomy Nov 15, 2026
f2cae4a
fix: File(File) does not exist; use the path for the sabotage file
xeriomy Nov 15, 2026
5884d62
fix: dex disallows spaces in SimpleName; camelCase the instrumented t…
xeriomy Nov 15, 2026
f3ea769
ci: try the aosp_atd image (runner has no KVM)
xeriomy Nov 15, 2026
7a839d1
ci: drop the emulator job; document instrumentation as compile-only
xeriomy Nov 15, 2026
96d9a97
docs: point the Sprint 1B verification at the latest green head (7a83…
xeriomy Nov 15, 2026
4ff2a34
docs: point the verification record at the PR description
xeriomy Nov 15, 2026
e89c5e6
chore: serialize pipeline attach/detach; drop an unused predicate
xeriomy Nov 15, 2026
1c7397b
docs: drop the stale run pointer for the review-fix section
xeriomy Nov 15, 2026
864d83e
docs: record the final green head in the sprint task lists
xeriomy Nov 15, 2026
6b883ef
fix: a stale freeze action must not leave an idle service behind
xeriomy Nov 15, 2026
e6d1de8
fix: landscape MLBB capture shows the whole game (PR #3 review pass)
xeriomy Nov 15, 2026
2597fdb
fix: CapturedFrame carries no sourceWidth/sourceHeight of its own
xeriomy Nov 15, 2026
856418b
fix: check the buffer limit, not remaining, for positioned plane buffers
xeriomy Nov 15, 2026
5252016
fix: dex disallows spaces in SimpleName; camelCase the pipeline tests
xeriomy Nov 15, 2026
9ca1c36
docs: landscape capture bugfix report, diagnostics and verification m…
xeriomy Nov 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 29 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,11 @@ working capture path into a real **image-bearing** foundation: frames carry pixe
is tested, preprocessing validates before it renders, and a Vision Lab lets you place manual
slot annotations and save calibration profiles.

**Maintainer-reported device smoke:** the Sprint 0 capture path was used on a physical Android
device and confirmed working. Sprint 1A itself has **not** been verified on a physical device —
see [docs/TESTING_PHYSICAL_DEVICE.md](docs/TESTING_PHYSICAL_DEVICE.md) and
[docs/VISION_TESTING.md](docs/VISION_TESTING.md).
**Maintainer-reported device smoke:** the Sprint 0 capture path and the Sprint 1A Vision
Lab interaction behavior were used on a physical Android device and confirmed working.
Device details, the exact build and test dates were not supplied, and the MediaProjection
lifecycle matrix remains **NOT RUN**. See [docs/TESTING_PHYSICAL_DEVICE.md](docs/TESTING_PHYSICAL_DEVICE.md)
and [docs/VISION_TESTING.md](docs/VISION_TESTING.md).

## What works in Sprint 1A
- MediaProjection consent flow (`ActivityResultContracts.CreateScreenCaptureIntent`)
Expand All @@ -26,6 +27,13 @@ see [docs/TESTING_PHYSICAL_DEVICE.md](docs/TESTING_PHYSICAL_DEVICE.md) and
unsupported-schema refusal, delete-as-escape-hatch
- Vision Lab tab: frame preview, freeze/resume, explicit processing, interactive annotation
canvas, text slot editor, profile controls
- Hero recognition v0 (Sprint 1B, PR #3): hero catalog sync from public static data,
explicit Identify action for the selected slot, ranked candidates with similarity scores,
manual confirm/correct — **accuracy on real game screens is not measured yet**
- Freeze the game frame from the notification (Sprint 1B): tap **Freeze current frame** on the
capture notification while you are in MLBB — no need to leave the game; Vision Lab shows the
pinned game frame, not the DraftLens screen; **Resume live capture** from the notification or
in the Vision Lab

## What is deliberately NOT here (later milestones)
ML hero recognition, recommendation engine, counter/synergy/role scoring, floating overlays,
Expand All @@ -49,14 +57,23 @@ build of record.** See `docs/VISION_TESTING.md` for exactly what ran.
1. Install the APK from the `draftlens-debug` CI artifact on a physical device (the emulator
validates compilation, not MediaProjection reliability).
2. Open the app; the **Debug Capture** tab is first.
3. Tap **Start Capture** → grant the system screen-capture consent dialog.
4. MLBB (or any app) — the debug screen shows live frames.
5. Switch to the **Vision Lab** tab. Capture keeps running while you are there.
6. Freeze a frame, drag out a slot rectangle, then edit it in the text editor below the
canvas. Press **Process Frame** to run the preprocessing engine on the selected frame.
7. Save the annotations as a calibration profile; apply it later on a matching frame.
8. **Stop Capture**, or the notification's **Stop**, or revoke consent in Settings to
exercise teardown.
3. Tap **Start Capture** → grant the system screen-capture consent dialog. When Android asks
what to capture, choose **Entire screen / full device display** — app-window capture shows
this app, not Mobile Legends, and the notification freeze workflow would be pointless.
4. Open Mobile Legends and wait for the draft screen you want to annotate.
5. Pull down the notification shade and tap **Freeze current frame** on the DraftLens capture
notification (notifications must be enabled — the app checks and tells you).
6. Switch back to **DraftLens → Vision Lab**. Capture kept running; the Vision Lab shows the
pinned MLBB frame instead of the DraftLens screen.
7. Draw a slot rectangle, then run **Identify hero** on the frozen frame. Candidates are
similarity scores, not probabilities — confirm or correct manually (the manual search is
available in every recognition state).
8. Tap **Resume live capture** in the notification, or **Resume** in the Vision Lab, to show
live frames again. No second capture session is created.
9. Press **Process Frame** to run the preprocessing engine on the selected frame.
10. Save the annotations as a calibration profile; apply it later on a matching frame.
11. **Stop Capture**, or the notification's **Stop**, or revoke consent in Settings to
exercise teardown.

## Docs
- `docs/ARCHITECTURE.md` — module layout, state machine, data flow, Vision boundary
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,233 @@
package com.draftlens

import android.graphics.Bitmap
import androidx.test.ext.junit.runners.AndroidJUnit4
import com.draftlens.models.FrameSnapshots
import com.draftlens.vision.annotation.DraftSlot
import com.draftlens.vision.annotation.SlotTeam
import com.draftlens.vision.annotation.SlotType
import com.draftlens.vision.geometry.NormalizedRect
import com.draftlens.vision.heroes.AndroidBitmapSlotImageExtractor
import com.draftlens.vision.model.FrameId
import com.draftlens.vision.model.FrameRotation
import com.draftlens.vision.model.OriginalFrame
import com.draftlens.vision.model.SlotImage
import com.draftlens.vision.model.android.BitmapFramePayload
import org.junit.After
import org.junit.Assert.assertArrayEquals
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNotSame
import org.junit.Assert.assertTrue
import org.junit.Test
import org.junit.runner.RunWith

/**
* Device-side verification of the Android extractor: exact pixels, ownership and failure
* paths. Run on an emulator or device; without one this class is compiled but not executed.
*/
@RunWith(AndroidJUnit4::class)
class AndroidBitmapExtractionInstrumentedTest {

private val ownedBitmaps = mutableListOf<Bitmap>()

@After fun tearDown() {
ownedBitmaps.forEach { if (!it.isRecycled) it.recycle() }
ownedBitmaps.clear()
}

private fun track(bitmap: Bitmap): Bitmap = bitmap.also { ownedBitmaps.add(it) }

private fun slot(rect: NormalizedRect, id: String = "slot-1") = DraftSlot(
id = id,
type = SlotType.BAN,
team = SlotTeam.ALLY,
index = 1,
rect = rect,
)

private fun frame(width: Int, height: Int, rotation: FrameRotation, pixels: IntArray): OriginalFrame {
val bitmap = track(Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888))
bitmap.setPixels(pixels, 0, width, 0, 0, width, height)
return OriginalFrame(
id = FrameId("ft-$width-$height-$rotation"),
payload = BitmapFramePayload(bitmap),
sourceWidth = width,
sourceHeight = height,
displayRotation = FrameRotation.ROTATION_0,
rotation = rotation,
capturedAtElapsedRealtimeMs = 1,
wallClockTimestampMs = 1,
captureLatencyMs = 1,
)
}

private fun pixelsOf(image: SlotImage): Pair<IntArray, Int> {
val bitmap = (image.payload as BitmapFramePayload).bitmap
assertFalse(bitmap.isRecycled)
val pixels = IntArray(bitmap.width * bitmap.height)
bitmap.getPixels(pixels, 0, bitmap.width, 0, 0, bitmap.width, bitmap.height)
return pixels to bitmap.width
}

// Asymmetric 3x2 source; values chosen so every rotation is distinguishable.
private val asymmetric = intArrayOf(10, 11, 12, 20, 21, 22)

// 5x3 source used for edge crops.
private val edgeFrame = intArrayOf(
0, 1, 2, 3, 4,
10, 11, 12, 13, 14,
20, 21, 22, 23, 24,
)

@Test fun allFourRotationsProduceExactExpectedPixels() {
val rect = NormalizedRect(0f, 0f, 1f, 1f)
val expected = mapOf(
FrameRotation.ROTATION_0 to Triple(3, 2, intArrayOf(10, 11, 12, 20, 21, 22)),
FrameRotation.ROTATION_90 to Triple(2, 3, intArrayOf(20, 10, 21, 11, 22, 12)),
FrameRotation.ROTATION_180 to Triple(3, 2, intArrayOf(22, 21, 20, 12, 11, 10)),
FrameRotation.ROTATION_270 to Triple(2, 3, intArrayOf(12, 22, 11, 21, 10, 20)),
)
for (rotation in FrameRotation.entries) {
val source = frame(3, 2, rotation, asymmetric.copyOf())
val result = AndroidBitmapSlotImageExtractor.extract(source, slot(rect))

assertTrue("rotation $rotation must succeed", result.isSuccess)
val image = result.getOrThrow()
val (pixels, width) = pixelsOf(image)
val (expectedWidth, expectedHeight, expectedPixels) = expected.getValue(rotation)
assertEquals("width[$rotation]", expectedWidth, width)
assertEquals("height[$rotation]", expectedHeight, (image.payload as BitmapFramePayload).bitmap.height)
assertArrayEquals("pixels[$rotation]", expectedPixels, pixels)
}
}

@Test fun topBottomEdgeCropsExtractExactlyThosePixels() {
val rect = NormalizedRect(0.2f, 0f, 0.6f, 1f) // left=1px, right=3px of 5
val expected = mapOf(
FrameRotation.ROTATION_0 to Triple(2, 3, intArrayOf(1, 2, 11, 12, 21, 22)),
FrameRotation.ROTATION_90 to Triple(3, 2, intArrayOf(21, 11, 1, 22, 12, 2)),
FrameRotation.ROTATION_180 to Triple(2, 3, intArrayOf(22, 21, 12, 11, 2, 1)),
FrameRotation.ROTATION_270 to Triple(3, 2, intArrayOf(2, 12, 22, 1, 11, 21)),
)
for (rotation in FrameRotation.entries) {
val source = frame(5, 3, rotation, edgeFrame.copyOf())
val result = AndroidBitmapSlotImageExtractor.extract(source, slot(rect))

assertTrue("rotation $rotation must succeed", result.isSuccess)
val (pixels, width) = pixelsOf(result.getOrThrow())
val (expectedWidth, expectedHeight, expectedPixels) = expected.getValue(rotation)
assertEquals("width[$rotation]", expectedWidth, width)
assertEquals("height[$rotation]", expectedHeight, (result.getOrThrow().payload as BitmapFramePayload).bitmap.height)
assertArrayEquals("pixels[$rotation]", expectedPixels, pixels)
}
}

@Test fun bottomRightEdgeCropExtractsExactlyThosePixels() {
val rect = NormalizedRect(0.6f, 1f / 3f, 1f, 1f) // left=3px, top=1px of 5x3
val expected = mapOf(
FrameRotation.ROTATION_0 to Triple(2, 2, intArrayOf(13, 14, 23, 24)),
FrameRotation.ROTATION_90 to Triple(2, 2, intArrayOf(23, 13, 24, 14)),
FrameRotation.ROTATION_180 to Triple(2, 2, intArrayOf(24, 23, 14, 13)),
FrameRotation.ROTATION_270 to Triple(2, 2, intArrayOf(14, 24, 13, 23)),
)
for (rotation in FrameRotation.entries) {
val source = frame(5, 3, rotation, edgeFrame.copyOf())
val result = AndroidBitmapSlotImageExtractor.extract(source, slot(rect))

assertTrue("rotation $rotation must succeed", result.isSuccess)
val (pixels, width) = pixelsOf(result.getOrThrow())
val (expectedWidth, expectedHeight, expectedPixels) = expected.getValue(rotation)
assertEquals("width[$rotation]", expectedWidth, width)
assertEquals("height[$rotation]", expectedHeight, (result.getOrThrow().payload as BitmapFramePayload).bitmap.height)
assertArrayEquals("pixels[$rotation]", expectedPixels, pixels)
}
}

@Test fun payloadDimensionMismatchIsRefused() {
val bitmap = track(Bitmap.createBitmap(2, 2, Bitmap.Config.ARGB_8888))
val frame = OriginalFrame(
id = FrameId("mismatch"),
payload = BitmapFramePayload(bitmap),
sourceWidth = 4,
sourceHeight = 4,
displayRotation = FrameRotation.ROTATION_0,
rotation = FrameRotation.ROTATION_0,
capturedAtElapsedRealtimeMs = 1,
wallClockTimestampMs = 1,
captureLatencyMs = 1,
)

val result = AndroidBitmapSlotImageExtractor.extract(frame, slot(NormalizedRect(0.1f, 0.1f, 0.9f, 0.9f)))

assertTrue(result.isFailure)
assertTrue(result.exceptionOrNull()!!.message!!.contains("does not match"))
}

@Test fun sourceBitmapStaysValidAfterExtraction() {
val source = frame(4, 3, FrameRotation.ROTATION_90, edgeFrame.copyOf())
val bitmap = (source.payload as BitmapFramePayload).bitmap
val before = IntArray(edgeFrame.size).also { bitmap.getPixels(it, 0, 4, 0, 0, 4, 3) }

val result = AndroidBitmapSlotImageExtractor.extract(source, slot(NormalizedRect(0.2f, 0f, 0.6f, 1f)))

assertTrue(result.isSuccess)
assertFalse("the source bitmap must not be recycled", bitmap.isRecycled)
val after = IntArray(edgeFrame.size).also { bitmap.getPixels(it, 0, 4, 0, 0, 4, 3) }
assertArrayEquals("the source pixels must be unchanged", before, after)
}

@Test fun minimumValidCropProducesOnePixel() {
val source = frame(128, 128, FrameRotation.ROTATION_0, IntArray(128 * 128) { it })
val result = AndroidBitmapSlotImageExtractor.extract(
source,
slot(NormalizedRect(0.125f, 0.125f, 0.1328125f, 0.1328125f)),
)

assertTrue(result.isSuccess)
val (pixels, width) = pixelsOf(result.getOrThrow())
assertEquals(1, width)
assertEquals(1, (result.getOrThrow().payload as BitmapFramePayload).bitmap.height)
assertEquals(16 * 128 + 16, pixels.single())
}

@Test fun subPixelRectangleIsRefused() {
val source = frame(10, 10, FrameRotation.ROTATION_0, IntArray(100) { 7 })

val result = AndroidBitmapSlotImageExtractor.extract(source, slot(NormalizedRect(0f, 0f, 0.05f, 0.05f)))

assertTrue(result.isFailure)
}

@Test fun recycledSourceBitmapIsRefused() {
val source = frame(4, 3, FrameRotation.ROTATION_0, edgeFrame.copyOf())
val bitmap = (source.payload as BitmapFramePayload).bitmap
bitmap.recycle()

val result = AndroidBitmapSlotImageExtractor.extract(source, slot(NormalizedRect(0f, 0f, 1f, 1f)))

assertTrue(result.isFailure)
}

@Test fun bitmapSnapshotSharesNoPixelsWithPipeline() {
val original = frame(4, 2, FrameRotation.ROTATION_0, asymmetric.copyOf())
val snapshot = FrameSnapshots.copyOf(original)

assertTrue(snapshot != null)
val sourceBitmap = (original.payload as BitmapFramePayload).bitmap
val snapshotBitmap = (snapshot!!.payload as BitmapFramePayload).bitmap
assertNotSame(sourceBitmap, snapshotBitmap)
ownedBitmaps.add(snapshotBitmap)
assertFalse("the source must not be recycled by snapshotting", sourceBitmap.isRecycled)
assertEquals(
snapshotBitmap.width,
sourceBitmap.width,
)
sourceBitmap.setPixels(IntArray(8) { -1 }, 0, 4, 0, 0, 4, 2)
val snapshotAfter = IntArray(8).also { snapshotBitmap.getPixels(it, 0, 4, 0, 0, 4, 2) }
assertTrue(
"mutating the live bitmap must not leak into the frozen snapshot",
snapshotAfter.contains(10),
)
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
package com.draftlens

import androidx.test.ext.junit.runners.AndroidJUnit4
import com.draftlens.capture.FramePipeline
import com.draftlens.capture.FrameSampler
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNotNull
import org.junit.Assert.assertNotSame
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
import org.junit.Test
import org.junit.runner.RunWith

/**
* Lifecycle invariants for the capture pipeline on a device or emulator: generation
* bumps, replacement surfaces and the frozen snapshot. No pixels are pushed, so this
* needs no MediaProjection consent. Compiled in CI; execution requires a device/emulator
* (see docs/TESTING_PHYSICAL_DEVICE.md).
*/
@RunWith(AndroidJUnit4::class)
class FramePipelineInstrumentedTest {

@Test fun startingAPipelineCreatesASurfaceSizedToTheRequest() {
val pipeline = FramePipeline(FrameSampler(FrameSampler.DEFAULT_INTERVAL_MS))
try {
pipeline.start(1080, 2400, 440)

assertNotNull(pipeline.surface())
assertEquals(1080, pipeline.displayWidth)
assertEquals(2400, pipeline.displayHeight)
} finally {
pipeline.stop()
}
}

@Test fun aResizeReplacesTheSurfaceAndBumpsTheGeneration() {
val pipeline = FramePipeline(FrameSampler(FrameSampler.DEFAULT_INTERVAL_MS))
try {
pipeline.start(1080, 2400, 440)
val before = pipeline.surface()
val generationBefore = pipeline.peekGeneration()

val replacement = pipeline.prepareResize(2400, 1080, 440,
com.draftlens.vision.model.FrameRotation.ROTATION_0)

assertNotNull(replacement)
replacement!!.closePrevious()
assertEquals(2400, pipeline.displayWidth)
assertEquals(1080, pipeline.displayHeight)
assertNotSame("resize must install a replacement surface", before, pipeline.surface())
assertTrue("resize must advance the generation", pipeline.peekGeneration() > generationBefore)
} finally {
pipeline.stop()
}
}

@Test fun freezeWithNoFramesReportsNoFrameAndKeepsPipelineAlive() {
val pipeline = FramePipeline(FrameSampler(FrameSampler.DEFAULT_INTERVAL_MS))
try {
pipeline.start(2400, 1080, 440)

val outcome = pipeline.freezeAndSnapshot()

assertEquals(FramePipeline.FreezeResult.NoFrame, outcome)
assertNotNull("the reader must survive a freeze request with no frames", pipeline.surface())
} finally {
pipeline.stop()
}
}

@Test fun stoppingThePipelineReleasesItsSurface() {
val pipeline = FramePipeline(FrameSampler(FrameSampler.DEFAULT_INTERVAL_MS))
pipeline.start(1080, 2400, 440)

pipeline.stop()

assertNull(pipeline.surface())
assertEquals(0, pipeline.displayWidth)
assertEquals(0, pipeline.displayHeight)
}
}
Loading
Loading