diff --git a/README.md b/README.md index cf78132..71f59cb 100644 --- a/README.md +++ b/README.md @@ -1,100 +1,100 @@ -# DraftLens — Sprint 1A Vision Foundation - -Native Android app that assists MLBB Draft Mode players. Sprint 1A turns the previously -working capture path into a real **image-bearing** foundation: frames carry pixels, geometry -is tested, preprocessing validates before it renders, and a Vision Lab lets you place manual -slot annotations and save calibration profiles. - -**Maintainer-reported device smoke:** the Sprint 0 capture path and the Sprint 1A Vision -Lab interaction behavior were used on a physical Android device and confirmed working. -Device details, the exact build and test dates were not supplied, and the MediaProjection -lifecycle matrix remains **NOT RUN**. See [docs/TESTING_PHYSICAL_DEVICE.md](docs/TESTING_PHYSICAL_DEVICE.md) -and [docs/VISION_TESTING.md](docs/VISION_TESTING.md). - -## What works in Sprint 1A -- MediaProjection consent flow (`ActivityResultContracts.CreateScreenCaptureIntent`) -- Foreground service (`mediaProjection` type) with persistent notification + Stop action -- Frame acquisition at 2 FPS default, configurable 334–2000 ms (hard cap 3 FPS) -- Live frame preview in the debug screen, frame count / resolution / latency / timestamp -- Pause / resume, rotation & resize handling, clean resource teardown on stop/revoke -- Non-null frame contract: `FramePayload`, `OriginalFrame`, `ProcessedFrame`, `SlotImage` -- Rotation-aware geometry with round-trip tests (`FrameTransform`) -- Validated preprocessing with honest failure codes (`PreprocessingMath`, - `PixelArrayFramePreprocessor`, `AndroidFramePreprocessor`) -- Manual slot annotations with stable ids, bounds/duplicate/overlap validation and a strict - JSON codec +# DraftLens + +DraftLens is a native Android app for players of Mobile Legends: Bang Bang draft mode. It +captures the game screen, lets you mark the ban/pick portrait slots by hand, freezes the +game frame without leaving the match, and runs a small on-device hero-identification +baseline that always asks you to confirm or correct the result. It is a calibration and +inspection tool, not an automatic draft advisor. + +Current code is the merge of three increments: **1A** (image-bearing frame and Vision Lab +foundation), **1B** (hero catalog, slot extraction, recognition v0) and **1C** (app-only +capture, visibility gating and the circular floating freeze control). The status of each, +including what has not been verified on hardware, is recorded in +[docs/IMPLEMENTATION_REPORT.md](docs/IMPLEMENTATION_REPORT.md). + +## What it can do + +**Capture** +- MediaProjection consent flow; a typed foreground service owns the projection and the + notification. +- Frames at 2 FPS by default, configurable 334–2000 ms (hard cap 3 FPS). +- Pause and resume, rotation and resize handling, explicit teardown on stop or revocation. +- Freeze from the notification or from one floating button; a pinned frame survives pause, + resize and visibility changes inside the session. +- App-only capture on Android 14+: when the captured window is hidden, images are still + acquired and closed but never published, so the last valid game frame is kept. +- One 52 dp overlay button, attached only while capture runs, with edge snapping and a + normalized position saved in DataStore. It is opt-in and never touches the game. + +**Vision Lab** +- Non-null frame contract with separate original, processed and slot images. +- Rotation-aware geometry and validated preprocessing with explicit failure codes. +- Manual slot annotation with stable ids, bounds/duplicate/overlap checks, undo/redo, + copy/paste/duplicate and a strict JSON codec. - Versioned calibration profiles: 1 % aspect-ratio tolerance, exact orientation match, - unsupported-schema refusal, delete-as-escape-hatch -- Vision Lab tab: frame preview, freeze/resume, explicit processing, interactive annotation - canvas, text slot editor, profile controls -- Hero recognition v0 (Sprint 1B, PR #3): hero catalog sync from public static data, - explicit Identify action for the selected slot, ranked candidates with similarity scores, - manual confirm/correct — **accuracy on real game screens is not measured yet** -- Capture notification with **Stop** and **Pause** (Sprint 1C): pausing keeps the latest - valid frame, blocks new frames replacing it, keeps MediaProjection running, and the same - Resume action recovers from pause or from a pinned frame; the notification reflects - LIVE / PAUSED / FROZEN and never shows both actions -- App-only capture guidance (Sprint 1C): prefer **Single app / app window → Mobile Legends** - (Android 14+ chooser); the captured window's visibility drives publication — - MLBB hidden ⇒ the last valid game frame is kept, never DraftLens/shade/system UI; - full-display capture remains the documented fallback (it may include the shade/system UI) -- Circular floating freeze button (Sprint 1C): one 52dp circular `TYPE_APPLICATION_OVERLAY` - button shown only while capture runs — tap toggles LIVE (green) / orange (paused or frozen), - long-press-and-drag moves it, it snaps to a screen edge, position persists normalized in - DataStore; notifications/in-app Freeze/Resume keep working without it; the overlay - permission is requested only when you enable the button - -## What is deliberately NOT here (later milestones) -Recommendation engine, counter/synergy/role scoring — and no recommendation overlays -(Sprint 1C ships exactly one capture-freeze overlay button, and nothing else). -ML hero recognition is Sprint 1B (PR #3). API sync, player profiling, builds. -The Vision Lab says **"Detection: not implemented"** — manual slots are calibration data, -never detections. See `tasks/plan.md` and `docs/ARCHITECTURE.md`. - -## Build -Requires JDK 17 and Android SDK platform 35, build-tools 35. + refusal of unsupported schemas, and a delete-all escape hatch. + +**Recognition v0** +- Public static hero catalog from Rone Arena, cross-checked against MLBBDex, synced only + when you ask, cached privately on the device. +- Upright slot crops for every rotation, then a dHash plus 3×3 mean-RGB similarity. +- Blank or detail-free crops are rejected; low similarity is reported as no match. +- Candidates are labelled as uncalibrated similarity, never probability. You always + confirm or correct manually. Accuracy on real MLBB screens has not been measured. + +**Deliberately not included:** recommendation, counter/synergy/role scoring, automatic +draft-phase detection, game-memory reading, injected touches, cloud recognition, ML models, +OpenCV, and any overlay beyond the single capture-freeze button. Captured pixels never +leave the device and are never written to disk. + +## Build and verify + +JDK 17 and Android SDK platform 35 with build-tools 35 are required. + ```sh -./gradlew test # JVM unit tests -./gradlew lint # Android lint -./gradlew assembleDebug # produces app/build/outputs/apk/debug/*.apk +./gradlew test # JVM unit tests +./gradlew lint # strict lint (abortOnError = true) +./gradlew assembleAndroidTest # compile instrumented tests +./gradlew assembleDebug # debug APK ``` -GitHub Actions (`.github/workflows/build.yml`) runs exactly those steps on push/PR and uploads -the APK artifact. **This authoring environment is aarch64 and the SDK's aapt2/build-tools are -x86_64-only, so nothing that touches resources can be assembled locally — the CI run is the -build of record.** See `docs/VISION_TESTING.md` for exactly what ran. + +GitHub Actions (`.github/workflows/build.yml`) runs these on every push and pull request and +uploads the `draftlens-debug` APK. CI is the build of record: the authoring host is aarch64 +while the SDK's `aapt2` is x86_64-only, so resource-touching tasks cannot run locally here. ## Run -1. Install the APK from the `draftlens-debug` CI artifact on a physical device (the emulator - validates compilation, not MediaProjection reliability). -2. Open the app; the **Debug Capture** tab is first. -3. Tap **Start Capture** → grant the system screen-capture consent dialog. When Android asks - what to capture, prefer **Single app / app window → Mobile Legends: Bang Bang** (Android 14+); - if that option is missing, choose **Entire screen / full device display** (it can capture the - shade and system UI). -4. Open Mobile Legends and wait for the draft screen you want to annotate. -5. With MLBB on screen, tap the circular **Floating freeze button** (enable it on the capture - screen first) to freeze the frame — or, if you never granted the overlay permission, pull - down the notification shade and tap **Pause** (notifications must be enabled). -6. Switch back to **DraftLens → Vision Lab**. Capture kept running; the Vision Lab shows the - pinned MLBB frame instead of the DraftLens screen. -7. Draw a slot rectangle, then run **Identify hero** on the frozen frame. Candidates are - similarity scores, not probabilities — confirm or correct manually (the manual search is - available in every recognition state). -8. Tap **Resume live capture** in the notification, or **Resume** in the Vision Lab, to show - live frames again. No second capture session is created. -9. Press **Process Frame** to run the preprocessing engine on the selected frame. -10. Save the annotations as a calibration profile; apply it later on a matching frame. -11. **Stop Capture**, or the notification's **Stop**, or revoke consent in Settings to - exercise teardown. - -## Docs -- `docs/ARCHITECTURE.md` — module layout, state machine, data flow, Vision boundary -- `docs/VISION_ARCHITECTURE.md` — Vision packages, ownership, processing rules -- `docs/VISION_FRAME_CONTRACT.md` — payload contract, ownership, original/processed/slot frames -- `docs/VISION_COORDINATES.md` — normalized coordinates, rotation tables, crop semantics -- `docs/VISION_LAB.md` — user-facing controls, freeze behavior, slot editing, profile rules -- `docs/VISION_TESTING.md` — test inventory, CI gates, what was and was not run -- `docs/SCREEN_CAPTURE.md` — MediaProjection lifecycle, Android 14+ rules, limits -- `docs/DATA_SOURCES.md` — Rone Arena vs MLBBDex audit (verified live 2026-11-04) -- `docs/decisions/ADR-001-android-native.md` — why native, single-module, and version pins -- `docs/IMPLEMENTATION_REPORT.md` — executed verification, limitations, next tasks + +1. Install the `draftlens-debug` APK from CI on a physical device. +2. Open **Debug Capture**, enable notifications, and optionally enable the floating control + (this is the only time the overlay permission is requested). +3. Tap **Start Capture**. When Android asks what to share, prefer **Single app → Mobile + Legends**; choose the whole display only as a fallback, since it can include the shade. +4. In MLBB, freeze with the floating button or with **Pause** in the notification. +5. Return to **Vision Lab**. Draw a slot, run **Identify hero**, and confirm or correct the + candidate. You can also process the frame and save the slots as a calibration profile. +6. Tap **Resume** in the notification or the Vision Lab to return to live frames. Stop from + the app, the notification, or by revoking consent. + +Physical-device behaviour is not yet verified end to end. The checklist and its honest +status are in [docs/TESTING_PHYSICAL_DEVICE.md](docs/TESTING_PHYSICAL_DEVICE.md). + +## Documentation + +Current state: +- [Architecture](docs/ARCHITECTURE.md) — modules, layers, flows and boundaries. +- [Screen capture](docs/SCREEN_CAPTURE.md) — MediaProjection lifecycle, freeze, visibility, resize, overlay. +- [Vision architecture](docs/VISION_ARCHITECTURE.md) — frame, geometry, preprocessing, annotation, recognition. +- [Vision frame contract](docs/VISION_FRAME_CONTRACT.md) — payload and ownership rules. +- [Vision coordinates](docs/VISION_COORDINATES.md) — normalized space, rotation and crop semantics. +- [Vision Lab](docs/VISION_LAB.md) — user-facing controls. +- [Data sources](docs/DATA_SOURCES.md) — hero catalog providers and their contracts. +- [Testing](docs/VISION_TESTING.md) — test inventory and CI gates. +- [Physical-device checklist](docs/TESTING_PHYSICAL_DEVICE.md) — what still needs a device. +- [Implementation report](docs/IMPLEMENTATION_REPORT.md) — delivered scope and verification status. +- [ADR-001](docs/decisions/ADR-001-android-native.md) — native, single-module decision. + +Historical records, kept for provenance: +- [Sprint 1B](docs/sprints/SPRINT_1B_HERO_RECOGNITION.md), [Sprint 1C](docs/sprints/SPRINT_1C_APP_ONLY_CAPTURE.md). +- [Sprint 1A plan](docs/superpowers/plans/2026-11-06-sprint-1a-vision-foundation.md), [Sprint 1C plan](docs/superpowers/plans/2026-11-16-sprint-1c-app-only-capture.md). +- [Sprint 1A design](docs/superpowers/specs/2026-11-06-sprint-1a-vision-foundation-design.md), [Sprint 1C design](docs/superpowers/specs/2026-11-16-sprint-1c-app-only-capture-design.md). +- [Task history](tasks/todo.md), [roadmap](tasks/plan.md). diff --git a/app/build.gradle.kts b/app/build.gradle.kts index 388f40d..d13d026 100644 --- a/app/build.gradle.kts +++ b/app/build.gradle.kts @@ -14,7 +14,7 @@ android { minSdk = 26 targetSdk = 35 versionCode = 1 - versionName = "0.1.0-sprint0" + versionName = "0.2.0" testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner" } @@ -41,9 +41,6 @@ android { buildFeatures { compose = true } - composeOptions { - kotlinCompilerExtensionVersion = "1.5.14" - } packaging { resources { excludes += "/META-INF/{AL2.0,LGPL2.1}" diff --git a/app/src/main/kotlin/com/draftlens/capture/CaptureFreeze.kt b/app/src/main/kotlin/com/draftlens/capture/CaptureFreeze.kt index 3519160..aa8162d 100644 --- a/app/src/main/kotlin/com/draftlens/capture/CaptureFreeze.kt +++ b/app/src/main/kotlin/com/draftlens/capture/CaptureFreeze.kt @@ -27,9 +27,6 @@ object CaptureFreeze { } } - /** True when a pipeline is installed (i.e. capture is running). */ - fun hasActivePipeline(): Boolean = pipeline != null - fun freezeCurrentFrame(): FreezeOutcome { val current = CaptureStateHolder.frozenFrame.value if (current != null) return FreezeOutcome.AlreadyFrozen diff --git a/app/src/main/kotlin/com/draftlens/capture/CaptureSessionService.kt b/app/src/main/kotlin/com/draftlens/capture/CaptureSessionService.kt index d40c9fc..cc7ab6c 100644 --- a/app/src/main/kotlin/com/draftlens/capture/CaptureSessionService.kt +++ b/app/src/main/kotlin/com/draftlens/capture/CaptureSessionService.kt @@ -616,11 +616,6 @@ class CaptureSessionService : Service() { } } - fun stopCapture() { - cleanupAfterTermination(forceIdle = true) - stopSelf() - } - override fun onTaskRemoved(rootIntent: Intent?) { // MediaProjection tokens cannot survive process death. Do not recreate them from a null intent. Log.i(TAG, "Activity task removed; capture session will not be restarted with stale consent") diff --git a/app/src/main/kotlin/com/draftlens/capture/FramePipeline.kt b/app/src/main/kotlin/com/draftlens/capture/FramePipeline.kt index 735eecc..ea615e3 100644 --- a/app/src/main/kotlin/com/draftlens/capture/FramePipeline.kt +++ b/app/src/main/kotlin/com/draftlens/capture/FramePipeline.kt @@ -52,9 +52,6 @@ class FramePipeline( @Volatile private var sessionId: Int = 0 - /** Effective gate state for the structured acquisition log. */ - fun visibilityState(): String = visibility.currentState().name - /** * Bumped every time a new reader/surface generation is created (start/resize). * Images captured with an older generation are dropped, so a late callback from a diff --git a/app/src/main/kotlin/com/draftlens/capture/RgbaPlaneCopy.kt b/app/src/main/kotlin/com/draftlens/capture/RgbaPlaneCopy.kt index 874f0a4..9b546fe 100644 --- a/app/src/main/kotlin/com/draftlens/capture/RgbaPlaneCopy.kt +++ b/app/src/main/kotlin/com/draftlens/capture/RgbaPlaneCopy.kt @@ -20,9 +20,6 @@ import java.nio.ByteOrder */ internal object RgbaPlaneCopy { - /** RGBA_8888 stores 4 bytes per pixel. */ - const val RGBA_PIXEL_BYTES = 4 - /** A validated plane layout, used for the conversion and for structured logs. */ data class PlaneLayout( val imageWidth: Int, @@ -35,8 +32,6 @@ internal object RgbaPlaneCopy { val cropWidth: Int, val cropHeight: Int, ) { - val rowPaddingPixels: Int get() = rowPaddingBytes / pixelStride - /** True when the crop covers the whole image. */ val isFullCrop: Boolean get() = cropLeft == 0 && cropTop == 0 && diff --git a/app/src/main/kotlin/com/draftlens/datastore/AppPreferencesRepository.kt b/app/src/main/kotlin/com/draftlens/datastore/AppPreferencesRepository.kt index 395a230..d38a8c2 100644 --- a/app/src/main/kotlin/com/draftlens/datastore/AppPreferencesRepository.kt +++ b/app/src/main/kotlin/com/draftlens/datastore/AppPreferencesRepository.kt @@ -5,10 +5,8 @@ import androidx.datastore.preferences.core.booleanPreferencesKey import androidx.datastore.preferences.core.edit import androidx.datastore.preferences.core.floatPreferencesKey import androidx.datastore.preferences.core.longPreferencesKey -import androidx.datastore.preferences.core.stringPreferencesKey import androidx.datastore.preferences.preferencesDataStore import kotlinx.coroutines.flow.Flow -import kotlinx.coroutines.flow.first import kotlinx.coroutines.flow.map import com.draftlens.capture.FrameSampler import com.draftlens.overlay.OverlayPosition @@ -19,8 +17,6 @@ class AppPreferencesRepository(private val context: Context) { private object Keys { val SAMPLING_INTERVAL = longPreferencesKey("sampling_interval_ms") - val LAST_CAPTURE_AT = longPreferencesKey("last_capture_at_ms") - val CONSENT_NOTE = stringPreferencesKey("consent_note") val FLOATING_CONTROL_ENABLED = booleanPreferencesKey("floating_control_enabled") val FLOATING_POSITION_X = floatPreferencesKey("floating_position_x") val FLOATING_POSITION_Y = floatPreferencesKey("floating_position_y") @@ -35,12 +31,6 @@ class AppPreferencesRepository(private val context: Context) { context.dataStore.edit { it[Keys.SAMPLING_INTERVAL] = clamped } } - suspend fun recordCaptureStarted() { - context.dataStore.edit { it[Keys.LAST_CAPTURE_AT] = System.currentTimeMillis() } - } - - suspend fun currentSamplingIntervalMs(): Long = samplingIntervalMs.first() - /** True only when the user asked for the floating freeze button (and it may be shown during capture). */ val floatingControlEnabled: Flow = context.dataStore.data.map { it[Keys.FLOATING_CONTROL_ENABLED] ?: false } diff --git a/app/src/main/kotlin/com/draftlens/detection/DetectionInterfaces.kt b/app/src/main/kotlin/com/draftlens/detection/DetectionInterfaces.kt deleted file mode 100644 index fe9d8f7..0000000 --- a/app/src/main/kotlin/com/draftlens/detection/DetectionInterfaces.kt +++ /dev/null @@ -1,80 +0,0 @@ -package com.draftlens.detection - -/** - * Sprint 0: interfaces only. Every impl below throws/returns NotImplemented. - * Production code must never fabricate a detection result. See docs/ARCHITECTURE.md. - */ - -/** Abstracts "there is a screen and frames are flowing". */ -interface ScreenCaptureProvider { - val isCapturing: Boolean - fun start() - fun stop() -} - -/** Consumes captured frames for ML or heuristic analysis. */ -interface FrameProcessor { - /** Returns a human-readable status; Sprint 0 always NOT_IMPLEMENTED. */ - fun process(frame: FrameRef): ProcessorResult -} - -data class FrameRef(val timestampMs: Long, val width: Int, val height: Int) - -enum class ProcessorResult { NOT_IMPLEMENTED, OK, ERROR } - -/** Detects MLBB draft phase from a frame (bans/picks/timer). */ -interface DraftPhaseDetector { - fun detect(frame: FrameRef): DetectionResult -} - -/** Recognizes which hero occupies a slot. */ -interface HeroRecognizer { - fun recognize(slot: HeroSlot): DetectionResult -} - -/** Locates hero portrait slots on the draft screen. */ -interface HeroSlotDetector { - fun findSlots(frame: FrameRef): DetectionResult> -} - -/** Cross-checks a detection against history/rules before it is trusted. */ -interface DetectionValidator { - fun validate(candidate: DetectionResult<*>): Boolean -} - -// ---- Value types (no behavior) ---- - -enum class DraftPhase { UNKNOWN, BANNING, PICKING, LOADING, IN_GAME, NOT_IMPLEMENTED } - -data class HeroSlot(val id: Int, val x: Int, val y: Int, val width: Int, val height: Int, val side: TeamSide) -enum class TeamSide { ALLY, ENEMY } - -data class HeroIdentity(val heroId: Int?, val displayName: String?, val confidence: Float) - -sealed interface DetectionResult { - data class Found(val value: T, val confidence: Float) : DetectionResult - data class NotImplemented(val reason: String = "No recognition model in Sprint 0") : DetectionResult - data class Unavailable(val reason: String) : DetectionResult - data class Failed(val error: String) : DetectionResult -} - -/** Sprint 0 stubs: always NotImplemented. Never fake a result. */ -class UnavailableDraftPhaseDetector : DraftPhaseDetector { - override fun detect(frame: FrameRef) = DetectionResult.NotImplemented() -} - -class UnavailableHeroRecognizer : HeroRecognizer { - override fun recognize(slot: HeroSlot) = DetectionResult.NotImplemented() -} - -class UnavailableHeroSlotDetector : HeroSlotDetector { - override fun findSlots(frame: FrameRef) = DetectionResult.NotImplemented>() -} - -class UnavailableFrameProcessor : FrameProcessor { - override fun process(frame: FrameRef) = ProcessorResult.NOT_IMPLEMENTED -} - -class UnavailableDetectionValidator : DetectionValidator { - override fun validate(candidate: DetectionResult<*>): Boolean = false -} diff --git a/app/src/main/kotlin/com/draftlens/feature/home/DebugCaptureScreen.kt b/app/src/main/kotlin/com/draftlens/feature/home/DebugCaptureScreen.kt index c904141..eab8ce6 100644 --- a/app/src/main/kotlin/com/draftlens/feature/home/DebugCaptureScreen.kt +++ b/app/src/main/kotlin/com/draftlens/feature/home/DebugCaptureScreen.kt @@ -39,7 +39,6 @@ import androidx.core.content.ContextCompat import androidx.lifecycle.viewmodel.compose.viewModel import com.draftlens.capture.CaptureController import com.draftlens.datastore.AppPreferencesRepository -import com.draftlens.models.CaptureStatus import android.content.Intent import android.net.Uri import android.provider.Settings diff --git a/app/src/main/kotlin/com/draftlens/feature/home/HomeViewModel.kt b/app/src/main/kotlin/com/draftlens/feature/home/HomeViewModel.kt index 295d6e7..c24a465 100644 --- a/app/src/main/kotlin/com/draftlens/feature/home/HomeViewModel.kt +++ b/app/src/main/kotlin/com/draftlens/feature/home/HomeViewModel.kt @@ -41,7 +41,6 @@ class HomeViewModel( fun onPermissionGranted() { _permissionStatus.value = "Granted (session-scoped)" - viewModelScope.launch { preferences.recordCaptureStarted() } } fun onPermissionDenied() { diff --git a/app/src/main/kotlin/com/draftlens/feature/visionlab/AnnotationGestureHint.kt b/app/src/main/kotlin/com/draftlens/feature/visionlab/AnnotationGestureHint.kt index 4365106..98d246c 100644 --- a/app/src/main/kotlin/com/draftlens/feature/visionlab/AnnotationGestureHint.kt +++ b/app/src/main/kotlin/com/draftlens/feature/visionlab/AnnotationGestureHint.kt @@ -27,8 +27,7 @@ enum class AnnotationWorkspaceMode { /** * The workspace fills the space it is given and the image is fitted inside it. - * Used by the fullscreen editor, where the image should be as large as possible and - * the user can zoom and pan. + * Used by the fullscreen editor, where the image should be as large as possible. */ Fill, } diff --git a/app/src/main/kotlin/com/draftlens/feature/visionlab/SlotEditor.kt b/app/src/main/kotlin/com/draftlens/feature/visionlab/SlotEditor.kt index db88213..4c09f5b 100644 --- a/app/src/main/kotlin/com/draftlens/feature/visionlab/SlotEditor.kt +++ b/app/src/main/kotlin/com/draftlens/feature/visionlab/SlotEditor.kt @@ -8,7 +8,6 @@ import androidx.compose.foundation.layout.Row import androidx.compose.foundation.layout.fillMaxWidth import androidx.compose.foundation.layout.padding import androidx.compose.foundation.layout.size -import androidx.compose.foundation.layout.width import androidx.compose.material3.AlertDialog import androidx.compose.material3.Card import androidx.compose.material3.CardDefaults diff --git a/app/src/main/kotlin/com/draftlens/feature/visionlab/VisionLabScreen.kt b/app/src/main/kotlin/com/draftlens/feature/visionlab/VisionLabScreen.kt index 7ffe652..e33d59a 100644 --- a/app/src/main/kotlin/com/draftlens/feature/visionlab/VisionLabScreen.kt +++ b/app/src/main/kotlin/com/draftlens/feature/visionlab/VisionLabScreen.kt @@ -40,8 +40,8 @@ import com.draftlens.vision.model.VisionFrameSource import com.draftlens.vision.preprocessing.AndroidFramePreprocessor /** - * Vision Lab: inspect frames, place manual draft-slot annotations, and manage calibration - * profiles. There is no recognizer in this sprint, so nothing here reports a detection. + * Vision Lab: inspect frames, place manual draft-slot annotations, run the identification + * baseline, and manage calibration profiles. Identification always needs manual confirmation. */ @Composable fun VisionLabScreen( @@ -89,8 +89,8 @@ fun VisionLabScreen( .padding(horizontal = 16.dp, vertical = 12.dp), verticalArrangement = Arrangement.spacedBy(12.dp), ) { - Text("DraftLens — Vision Lab (Sprint 1A)", style = MaterialTheme.typography.titleLarge) - Text("Detection: not implemented. Manual slots are calibration data, not detections.") + Text("DraftLens — Vision Lab", style = MaterialTheme.typography.titleLarge) + Text("Recognition is an uncalibrated baseline. Always confirm or correct the result.") FramePreviewCard( state = state, @@ -253,9 +253,8 @@ private fun ProcessingInspectorCard( Card(modifier = Modifier.fillMaxWidth()) { Column(modifier = Modifier.padding(12.dp), verticalArrangement = Arrangement.spacedBy(6.dp)) { Text("Processing Inspector", style = MaterialTheme.typography.titleMedium) - Text("Detection: not implemented") Text( - "Process Frame resizes and prepares the frame only; no recognition exists yet.", + "Process Frame prepares pixels for inspection. Identification is a separate action.", style = MaterialTheme.typography.bodySmall, ) diff --git a/app/src/main/kotlin/com/draftlens/models/CaptureSessionState.kt b/app/src/main/kotlin/com/draftlens/models/CaptureSessionState.kt index 626c9b8..679f7e6 100644 --- a/app/src/main/kotlin/com/draftlens/models/CaptureSessionState.kt +++ b/app/src/main/kotlin/com/draftlens/models/CaptureSessionState.kt @@ -16,8 +16,7 @@ data class CaptureSessionState( val lastFrameProcessingLatencyMs: Long? = null, val isPaused: Boolean = false, val lastError: String? = null, - val startedAtMs: Long? = null, - val detectionStatus: String = "Not implemented in Sprint 0" + val startedAtMs: Long? = null ) { val isCapturing: Boolean get() = status == CaptureStatus.CAPTURING || status == CaptureStatus.PAUSED val isStartBlocking: Boolean get() = status != CaptureStatus.IDLE && status != CaptureStatus.ERROR diff --git a/app/src/main/kotlin/com/draftlens/vision/heroes/VisualInformation.kt b/app/src/main/kotlin/com/draftlens/vision/heroes/VisualInformation.kt index ba2a0fb..530f792 100644 --- a/app/src/main/kotlin/com/draftlens/vision/heroes/VisualInformation.kt +++ b/app/src/main/kotlin/com/draftlens/vision/heroes/VisualInformation.kt @@ -29,9 +29,7 @@ data class VisualInformationReport( val meaningfulDifferences: Int, /** Null when the crop is usable; otherwise a short reason it is not. */ val insufficientReason: String?, -) { - val isSufficient: Boolean get() = insufficientReason == null -} +) object VisualInformationAnalyzer { diff --git a/app/src/test/kotlin/com/draftlens/capture/CaptureDiagnosticsTest.kt b/app/src/test/kotlin/com/draftlens/capture/CaptureDiagnosticsTest.kt index bf116b7..d055bcc 100644 --- a/app/src/test/kotlin/com/draftlens/capture/CaptureDiagnosticsTest.kt +++ b/app/src/test/kotlin/com/draftlens/capture/CaptureDiagnosticsTest.kt @@ -1,8 +1,6 @@ package com.draftlens.capture import org.junit.Assert.assertEquals -import org.junit.Assert.assertFalse -import org.junit.Assert.assertTrue import org.junit.Test class CaptureDiagnosticsTest { diff --git a/app/src/test/kotlin/com/draftlens/capture/PendingResizeTest.kt b/app/src/test/kotlin/com/draftlens/capture/PendingResizeTest.kt index 7ed9b65..2267eea 100644 --- a/app/src/test/kotlin/com/draftlens/capture/PendingResizeTest.kt +++ b/app/src/test/kotlin/com/draftlens/capture/PendingResizeTest.kt @@ -1,8 +1,6 @@ package com.draftlens.capture import org.junit.Assert.assertEquals -import org.junit.Assert.assertFalse -import org.junit.Assert.assertTrue import org.junit.Before import org.junit.Test diff --git a/app/src/test/kotlin/com/draftlens/datastore/HeroRecognitionStoreTest.kt b/app/src/test/kotlin/com/draftlens/datastore/HeroRecognitionStoreTest.kt index ce5dd18..0421c98 100644 --- a/app/src/test/kotlin/com/draftlens/datastore/HeroRecognitionStoreTest.kt +++ b/app/src/test/kotlin/com/draftlens/datastore/HeroRecognitionStoreTest.kt @@ -2,7 +2,6 @@ package com.draftlens.datastore import androidx.datastore.preferences.core.PreferenceDataStoreFactory import androidx.datastore.preferences.core.edit -import androidx.datastore.preferences.core.preferencesOf import androidx.datastore.preferences.core.stringPreferencesKey import kotlinx.coroutines.CoroutineScope import kotlinx.coroutines.Dispatchers diff --git a/app/src/test/kotlin/com/draftlens/detection/DetectionStubsTest.kt b/app/src/test/kotlin/com/draftlens/detection/DetectionStubsTest.kt deleted file mode 100644 index eff4f03..0000000 --- a/app/src/test/kotlin/com/draftlens/detection/DetectionStubsTest.kt +++ /dev/null @@ -1,27 +0,0 @@ -package com.draftlens.detection - -import org.junit.Assert.* -import org.junit.Test - -class DetectionStubsTest { - - @Test fun `draft phase detector is explicitly not implemented`() { - val r = UnavailableDraftPhaseDetector().detect(FrameRef(0, 1080, 2400)) - assertTrue(r is DetectionResult.NotImplemented) - } - - @Test fun `hero recognizer never fabricates a hero`() { - val r = UnavailableHeroRecognizer().recognize(HeroSlot(0, 0, 0, 100, 100, TeamSide.ALLY)) - assertTrue(r is DetectionResult.NotImplemented) - assertNull((r as DetectionResult.NotImplemented).let { null }) - } - - @Test fun `validator rejects not-implemented results`() { - val v = UnavailableDetectionValidator() - assertFalse(v.validate(DetectionResult.NotImplemented())) - } - - @Test fun `frame processor reports not implemented`() { - assertEquals(ProcessorResult.NOT_IMPLEMENTED, UnavailableFrameProcessor().process(FrameRef(0, 1, 1))) - } -} diff --git a/app/src/test/kotlin/com/draftlens/feature/visionlab/AnnotationInteractionTest.kt b/app/src/test/kotlin/com/draftlens/feature/visionlab/AnnotationInteractionTest.kt index 41909b9..b90174e 100644 --- a/app/src/test/kotlin/com/draftlens/feature/visionlab/AnnotationInteractionTest.kt +++ b/app/src/test/kotlin/com/draftlens/feature/visionlab/AnnotationInteractionTest.kt @@ -1,7 +1,6 @@ package com.draftlens.feature.visionlab import androidx.compose.ui.geometry.Offset -import androidx.compose.ui.geometry.Rect import androidx.compose.ui.geometry.Size import com.draftlens.vision.annotation.DraftSlot import com.draftlens.vision.annotation.SlotTeam diff --git a/app/src/test/kotlin/com/draftlens/heroes/FileHeroCatalogRepositoryTest.kt b/app/src/test/kotlin/com/draftlens/heroes/FileHeroCatalogRepositoryTest.kt index d6e8790..2f76dd4 100644 --- a/app/src/test/kotlin/com/draftlens/heroes/FileHeroCatalogRepositoryTest.kt +++ b/app/src/test/kotlin/com/draftlens/heroes/FileHeroCatalogRepositoryTest.kt @@ -12,7 +12,6 @@ import org.junit.Assert.assertEquals import org.junit.Assert.assertNotNull import org.junit.Assert.assertNull import org.junit.Assert.assertTrue -import org.junit.Assert.assertFalse import org.junit.Test import java.io.File import java.nio.file.Files diff --git a/app/src/test/kotlin/com/draftlens/heroes/HeroCatalogSyncTest.kt b/app/src/test/kotlin/com/draftlens/heroes/HeroCatalogSyncTest.kt index 99c08ad..6bf5139 100644 --- a/app/src/test/kotlin/com/draftlens/heroes/HeroCatalogSyncTest.kt +++ b/app/src/test/kotlin/com/draftlens/heroes/HeroCatalogSyncTest.kt @@ -1,6 +1,5 @@ package com.draftlens.heroes -import com.draftlens.vision.heroes.HeroCatalogEntry import com.draftlens.vision.heroes.HeroCatalogSource import com.draftlens.vision.heroes.HeroCatalogSync import com.draftlens.vision.heroes.HeroHttpResponse diff --git a/app/src/test/kotlin/com/draftlens/heroes/HeroMatcherTest.kt b/app/src/test/kotlin/com/draftlens/heroes/HeroMatcherTest.kt index 1d33bc4..cb3b316 100644 --- a/app/src/test/kotlin/com/draftlens/heroes/HeroMatcherTest.kt +++ b/app/src/test/kotlin/com/draftlens/heroes/HeroMatcherTest.kt @@ -10,7 +10,6 @@ import com.draftlens.vision.heroes.HeroMatchResult import com.draftlens.vision.heroes.HeroMatcher import com.draftlens.vision.heroes.HeroReference import org.junit.Assert.assertEquals -import org.junit.Assert.assertFalse import org.junit.Assert.assertTrue import org.junit.Test diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index f6da2f4..39a7fa9 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,64 +1,81 @@ -# Architecture — DraftLens +# Architecture ## Stack -Kotlin 2.0.20 · AGP 8.6.1 · Gradle 8.7 · compileSdk/targetSdk 35 · minSdk 26 · Jetpack Compose (material3) · -Coroutines + Flow · DataStore Preferences. Single Gradle module `:app`. -## Package map -``` +Kotlin 2.0.20 · AGP 8.6.1 · Gradle 8.7 · JDK 17 · compileSdk/targetSdk 35 · minSdk 26 · +Jetpack Compose (Material 3) · Coroutines and Flow · DataStore Preferences · +kotlinx-serialization. One Gradle module, `:app`. Versions are centralised in +`gradle/libs.versions.toml`. + +## Layout + +```text com.draftlens - app/ MainActivity (tab host), DraftLensApp (DI root, no framework) - capture/ CaptureSessionService (foreground), CaptureController, FramePipeline, FrameSampler, - CaptureStateHolder, CaptureVisionFrameSource (capture -> Vision adapter) - models/ CaptureSessionState + CaptureTransitions, CapturedFrame - permissions/ ProjectionPermission (consent intent factory) - datastore/ AppPreferencesRepository (sampling interval), - CalibrationProfileDataStore (one JSON calibration document) - detection/ Interfaces only — ScreenCaptureProvider, FrameProcessor, DraftPhaseDetector, - HeroRecognizer, HeroSlotDetector, DetectionValidator (all NotImplemented) - vision/ - model/ FrameId, FrameEnums, FramePayload, VisionFrame, VisionFrameSource (pure Kotlin) - model/android/BitmapFramePayload - geometry/ NormalizedPoint/Rect, PixelPoint/Rect, FrameTransform (pure Kotlin) - preprocessing/PreprocessTypes, PreprocessingMath, PixelArrayFramePreprocessor, - AndroidFramePreprocessor - annotation/ DraftSlot, SlotAnnotationStore, SlotAnnotationCodec (pure Kotlin) - calibration/ CalibrationProfile, CalibrationCompatibility, CalibrationProfileCodec, - CalibrationRepository (pure Kotlin) - feature/home/ HomeViewModel, DebugCaptureScreen (tab 0) - feature/visionlab/ VisionLabViewModel, VisionLabScreen, AnnotationCanvas, SlotEditor, - ProfileControls (tab 1) + MainActivity two-tab host; DraftLensApp holds process-scoped dependencies + capture/ foreground service, controller, pipeline, sampler, freeze, visibility, + generations, plane copy, diagnostics, the capture->Vision adapter + models/ capture session state and transitions, captured frame, snapshots + datastore/ preferences, calibration document, hero confirmations + permissions/ projection consent intent factory + overlay/ floating control view, controller, enablement, position math + vision/model/ ids, enums, payloads, frames, VisionFrameSource + vision/model/android/ BitmapFramePayload + vision/geometry/ normalized and pixel value types, FrameTransform + vision/preprocessing/ plan/result, pure math, renderers, diagnostics + vision/annotation/ slots, store, codecs, validator + vision/calibration/ profiles, compatibility, codec, repository + vision/heroes/ catalog, transport, cache, extraction, visual gate, matcher + feature/home/ capture debug screen and ViewModel + feature/visionlab/ Vision Lab ViewModel, screens, canvas and panels ``` -Room and a separate Gradle module are **not** created — Sprint 1A has no independent feature -module, so empty scaffolding is omitted. -## State machine -`IDLE → AWAITING_PERMISSION → STARTING → CAPTURING ⇄ PAUSED → STOPPING → IDLE`, any → `ERROR → IDLE`. -`CaptureTransitions.canTransition()` is the single authority. UI and service both speak -`CaptureSessionState`; the service is the source of truth, the ViewModel mirrors actions. +`vision/**` is pure Kotlin except the explicitly Android-named adapters. This is what makes +geometry, codecs, policy and matching unit-testable without an emulator. + +## State and ownership + +The capture state machine is +`IDLE → AWAITING_PERMISSION → STARTING → CAPTURING ⇄ PAUSED → STOPPING → IDLE`, with any state +able to enter `ERROR → IDLE`. The service owns the projection and publishes state; the +ViewModel mirrors it. `CaptureStateHolder` is process-local by design: process death clears it +because MediaProjection tokens cannot survive. + +Publication is gated by three independent conditions: the session is capturing, no frame is +pinned, and the captured window is visible (where the platform reports visibility). Freeze, +pause and hide all stop conversion without tearing down the projection. ## Data flow -MediaProjection → VirtualDisplay → ImageReader(maxImages=3) → HandlerThread("draftlens-capture") -→ FrameSampler(monotonic elapsed time, ≥interval) → Bitmap (latest-only, stale ref cleared on stop/reset) → `StateFlow` -→ ViewModel → Compose preview. Main thread never decodes pixels; UI stays responsive. - -## Vision boundary -Vision consumes frames through `VisionFrameSource` only. It never imports `CaptureSessionService` -or `CaptureStateHolder`; `capture/CaptureVisionFrameSource.kt` is the single adapter between the -two, mapping `CaptureStatus` to `VisionSourceState` and `CapturedFrame` to `OriginalFrame`. - -Processing is explicit: the user presses *Process Frame*, one job runs on `Dispatchers.Default`, -a generation counter discards stale results, and failures are reported instead of published as -partial frames. There is no recognizer, so nothing reports a detection. See -`docs/VISION_ARCHITECTURE.md`. - -## Key decisions -- Latest-frame-only slot because stale frames are worthless for draft detection. -- Service foregrounds before `getMediaProjection()`. -- Consent re-requested and token used once per session—never cached. -- Capture dimensions prefer `maximumWindowMetrics` on API 30+; legacy fallback is retained below that. -- Bitmaps are never recycled while Compose may still render them; stopping/resetting clears the state reference. -- Self-capture: no `FLAG_SECURE`; users see system scope options to choose “Entire screen” (debug UI included) or “A single app”. -- MediaProjection buffers are display-oriented, so production frames keep `rotation = ROTATION_0` and only record `displayRotation`. -- Calibration profiles store normalized rectangles and metadata only — never pixels or screenshots. -- No network code. Data-source audit is documentation-only (`docs/DATA_SOURCES.md`). + +```text +MediaProjection → VirtualDisplay → ImageReader (maxImages 3) + → RgbaPlaneCopy → Bitmap (latest only, generation-checked) + → CaptureStateHolder → CaptureVisionFrameSource → VisionLabViewModel + → Compose preview / annotation canvas + → explicit Process Frame → Dispatchers.Default → ProcessedFrame + → select slot → extract → visual gate → match → confirm → store +``` + +The reader runs on a dedicated `HandlerThread`; the main thread never decodes pixels. Stale +callbacks from replaced reader generations are dropped at the listener and again before +publication. A resize replaces the ImageReader and VirtualDisplay at the reported size; a pure +rotation change only updates metadata. + +## Boundaries + +- **Capture ↔ Vision:** `VisionFrameSource` is the only seam. `CaptureVisionFrameSource` is the + single file that imports both packages. +- **Pure ↔ Android:** geometry, codecs, policy and matching have no Android imports; platform + adapters are thin and named. +- **Frames ↔ disk:** captured pixels and slot crops never leave memory. Only metadata, + calibration profiles and public catalog assets are persisted. + +## Build and verification + +Gradle tasks in CI (`.github/workflows/build.yml`): `test`, `assembleAndroidTest` (compile +only), `lint` with `abortOnError = true`, `assembleDebug`, APK upload and manifest inspection. +`main` is protected by the build check. CI is the build of record because the authoring host is +aarch64 and the SDK's `aapt2` is x86_64-only. + +See [Screen capture](SCREEN_CAPTURE.md) for the platform lifecycle, +[Vision architecture](VISION_ARCHITECTURE.md) for the processing pipeline, and +[Physical-device checklist](TESTING_PHYSICAL_DEVICE.md) for what CI cannot prove. diff --git a/docs/DATA_SOURCES.md b/docs/DATA_SOURCES.md index 7ec537b..2e9398f 100644 --- a/docs/DATA_SOURCES.md +++ b/docs/DATA_SOURCES.md @@ -1,81 +1,63 @@ -# Data Source Audit — DraftLens (verified live 2026-11-04) +# Hero data sources -Both candidates were fetched live during Sprint 0. **Neither provider's rates are hard-coded -anywhere in the app** — Sprint 0 contains no recommendation code. Findings below are the -contract a future sync milestone must code against. +The audit below was verified live on 2026-11-04. Sprint 1B implements only the catalog part: +Rone Arena supplies hero identities and portraits, and MLBBDex supplies a completeness +cross-check plus role and lane tags. No win rate, counter, synergy, rating or player endpoint +is integrated. The app has no recommendation or scoring code. + +## Primary — Rone Arena + +`https://arena.rone.dev/api/docs` · OpenAPI 3 spec `https://arena.rone.dev/api/openapi.json`, +v1.6.2, 56 paths. License per spec: BSD 3-Clause, with a non-affiliation notice requested for +derivative UIs. -## Primary: Rone Arena — https://arena.rone.dev/api/docs -- OpenAPI 3 spec: `https://arena.rone.dev/api/openapi.json`, **v1.6.2**, 56 paths. -- License (per spec): **BSD 3-Clause** (not affiliated with/endorsed by Moonton; all game - names/marks belong to their owners). - Hero data: `GET /api/heroes`, `/api/heroes/{id}`, `/api/heroes/rank`, `/api/heroes/positions`. -- Stats: `/api/heroes/{id}/stats`, `/trends`, `/skill-combos`. -- **Counters/relations**: `/api/heroes/{id}/relations` returns `{"strong":[...ids],"weak":[...],"assist":[...]}`; - `/counters`, `/compatibility` also exist — the richest counter/synergy surface of the two. -- Win/pick/ban: via `/api/heroes/rank` (rank-scoped); **statistical definitions are upstream's - and must be quoted, not reinterpreted.** -- Ranks/roles/lanes/items/emblems/spells: under `/api/academy/*` (e.g. `/api/academy/roles`, - `/api/academy/heroes/{id}/lane`, `/api/academy/ranks/{rank_id}`). -- Portraits: per-hero `head`/`smallmap` image URLs (proxied through wsrv.nl at fetch time). -- Esports: `/api/esports/{id,ph,my,kh}/*` schedules/standings. -- **User endpoints (`/api/user/*` — auth, profiles, match history) exist but are explicitly - out of scope:** DraftLens must not add player accounts or collect player data. -- Rate limits / availability guarantee: **none published** in the spec. Treat as best-effort; - cache locally (Room) and degrade silently offline. -- Attribution: BSD-3 requires preserving the notice; the spec also asks for the - non-affiliation disclaimer in any derivative UI. +- Roles, lanes, ranks and items under `/api/academy/*`. +- Portraits are per-hero `head` and `smallmap` URLs, proxied through an image service at fetch + time. The app downloads and caches the bytes; it never hot-links them. +- Counters and relations exist (`/relations`, `/counters`, `/compatibility`) but are **not** + used by DraftLens. +- `/api/user/*` endpoints exist and are explicitly out of scope: no player accounts or player + data will ever be collected. +- Rate limits and availability guarantees are unpublished. Treat the service as best-effort; + sync is manual and the cached snapshot works offline. + +## Cross-check — MLBBDex + +`https://mlbbdex.com/en/api-doc` · "No key, no quota", static cached responses, informal +"be reasonable" guidance. On-page counts at audit time: 134 heroes, 133 ranked, 102 items, +276 patches. -## Secondary: MLBBDex — https://mlbbdex.com/en/api-doc -- No key, no quota ("No key, no quota. Responses are static and cached."), rate guidance is - informal ("Be reasonable"). Hero count surfaced on-page: **134 heroes, 133 ranked, 102 items, - 276 patches.** -- Endpoints: `GET /api/v1/heroes` (role/lane filters), `/api/v1/heroes/{slug}` (profile = - skills, skins, images, written analysis), `/api/v1/items`, `/api/v1/patches`, `/api/v1/rankings`. -- Response envelope: `{"data": …, "total": …, "source": {"name": "Mobile Legends Wiki", - "url": "https://mobilelegends.fandom.com", "license": "CC BY-SA"}}` — **attribution travels - with every response and must be displayed.** -- `/api/v1/rankings` (verified live, `measuredAt: 2026-10-19`): per-hero `{slug, tier (S+..C), - winRate, banRate, pickRate, score, lowSample}`. This is the cleanest win/ban/pick surface — - but **`score` is MLBBDex's proprietary composite** (undocumented formula) and `lowSample` - must be respected by any consumer. -- Hero shape: `{slug, name, roles[], lanes[], specialties[], release, ratings{offense, - durability, abilityEffects, difficulty}, skins, skills[]}` — roles and lanes here are **static - tags**, not ranked-performance data. -- Counters: present on-site as written analysis, **not** as a structured endpoint; no synergy - endpoint. Weaker than Rone for counter/synergy. -- Upstream honesty: MLBBDex states its win/ban/pick stats come "through the arena.rone.dev - community API" — **the two providers share an upstream and are not independent corroboration.** -- License: game data CC BY-SA (Mobile Legends Wiki); MLBBDex's written analyses MIT. - One sync cadence: "weekly sync," latest shown 2026-10-21. -- No player data anywhere ("No player data… this API offers none"). +- `GET /api/v1/heroes`, `/api/v1/heroes/{slug}`, `/api/v1/items`, `/api/v1/patches`, + `/api/v1/rankings`. +- Every response carries a source block with name, URL and CC BY-SA license; that attribution + must travel with any use. +- Roles and lanes on the hero object are **static tags**, not performance data. +- Rankings carry a proprietary `score` with an undocumented formula and a `lowSample` flag. + Neither is used by DraftLens. +- Counters are written analysis on the site, not a structured endpoint; there is no synergy + endpoint. +- MLBBDex states its statistics come from the Rone Arena community API, so the two providers + share an upstream and are **not independent corroboration**. +- No player data is offered. -## Comparison +## How DraftLens uses them -| Dimension | Rone Arena | MLBBDex | -|---|---|---| -| Hero IDs | numeric `hero_id`; id-or-name accepted | slug (`khufra`, `yi-sun-shin`) | -| Catalog completeness | full incl. new heroes (observed ids ≥133) | 134 heroes, 133 ranked | -| Portraits | head/smallmap URLs | images in hero profile | -| Role/lane | positions + academy lane endpoints | role/lane tags on hero | -| Win/pick/ban | `/heroes/rank`, rank-scoped | `/rankings`, with `lowSample` flag | -| Counters | structured strong/weak/assist | written analysis only | -| Synergy/compatibility | `/compatibility`, relations.assist | not exposed | -| Rank filters | academy ranks | page-level rank brackets | -| Patch metadata | academy meta/version | `/patches` (276) | -| Attribution | BSD-3 + non-affiliation notice | CC BY-SA source block per response | -| Auth | has user auth (unused by us) | none, none needed | -| Rate limits | unpublished | "no quota, be reasonable" | -| Availability guarantee | none | none | +- One identity per hero, derived from the name as a slug, independent of provider-specific ids. +- Rone Arena is the primary catalog and portrait source; MLBBDex is read for completeness and + role/lane tags. +- Sync is explicit, HTTPS-only, and writes a snapshot plus a private image cache atomically. +- Assets are revalidated when served: URL, SHA-256, size, content type, magic bytes and + container structure, with a real decode on Android. Changed URLs force a new download; a + failed refresh keeps the previous asset rather than exposing it under a changed URL. +- No statistics from either provider are shown as facts about the current match. The only + similarity shown is the local matcher's uncalibrated ranking of reference images. -## Do-not-assume list (for the future sync milestone) -1. Win/pick/ban numbers are **not comparable** across providers — different measurement windows, - rank pools, and (for MLBBDex `score`) proprietary formulas. -2. `tier` (S+…C) is MLBBDex-calculated; Rone uses ranks — never merge them into one ladder - without re-deriving from raw rates. -3. Portrait URLs may be proxied/expire — cache bytes locally, don't hot-link. -4. MLBBDex hero counts (134/133) and Rone hero ids are snapshots; treat both as volatile. -5. No player-account endpoints will ever be integrated (privacy rule, Sprint 0 brief §7). +## Do not assume -**Recommended default for later milestones:** Rone Arena as the structured-stats/tier source -(counters + compatibility), MLBBDex rankings envelope for win/ban/pick cross-checks — but only -after re-verifying definitions, and with attribution UI for both. +1. Win, pick and ban numbers are not comparable across providers: different windows, rank + pools and proprietary formulas. +2. Provider tiers and ranks are not one ladder; never merge them without re-deriving raw rates. +3. Portrait URLs may expire — cache bytes, never hot-link. +4. Hero counts and ids are snapshots and will change. +5. Player-account endpoints are out of scope for privacy reasons. diff --git a/docs/IMPLEMENTATION_REPORT.md b/docs/IMPLEMENTATION_REPORT.md index 76a22a9..817e14d 100644 --- a/docs/IMPLEMENTATION_REPORT.md +++ b/docs/IMPLEMENTATION_REPORT.md @@ -1,730 +1,62 @@ -# DraftLens — Sprint 1A Vision Foundation Report +# Implementation report -**Branch:** `sprint-1-vision-foundation` · **PR:** #2 (draft, **not merged**) +**Verification head:** `4eb9948` (PR #5) · **CI:** green, run `43942122777` +(918 unit tests, 0 failures, 0 errors, 0 skipped) -## What Sprint 1A delivers +This report describes what the merged code contains and what has actually been verified. The +sprint documents in `docs/sprints/` and `docs/superpowers/` are historical records. -An image-bearing foundation for vision work. Frames carry pixels, geometry is tested, -preprocessing validates before it renders, and a Vision Lab lets a user place manual slot -annotations and save calibration profiles. +## Delivered -**No recognizer.** No OCR, no hero identification, no automatic ban/pick detection, no -confidence values, no ML model, no network client, no screenshot persistence. The Vision Lab -states "Detection: not implemented"; manual slots are calibration data, never detections. +**Capture (Sprints 0–1C).** MediaProjection consent, a typed foreground service with Stop and +one state action, frame sampling, pause/resume, freeze with an independently owned snapshot, +captured-window visibility gating, resize and rotation handling with numbered reader +generations, `cropRect`-aware plane copying, and one opt-in circular overlay control. -## Changed areas +**Vision foundation (1A).** Non-null frame payloads, rotation-aware geometry with round-trip +tests, preprocessing that returns failure codes, manual slot annotation with validation, +undo/redo and strict codecs, calibration profiles with compatibility checks, and the Vision Lab. -| Area | Files | Notes | -|---|---|---| -| Frame contract | `vision/model/*`, `models/CapturedFrame.kt` | non-null payload; `displayRotation` recorded, `rotation` left at `ROTATION_0` for MediaProjection buffers | -| Geometry | `vision/geometry/FrameTransform.kt` | source→oriented→crop→processed round trips, crop and downscale | -| Preprocessing | `vision/preprocessing/*` | validation, aspect-preserving sizing, single-pass Android render, honest failure codes | -| Annotations | `vision/annotation/*` | `SlotAnnotationStore` with stable ids, duplicate/overlap/bounds validation, strict JSON codec | -| Calibration | `vision/calibration/*`, `datastore/CalibrationProfileDataStore.kt` | 1 % aspect tolerance, exact orientation match, unsupported-schema refusal, delete-as-escape-hatch | -| Capture adapter | `capture/CaptureVisionFrameSource.kt` | the only code that knows about both capture and Vision | -| Vision Lab | `feature/visionlab/*` | preview, freeze/resume, explicit processing, annotation canvas, text slot editor, profile controls | -| Tabs | `MainActivity.kt`, `DraftLensApp.kt`, `feature/home/DebugCaptureScreen.kt` | Material 3 `TabRow`; capture stays alive across tabs | +**Recognition baseline (1B).** Public static hero catalog with manual sync and a private, +validated cache; upright slot extraction; a visual-information gate; dHash plus mean-RGB +similarity reported as uncalibrated ranking; manual confirmation and correction; per-slot +confirmation storage. -## Sprint 1A.1 — Vision Lab interaction redesign +## Verification evidence -The annotation workflow was rebuilt around an explicit gesture model and a contextual -property editor. The frame sizing and rendering fix was confirmed working on a real device -by the user before this work started and was left untouched. - -### The interaction model - -`AnnotationInteraction` is a pure state machine that decides what every gesture means, so -two gestures can never compete and the rules are unit-tested: - -| Interaction | Action | +| Gate | Result at head `4eb9948` | |---|---| -| Short tap on a rectangle | select it | -| Drag inside a selected rectangle | move it | -| Long press on a rectangle | open its property editor | -| Drag a resize handle | resize it | -| Long press on empty image space, then drag | draw a new slot | -| Short tap on empty image space | nothing | -| Tap the X control | delete it | - -A tap never moves anything, a drag on an unselected rectangle only selects it, a stationary -long press is not a move, and a drag on empty space never creates a rectangle. - -### The contextual property editor - -A long press opens a compact editor next to the slot: type, team, index and an optional -label, plus **Copy**, **Paste** and **Duplicate**. It positions itself inside the window, so -it is never clipped off-screen in either orientation, and it no longer occupies a permanent -part of the canvas. The old bottom property controls and the "Add slot" card with its -hardcoded rectangle are gone; creating a slot is now a long press and a drag on the image. -Opening the menu selects the slot first, so the menu and the selection always agree. A type, team or index change that -would produce a duplicate logical identity is refused with the reason shown next to the -controls, the slot keeps its previous values and the editor stays open; invalid index text is -flagged inline. - -Copy and paste use the application's own clipboard, never the Android system clipboard. The -copied slot is always the source: a pasted slot gets a new unique id, takes the copied -slot's type, team, label and rectangle dimensions, is offset a little from the anchor (the -target slot when paste is invoked from its menu, the copied slot otherwise), stays inside -the image and takes the next free index in the copied slot's `(type, team)` scope. The -source and the target are never modified or deleted. - -### Deletion and undo - -The selected slot shows an X control at its top-right corner, drawn in red and distinct from -the resize handles. Deleting uses the slot's stable id. **Undo** covers a deletion, a -creation and a move or resize; a deletion is undone by restoring the slot with its own id. - -### The slot list - -The list is grouped into ALLY · BAN, ENEMY · BAN, ALLY · PICK, ENEMY · PICK and GLOBAL. -Every row is keyed by `key(slot.id)`, so the list never relies on positions. Tapping a row -selects that rectangle, and selecting a rectangle highlights its row. - -Investigating the deletion path: `SlotAnnotationStore.delete(id)` already removed by id and -was never deleting the first item. The real risk was Compose list identity, which is now -removed by keying every row with the slot's own id; a regression test deletes the second of -three slots and checks that the other two survive. - -### Fullscreen editor - -*Fullscreen* opens the image on its own screen in either orientation: a compact expand icon -in the corner of the canvas card opens it, the image and canvas fill the full available -phone workspace, and a compact collapse icon in the corner leaves it. There is no zoom, no -pan, no *Fit to screen* and no *Reset zoom* — the frame is fitted into the screen and the -slot overlay stays aligned with it. Android Back leaves the fullscreen editor first, and -only then the Vision Lab. The gesture hint appears when the editor is entered and can be -dismissed. - -### Slot colours - -`ALLY` is blue, `ENEMY` is red and `GLOBAL` is neutral grey, and every rectangle also -carries a readable label, so colour is never the only way to tell teams apart. - -### Unchanged on purpose - -The frame sizing and rendering fix, the letterbox rejection, the source-normalized -coordinate semantics, the display-rotation handling, the calibration codecs and the capture -lifecycle were not rewritten. - -### Sprint 1A.1 corrections - -Four corrections were applied on top of the interaction rework. - -1. **The delete control replaces the top-right resize handle.** The top-right handle is gone - and the X sits on exactly that corner, so the corner carries one control and not two. - The remaining handles — top-left, bottom-left, bottom-right — still change the width and - the height from either side, so nothing became unreachable. -2. **The long press opens the property menu immediately.** The long press is detected on its - own timeout instead of waiting for a pointer event, so the menu no longer needs the - finger to move first, and a small movement afterwards neither delays nor repeats it. -3. **Fullscreen editing was simplified.** Pinch-to-zoom, pan, *Fit to screen* and *Reset - zoom* were removed together with the view-transform state they needed. The fullscreen - entry is now a compact icon, the editor fills the screen, a collapse icon leaves it, and - Android Back leaves the editor before the Vision Lab. -4. **The page no longer scrolls during annotation.** The canvas takes the pointer from the - first touch, so moving, resizing and drawing a slot never scrolls the page. Scrolling - stays available everywhere else, and the fullscreen canvas is not inside a scrolling - parent at all. - -### Sprint 1A.2 — bug-fix pass - -Six focused fixes on top of 1A.1, each with regression tests: - -1. **Paste reads from the clipboard.** `pasteSlot()` used to read the target slot as the - source, so pasting from another slot's menu duplicated that slot instead of the copied - one. The clipboard slot is now always the source of type, team, label and rectangle - dimensions; the target only anchors the placement, and the pasted rectangle is offset from - the anchor (or from the copied slot when no target exists) and kept inside the image. - The copied source and the target are never modified. -2. **The menu and the selection are the same slot.** `openSlotMenu()` validates the id, - selects that exact slot and only then opens the menu, so the outline, the handles, the - delete control, the list highlight and the menu always refer to one slot. -3. **An early drag cancels the pending long press.** Moving past the touch slop before the - timeout now cancels it: a drag on an unselected slot only selects it, a drag on a - selected slot moves it, and a drag on empty space never turns into slot creation. Once - the timeout fires, the deadline is abandoned, so later movement can neither delay nor - re-open the menu. The 450 ms threshold is unchanged. -4. **One rotation snapshot per captured frame.** `FramePipeline.displayRotation` stays - `@Volatile`, and the ImageReader thread reads it once per frame; the same local value is - stamped on the frame and used for the orientation correction, so the two can never - disagree. The MediaProjection pixel-orientation behaviour is unchanged. -5. **Validation failures are visible.** A type, team or index change refused because of a - duplicate logical identity now shows the reason next to the controls, keeps the slot's - previous values and keeps the editor open. Invalid index text is flagged inline in the - index field. -6. **This report.** The fullscreen section no longer describes the removed pinch-to-zoom, - pan, *Fit to screen* and *Reset zoom* controls. - -Note: an earlier revision of this pass reworked the delete control (a density-aware 48 dp -touch target drawn outside the canvas clip region) and made slot creation open the property -editor automatically. Both were reverted at the maintainer's request and are **not** part of -the current build; the delete control and the creation flow behave exactly as they did in -1A.1. - ---- - -## Verification actually performed - -Everything below ran in **GitHub Actions** (`ubuntu-latest`, x86_64, JDK 17). The authoring -host is aarch64 and AGP 8.6.1 publishes only an x86_64 `aapt2`, so no Gradle task that -touches resources can run there — verified as -`Cannot run program ".../aapt2": error=2`. CI is the build of record. - -### Latest green run - -| | | -|---|---| -| Run | **43643640990** — | -| Tested head SHA | `5ffb0c193cb84da77fb49984c81e219178c7ef3f` | -| Unit tests | 269 tests · 0 failures · 0 errors · 0 skipped (538 executions, debug + release variants) | -| Other gates | `assembleAndroidTest`, strict `lint`, `assembleDebug` — all green | -| APK artifact | `draftlens-debug` · 10 215 534 bytes · digest `sha256:fef08a8b0233e78b30fb18094fd31e90c17557766954fc8fbf3d6a2d20db8a8f` · APK SHA-256 `937541942ff13d54b22d5fae28dbb7343ecb330ccc7e8b0d16f5048dbec758c5` | - -### Run history - -| CI run | Head SHA | Result | What it showed | -|---|---|---|---| -| `42905335945` | `34ebb10` | success | baseline before this sprint's work | -| `42905781209` | `b84b69f` | failure (expected) | Task 6 red: the store had no injectable constructor | -| `42906245805` | `85f04c8` | success | Task 6 green | -| `42906470342` | `473b632` | failure (expected) | Task 7 red: the seam and ViewModel did not exist | -| `42907112482` | `fedaa4e` | failure | nested test fakes could not reach an outer helper | -| `42907516553` | `82c330a` | failure | `advanceUntilIdle` does not drive `backgroundScope` collectors | -| `42907916398` | `0c7d286` | failure | the stale-result fake blocked by call order while the two jobs race | -| `42908409411` | `606206d` | failure | the first request can be cancelled before it reaches the preprocessor | -| `42908944687` | `b4c8c78` | success | Task 7 green | -| `42909440502` | `303b818` | failure | `MainActivity`: `this` resolved to `ColumnScope`, not the Activity | -| `42909895639` | `45e2f2c` | success | Task 8 green — Vision Lab UI compiles, lints and assembles | -| `42917700880` | `90cb807` | failure (expected) | review-fix red: the new pins had no implementation | -| `42918290133` | `691b9d2` | success | review fixes green | -| `42918914137` | `da35ca0` | success | documentation-only | -| `42919023498` | `8d3a330` | success | documentation-only | -| `42919485331` | `3e9e88c4be76933f0c6cc060f51acf3ab0444b14` | success | pre-fix state; artifact digest `sha256:ae7264df1ac2ec1f9c7091baf0ad0bb865f35db8f65d84bcb86bbfe398633eae`, APK SHA-256 `c9936bdbf64863c6baec7ceff227ea73d2b8be9a32593c748ef8d33321de710f` | -| `42925456760` | `1789d44a37143fcb59fbdefa6a43b876a9022773` | failure (expected) | focused-fix red: the new pins had no implementation | -| `42925901901` | `8f046c0` | failure | 5 tests still assumed the old profile frame policy | -| `42926242828` | `313775d` | success | focused fixes green | -| `42926731121` | `007621f` | success | profile-controls hint and documentation | -| `42942937802` | `d978309` | failure (expected) | canvas-fix red: the new pins had no implementation | -| `42943163574` | `01603de` | failure | 1 test used an out-of-range `NormalizedPoint`, which the type forbids | -| `42943730823` | `4c42513` | success | canvas rendering and workspace fixes green | -| `42953552163` | `927e8e0` | failure | the interaction rework did not compile: wrong pointer imports and a rectangle type mismatch | -| `42954140054` | `0762e96` | failure | 1 test expected a duplicate index that was not actually taken | -| `42955353155` | `dffc12d` | success | interaction rework green | -| `43513876694` | `24a43af` | failure | the annotation icons gave a `StrokeCap` where a `StrokeJoin` was expected | -| `43514508049` | `593eeb4` | success | corrections green | -| `43515376361` | `e1aca69` | **success** | final: corrections plus a small cleanup | -| `43516005404` | `c9a2cef` | **success** | documentation-only pointer before the 1A.2 pass | -| `43641588245` | `25307b8` | **success** | Sprint 1A.2 bug fixes green | -| `43642050380` | `f4df20f` | **success** | documentation-only pointer to the 1A.2 run | -| `43643640990` | `5ffb0c1` | **success** | 1A.2 with the delete control and creation flow reverted to 1A.1 | -| `42943730823` | `4c42513` | **success** | canvas rendering and workspace fixes green | - -Green run tasks: `./gradlew test`, `./gradlew assembleAndroidTest`, `./gradlew lint` -(strict, `abortOnError = true`), `./gradlew assembleDebug`, APK uploaded as the -`draftlens-debug` artifact, and the packaged manifest re-inspected for the foreground-service -permissions and the `mediaProjection` service type. - -**Instrumentation tests were compiled, not executed.** No device and no emulator was -available. - -## Focused review fixes (second pass) - -Six areas were reworked on top of the first green state. Each has a regression test that -failed before the fix. - -1. **Display rotation metadata updates.** `CaptureSessionService.recreateDisplay()` used to - return early when the capture size was unchanged, so a rotation that kept the size was - never recorded. The rotation is now updated independently of the ImageReader resize, a - `DisplayManager.DisplayListener` watches the default display (and is unregistered during - cleanup), and `FramePipeline` reads the current rotation when a frame is emitted instead - of capturing the value from start-up. The pixel correction stays `ROTATION_0`: MediaProjection - buffers are display-oriented, so no double rotation is introduced. -2. **Annotation canvas letterbox.** `CanvasGeometry.canvasToNormalized()` rejects a point - outside the fitted image rectangle instead of clamping it onto the image edge, so a touch - on a letterbox band selects nothing and cannot start a drag. -3. **Clear cancels in-flight processing.** `VisionLabViewModel.clear()` bumps the processing - generation, cancels the job and drops the references, so a late result cannot restore - itself. -4. **Calibration slot semantics.** `SlotValidator` is now the single validator used by both - the annotation codec and the calibration codec. A stored document with an unusable slot — - too small a rectangle, an index below 1, a duplicate slot id, a duplicate - `(type, team, index)` — or with a duplicate profile id is reported as `Corrupt` and is - never silently replaced. Unsupported-schema refusal and the delete-only recovery are - unchanged. -5. **Profile frame policy.** Saving, updating and applying a profile now require a frozen - frame; a moving live frame is never measured. -6. **Documentation and CI.** This report, `docs/VISION_TESTING.md` and PR #2 name the exact - green run, its head SHA and the artifact digest. - -## Canvas rendering and workspace fixes (third pass) - -A user reported that drawing rectangles was restricted to a small central area after the -letterbox fix. Two causes, both fixed. - -1. **The bitmap was drawn at its raw pixel size.** `drawOrientedBitmap()` passed the - bitmap's source pixel dimensions straight through as the Compose destination size, while - the geometry mapped annotations against the smaller fitted rectangle — so the pixels and - the overlay did not share the same bounds. `OrientedBitmapLayout` now computes the draw - destination: the oriented dimensions for all four rotations, one uniform scale into the - fitted rectangle, the bitmap centred on `fitted.center`, and the rotation applied around - that same centre. The image is never stretched, and the axis-aligned bounds of the - rotated bitmap are exactly the fitted rectangle. The preview card and the annotation - canvas use the same renderer. -2. **The workspace was a fixed 260 dp.** The canvas is now sized from the oriented aspect - ratio: a portrait capture gets a tall workspace that grows inside the scrollable page, a - landscape capture keeps a 240 dp minimum. The frame is never cropped and never stretched. - -Coordinate behaviour is unchanged and still pinned by tests: a letterbox touch is rejected at -gesture start, a tap outside the fitted image is rejected, an in-progress move or resize is -clamped to the image edge, and the create drag is deterministic. - -`FramePipeline.displayRotation` is now `@Volatile`, because it is written by the display -listener's thread and read by the ImageReader thread. The pixel correction stays separate -from the display metadata and remains `ROTATION_0` — no double rotation. - -The Vision Lab now explains its controls in place: *Freeze* locks the current frame before -annotation, *Resume* returns to live frames, a *slot* is a manually marked region where a -ban/pick hero portrait is expected, *Process Frame* resizes and prepares the frame only -because no recognition exists yet, and a *calibration profile* saves and restores normalized -slot coordinates for matching screen layouts. The "Detection: not implemented" notice stays. - -## Physical-device status - -- **Sprint 1A Vision Lab interaction: maintainer-reported PASS (device smoke).** The Vision - Lab interaction behavior in the retained 1A.2 state was confirmed working on a physical - Android device: slot selection, movement and resizing; long-press property menu; - long-press-and-drag creation; the original compact red X deletion control; copy, paste and - duplicate; fullscreen entry/exit and Android Back; annotation gestures isolated from page - scrolling; and existing portrait/landscape rendering. Device model, Android version, - install source/APK artifact, test date, logs and recordings were **not** supplied, so they - are not recorded here and no measurements are claimed. -- **MediaProjection lifecycle matrix remains NOT RUN.** The repeated start/stop, projection - revocation, app-process interruption, rotation/resize, background stability and sustained - capture scenarios in `docs/TESTING_PHYSICAL_DEVICE.md` are untested. -- **Sprint 0: PARTIAL, maintainer-reported.** Capture and live preview were reported working - on a physical device; device model, Android version, frame counts and measurements were not - supplied, so full PASS is not claimed. - -## Open limitations - -- No recognition, OCR, recommendations, overlays, API sync, or hero data. -- No automatic processing: the user must press *Process Frame*. -- No automatic profile application; an incompatible profile is refused with the expected and - actual ratio and orientation. -- Applying a profile regenerates slot ids (the store owns id generation). -- Native-memory reclamation after reference clearing is not guaranteed and is not a strict - one-bitmap bound. -- Self-capture: no `FLAG_SECURE` by maintainer preference; the user can choose "Entire - screen" or "A single app". -- OEM/background/store policies and review gates remain for release. -- Physical-device matrix needs execution before any store submission. - -## Next - -1. Execute the physical-device matrix, including the Vision Lab workflow. -2. Recognizer seam: a real hero-ID lookup over the slot images, with no fabricated results. -3. Recommendation scoring over detected slots, still behind an explicit user action. -4. Only then: floating overlay and API sync. - ---- - -# DraftLens — Sprint 0.1.3 Capture Lifecycle Report - -**Branch:** `sprint-0-android-foundation` · **PR:** #1 - -## Physical-device status -- **Baseline maintainer-reported smoke (Sprint 0): PARTIAL.** Capture and live preview were reported working on a physical device. Device model, Android version, frame counts and measurements were not supplied, so full PASS is not claimed. -- Expanded physical-device lifecycle matrix in `docs/TESTING_PHYSICAL_DEVICE.md` remains **NOT RUN** except the one PARTIAL row. Expanded scenarios not claimed executed. - -## Verified fixes (Sprint 0.1 set) -- AGP **8.6.1**, Gradle 8.7, Kotlin 2.0.20, compile/target SDK 35, minSdk 26 -- Strict lint enforced and used as CI gate -- Foreground service promoted synchronously with `mediaProjection` type before projection acquisition -- `START_NOT_STICKY`; null `Intent` is ignored; fresh consent per session -- `CaptureStartupExecutor` ordering: promotion → projection → initialization -- One `VirtualDisplay` per session; rotation uses `setSurface()` + `resize()` -- Capture dimensions prefer `WindowManager.maximumWindowMetrics` on API 30+, fallback below -- `VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR` -- Monotonic sampling; wall-clock retained separately; UI reports frame-processing latency only -- Latest frame is published by `CaptureStateHolder`; Compose bitmap is not recycled while rendering -- Termination paths are unified through `terminateSession()`: - - normal Stop → IDLE - - projection revocation/startup failure → ERROR, with frame cleared and metrics reset - - task removal/process end uses the same state cleanup - - permission and dimensions are cleared on termination, ERROR is preserved -- Consent intent is requested fresh at Start time - -## Tests covered by CI -- Startup order and failure propagation -- Target sampled time versus wall clock: monotonic reset, pause/resume semantics -- Clear/retained-frame removal tests using shared `terminateSession()` -- Session state reset and start-blocking rules -- Projection consent-result validation -- Detection stubs explicitly `NotImplemented` - -## CI (source of truth) -Local Gradle assembly is unavailable on this host because Android SDK build tools are x86_64 and host is aarch64. GitHub Actions is the build of record for `test`, `lint`, `assembleAndroidTest`, `assembleDebug`, APK artifact upload and merged-manifest inspection. - -Note: invocation `./gradlew assembleDebug` is intentionally not claimed locally; CI output is the deliverable. - -## Open limitations -- No recognition, OCR, recommendations, overlays, API sync, or hero data -- Analysis frame pipeline not added yet; preview is one full-size frame -- Self-capture: no `FLAG_SECURE` by maintainer preference; user can choose “Entire screen” or “A single app” -- Native-memory reclamation after reference clearing is not guaranteed and is not a strict one-Bitmap bound -- OEM/background/store policies and review gates remain for release -- Physical-device matrix needs execution before store release - -## Sprint 1B — review fixes and game-frame freeze (current) - -Branch `sprint-1b-hero-recognition`, PR #3 (draft, **not merged**). The full report is -`docs/sprints/SPRINT_1B_HERO_RECOGNITION.md`; this section records what changed for the -review-fix pass and how it was verified. - -### Changes made - -- **Part A — Freeze from the notification.** New `CaptureFreeze` bridge (service command + - in-app button are one code path). The ongoing capture notification now exposes - **Freeze current frame** (LIVE state) / **Resume live capture** (FROZEN state) plus - (superseded by the Sprint 1C follow-up: the notification now carries **Pause**/**Resume**) - - **Stop**. The command goes to the running service via an explicit, immutable - `PendingIntent.getService` (`ACTION_FREEZE` / `ACTION_RESUME_LIVE`); it never launches an - Activity, never re-promotes the service, never re-requests consent and never creates a - VirtualDisplay. `FramePipeline` gates publication on a `freezeGate` lock shared with the - image listener: while frozen, images are still acquired and closed (the reader stays - healthy) but not converted or published; a freeze that lands mid-conversion wins and the - in-flight frame is dropped. `FrameSnapshots.copyOf` copies pixels (bitmap `copy()` or - `IntArray.copyOf`) preserving dimensions, rotation, display rotation, timestamps and the - capture-latency metadata; the live pipeline never shares a payload buffer with a - snapshot; only one frozen frame is kept; a new capture session drops it; an Android-initiated - stop keeps it in process memory for editing (not across process death). The Vision Lab - in-app Freeze/Resume went through the same path. Guidance for full-device capture is in - the capture screen, the Vision Lab card and the README; `POST_NOTIFICATIONS` is checked - and surfaced (request on Android 13+, settings deep-link otherwise); the notification is - re-posted whenever the frozen state changes. -- **Part B — Pixel rotation fixed.** `PixelSlotImageExtractor` now follows - `FrameTransform.sourceToOriented()` exactly; tests rewritten with asymmetric non-square - fixtures and hand-computed expected pixel orders for all four rotations, crops touching - every edge, minimum-size and refused rectangles, and source-immutability checks. -- **Part C — Android extractor.** Validates Bitmap dimensions against - `OriginalFrame.sourceWidth/sourceHeight`; rotation rewritten to pixel-exact mappings; - source Bitmap never mutated/recycled; every temporary created in `extract()` is recycled - exactly once; failures recycle their temporaries. Instrumented tests added (all four - rotations, non-square edge crops, payload mismatch, source validity, minimum sizes, - recycled-bitmap refusal, bitmap snapshot ownership). -- **Part D — Blank rejection.** New pure-Kotlin `VisualInformationAnalyzer` (luma/color - variance, edge density, meaningful differences, minimum dimension 12×12) runs before - matching. Deterministic, no Android APIs. New distinct outcome - `InsufficientVisualInformation` and `INSUFFICIENT_INFO` UI state with - "Not enough visual information. Select a clearer hero crop." — uniform/near-uniform/ - detail-free crops never produce candidates. Thresholds are conservative heuristics, - documented as uncalibrated for MLBB. -- **Part E — Manual correction everywhere.** Manual name search and confirmation are now - reachable from CANDIDATES, NO_MATCH, FAILED, INSUFFICIENT_INFO and CONFIRMED. - **Change…** in CONFIRMED opens the manual correction search instead of re-running - Identify; a candidate can be corrected without clearing first. Failed persistence - operations are reported and never shown as success (confirm already reported; clear did - not — fixed with tests). -- **Part F — Cache freshness.** Per-asset `*.meta` sidecar records source URL, SHA-256, - size, content type. Serve-time validation: size cap 4 MiB, sidecar URL == catalog URL, - size and SHA-256 match, magic bytes, container structure (PNG chunks + CRCs, JPEG/GIF - trailers, WebP RIFF), content type, plus a real `BitmapFactory` decode on Android - (`AndroidReferenceImageValidator`, wired in `DraftLensApp`). Changed URLs force - re-download; the old asset is never served under a changed URL; a failed refresh keeps - the previous asset (unchanged URL) and never exposes it as the new one. Redirects are - followed manually with per-hop HTTPS re-validation (`HeroRedirectPolicy`); HTTP - downgrades and non-HTTP schemes refused. Snapshot writes flush/sync and `ATOMIC_MOVE`. -- **Part G — Atomic persistence.** `writeAtomically()` is now: temp sibling in the same - directory → flush + `FileDescriptor.sync()` (best effort) → `Files.move(ATOMIC_MOVE, - REPLACE_EXISTING)` → on any failure the temp file is deleted and the previous target is - preserved. No direct-write fallback. Stale `*.tmp` siblings older than an hour are swept - on init. Failure-path tests added. -- **Part H — Reference loader.** Dimensions are read before recycling; pixel reads - complete before release; exceptional paths recycle. -- **Part I — Matcher honesty.** The matcher remains clearly labelled a prototype ranking - baseline; weights and the 0.55 threshold are marked uncalibrated; an honest evaluation - section (synthetic fixtures, blank rejection, top-1/top-3 methodology, false-match - methodology, real-screen status, known limitations) is in the Sprint 1B report. No - production-accuracy or probability claims anywhere. -- **Part J — Scope.** Sprint 1A behavior preserved as-is (compact X delete control, 450 ms - long-press, early-drag cancellation, paste semantics, fullscreen without zoom/fit, gesture - isolation, portrait/landscape rendering, MediaProjection lifecycle apart from the freeze - integration). -- **Part K — Verification.** See the exact run below. Instrumented tests run on an - emulator job when available. - -### Verification actually performed (this revision) - -Docs-only commits after the code freeze do not change the build; the exact final head and its CI run are recorded in PR #3's description. - -- **Unit tests, debug and release: 792 executions, 0 failures, 0 errors, 0 skipped** — - , - head SHA `7a839d1422ee5ba53f65a78ef746d323091d3b40`. -- strict `lint` (abortOnError = true) — green; `assembleDebug` — green; public artifact - `draftlens-debug` 10 362 309 bytes, digest - `sha256:53c07492517fdfc2837ebb179393e526d6b397db2146228aa853eb4914221677`. -- `assembleAndroidTest` — instrumentation sources compile; **the suite was not executed**. - The CI runner has no KVM access: the default API-30 image did not boot within the - timeout, and the `aosp_atd` image boots but Gradle reports it as an "Unknown API Level" - and skips it. The emulator job was removed rather than left flaky; instrumented tests - must run on a physical device or a KVM-enabled emulator. Nothing here is claimed as - executed. - -### Claims NOT made - -- No production-grade recognition, no calibrated confidence, no MLBB accuracy claim, no - accuracy-based-on-synthetic-fixtures claim. The threshold and weights are uncalibrated. -- No physical-device freeze verification yet (rows 14–22 in - `docs/TESTING_PHYSICAL_DEVICE.md`). Device model, install source, test date and logs are - not recorded; the MediaProjection lifecycle matrix remains NOT RUN. -- The instrumentation suite does not and cannot simulate the consent UI; the notification - action end-to-end (tap in the shade while the game is foreground) needs a device. -- No screenshots or captured frames are committed; none were uploaded anywhere. - -### Remaining acceptance criteria - -- Physical-device smoke of the freeze workflow (notification tap while MLBB is foreground; - Vision Lab displays the pinned frame; Resume returns to live frames; stop keeps the pinned - frame for editing). -- On-device accuracy evaluation with manually labeled samples; threshold and - visual-information-heuristic calibration against real captures. -- Catalog entries loaded at device sync time recorded here when a device sync happens. -- PR #3 stays Draft and unmerged; do not merge without this verification. - -Historical green runs from before the review-fix pass (kept as history; do not cite these -as the current verification): `e58611c` — run 43732908605 (646 executions); `7d0c903` — run -43731484922 (same totals). - -## Critical fix — landscape capture shows only part of the game (device-reported) - -Reported by the maintainer on a physical phone: a normal screenshot captures MLBB correctly, -but the MediaProjection preview/capture shows only part of the game with the rest black; -portrait apps capture fine. (PR #3, still Draft and unmerged.) - -### Root cause analysis - -The visible path from Android's capture-resize callback to the frame on screen was inspected: -`CaptureSessionService` → `FramePipeline` → `ImageReader` → `imageToPayload` → -`PixelArrayFramePreprocessor`/`AndroidFramePreprocessor` → Vision Lab. - -Three defects combined, any one sufficient to produce the symptom: - -1. **The capture size was fixed at session start and never re-derived.** The only resize - trigger was `MediaProjection.Callback.onCapturedContentResize` (API 34+), so on pre-34 - devices — and on API 34+ whenever the callback does not fire for whole-screen captures — - rotating to landscape left the VirtualDisplay at the portrait session-start size forever. - The mirrored landscape content was clipped/letterboxed into the stale buffer and the rest - stayed black. The `DisplayManager` listener only updated rotation *metadata*, never size. -2. **Late frames from a released reader could publish into the replacement reader's stream** - (no generation guard) — stale frames could land on top of fresh ones after a resize. -3. **`image.cropRect` was ignored** in the plane conversion, so letterboxed/partial content - (e.g. before the first resize after a rotation) produced shifted garbage rather than the - full game. - -What was NOT the cause: the game does not block screen capture (a hardware screenshot works); -the matcher/Vision Lab are unaffected (this bug is in the capture pipeline only). - -### Fix - -- The captured-content resize callback now feeds a shared, authoritative resize path - (`recreateDisplay`) that re-creates the ImageReader/Surface generation at the *reported* - dimensions; `DisplayManager` changes re-derive the size from the current window metrics - and funnel into the same path — covered on all API levels ≥ 26; both paths are idempotent - and deduped by `DisplayChangePolicy`. -- Every reader/surface generation is numbered (`GenerationCounter`); images captured with a - stale generation are dropped at the listener and again at publish time, so a late frame - from a released reader can never reach the UI or the frozen snapshot. -- The plane conversion is rewritten into a pure, JVM-tested `RgbaPlaneCopy`: it honors - padded rows (`rowStride > width * pixelStride`), `image.cropRect` (letterboxed/offset - content produces a tightly packed image of the content), arbitrary `pixelStride >= 1` - and both byte orders; it never mutates the caller's buffer and drops malformed frames - instead of faking pixels. Bitmap dimensions used by the vision pipeline equal the - payload dimensions exactly. -- Structured `key=value` diagnostics cover every stage (`session=` id, `generation=`, - image dims, cropRect, rowStride, pixelStride, padding, bitmap dims, preprocess in/out). - No image contents are logged. See `docs/SCREEN_CAPTURE.md` → "Diagnosing the landscape bug". -- Freeze continues to work unchanged: the pinned frame is never overwritten by newer frames; - a resize during freeze pins the newest *valid* frame; Resume continues with the current - generation; the frozen frame is an independently owned copy with no shared payload buffer. - -### Evidence - -- CI at head `5252016cd15c824009368e831ed499f24209980d` (run 43773398149): **840 unit tests - (debug+release), 0 failures / 0 errors / 0 skipped**; strict lint green; `assembleDebug` - green; artifact `draftlens-debug`, digest - `sha256:c46580570a85054f225bd4d927350b153991f98956b1e3ee9a877d5ed9c1b3f1`. -- Letterbox/stride/offset behavior verified by deterministic JVM tests - (`RgbaPlaneCopyTest` — 22 cases) and the orientation swap in `DisplayChangePolicyTest`. -- Instrumented tests compiled, not executed (no KVM on the CI runner); the new - `FramePipelineInstrumentedTest` covers lifecycle invariants and must run on a device. -- On-device verification is **pending**: rows 23–32 of `docs/TESTING_PHYSICAL_DEVICE.md` - (portrait→landscape, resize storm, freeze-during-rotate, letterbox transient, API-34+ - single-app resize). **The fix is not claimed to work on a device until those rows are run.** -- If anything still shows black after this fix, the new logs pin down whether acquisition - (`generation=... plane:` line) or conversion (`... frame stored: ... bitmap:` lines) is - the source — see `docs/SCREEN_CAPTURE.md` → "Diagnosing the landscape bug with logs". - ---- - -# Sprint 1C — App-Only Capture & Circular Floating Control - -**Branch:** `sprint-1c-app-only-capture` · stacked on PR #3 (`sprint-1b-hero-recognition`, unmerged) · **PR:** #4 (base `sprint-1b-hero-recognition`) -**Spec:** maintainer Sprint 1C brief · **Design:** `docs/superpowers/specs/2026-11-16-sprint-1c-app-only-capture-design.md` - -## What shipped - -- Single-app (API 34+ partial) capture is now the recommended MLBB mode; the system chooser - is never forced. `onCapturedContentVisibilityChanged` drives frame publication: while the - selected window is hidden, images are acquired and closed but never converted or published; - the last valid game frame (and, if pinned, the frozen snapshot) is preserved. Publication - resumes on visibility return unless a frame is pinned. Full-display capture remains the - documented fallback with an explicit no-notification-free-promise note. -- One circular 52dp floating button (`TYPE_APPLICATION_OVERLAY`), shown only while a capture - session runs and only after the user grants the overlay permission (requested from the - capture screen, never at install time). Tap toggles LIVE (green) / FROZEN (orange) by calling - the shared `CaptureFreeze` path; 400 ms long-press arms drag; release snaps to the nearest - horizontal edge and persists normalized coordinates (DataStore). Rotation re-anchors it. -- Overlay is non-focusable, exactly button-sized, and removed on every teardown path - (stop, error, revoke, task removal, resize failure). -- Guidance rewritten: the "choose Entire screen" instruction is gone from the debug and - Vision Lab screens. -- No recommendation UI, no panel, no menu, no accessibility service; `SYSTEM_ALERT_WINDOW` - declared, requested only on opt-in. - -## Verification - -- Implementation head `f992d45`: CI run 43784157138 — **886 unit tests, 0 failures / 0 errors / 0 skipped**; - strict lint green; `assembleDebug` green; APK `app-debug.apk` - sha256 `1d1f50629e145d48e64ed56d0cf74228e7d48d4c6f94fde81eb6c22fc4795810`. Docs-only commits after - it change no build inputs and are re-checked by CI on every push. -- Instrumentation: compiled via `assembleAndroidTest` in CI; not executed — no KVM on the runner -- Physical device (rows 33–43 of `docs/TESTING_PHYSICAL_DEVICE.md`): NOT RUN — maintainer task; - in particular the overlay-vs-capture isolation finding (rows 34/35) is an on-device question - that CI cannot answer; no exclusion claim is made. - -## Known limitations / compatibility notes - -- API < 34: no visibility signal; app-only windows are undetectable — behavior degrades to - full-display semantics. -- If the captured window is already hidden when a session starts, publication stays open - until Android's first visibility callback arrives (API 34+; typically milliseconds). The - gate defaults to open because full-display projections may never call the callback at all. -- Full-display fallback may capture the shade/system UI/other apps/DraftLens; the freeze - controls still work. -- Whether the overlay appears inside captures is unverified per device (see sprint doc). -- A resume requested while MLBB is hidden flips the user's intent to LIVE and takes effect - when MLBB returns; publication stays gated while hidden (no black/DraftLens frames). -- Resize during freeze is enforced by the pipeline's freeze gate (policy truth-table + - `CaptureVisibilityPolicyTest`) and by on-device rows 25/30; no synthetic-pixel - instrumented test exists — an ImageReader surface cannot be filled from a test Canvas. - ---- - -# Sprint 1C — Follow-up fixes (notification Pause, permissions-only UI) - -**Branch:** `sprint-1c-app-only-capture` (stacked on PR #3, unmerged) · **PR:** #4 - -## Root cause of the overlay misbehaving during app-only capture - -Full trace (callback → policy → pipeline → freeze → overlay) shows the overlay's lifecycle -was never coupled to capture visibility — render, touch and window lifetime have no frame -input. Three real defects were causing the observed "frozen" control: - -1. silent no-op tap on `NoFrame` in fresh app-only sessions (nothing published yet); -2. overlay glyph and notification derived only from `frozenFrame`, so a non-pin publication - block (PAUSED) showed LIVE/green everywhere while publishing nothing; -3. no single derivation for "is live publication happening" across surfaces. - -Fixed by: unified `CapturePublishState` (publishStateFor / publishActionFor / -floatingTapFor) shared by notification and overlay; combined frozen-frame + session-state -collector; toast feedback on `NoFrame`; OEM/immersive-game overlay coverage documented as a -hardware finding (device rows 44–53, NOT RUN), with no masking or mode fallback shipped. - -## Notification: Freeze → Pause - -Stop + one state action: Pause (LIVE) / Resume (PAUSED or FROZEN). Pause maps onto the -existing `CaptureStatus.PAUSED` pipeline gate — no second projection, no second state, latest -valid frame preserved. Resume = `resumeLive()` + `setPaused(false)`. - -## UI cleanup - -General instructions removed: capture-mode tutorial card, floating-control tutorial card, -in-game freeze help texts, stale Sprint-0 marker. Kept: notification permission (request, -status, settings action) and overlay permission (request, status, settings action) in one -compact card. - -## Verification - -- Implementation head `130ce6c`: CI run 43922482183 — **904 unit tests, 0 failures / 0 errors / 0 skipped**; - strict lint green; `assembleAndroidTest` green; `assembleDebug` green; APK `app-debug.apk` - sha256 `48b510673919fe2caa9c906747d9dc766eb27bb170c589f67b58c9a175dd7b72`. Docs-only commits after - it change no build inputs and are re-checked by CI on every push. -- Instrumented tests: compiled via `assembleAndroidTest`; not executed (no KVM). -- Physical device rows 44–53: **NOT RUN** — see `docs/TESTING_PHYSICAL_DEVICE.md`. - ---- - -# Sprint 1C — Single-app first-frame bug fix - -**Branch:** `sprint-1c-app-only-capture` (stacked on PR #3, unmerged) · **PR:** #4 - -## Root cause (code-level evidence) - -The Sprint 1B capture path had no visibility gate. Sprint 1C introduced one and registered -`MediaProjection.Callback` **before** `FramePipeline.start()`, `createVirtualDisplay()` and -`CaptureStateHolder.beginCapture()`. On single-app capture both callbacks can arrive in -that STARTING window: - -1. `onCapturedContentVisibilityChanged(false)` — the selected MLBB window is hidden while - the user is still in DraftLens. With the old boolean gate, that value either got - overwritten by `pipeline.start()`'s unconditional `true` or closed the gate before any - frame existed. A missed/first-only callback leaves publication blocked with no valid - frame to protect — the "No frame captured yet" toast. -2. `onCapturedContentResize(W,H)` with the *selected app's* dimensions — `recreateDisplay` - either ignored it (status STARTING) or terminated the session as "VirtualDisplay - disappeared during resize" when it arrived before the VirtualDisplay existed. The - reader stayed at DraftLens' window metrics. -3. `frameJob` also dropped frames published before `beginCapture()` from the holder. - -This is one ordering defect across both callbacks, not three unrelated bugs: startup events -were not buffered and the gate's initial boolean could be confused with a real callback. - -## Diagnostics (added before changing the policy) - -`CaptureDiagnostics` rate-limits every event key (first 4 + every 20; no pixel data). -`CaptureSessionService` logs callback seq/value/pipelineInitialized/status and resize -seq/dimensions/readiness. `FramePipeline` logs acquisition count, image dimensions, -generation, isActive/generationCurrent/frozen/contentVisible, the exact gate-reject reason, -sampling accept/reject, conversion success/failure and bitmap dimensions, publication and -holder delivery. `freezeAndSnapshot` logs the state behind `NoFrame`. These log lines are -the evidence the maintainer needs to answer: "did ImageReader receive images that our code -rejected, or did the platform deliver none?" — see `docs/TESTING_PHYSICAL_DEVICE.md`, -"Focused first-frame test". - -## Fix - -- `CaptureVisibilityPolicy` now uses explicit `UNKNOWN` / `VISIBLE` / `HIDDEN`. UNKNOWN - deliberately allows publication (full-display never calls back; an early single-app - `false` must not block the first frame). The first published frame promotes - UNKNOWN → VISIBLE, after which `false` is authoritative and protects the valid frame. - The policy is owned by the service, so callbacks before pipeline creation are preserved; - stale-session callbacks are rejected; duplicate callbacks are idempotent. -- `FramePipeline` no longer resets visibility in `start()`/`prepareResize`; it consults the - injected policy, promotes on publish, and no longer duplicates callback state. -- `PendingResize` buffers the latest valid captured-content dimensions during startup. - `initializeProjection` sizes both ImageReader and VirtualDisplay from the buffered value - when present, applies any later value after `beginCapture()`, stays idempotent, and no - longer fails the session for a resize that arrived before the VirtualDisplay existed. -- `frameJob` records every published frame (the pipeline clears `latestFrame` on stop). - -## Verification - -- Implementation head `3c7b58c`: CI run 43936524131 — **926 unit tests, 0 failures / 0 errors / 0 skipped**; - strict lint green; `assembleAndroidTest` green; `assembleDebug` green; APK `app-debug.apk` - sha256 `ade9ec10c5d6b9d4e372b325e6b521f87170d0fa5273c9b30e9ea9063a763070`. Docs-only commits - after it change no build inputs and are re-checked by CI on every push. -- Instrumented tests: compiled, not executed (no KVM on the runner). -- Physical device: the A–E first-frame checklist and rows 44–53 of - `docs/TESTING_PHYSICAL_DEVICE.md` are **NOT RUN**. This fix is not claimed to work on a - device from JVM tests or CI alone. +| `./gradlew test` | 918 unit tests, 0 failures, 0 errors, 0 skipped | +| `./gradlew lint` | passes with `abortOnError = true` | +| `./gradlew assembleAndroidTest` | instrumented tests compile | +| `./gradlew assembleDebug` | debug APK produced and uploaded | +| Manifest inspection | `FOREGROUND_SERVICE_MEDIA_PROJECTION` and type `0x20` present | + +Instrumented tests are **compiled, not executed**: the hosted runner has no KVM, so the +emulator image either does not boot in time or is skipped as an unknown API level. CI proves +compilation, pure logic and lint only. + +## Not verified + +- The MediaProjection lifecycle matrix and the freeze workflow on a physical device + (rows 1–22 of the checklist). +- The landscape capture fix, rotation, resize storms and letterbox transients (rows 23–32). +- App-only visibility, the floating control and its OEM-specific capture isolation + (rows 33–53). +- Identification accuracy on real MLBB screens. Top-1/top-3 and rejection behaviour are + evaluated on synthetic fixtures only; the no-match threshold and visual-information + heuristics are starting defaults, not calibrated values. +- Live network behaviour: transport tests use fake responses, and the number of catalog + entries loaded is only known at device sync time. + +No device model, Android version, APK artifact or measurement was supplied for the +maintainer-reported smoke tests, so those are recorded as reports rather than evidence. See +[TESTING_PHYSICAL_DEVICE.md](TESTING_PHYSICAL_DEVICE.md) for the full status and +[VISION_TESTING.md](VISION_TESTING.md) for the test inventory. + +## Scope boundaries + +No recommendation, counter or synergy scoring; no automatic draft-phase detection; no +game-memory reading, injected touches, cloud recognition, ML model or OpenCV; no overlay other +than the single capture-freeze control. Captured frames and slot crops stay in memory and are +never uploaded. diff --git a/docs/SCREEN_CAPTURE.md b/docs/SCREEN_CAPTURE.md index dafd403..4391e01 100644 --- a/docs/SCREEN_CAPTURE.md +++ b/docs/SCREEN_CAPTURE.md @@ -1,190 +1,83 @@ -# Screen Capture — DraftLens Sprint 0.1.3 + 1B + 1C +# Screen capture ## Mechanism + `MediaProjectionManager.createScreenCaptureIntent()` → user consent → `ServiceCompat.startForeground()` -with `ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION` → `getMediaProjection()` → -`createVirtualDisplay()` → `ImageReader` surface. Consent acquisition stays in Activity; -`CaptureSessionService` performs foreground promotion, projection creation, VirtualDisplay and all frame work. - -## The floating freeze button (Sprint 1C) - -A single circular 52dp button rendered via `WindowManager` + `TYPE_APPLICATION_OVERLAY`, -attached by the capture foreground service **only while a capture session is running** and -removed on every stop/terminate path — a detach() invalidates any in-flight attach, so a -teardown can never orphan the window. It is the primary in-game freeze control; the -notification carries Stop and Pause and is the no-overlay fallback. - -Gestures: **tap** toggles LIVE (green, publishing) ↔ orange (publication paused or a frame -pinned); the glyph and color are driven only by the shared LIVE/PAUSED/FROZEN presentation -(`CapturePublishState`), never by incoming frames — hiding the captured window cannot pause, -detach or stale the button. **Long-press (400 ms) then drag** moves it; any movement before -the long-press fires cancels the gesture entirely. On release it **snaps to the nearest -horizontal edge** and persists its position as normalized coordinates in DataStore -(`floating_control_enabled`, `floating_position_x/y`). Rotation re-anchors it via the -display-metrics path; no re-consent, no re-attachment races. The overlay lifecycle is fully -independent of capture visibility: `onCapturedContentVisibilityChanged` gates only the -frame pipeline. - -`SYSTEM_ALERT_WINDOW` is requested only when the user enables the floating button on the -capture screen (Android overlay-settings screen); the capture itself never needs it. If the -permission is missing or revoked: capture and the in-app Freeze/Resume and the notification -Stop/Pause actions are unaffected, the capture screen explains how to grant it, and no -button is shown. The button calls the exact same shared pause/freeze path as everything -else — it never creates a second VirtualDisplay or re-requests consent. - -**Isolation caveat:** the overlay sets no flags to hide itself, and no capture region is -masked. Whether it appears inside single-app or full-display captures is a -physical-device finding per OEM — not verified in CI. See -`docs/sprints/SPRINT_1C_APP_ONLY_CAPTURE.md` and rows 33–43 of -`docs/TESTING_PHYSICAL_DEVICE.md`. - -## Capture notification: Stop and Pause (Sprint 1C) -The ongoing capture notification carries **Stop** and one state action — **Pause** while live, -**Resume** when paused or frozen — implemented as explicit `PendingIntent.getService`s back to -the running service — no Activity launch, no consent re-request, no second projection or -VirtualDisplay. **Pause** maps onto the shared capture pause state (`CaptureStatus.PAUSED`): -the pipeline stops converting/publishing (images are acquired and closed), the latest valid -frame is preserved, MediaProjection and the VirtualDisplay keep running, and nothing replaces -the paused frame. **Resume** clears any pinned frame (`CaptureFreeze.resumeLive()`) and clears -the pause (`setPaused(false)`), returning to live publication. The notification shows exactly -one of Pause/Resume plus Stop, reflects LIVE / PAUSED / FROZEN, and stays in sync with the -floating overlay and the in-app controls through the same `CapturePublishState` derivation. -Images arriving while frozen are acquired and closed but not converted or published. The -in-app Freeze/Resume (pinning an independently owned snapshot via `CaptureFreeze` / -`FrameSnapshots`) is unchanged. After an Android-initiated stop the pinned frame survives in -process memory for editing, but not process death. - -## Android version notes (compile/target SDK 35) -- Android 13+ `POST_NOTIFICATIONS` may be denied; capture still proceeds with a valid ongoing - foreground notification that Android may hide. In-app status and Stop control remain - available. The capture notification's **Stop** and **Pause** actions are not reachable while - the notification is hidden, so the app checks the permission (and requests it on Android 13+) - and says so on the capture screen; without it, pausing/freezing is done from the in-app - controls (or with the notification enabled first). -- **Android 14+**: service calls `startForeground` with the `mediaProjection` type **before** - `getMediaProjection()`; the consent token is single-session and dies with the service. - `MediaProjection.Callback.onStop()` (revocation) and `onCapturedContentResize()` (rotation) - are both handled. A resize signal dispatches to one authoritative path: the VirtualDisplay - keeps running, but the ImageReader/Surface generation is replaced and re-attached; - no second consent prompt is requested. -- **Orientation changes without the resize callback** (pre-API-34 devices, and any API level - where the callback does not fire) are handled by a `DisplayManager` fallback: when the - default display's metrics change, the capture size is re-derived from the current window - metrics and the same resize path runs. Both paths are idempotent and deduped by - `DisplayChangePolicy`. -- **Single-app capture (API 34+ partial capture)**: supported and preferred for the MLBB workflow — - the system chooser offers it; DraftLens never forces it. Prefer **Single app / app window → - Mobile Legends: Bang Bang**. The captured window's size is known through - `onCapturedContentResize` (API 34+) and flows through the resize path; its on-screen - visibility flows through `onCapturedContentVisibilityChanged` (see below). Full-display - capture remains the fallback for older Android and behaves exactly as before. -- **onCapturedContentVisibilityChanged (API 34+) IS handled (Sprint 1C)**: while the - selected window is hidden, images are acquired and closed but never converted or - published; the last valid published game frame is preserved. When the window returns, - publication resumes automatically unless a frame is pinned. Full-display captures never - report hidden, so this path is inert there. Below API 34 the signal does not exist and - the behavior degrades to the previous full-display semantics. The callback is fed to the - pipeline regardless of session status, so a hide during PAUSED cannot stale the gate on - resume. If the captured window is already hidden when a session starts, publication stays - open until the first visibility callback arrives (typically milliseconds) — the gate - defaults to open because full-display projections may never call the callback at all. -- **Capture dimensions:** API 30+ uses `WindowManager.maximumWindowMetrics`; lower APIs retain - the legacy `defaultDisplay.getSize()` path. -- **Android 12+**: partial screen capture (share one app) is possible in the system dialog. - For the MLBB workflow, **prefer Single app → Mobile Legends**; choose **Entire screen / - full device display** only when that option is missing. Full-display capture may include - the notification shade, Quick Settings, other apps and DraftLens itself — DraftLens - cannot promise notification-free capture on that path. The app says this on screen too. -- **minSdk 26**: `ImageReader`, `VirtualDisplay`, and foreground-service APIs used are all - available since API 21–26; no version-branched capture code is needed. - -## Landscape capture (MLBB) - -Full-screen captures mirror the display. If the device rotates from portrait to landscape -while a capture is running, the mirror's VirtualDisplay must change size with it. Three -inputs cover every supported API level, and all three funnel into one idempotent resize -path (`CaptureSessionService.recreateDisplay` → `FramePipeline.prepareResize`): - -1. `MediaProjection.Callback.onCapturedContentResize(width, height)` — the authoritative - content dimensions (API 34+). Both the buffer and the VirtualDisplay move to those dims. -2. `DisplayManager.DisplayListener.onDisplayChanged` — re-derives the size from the current - window metrics; the standard path on pre-34 devices. -3. `Resources.configuration` changes — the resize path itself is configuration-independent. - -While the capture size is stale (typically before the first callback after a rotation), -letterboxed or partial frames can be delivered. The conversion honors `image.cropRect`, so -such a frame shows the full game at its content size instead of garbage, until the resize -completes (typically the next frame). Nothing is ever stretched, padded with invented pixels, -or hidden by invented crops. - -**Reader generations.** Every ImageReader/Surface generation is numbered. Images captured -with an older generation are dropped at the listener and again at publish time, so a late -frame from a released reader can never overwrite the replacement reader's frames. A freeze -snapshot taken during a resize pins the newest *valid* published frame; resuming continues -with the current generation. Stale frames never reach the pinned slot either. - -### Diagnosing the landscape bug with logs - -All capture diagnostics are structured `key=value` lines under the tags -`CaptureSessionService`, `FramePipeline` and `VisionPreprocessor`. To collect them: - -```sh -adb logcat -c && adb logcat -v time -s CaptureSessionService FramePipeline VisionPreprocessor -``` - -Look for, in order: - -1. `session=N capture-start: dimensions=WxH dpi=D rotation=R source=windowMetrics` — the - size chosen at session start. If it is `1080x2400` while the game is landscape, this is - the session-start case, and the resize must follow (see 3). -2. `session=N default-display changed` and/or `captured-content-resize: CxC -> WxH` — which - signal fired and what dimensions it carried. If neither line appears on rotation, the - platform delivered no signal; the capture stays at the session-start size forever. -3. `session=N surface-detach ... surface-attach ... virtual-display resized: WxH dpi=D generation=G` - — the resize sequence completed; generation G is attached to the VirtualDisplay. - `pipeline resize: detaching previous generation=G-1 ...` shows the old generation detaching first. -4. `generation=G plane: image=WxH crop=.. rowStride=R pixelStride=4 paddingBytes=P fullCrop=true` - — one frame parsed straight out of the plane. Compare `rowStride` with `image width x 4` - and `crop` with the game's content rect. A mismatch here means the black region originates - in the acquired image (acquisition-side), not in the conversion. -5. `generation=G frame stored: source=WxH rotation=...` and later - `preprocess(Bitmap): source=... output=WxH` — the frame left the capture pipeline and the - preprocessing step; if (5) disagrees with (4), the black region is introduced in - conversion or transformation, not acquisition. - -Image contents are never logged, written to disk or uploaded. - -## Sampling -Default 500 ms (2 FPS), configurable 334–2000 ms via DataStore, hard-capped at ~3 FPS -(`FrameSampler`). `ImageReader(maxImages=3)` bounds backpressure; the listener drops (does not -queue) frames arriving faster than the sampler. The app publishes and retains the latest frame. -When a newer frame replaces it, the older frame becomes eligible for garbage collection if no -other component references it. DraftLens does not manually recycle a Bitmap that Compose may -still be rendering. **Stopping clears references; it does not guarantee immediate native-memory reclamation.** This is not a strict one-Bitmap memory bound. No frames touch disk. - -## Lifecycle / teardown -Normal user Stop produces IDLE. External revocation, startup failure, resize failure, process -termination and task removal all use the same terminate path: - -- clear the latest-frame reference -- reset session counters, timing, dimensions and density -- clear stale permission/session metadata -- preserve ERROR and its original message when applicable -- release VirtualDisplay → MediaProjection → ImageReader/Surface → HandlerThread → notification -- if no ERROR already exists, transition to IDLE - -Stopping does **not** clear the pinned (frozen) frame: it survives the projection session in -process memory for editing, and only disappears with process death, an explicit Resume/Clear, -or a new capture session. - -Any exception in the acquisition path is caught per frame so one bad frame never kills the session. - -## Limitations (honest list) -- Frame→Bitmap conversion honors the RGBA_8888 plane layout, padded rows and `image.cropRect`; - offset crops produce tightly packed images of the content region. A malformed plane drops - the frame instead of producing a partial or shifted one. -- The ViewModel mirrors service state through explicit calls (Sprint 0 uses a single - activity); a Binder/AIDL API is the documented upgrade if the service must survive longer. -- Emulator testing proves compile + logic, **not** MediaProjection behavior on real devices. - Physical-device runbook: `CaptureLifecycleInstrumentedTest` class comment and rows 14–33 - of `docs/TESTING_PHYSICAL_DEVICE.md`. +with `FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION` → `getMediaProjection()` → +`createVirtualDisplay()` → `ImageReader` surface. The Activity acquires consent; +`CaptureSessionService` owns foreground promotion, projection, display, frames and teardown. +The service starts before the projection is obtained, and the consent token is used once per +session. + +## Capture modes + +Prefer **Single app / app window → Mobile Legends** on Android 14+. DraftLens never forces the +choice. Whole-display capture remains the fallback and may include the notification shade, +Quick Settings, other apps and DraftLens itself; the app says this on screen. + +When the selected window is hidden (API 34+ `onCapturedContentVisibilityChanged`), images are +still acquired and closed but never converted or published, so the last valid game frame is +kept instead of a screenshot of the shade. Below API 34 there is no visibility signal and +whole-display semantics apply. + +## Freeze, pause and resume + +- **Freeze** pins an independently owned snapshot (`CaptureFreeze` / `FrameSnapshots`). While + frozen, the reader keeps acquiring and closing images but performs no conversion or + publication, so a new frame can never overwrite the pinned one. +- **Pause** maps to `CaptureStatus.PAUSED`: publication stops, the latest valid frame is kept, + and the projection and VirtualDisplay keep running. +- **Resume** releases the pin and clears the pause. No second projection or display is created. + +The notification carries **Stop** and exactly one state action — Pause while live, Resume when +paused or frozen — as immutable `PendingIntent.getService`s to the running service. The +in-app controls and the overlay use the same `CaptureFreeze` and shared `CapturePublishState` +derivation, so the LIVE / PAUSED / FROZEN badge, the overlay glyph and the notification action +cannot disagree. + +## Floating control + +One 52 dp circular `TYPE_APPLICATION_OVERLAY` button, attached by the capture service only +while a session runs and removed on every teardown path. A detach invalidates any in-flight +attach, so teardown cannot orphan the window. Tap toggles publication state; long-press then +drag moves the button; on release it snaps to the nearest horizontal edge. Its normalized +position is stored in DataStore and re-anchored on rotation. The button's lifecycle is +independent of capture visibility. + +`SYSTEM_ALERT_WINDOW` is requested only when the user enables the control from the capture +screen. Denial leaves capture, in-app controls and notification actions working. There is no +flag hiding the button from captures and no masked region: whether it appears inside +single-app or full-display capture is an OEM-specific physical-device finding. + +## Resize and rotation + +Three inputs funnel into one idempotent resize path +(`CaptureSessionService.recreateDisplay` → `FramePipeline.prepareResize`): + +1. `onCapturedContentResize(width, height)` — authoritative content size, API 34+. +2. `DisplayManager.DisplayListener.onDisplayChanged` — re-derives size from current window + metrics; the standard path below API 34. +3. Configuration changes — handled by the same path. + +Every reader and surface generation is numbered, and stale callbacks are dropped at the +listener and again at publication, so a frame from a released reader cannot reach the UI or +the pinned snapshot. Plane conversion honours `image.cropRect` and padded rows, with pure +JVM-tested logic. While the size is stale, a letterboxed frame is shown complete rather than +garbage; nothing is stretched or padded with invented pixels. + +A rotation that leaves the capture size unchanged updates rotation metadata only. API 30+ +uses `WindowManager.maximumWindowMetrics`; lower APIs keep the legacy `defaultDisplay.getSize()` +fallback. + +## Android notes + +- **13+**: `POST_NOTIFICATIONS` may be denied. Capture still runs, but shade actions are + unreachable; the capture screen explains this and in-app controls remain available. +- **14+**: `startForeground` with the mediaProjection type happens before `getMediaProjection()`. + `onStop()` (consent revoked) is handled as session termination. +- **minSdk 26**: the ImageReader, VirtualDisplay and foreground-service APIs used are available. +- There is no `FLAG_SECURE`, no root access, no game-memory reading, no injected touches and no + network upload of captured pixels. + +Hardware behaviour of all of the above is tracked in +[TESTING_PHYSICAL_DEVICE.md](TESTING_PHYSICAL_DEVICE.md) and is largely still NOT RUN. diff --git a/docs/TESTING_PHYSICAL_DEVICE.md b/docs/TESTING_PHYSICAL_DEVICE.md index 540872a..0f9f6a0 100644 --- a/docs/TESTING_PHYSICAL_DEVICE.md +++ b/docs/TESTING_PHYSICAL_DEVICE.md @@ -1,21 +1,27 @@ -# Physical-device lifecycle checklist — DraftLens - -Status values: **PASS**, **FAIL**, **PARTIAL**, **NOT RUN**. Record the commit SHA, APK artifact run ID, device model and Android version for each execution. Do not mark a row PASS without evidence from that run. - -> Maintainer-reported baseline: Sprint 0 capture and preview were smoke-tested on a physical Android device and confirmed working. Device details, versions, frame counts and latency measurements from that test were not provided, so they are not recorded here. First capture is **PARTIAL** on that evidence; all expanded lifecycle rows below remain **NOT RUN**. - -> Maintainer-reported (Vision Lab, 1A.2 retained state): the Vision Lab interaction behavior in the currently shipped build was smoke-tested on a physical Android device and confirmed working. Device model, Android version, install source/APK artifact, test date, logs and recordings were **not** supplied, so they are not recorded here and no measurements are claimed. Tested and confirmed working: -> -> - Slot selection, movement and resizing. -> - Long-press property menu. -> - Long-press-and-drag slot creation (creation selects the new slot; the property editor does not open automatically). -> - The original compact red X deletion control at the top-right corner. -> - Copy, paste and duplicate. -> - Fullscreen entry/exit and Android Back. -> - Annotation gesture isolation from page scrolling. -> - Existing portrait/landscape rendering behavior. -> -> This does **not** cover the MediaProjection lifecycle matrix. Repeated start/stop cycles, projection revocation, app-process interruption, rotation/resize handling, background stability, sustained capture and the remaining lifecycle rows below remain **NOT RUN**. +# Physical-device checklist + +CI proves compilation, pure logic and lint. It cannot prove MediaProjection, visibility or +overlay behaviour on hardware, so this checklist is the remaining verification work. Status +values are **PASS**, **FAIL**, **PARTIAL** and **NOT RUN**. Record the commit SHA, APK artifact +run ID, device model and Android version for every execution. Do not mark a row PASS without +evidence from that run. + +## Maintainer-reported baseline (not evidence) + +Two smoke tests were reported on a physical device. No device model, Android version, install +source, APK artifact, test date, logs, recordings or measurements were supplied, so neither is +recorded as a verified result: + +- Sprint 0 capture and live preview were confirmed working. Row 1 is therefore **PARTIAL** on + that report alone. +- Vision Lab interactions were confirmed working: slot selection, movement and resizing; + long-press property menu; long-press-and-drag creation; the compact red X delete control; + copy, paste and duplicate; fullscreen entry and Android Back; gesture isolation from page + scrolling; and portrait/landscape rendering. + +Neither report covers the MediaProjection lifecycle matrix, freeze workflow, landscape fix, +app-only visibility or the floating control. All rows below remain **NOT RUN** until executed +with the evidence described at the end of this file. | # | Scenario | Expected behavior | Status | |---:|---|---|---| diff --git a/docs/VISION_ARCHITECTURE.md b/docs/VISION_ARCHITECTURE.md index 6a16048..d36bd75 100644 --- a/docs/VISION_ARCHITECTURE.md +++ b/docs/VISION_ARCHITECTURE.md @@ -1,55 +1,33 @@ -# Vision Architecture — Sprints 1A + 1B +# Vision architecture -## What this layer is - -Sprint 1A gives the app a real image-bearing pipeline. Frames carry pixels, geometry is -tested in pure Kotlin, preprocessing validates before it renders, and a Vision Lab lets a -user place manual slot annotations and save calibration profiles. Sprint 1B adds the hero -recognition pipeline on top — still offline, still manual-confirmation only. - -**What it is not:** there is no recognizer that selects a hero by itself. No OCR, no automatic -ban/pick detection, no calibrated confidence values, no ML model. Manual slots are calibration -data, never detections. +Vision turns captured frames into inspectable pixels and geometry, then applies a deliberately +small identification baseline whose results always require manual confirmation. The layer is +mostly pure Kotlin so the risky logic is testable on the JVM. ## Package map -```text -com.draftlens - capture/ - CaptureSessionService foreground service, MediaProjection owner, notification actions - CaptureController start/stop/pause entry points - CaptureFreeze freeze/resume command bridge (service commands + in-app button) - FramePipeline ImageReader -> Bitmap, latest-frame-only, freeze-aware - FrameSampler monotonic sampling - CaptureStateHolder process-local state holder (live + frozen frame) - CaptureVisionFrameSource adapter: capture -> VisionFrameSource (the only bridge) - models/ - CaptureSessionState, CaptureTransitions, CapturedFrame, FrameSnapshots - datastore/ - AppPreferencesRepository sampling interval - CalibrationProfileDataStore one JSON document under `profiles_json` - vision/ - model/ FrameId, FrameEnums, FramePayload, VisionFrame, VisionFrameSource - model/android/ BitmapFramePayload (the only Android type in vision/model) - geometry/ NormalizedPoint/Rect, PixelPoint/Rect, FrameTransform - preprocessing/ PreprocessTypes, PreprocessingMath, PixelArrayFramePreprocessor, - AndroidFramePreprocessor - annotation/ DraftSlot, SlotAnnotationStore, SlotAnnotationCodec - calibration/ CalibrationProfile, CalibrationCompatibility, CalibrationProfileCodec, - CalibrationRepository - feature/ - home/ HomeViewModel, DebugCaptureScreen (tab 0) - visionlab/ VisionLabViewModel, VisionLabScreen, AnnotationCanvas, - SlotEditor, ProfileControls (tab 1) -``` - -`vision/**` is pure Kotlin except `vision/model/android/BitmapFramePayload.kt` and -`vision/preprocessing/AndroidFramePreprocessor.kt`, which touch `android.graphics`. +| Package | Responsibility | +|---|---| +| `vision/model` | Frame ids, enums, payloads, original/processed/slot frames, `VisionFrameSource` | +| `vision/model/android` | `BitmapFramePayload` — the only Android type in the model package | +| `vision/geometry` | Normalized and pixel value types, `FrameTransform` | +| `vision/preprocessing` | Plan/result types, pure math, pixel and Android renderers, diagnostics | +| `vision/annotation` | `DraftSlot`, validating store, strict codecs, shared validator | +| `vision/calibration` | Profiles, compatibility check, codec and repository interface | +| `vision/heroes` | Catalog, transport, cache, slot extraction, visual-information gate, matcher | +| `capture` | Capture service, freeze, visibility policy, generations, plane copy, the only adapter | +| `datastore` | Preferences, calibration document, hero confirmation store | +| `feature/visionlab` | ViewModel, screens, canvas, geometry helpers, panels | +| `feature/home` | Capture debug screen and ViewModel | +| `overlay` | Floating control view, controller, enablement decision, position math | + +Everything under `vision/**` is pure Kotlin except `BitmapFramePayload`, +`AndroidFramePreprocessor`, `AndroidBitmapSlotImageExtractor`, +`AndroidHeroCatalogTransport` and `AndroidReferenceImageValidator`, which touch Android APIs. ## The boundary -Vision never imports `CaptureSessionService` or `CaptureStateHolder`. It consumes a narrow -seam: +Vision never imports `CaptureSessionService` or `CaptureStateHolder`. It sees only: ```kotlin interface VisionFrameSource { @@ -61,129 +39,54 @@ interface VisionFrameSource { } ``` -`CaptureVisionFrameSource` (in `capture/`) is the only file that knows about both sides. -It maps `CaptureStatus` to `VisionSourceState` and delegates the freeze commands to -`CaptureFreeze`, the same bridge the service's notification action uses — in-app and -notification freezes are one code path. - -| CaptureStatus | VisionSourceState | -|---|---| -| `CAPTURING` | `CAPTURING` | -| `PAUSED` | `PAUSED` | -| `IDLE` | `IDLE` | -| `STOPPING` | `STOPPED` | -| `ERROR` | `ERROR` | -| `AWAITING_PERMISSION`, `STARTING` | `UNKNOWN` | - -`AWAITING_PERMISSION` and `STARTING` map to `UNKNOWN` because the system consent dialog and -service foregrounding are not yet meaningful to Vision. +`CaptureVisionFrameSource` is the single file that knows both sides. It maps capture status to +`VisionSourceState` and delegates freeze commands to `CaptureFreeze`, the same bridge used by +the notification and the overlay. ## Data flow ```text -MediaProjection -> VirtualDisplay -> ImageReader -> capture HandlerThread - -> Bitmap (latest only) -> CaptureStateHolder -> CaptureVisionFrameSource - -> VisionLabViewModel (live or frozen reference) -> VisionLabScreen - -> explicit Process Frame -> Dispatchers.Default job -> ProcessedFrame - -> Compose overlay and inspectors -``` - -The freeze workflow is a side channel on the same pipeline — never a second capture system: - -```text -in-app Freeze / the floating button (pinning via CaptureFreeze) or the Vision Lab Freeze button - -> CaptureFreeze.freezeCurrentFrame() (the notification issues Pause = ACTION_PAUSE) - - -> CaptureFreeze.freezeCurrentFrame() - -> FramePipeline freezes publication; copies the latest published frame - -> CaptureStateHolder.frozenFrame (independently owned copy, at most one) - -> CaptureVisionFrameSource.frozenFrame -> VisionLabViewModel -> Vision Lab canvas - -> notification re-posted as FROZEN with a "Resume live capture" action +ImageReader -> RgbaPlaneCopy -> Bitmap -> capture holder + -> CaptureVisionFrameSource -> VisionLabViewModel (live or pinned) + -> explicit Process Frame -> ProcessedFrame -> inspector + -> select slot -> SlotImageExtractor -> VisualInformation gate + -> HeroMatcher against cached reference images -> candidates + -> manual confirm or correct -> HeroRecognitionStore ``` -While frozen, the ImageReader keeps acquiring and closing incoming `Image`s (the reader and -VirtualDisplay stay healthy) but performs no conversion or publication, so new frames can -never replace the pinned one. Capture keeps running when the Vision Lab tab is not visible: -the service owns the projection and the process-local holder owns the latest frame. The -Vision Lab only reads. - -## Processing rules - -- Processing is **explicit**. Nothing processes automatically; the user presses *Process Frame*. -- At most **one** job runs. A new request increments a generation counter and cancels the - previous job. A result whose generation is no longer current is discarded, so a slow - frame can never overwrite a newer one. -- The frame is captured at press time (frozen frame if pinned, otherwise the latest live - frame), so a frame that arrives mid-flight cannot be substituted. -- Duration uses `System.nanoTime()`. -- Failure is returned as `ProcessFrameResult.Failure` with a reason code. No partial frame is - ever published as a success. - -## Ownership - -- The capture pipeline owns the bitmap. `CaptureStateHolder` keeps only the latest frame. -- No path calls `Bitmap.recycle()`. References are dropped; native memory is reclaimed by GC. -- `BitmapOwnership.SHARED_READ_ONLY` is descriptive, not a synchronized lifetime system: - consumers may draw the bitmap but must not mutate or recycle it. -- A frozen frame is an **independently owned copy** (`FrameSnapshots`): pixels are copied and - metadata (dimensions, rotation, display rotation, timestamps, latency) is preserved - verbatim; the live pipeline never shares a payload buffer with it. The copy survives the - projection stopping (for editing while the process lives) but not process death. A new - capture session drops it. `Clear` releases it. -- Calibration profiles store normalized rectangles, labels and metadata only — never pixels, - bitmaps, transform ids or screenshots. +Processing and identification are separate explicit actions. At most one processing job runs: +a generation counter discards stale results, and failures are reported rather than published as +partial frames. + +## Frame and geometry rules + +- Payloads are non-null and immutable from the consumer's point of view. +- `displayRotation` is a record; `rotation` is the correction actually applied. Production + frames use `ROTATION_0` because MediaProjection buffers are already display-oriented. +- Normalized coordinates are in source space, bounded to `[0, 1]`; pixel rectangles have + exclusive far edges. +- The processing order is validate → orient → crop in oriented space → resize without + distortion. Crops are stored in oriented normalized coordinates. +- The annotation canvas maps pointers and rectangles through an identity-crop transform so + overlays align with the rendered image for every rotation. + +## Recognition rules + +- The catalog has one identity per hero derived from the name. Rone Arena is primary; MLBBDex + cross-checks completeness and supplies role and lane tags. Sync and image download happen + only on explicit request and are cached privately. +- Every asset is validated when served and re-downloaded if its URL, hash or structure changes; + a failed refresh keeps the previous asset. Snapshot writes are atomic. +- Slot crops are extracted upright for every rotation. Blank, near-uniform and detail-free + crops are rejected before matching. +- The matcher combines dHash and a mean-RGB descriptor into an uncalibrated similarity. It is + a ranking baseline, not a probability and not an ML model. Below threshold the result is no + match. +- Manual confirmation is mandatory and available from every state, including correction by + name search. Confirmations are stored per slot; failed writes are reported honestly. ## Persistence -`CalibrationProfileDataStore` writes one JSON string under `profiles_json` in the -`draftlens_calibration` preferences store. Every mutating operation reads and decodes the -current document inside the same transaction, so an unsupported or corrupt document is never -overwritten. `clearDocument()` is the only escape route out of a document this build cannot -read; it is what the "Delete all calibration data" action calls. - -## Failure model - -| Situation | Result | -|---|---| -| Frame dimensions ≤ 0 | `INVALID_DIMENSIONS` | -| Payload size ≠ source dimensions | `PAYLOAD_MISMATCH` | -| Source pixels above `maxPixels` | `TOO_MANY_PIXELS` | -| Crop empty or out of range | `INVALID_CROP` | -| Output above the pixel budget | `OUTPUT_TOO_LARGE` | -| Non-bitmap payload on the Android path | `UNSUPPORTED_FORMAT` | -| Bitmap drawing threw | `RENDER_FAILED` | -| Calibration document newer than schema 1 | `CalibrationLoadResult.UnsupportedSchema`, writes refused | -| Calibration document unparseable | `CalibrationLoadResult.Corrupt`, writes refused | -| Calibration slot with a too-small rectangle, an index below 1, a duplicate id or a duplicate `(type, team, index)` | `CalibrationLoadResult.Corrupt`, writes refused | -| Two calibration profiles with the same id | `CalibrationLoadResult.Corrupt`, writes refused | - -Slot semantics are validated once, in `vision/annotation/SlotValidator`, and used by both -the annotation codec and the calibration codec. A slot the editor would refuse cannot be -persisted as calibration data, and a stored document that contains one is reported as -`Corrupt` and left untouched until it is deleted. - -## Sprint 1B — hero recognition boundary - -`vision/heroes/` holds the recognition pipeline. It consumes `VisionFrame` inputs (the same -`OriginalFrame`/`SlotImage` contract as the annotation canvas) and never imports capture code. -Everything network-facing goes through `HeroCatalogTransport`; tests use fake transports. - -- **Catalog** (`HeroCatalog*`) is public, static metadata only: hero id/name/images/roles/lanes. - Downloaded only on the user's explicit **Sync catalog** action; cached in the app's private - storage; works offline after sync. The cache binds each asset to its source URL and a - SHA-256 of the bytes, validates before reuse, and flushes/syncs snapshots atomically. -- **Extraction** (`SlotImageExtractor`) maps the slot's normalized source rect to source pixels - and produces an upright in-memory `SlotImage`. Never written to disk. -- **Visual information** (`VisualInformationAnalyzer`) runs before matching and refuses - blank/near-uniform/detail-free crops as `InsufficientVisualInformation`; it is a pure - Kotlin, deterministic gate, with conservative uncalibrated heuristics — never presented as - a recognition accuracy. -- **Matching** (`HeroMatcher`) is a deterministic dHash + color descriptor, an uncalibrated - prototype ranking baseline. Output is a ranked list of candidates or an explicit no-match; - neither says the hero exists in the crop. The UI turns that into manual confirm/correct — - never an automatic selection. -- **Confirmations** (`HeroRecognitionStore`) persist `slotId → {heroId, heroName, confirmedAtMs}` - in a separate preferences document; it never touches the annotation schema or the calibration - store, and it contains no images. -- **Privacy**: frames and crops stay in memory on-device; only the public static catalog and - its reference images are downloaded; INTERNET is the only new permission. +Calibration profiles and hero confirmations are metadata only. No captured pixels, slot crops +or screenshots are persisted. The app's only network client fetches public static catalog data +and reference images over HTTPS. diff --git a/docs/VISION_COORDINATES.md b/docs/VISION_COORDINATES.md index 1f0be19..8006c7f 100644 --- a/docs/VISION_COORDINATES.md +++ b/docs/VISION_COORDINATES.md @@ -1,139 +1,82 @@ -# Vision Coordinates +# Vision coordinates -## Normalized coordinates +This document defines the coordinate spaces used by capture, preprocessing, annotation and +recognition. The rules are implemented in `com.draftlens.vision.geometry` and are pure Kotlin +so they can be tested on the JVM. -All annotation geometry is stored in **normalized source coordinates**: +## Spaces -- `x` runs left to right, `y` runs top to bottom, both in `[0.0, 1.0]`. -- The axes are the source image's axes **as captured** — before any rotation, crop or scale. -- They are not display dp, not density-scaled, and not affected by a crop origin. +| Space | Type | Units | Origin | +|---|---|---|---| +| Source | `NormalizedRect`, `NormalizedPoint` | `0..1` per axis, as captured | top-left of the captured pixels | +| Source pixels | `PixelPoint`, `PixelRect` | integer pixels | top-left of the captured pixels | +| Oriented pixels | `FloatPoint` | continuous pixels | top-left after rotation correction | +| Crop pixels | `FloatPoint` | continuous pixels | top-left of the crop rectangle | +| Processed pixels | `PixelRect` | integer pixels | top-left of the processed output | -`NormalizedRect(left, top, right, bottom)` is valid only when every value is finite, -`left < right`, `top < bottom`, and all four are inside `[0, 1]`. An empty rectangle is not -storable. +Normalized values are not display dp, are not density-scaled, and do not change when a crop +is applied. A `NormalizedRect` is valid only when every value is finite, `left < right`, +`top < bottom`, and all four values are inside `[0, 1]`. A `PixelRect` has exclusive right and +bottom edges, so it always keeps at least one pixel between opposite edges. -`PixelRect(left, top, right, bottom)` uses integer pixels with **exclusive** right and bottom -edges, so at least one pixel always remains between opposite edges. - -Conversions: +Conversions round outward so the pixel rectangle covers the requested area: ```kotlin -fun NormalizedRect.toPixelRect(width: Int, height: Int): PixelRect // rounds outward +fun NormalizedRect.toPixelRect(width: Int, height: Int): PixelRect fun PixelRect.toNormalized(width: Int, height: Int): NormalizedRect ``` -Rounding outward means the pixel rectangle always covers the requested area. - -## Orientation categories +`OrientationCategory.from(width, height)` returns `LANDSCAPE`, `PORTRAIT` or `SQUARE`; square +is its own category. -`OrientationCategory.from(width, height)` returns `LANDSCAPE` when `width > height`, -`PORTRAIT` when `height > width`, and `SQUARE` when they are equal. Square is a category of -its own rather than being squeezed into portrait or landscape. - -## Two rotations +## Two different rotations | Field | Meaning | Effect on pixels | |---|---|---| -| `displayRotation` | device display rotation at acquisition | none — it is a record | -| `rotation` | pixel-orientation correction | the transform is applied | +| `displayRotation` | device rotation at acquisition | none — a record only | +| `rotation` | pixel-orientation correction | applied by the transform | -MediaProjection buffers are already display-oriented, so production frames set -`rotation = ROTATION_0`. A non-zero `rotation` appears only in synthetic fixtures or in an -authorized import that genuinely needs straightening. +MediaProjection buffers are already display-oriented, so production frames carry +`rotation = ROTATION_0`. Non-zero corrections exist for synthetic fixtures and for imports +that genuinely need straightening. -Source-to-oriented mappings, applied to a continuous source point `(x, y)`: +Source-to-oriented mappings for a continuous point `(x, y)` in a `W × H` image: -| Rotation | Oriented size | Source point → oriented point | +| Rotation | Oriented size | Oriented point | |---|---|---| | `ROTATION_0` | `W × H` | `(x, y)` | | `ROTATION_90` | `H × W` | `(H - y, x)` | | `ROTATION_180` | `W × H` | `(W - x, H - y)` | | `ROTATION_270` | `H × W` | `(y, W - x)` | -`ROTATION_90` and `ROTATION_270` swap the axes; `ROTATION_180` preserves them. +The 90° and 270° cases swap axes; 180° does not. -## Pipeline order +## Processing order ```text -validate -> normalize orientation -> optional crop on the oriented image -> resize without distortion +validate → orient → crop in oriented space → resize without distortion ``` -The crop rectangle is a normalized rectangle in **oriented** coordinates. The crop is applied -after the orientation correction and before the resize. - -## FrameTransform - -`FrameTransform` is the record of one preprocessing run: - -| Field | Meaning | -|---|---| -| `sourceWidth`, `sourceHeight` | captured size | -| `rotation` | the correction that was applied | -| `crop` | the oriented crop, or the full oriented frame when nothing was cropped | -| `orientedWidth`, `orientedHeight` | size after the correction | -| `cropPixelRect` | the integer pixel rectangle of the crop inside the oriented image | -| `scale` | processed pixels per crop pixel, horizontally | -| `processedWidth`, `processedHeight` | the output size | - -`FrameTransform.create(sourceWidth, sourceHeight, rotation, crop, processedWidth, processedHeight)` -validates positive dimensions and requires the output to preserve the oriented crop's aspect -ratio within one pixel of width or height. - -The mapping chain is continuous (floating point) and reversible: - -```text -source pixels -> oriented pixels -> crop pixels -> processed pixels -``` - -Round trips hold up to floating-point error for the continuous mappings; the public integer -conversions round outward and stay within one pixel. A downscale discards detail that cannot -be recovered by scaling back up. - -## How the canvas uses this - -The Vision Lab canvas draws the **oriented, uncropped** frame. Crop belongs to processing and -inspection, not to the annotation viewport. - -Because slot rectangles live in source-normalized coordinates while the canvas shows oriented -pixels, the canvas builds the identity transform - -```kotlin -FrameTransform.create( - sourceWidth = frame.sourceWidth, - sourceHeight = frame.sourceHeight, - rotation = frame.rotation, - crop = null, - processedWidth = orientedWidth, - processedHeight = orientedHeight, -) -``` - -and maps every pointer position and every rectangle through it. The overlay therefore stays -aligned with the preview for portrait, landscape and square frames, and for any -`rotation` — no screen-dp guessing is involved. - -The preview box itself fits the frame with `ContentScale.Fit`; the fitted rectangle is -computed from the canvas size and the oriented aspect ratio, and all drawing and hit testing -use that rectangle. - -## How the bitmap and the overlay share one rectangle - -The captured bitmap's pixel size is **not** a canvas size. `OrientedBitmapLayout` computes -where the bitmap is actually drawn: - -- The oriented dimensions are `sourceWidth × sourceHeight`, or the swapped pair for 90° and - 270°. -- One uniform scale maps the oriented size into the fitted rectangle, so the image is never - stretched. -- The bitmap is drawn centred on `fitted.center` at that scale, and the rotation is applied - around the same centre. -- The axis-aligned bounds of the rotated bitmap are exactly the fitted rectangle. - -Because the overlay is mapped into the same fitted rectangle, the annotation of the whole -frame (`0,0 → 1,1`) lands exactly on the rendered bitmap bounds, for every rotation. The -same renderer draws the preview card and the annotation canvas. - -The workspace itself is sized from the oriented aspect ratio, so a portrait capture gets a -tall workspace and a landscape capture keeps a usable minimum height. The image is never -cropped and never stretched; a workspace taller than the image's aspect simply leaves -letterbox bands, and a touch on a band is not on the image. +The crop stored in `FrameTransform` is normalized in **oriented** coordinates; a null crop +means the full oriented frame. The output aspect ratio must match the integer crop rectangle +within one pixel of width or height. + +`FrameTransform` records the source size, correction, crop, oriented size, integer crop +rectangle, scale and output size. Its continuous mapping chain is +`source → oriented → crop → processed` and is reversible up to floating-point error; the +integer conversions round outward and stay within one pixel. Downscaling discards detail, so +scaling back up cannot recover it. + +## Canvas and preview + +The Vision Lab shows the **oriented, uncropped** frame. Cropping belongs to processing and +inspection, not to the annotation viewport. The canvas builds an identity-crop transform +from the frame's oriented size and maps both pointer positions and slot rectangles through +it, so overlays stay aligned in portrait, landscape and square without screen-dp guessing. + +`OrientedBitmapLayout` computes where the bitmap is really drawn: one uniform scale maps the +oriented size into the fitted rectangle, the bitmap is centred there, and rotation is applied +around the same centre. The rotated bitmap's axis-aligned bounds equal the fitted rectangle, +so a slot covering `0,0 → 1,1` lands exactly on the rendered image. The workspace height +comes from the oriented aspect ratio with a usable minimum; touches on letterbox bands are +not touches on the image. diff --git a/docs/VISION_FRAME_CONTRACT.md b/docs/VISION_FRAME_CONTRACT.md index 4829bc6..2d875fa 100644 --- a/docs/VISION_FRAME_CONTRACT.md +++ b/docs/VISION_FRAME_CONTRACT.md @@ -1,12 +1,14 @@ -# Vision Frame Contract +# Vision frame contract + +The contract in `com.draftlens.vision.model` is the only way Vision receives pixels. Capture +and Vision meet at `VisionFrameSource`, which now also carries the freeze commands. ## Payload -Every frame boundary carries a non-null payload. The nullable `CapturedFrame.bitmap` of -Sprint 0 is gone. +Every frame boundary carries a non-null payload. ```kotlin -sealed interface FramePayload { +interface FramePayload { val width: Int val height: Int val format: PixelFormat @@ -14,111 +16,70 @@ sealed interface FramePayload { } ``` -Two implementations exist: - -- `PixelArrayFramePayload(width, height, pixels)` — pure Kotlin, for JVM tests and synthetic - fixtures. It copies the input array in `init` and returns a copy from `pixels()`, so the - caller cannot mutate a frame after the fact. `byteCountEstimate` assumes 4 bytes per pixel - (ARGB_8888). It rejects empty or mismatched input. -- `BitmapFramePayload(bitmap, ownership = SHARED_READ_ONLY)` — the production path. It reads - width, height and `allocationByteCount` straight from the bitmap and reports zero once the - bitmap has been recycled, so a recycled bitmap never looks like a valid frame. +- `PixelArrayFramePayload` is pure Kotlin for JVM tests and synthetic fixtures. It copies the + input array on construction and returns a copy from `pixels()`, so a frame cannot be mutated + after the fact. It rejects empty or mismatched input. +- `BitmapFramePayload` is the production path. It reads size and allocation from the bitmap + and reports zero once the bitmap has been recycled, so a recycled bitmap never looks like a + valid frame. ## Frames -```kotlin -interface VisionFrame - -data class OriginalFrame( - val id: FrameId, - val payload: FramePayload, - val sourceWidth: Int, - val sourceHeight: Int, - val displayRotation: FrameRotation, - val rotation: FrameRotation, - val capturedAtElapsedRealtimeMs: Long, - val wallClockTimestampMs: Long, - val captureLatencyMs: Long?, -) - -data class ProcessedFrame( - val id: FrameId, - val sourceId: FrameId, - val payload: FramePayload, - val sourceWidth: Int, - val sourceHeight: Int, - val displayRotation: FrameRotation, - val rotation: FrameRotation, - val capturedAtElapsedRealtimeMs: Long, - val wallClockTimestampMs: Long, - val transform: FrameTransform, - val processingDurationMs: Long, -) - -data class SlotImage( - val id: FrameId, - val sourceId: FrameId, - val slotId: String, - val payload: FramePayload, - val sourceRect: PixelRect, - val createdAtElapsedRealtimeMs: Long, -) -``` - -`SlotImage` is part of the contract but has no UI or crop-export flow in this sprint. - -## Two rotations, two meanings - -- `displayRotation` is the device display rotation recorded at acquisition. It is - **information**, not a transform. -- `rotation` is the pixel-orientation correction needed to make the pixels upright. +- `OriginalFrame` — captured pixels plus id, source size, `displayRotation`, pixel-correction + `rotation`, monotonic and wall-clock timestamps and acquisition latency. +- `ProcessedFrame` — an original plus a `FrameTransform`, the processed payload and the + processing duration. It keeps the source id so the result can be traced back. +- `SlotImage` — one extracted slot crop, with its own id, source id, slot id, payload and + `sourceRect`. It is the input to identification and is produced by `SlotImageExtractor`. -MediaProjection delivers buffers that are already display-oriented, so production frames -carry `rotation = ROTATION_0` and only record `displayRotation`. Rotating an upright capture -is never implicit. Synthetic fixtures and any future authorized import may set a non-zero -`rotation`, and the geometry and preprocessing code apply that correction normally. +## Two rotations -## Capture integration +`displayRotation` records the device rotation at acquisition; it is information only. +`rotation` is the correction applied to make pixels upright. MediaProjection buffers are +already display-oriented, so production frames carry `rotation = ROTATION_0`. Non-zero +corrections are used by synthetic fixtures and imports that genuinely need straightening. -`CapturedFrame` is the capture-layer model. It embeds an `OriginalFrame`, forwards the -acquisition metadata cheaply (`width`, `height`, `displayRotation`, `wallClockTimestampMs`, -`sampledAtElapsedRealtimeMs`) and exposes `toVisionFrame(): OriginalFrame`. - -`FramePipeline.start(width, height, densityDpi, displayRotation)` passes the rotation into -every frame it produces. `CaptureSessionService` reads the rotation of -`DisplayManager.DEFAULT_DISPLAY` at capture start and whenever the display changes, and maps -`Surface.ROTATION_*` to `FrameRotation`; if the display is unavailable it falls back to -`ROTATION_0` and logs. - -The rotation is tracked **independently of the capture size**. A device can rotate while -the captured content keeps its dimensions, so: - -- `CaptureSessionService` watches the default display with a `DisplayManager.DisplayListener` - and updates the rotation on every display change, whether or not MediaProjection reports a - resize. The listener is unregistered during session cleanup. -- `FramePipeline` records the current rotation in a field that the image listener reads when - a frame is emitted, so a frame never carries a rotation that went stale at start-up. -- A change in capture size still replaces the ImageReader and the VirtualDisplay; a change - in rotation alone does not. +The service reads the default display rotation at session start and on display changes, +maps `Surface.ROTATION_*` to `FrameRotation`, and falls back to `ROTATION_0` with a log entry +if the display is unavailable. Rotation is tracked independently of capture size: a rotation +with unchanged dimensions updates metadata only, while a size change replaces the ImageReader +and VirtualDisplay. ## Timing -Two clocks, kept apart on purpose: +Two clocks are kept apart: + +- `capturedAtElapsedRealtimeMs` — monotonic elapsed realtime, used for sampling and latency. +- `wallClockTimestampMs` — wall clock, used for display and profile timestamps. -- `capturedAtElapsedRealtimeMs` — monotonic `SystemClock.elapsedRealtime()`. Used for - sampling intervals and latency arithmetic, immune to wall-clock changes. -- `wallClockTimestampMs` — `System.currentTimeMillis()`. Used only for display and for - calibration profile timestamps. +`captureLatencyMs` and `processingDurationMs` describe acquisition and preprocessing, **not** +identification. There is no calibrated confidence value anywhere in the contract. -`captureLatencyMs` and `processingLatencyMs` are acquisition-to-payload costs. They are not -detection latency, because there is no detection. +## Freeze in the seam -## Ownership rules +```kotlin +interface VisionFrameSource { + val frames: StateFlow + val state: StateFlow + val frozenFrame: StateFlow + fun freeze(): Boolean + fun resume() +} +``` -1. The capture pipeline owns the bitmap it produced. -2. `CaptureStateHolder` keeps only the latest frame; older frames become unreferenced. -3. No code path calls `recycle()`. References are dropped and native memory is reclaimed by - the garbage collector. -4. `SHARED_READ_ONLY` consumers may inspect and draw a bitmap but never mutate or recycle it. -5. The Vision Lab pins one extra reference while a frame is frozen, and *Clear* releases it. -6. Frames are never written to disk and never uploaded. There is no network client in the app. +`frozenFrame` is an independently owned snapshot published by capture. `freeze()` pins the +newest valid frame and stops publication of live frames; it returns false when there is no +frame or no session. `resume()` releases the pin and restores live publication. Neither call +creates a second projection or display. + +## Ownership + +- The pipeline owns the bitmap it created and keeps only the latest frame; older references + become garbage. +- The Vision Lab keeps one extra reference while a frame is pinned; Clear releases it. +- Frame payloads and slot crops are shared read-only. Consumers may inspect and draw them but + must not mutate them. +- Short-lived temporaries created during slot extraction and reference-image validation are + recycled exactly once by their owners after their pixels have been copied. +- Captured pixels and slot crops are never written to disk and never uploaded. The app's only + network client downloads public static hero catalog data and reference images. diff --git a/docs/VISION_LAB.md b/docs/VISION_LAB.md index 021d933..1c94317 100644 --- a/docs/VISION_LAB.md +++ b/docs/VISION_LAB.md @@ -1,175 +1,100 @@ # Vision Lab -The Vision Lab is the second tab of the app. The first tab, Debug Capture, stays one tap away -and capture keeps running while the Vision Lab is open — the projection lives in a foreground -service, not in a screen. +The Vision Lab is the second tab. Capture lives in the foreground service, so switching tabs +never stops it. The page has six cards: frame preview, annotation canvas, manual slots, +hero recognition, processing inspector and calibration profiles. -## Frame Preview +## Frame preview -Shows the frozen frame when one is pinned, otherwise the latest live frame. +Shows the pinned frame when one is frozen, otherwise the latest live frame, and reports source +size, display rotation, correction and source state. Controls: -- Original width × height -- Display rotation and the pixel-orientation correction -- Source state (`CAPTURING`, `PAUSED`, `IDLE`, `STOPPED`, `ERROR`, `UNKNOWN`) -- A `LIVE` / `FROZEN` badge -- *Freeze* / *Resume* — *Freeze* locks the current frame before annotation; *Resume* returns to - live frames. Freezing is also possible from the capture notification (**Freeze current - frame**) while you are in another app; the pinned frame is independently owned and newer - live frames never replace it. -- *Process Frame* — processes the pinned frame if there is one, otherwise the latest live - frame at the moment of the press. It resizes and prepares the frame only; no recognition - exists yet. -- *Clear* — releases the pinned reference and the processed frame, and cancels any - processing that is still in flight so a late result cannot come back. +- **Freeze / Resume** — pin the current frame for inspection or return to live frames. Freezing + is also available from the capture notification and from the floating control while you are + in another app; the pinned frame is an independent snapshot that live frames never replace. +- **Process Frame** — run preprocessing on the pinned frame, or on the latest live frame at the + moment of the press. This only prepares pixels and geometry; identification is a separate + action in the hero recognition card. +- **Clear** — release the pinned frame and processed result and cancel any processing in flight. -Nothing is written to disk. Frames live in memory only. +Frames stay in memory; nothing is written to disk from this screen. -## Annotation Canvas +## Annotation canvas -The canvas draws the oriented, uncropped frame inside a dark card. The workspace is sized -from the frame's own aspect ratio, so a portrait capture gets a tall workspace and a landscape -capture keeps a usable minimum height. The image is never cropped and never stretched. +The oriented, uncropped frame is drawn inside a dark card. The workspace is sized from the +frame's aspect ratio with a usable minimum, so the image is never cropped or stretched. -### The interaction model - -The rules live in one explicit state machine (`AnnotationInteraction`) so two gestures can -never compete: +The gesture rules live in one state machine (`AnnotationInteraction`): | Interaction | Action | |---|---| | Short tap on a rectangle | Select that slot | | Drag inside a selected rectangle | Move the selected slot | -| Long press on a rectangle | Open its contextual property editor | +| Long press on a rectangle | Open its contextual editor | | Drag a resize handle | Resize the selected slot | -| Long press on empty image space, then drag | Draw a new slot | -| Short tap on empty image space | Nothing | +| Long press on empty image, then drag | Draw a new slot | +| Short tap on empty image | Nothing | | Tap the X control | Delete the selected slot | -Rules that follow from it: - -- Only one slot is selected at a time, and the selection is the slot's stable id. -- A tap never moves anything. A drag on an **unselected** rectangle selects it and stops - there, so a drag can never move the wrong slot. -- A stationary long press is never read as a move, and it opens the property editor - immediately when the timeout is reached. -- Moving past the touch slop **before** the timeout cancels the pending long press: a drag - on an unselected rectangle only selects it, a drag on a selected rectangle moves it, and a - drag on empty space never turns into slot creation. Once the threshold is reached, - movement can neither delay nor cancel the already-open menu. -- A drag on empty space never creates a rectangle: creating requires the long press first. -- A rectangle is never created outside the displayed image. -- The canvas takes the gesture from the first touch, so **the page never scrolls while a - slot is being moved, resized or drawn**. The rest of the page scrolls normally. - -### The selection - -The selected rectangle gets a stronger outline, a translucent fill and three resize handles: -top-left, bottom-left and bottom-right. - -The **top-right corner carries exactly one control: the X that deletes the slot.** It sits -on the corner where the top-right resize handle used to be — there is no handle behind or -beside it, and nothing floats above the rectangle. The X is drawn in red, so it can never be -mistaken for a handle, and its touch target is larger than the drawn circle. - -The missing top-right handle does not limit resizing: the top-left handle changes the width -*and* the height, the bottom-right handle does the same from the other side, and the -bottom-left handle changes the width alone. - -### Colours and labels - -| Team | Colour | -|---|---| -| `ALLY` | blue | -| `ENEMY` | red | -| `GLOBAL` | neutral grey | - -Colour is never the only cue: every rectangle carries a readable label such as `A-B1`, -`E-P2` or `G-B1`, followed by the user's own label when there is one. - -## The contextual property editor - -A long press on a rectangle opens a small editor next to it. The menu appears **as soon as -the long-press timeout is reached** — the finger does not have to move at all, and a small -movement afterwards does not delay it either. It never occupies a permanent part of the -canvas, and it positions itself inside the window, so it is never clipped off-screen in -either orientation. +Consequences worth knowing: a tap never moves anything; a drag on an unselected rectangle only +selects it; moving past the touch slop before the timeout cancels a pending long press; and a +drag on empty space never creates a slot without the long press first. The canvas takes the +gesture from the first touch, so the page never scrolls during an annotation gesture. -The menu and the selection always refer to the same slot: opening the menu selects that -slot first, so the outline, the handles, the delete control and the list highlight all -agree. +The selected rectangle gets three resize handles (top-left, bottom-left, bottom-right) and a +red X at the top-right corner for deletion. The missing top-right handle does not limit +resizing: the other corners change both dimensions and the bottom-left changes the width. -Validation failures are never silent. A type, team or index change that would produce a -duplicate logical identity is refused with the reason shown next to the controls, the slot -keeps its previous values, and the editor stays open. Invalid index text is flagged inline -in the index field. +Team colours are blue for `ALLY`, red for `ENEMY` and grey for `GLOBAL`, always accompanied by +a label such as `A-B1` or `E-P2`. -It edits: +The **fullscreen editor** fits the image to the screen without zoom or pan, keeps overlays +aligned, and uses Android Back to leave fullscreen before leaving the tab. Orientation is +never forced. -- Slot type: `BAN` / `PICK`. -- Team: `ALLY` / `ENEMY` / `GLOBAL`. -- Index. -- Optional label. +## Contextual editor and clipboard -And it offers three actions: +A long press opens a compact editor for the selected slot: type (`BAN`/`PICK`), team +(`ALLY`/`ENEMY`/`GLOBAL`), index and optional label. Invalid changes are refused with the +reason shown inline and the previous values kept. -- **Copy** — puts the slot into the application's own clipboard. This is deliberately *not* - the Android system clipboard: a slot is calibration data that belongs to this app. -- **Paste** — creates a copy of the **clipboard** slot next to the slot the menu was opened - on (or next to the copied slot itself when Paste is invoked without a target). The - clipboard slot is always the source: type, team, label and rectangle dimensions are copied - from it. The target slot only anchors the placement. The copy gets a **new unique id**, is - offset a little from the anchor, stays inside the image and takes the next free index in - its `(type, team)` scope so it never collides with an existing logical slot. The index can - be changed afterwards. -- **Duplicate** — the same thing for the slot the editor belongs to. +- **Copy** puts the slot in the app's own clipboard, not the system clipboard. +- **Paste** creates a copy of the clipboard slot next to the target slot, which only anchors + placement. The copy gets a new id, the next free index and a small offset. +- **Duplicate** does the same for the slot the editor belongs to. -The original slot is always preserved by a copy, a paste or a duplicate. +The original is always preserved. -## Manual slots +## Manual slots and profiles -The list under the canvas is the text view of the same annotation set. +The list under the canvas is the text view of the same annotation set, grouped by team and +type and keyed by slot id. Tapping a row selects the rectangle and vice versa. **Undo** and +**Redo** cover creation, deletion and geometry changes; a deleted slot is restored with its own +id. There is no fixed number of slots. -Slots are grouped into **ALLY · BAN**, **ENEMY · BAN**, **ALLY · PICK**, **ENEMY · PICK** and -**GLOBAL**. Every row shows the type, team, index, the user's label and the normalized -rectangle, and every row is keyed by the slot's own id — the list never relies on positions. +Calibration profiles save normalized rectangles and metadata for matching layouts. Saving, +updating and applying require a pinned frame, and applying first checks aspect ratio and +orientation. Corrupt or unsupported documents are never silently emptied; the only recovery is +**clear all calibration data**. -- Tapping a row selects that rectangle on the canvas. -- Selecting a rectangle on the canvas highlights its row. -- *Select* and *Delete* act on that exact slot. -- *Edit geometry* reveals the `left` / `top` / `right` / `bottom` fields for typing exact - coordinates. +## Hero recognition -*Undo* and *Redo* sit above the list. Undo covers a deletion, a creation and a move or -resize; it restores the correct slot id and metadata. A deletion is undone by putting the -slot back with its own id, never with a new one. +This card contains the identification flow added in Sprint 1B. -There is no fixed number of bans or picks: the user creates as many slots as the layout -needs. The index is always configurable and a duplicate `(type, team, index)` is refused. +1. **Sync catalog** downloads public static hero data and reference images on explicit request. + The catalog is cached privately for offline browsing. +2. Select a slot on the canvas, then tap **Identify hero**. The slot crop is extracted upright + for the current rotation and rejected if it has too little visual information. +3. Candidates are shown as **similarity, not a probability**. The matcher is an uncalibrated + baseline; a low score is reported honestly as no match. +4. **Confirm** stores the chosen hero for that slot. **Change…** opens manual correction by + name, available from candidates, no-match, insufficient information, failed and confirmed + states. **Clear** removes a confirmation. -## Fullscreen annotation editor +Manual confirmation is always required; the app never selects a hero by itself. -*Fullscreen* opens the image on its own screen, in whichever orientation the device is in. -The button is a compact icon in the corner of the canvas card. - -- The image and the canvas fill the available phone screen, using the maximum width and - height, with the aspect ratio preserved. -- The slot overlays stay perfectly aligned with the image. -- There is no zoom and no pan: the frame is fitted into the screen, and nothing about the - view is adjusted while editing. -- Only the essential controls are visible: a compact collapse icon in the corner and the - dismissible gesture hint. -- The collapse icon returns to the embedded editor and the previous screen arrangement. -- Android Back leaves the fullscreen editor first, and only then the Vision Lab. -- The orientation is never forced. - -## What the controls mean - -| Control | What it does | -|---|---| -| **Freeze** | locks the current frame before annotation (the capture notification carries **Pause**/**Resume** for the same job while in another app; an independently pinned snapshot comes from the floating button or this button) | -| **Resume** | returns to live frames (also as **Resume live capture** in the notification) | -| **Slot** | a manually marked region where a ban/pick hero portrait is expected | -| **Process Frame** | resizes and prepares the frame only; no recognition exists yet | -| **Calibration profile** | saves and restores normalized slot coordinates for matching screen layouts | +## Processing inspector -The screen keeps stating **"Detection: not implemented"**. +Shows source and output dimensions, rotation, crop, scale, duration, status and errors, plus a +maximum processed dimension setting. Acquisition latency and processing duration are distinct +from identification time. diff --git a/docs/VISION_TESTING.md b/docs/VISION_TESTING.md index a8af604..90d20f0 100644 --- a/docs/VISION_TESTING.md +++ b/docs/VISION_TESTING.md @@ -1,177 +1,58 @@ -# Vision Testing - -## Where the tests run - -GitHub Actions is the build of record for this sprint. The authoring host is aarch64 and -AGP 8.6.1's `aapt2` is x86_64-only, so nothing that touches resources can be assembled -there. The current, authoritative verification record for this branch is the -**Sprint 1B review-fix pass** section below; the older Sprint 1A/1B run -records further down remain as history. - -See the sections below for the exact commands CI runs. - -## Test inventory - -| Suite | File | Covers | -|---|---|---| -| Vision model | `vision/model/VisionFrameTest.kt` | payload copying, byte estimates, empty/mismatched input, orientation categories, invalid rectangles | -| Geometry | `vision/geometry/GeometryTest.kt` | all four rotations, portrait/landscape/square, source↔normalized↔source, source↔processed↔source, crop-aware and downscaled mappings, invalid dimensions, NaN/infinity | -| Preprocessing math | `preprocessing/PreprocessingMathTest.kt` | validation, aspect-preserving output sizing, long-edge and pixel budgets, no upscale | -| Preprocessing render | `preprocessing/PixelArrayFramePreprocessorTest.kt` | rotation, crop and downscale of synthetic frames, identity path, failure reporting | -| Matrix | `preprocessing/SourceToProcessedMatrixTest.kt` | the Android matrix matches the pure-Kotlin elements | -| Annotation store | `annotation/SlotAnnotationStoreTest.kt` | add/move/resize/update/delete/select, stable ids, duplicate `(type, team, index)`, overlap severity, bounds clamping | -| Annotation codec | `annotation/SlotAnnotationCodecTest.kt` | JSON round trip, schema versions, unknown fields, invalid rectangles | -| Calibration | `calibration/CalibrationCodecCompatibilityTest.kt` | ratio tolerance, exact orientation match, unsupported envelope/profile/slot schema, malformed JSON, blank id, round trip, unusable slots, duplicate slot ids, duplicate `(type, team, index)`, duplicate profile ids | -| Calibration store | `datastore/CalibrationProfileDataStoreTest.kt` | empty store, upsert, replace, delete, unsupported document refused and left untouched, corrupt document refused and not reset, a document with an unusable slot is corrupt and not replaced, clear-as-escape-hatch | -| Vision source | `capture/CaptureVisionFrameSourceTest.kt` | capture-status mapping, later state changes, latest-frame publication | -| Display changes | `capture/DisplayChangePolicyTest.kt` | a rotation with unchanged dimensions needs only a rotation update; a changed size needs a resize; the pipeline records and updates its rotation | -| Canvas geometry | `feature/visionlab/CanvasGeometryTest.kt` | points inside the image map to normalized source coordinates; all four letterbox boundaries are rejected; the image edges themselves are accepted | -| Canvas rendering | `feature/visionlab/OrientedBitmapLayoutTest.kt` | the rendered bitmap bounds match the fitted rectangle for all four rotations; the scale is uniform so the image is never stretched; portrait and landscape frames map their image edges onto the workspace edges; the overlay of the whole frame aligns with the rendered bitmap bounds; the bitmap is drawn around the fitted centre | -| Canvas gestures | `feature/visionlab/AnnotationGeometryTest.kt` | a created rectangle spans the two drag points in either direction, is clamped to the unit square and is discarded below the minimum size; a resize moves only the dragged corner and is refused when it would collapse the rectangle | -| Canvas workspace | `feature/visionlab/AnnotationGeometryTest.kt` | a portrait frame gets a tall workspace; a landscape frame keeps the minimum workspace height; an unknown ratio falls back to the minimum | -| Annotation interaction | `feature/visionlab/AnnotationInteractionTest.kt` | a tap selects a slot; a tap on empty space does nothing; a drag on the selected slot moves it; a drag on an unselected slot selects it and never moves it; a movement below the touch slop never starts a move; a long press on a slot opens its menu and is not a move; a long press on empty space starts drawing; drawing produces a rectangle on release and nothing without one; the delete control deletes that slot; a handle drag resizes and never moves; a press outside the image does nothing | -| Annotation targets | `feature/visionlab/AnnotationInteractionTest.kt` | a point on a slot resolves to that slot; empty image space resolves to empty; a point outside the image resolves to outside; the delete control wins over the slot itself; the delete control sits exactly where the top-right handle was; the top-right corner has no resize handle; the remaining handles are still there; a resize handle resolves to that handle | -| Long press | `feature/visionlab/AnnotationInteractionTest.kt` | a long press opens the menu without any movement; a slight movement after the long press neither delays nor repeats the menu; a normal tap never opens the menu; moving a selected slot never opens the menu; an early drag past the slop cancels the pending long press (unselected drag selects, selected drag moves, empty-space drag never draws); the deadline stays armed within the slop; after the timeout the deadline is abandoned and movement can neither delay nor cancel the menu; creating requires hold, threshold, drag | -| Slot placement | `feature/visionlab/SlotPlacementTest.kt` | an offset slot stays inside the image and keeps its size; an offset slot is moved away from its source | -| Slot restore | `annotation/SlotAnnotationStoreTest.kt` | a deleted slot is restored with its own id and metadata; restoring a slot whose index or id is taken is refused | -| Canvas hit testing | `feature/visionlab/AnnotationHitTestTest.kt` | the topmost slot wins where rectangles overlap | -| Vision Lab state | `feature/visionlab/VisionLabViewModelTest.kt` | live frame exposure, freeze/resume/clear, clear cancels in-flight processing, processing success and failure, stale-result discard, profile save/update/apply require a frozen frame, profile save refusal, delete-all escape hatch, apply compatibility, selecting a slot by id, deleting an arbitrary slot, undo of a deletion and of a creation, copy/paste with a new id and the next free index, duplicate keeping the source, the pasted slot staying inside the image, paste using the copied slot as the source with the target only anchoring the placement, paste without a target offsetting from the copied slot, the menu opening for an existing slot, opening the menu on an unselected slot selecting it, an undoable move | -| Capture regression | `capture/CaptureStateHolderTest.kt`, `capture/FrameSamplerTest.kt`, `capture/CaptureStartupExecutorTest.kt` | the Sprint 0 capture lifecycle, sampling and startup behaviour still hold | -| Models | `models/CaptureSessionStateTest.kt`, `permissions/ProjectionPermissionTest.kt` | state machine transitions, consent intent | -| Detection stubs | `detection/DetectionStubsTest.kt` | the Sprint 0 seams still report `NotImplemented` and never fake a result | - -| Plane/stride conversion | `capture/RgbaPlaneCopyTest.kt` | landscape 2400x1080 & portrait 1080x2400 exact pixels, non-square frames, padded rows (full-pixel pads and one-pixel pads), offset & edge-touching `cropRect`s, `cropRect` smaller than the frame (letterbox), pixelStride 4/8, big-endian buffers, caller-buffer position independence, malformed inputs refused, full-crop equals direct read | -| Generation counter | `capture/GenerationCounterTest.kt` | generations monotonic, stale after bump, frozen resume does not advance it | -| Display policy | `capture/DisplayChangePolicyTest.kt` | portrait↔landscape always a size-swap resize, identical resize callbacks idempotent | -| Instrumentation | `androidTest/.../FramePipelineInstrumentedTest.kt` | pipeline starts/resizes with new surface and generation bump, freeze with no frames keeps the reader alive, stop releases the surface | - -## Fixture policy - -- **Synthetic images only.** No real game screenshots, no captured frames, no image assets are - committed. Fixtures use `PixelArrayFramePayload` with generated pixel values. -- The shared fixture is `com.draftlens.vision.testCapturedFrame()`. -- The calibration store test writes to a temporary directory under the system temp folder and - removes it afterwards. - -## What is not covered - -- **Physical-device smoke: maintainer-reported PASS for the Vision Lab interaction behavior.** - Slot selection, movement and resizing; the long-press property menu; long-press-and-drag - creation; the original compact red X deletion control; copy, paste and duplicate; fullscreen - entry/exit and Android Back; annotation gestures isolated from page scrolling; and existing - portrait/landscape rendering were confirmed working on a physical device. Device model, - Android version, install source/APK artifact, test date, logs and recordings were **not** - supplied, so they are not recorded here. -- **MediaProjection lifecycle matrix remains NOT RUN.** Repeated start/stop, projection - revocation, app-process interruption, rotation/resize, background stability and sustained - capture scenarios in `docs/TESTING_PHYSICAL_DEVICE.md` are untested. -- **Instrumentation tests are compiled, not executed.** `assembleAndroidTest` proves the - instrumentation sources compile; it does not run them. -- **No performance claim.** The JVM suite measures nothing about real hardware. Processing - durations are recorded per run in the UI; no 24/7 stability or memory bound is claimed. -- **No recognizer**, so there are no detection-accuracy tests to write. - -## Honest failure behaviour - -The suites assert that failures are reported rather than hidden: invalid dimensions, payload -mismatches, oversized output, unsupported formats and render errors all produce a -`ProcessFrameResult.Failure` with a reason code, and no partial frame is published. The -calibration suites assert that an unsupported or corrupt document is reported and left -untouched, and that only clearing the document recovers from it. - -## Sprint 1B review-fix pass (this revision) - -GitHub Actions is the build of record. The authoring host is **aarch64**, and AGP 8.6.1 -publishes only an **x86_64** `aapt2`, so nothing that touches resources can be assembled -locally; every verification below ran in CI on `ubuntu-latest` (x86_64) with JDK 17. - -Workflow: `.github/workflows/build.yml` (one `build` job), triggered by `pull_request` -(and pushes to `main`/`master`). - -Docs-only commits after the code freeze do not change the build; the exact final head and its CI run are recorded in PR #3's description. - -Latest green run: -— head SHA `5252016cd15c824009368e831ed499f24209980d` (docs-only commits after the code freeze -are not part of the 840-tests green state; PR #3's description records the final head). - -- **Unit tests: 840 executions (debug + release), 0 failures, 0 errors, 0 skipped.** -- `assembleAndroidTest` — instrumentation sources compile. -- strict `lint` (`abortOnError = true`) — green. -- `assembleDebug` — green; artifact `draftlens-debug`, 10 369 786 bytes, - digest `sha256:c46580570a85054f225bd4d927350b153991f98956b1e3ee9a877d5ed9c1b3f1`. - -The instrumentation suite is **compiled but not executed**. The CI runner has no KVM -access: the default API-30 image does not boot within the runner's timeout, and the -`aosp_atd` image boots but is reported to Gradle as an "Unknown API Level" and is skipped -(`Skipping device 'emulator-5554' ... No compatible devices connected`). Instrumented -tests must be run on a physical device or an emulator with KVM (see -`docs/TESTING_PHYSICAL_DEVICE.md`); they are not claimed as executed anywhere. - -The run also re-reads the packaged manifest to confirm the foreground-service permissions -and the `mediaProjection` service type survived the build. - -## Test inventory (Sprint 1B additions) - -| Suite | File | Covers | -|---|---|---| -| Pixel extraction | `heroes/SlotImageExtractorTest.kt` | asymmetric 3x2/5x3 fixtures, exact pixel order for all four rotations (hand-computed), edge-touching crops, minimum-size and refused crops, source immutability | -| Matcher | `heroes/HeroMatcherTest.kt` | identical images match, ranking, no-match threshold, ambiguous gap, corrupt references skipped, determinism | -| Recognition orchestration | `heroes/HeroRecognitionTest.kt` | candidates path, unusable references → failure, below-threshold → no-match (inverse-gradient fixture), uniform crop → `InsufficientVisualInformation` | -| Visual information | `heroes/VisualInformationTest.kt` | uniform gray/black/white, saturated solids, near-uniform noise, low-resolution crops, smooth & low-contrast gradients (not rejected), textured & pseudo-random detail (accepted), dimension mismatch, end-to-end blank → `InsufficientVisualInformation`, `NoMatch` kept distinct | -| Catalog sync | `heroes/HeroCatalogSyncTest.kt` | unchanged behaviour (pagination, mirror, duplicates, cross-check) | -| Catalog repository | `heroes/FileHeroCatalogRepositoryTest.kt` | snapshot + assets, offline restart, **unchanged URL → no re-download**, **URL change → re-download & new asset served**, **URL change + failed refresh → old asset not exposed**, **corrupt file → not served + re-downloaded**, metadata forgery rejected, missing/corrupt sidecar → re-download, oversized cached file rejected, truncated/HTML-as-PNG rejected, null content-type accepted, oversized download rejected, fresh assets served via network outage | -| Atomic writes | `heroes/WriteAtomicallyTest.kt` | exact-bytes write, failure leaves a directory target untouched and removes the temp file, blocked parent reports failure without touching it, hourly `*.tmp` sweep on init, no temp leftovers on a failed refresh | -| Redirect policy | `heroes/HeroRedirectPolicyTest.kt` | HTTPS targets accepted (absolute/relative), HTTPS→HTTP downgrade refused, non-HTTP schemes refused, missing/blank/over-long targets refused | -| Structural validator | `heroes/StructuralImageValidatorTest.kt` | real PNG passes, truncation/CRC corruption/HTML fail, GIF/WebP framing checks | -| Freeze flow (JVM) | `capture/CaptureFreezeTest.kt` | freeze without an active capture → `NoPipeline`; resume drops the pin; a duplicate freeze keeps the original frame | -| Freeze flow (JVM) | `capture/CaptureStateHolderTest.kt` | pinned frame survives `terminateSession` (stop and error), new session drops it, clear drops it, reset drops it | -| Freeze flow (JVM) | `models/FrameSnapshotsTest.kt` | copy preserves dimensions/rotation/timestamps/id, pixel values equal, caller-array mutation does not affect the copy, capture latency wrapper preserved | -| Vision source | `capture/CaptureVisionFrameSourceTest.kt` | adapter publishes capture-side frozen frames; freeze with no capture → false; resume is a safe no-op | -| ViewModel | `feature/visionlab/VisionLabViewModelTest.kt` | **capture-side freeze shows the pinned frame while newer live frames keep arriving**, resume reveals live frames again, freeze with no frame reports honestly, uniform-gray Frame produces `INSUFFICIENT_INFO` instead of candidates, inverse-gradient → `NO_MATCH`, confirmation/correct of candidates works, direct correction without clearing first, **confirm/clear persistence failure reported, never shown as success** | -| Instrumentation | `androidTest/.../AndroidBitmapExtractionInstrumentedTest.kt` | all four rotations pixel-exact (asymmetric non-square fixtures), edge crops, payload/source dimension mismatch refused, source Bitmap valid after extraction, minimum-size and refused crops, recycled Bitmap refused, bitmap snapshot copies pixels independently | -| Instrumentation | `androidTest/.../ReferenceImageValidatorInstrumentedTest.kt` | real PNG decodes on device, truncated PNG and HTML do not decode, JPEG structure-vs-decode layering | -| Instrumentation | `androidTest/.../CaptureLifecycleInstrumentedTest.kt` | manifest declares the service, `mediaProjection` foreground type, required permissions | - -## Fixture policy - -- **Synthetic images only.** No real game screenshots, no captured frames, no image assets are - committed. Fixtures are generated pixel arrays or the built-in 1x1 red/green PNGs - (`com.draftlens.vision.testRealPng()` / `testRealGreenPng()`), which are real, decodable - containers with correct CRCs. -- The calibration store test writes to a temporary directory under the system temp folder and - removes it afterwards. - -## What is not covered - -- **Instrumentation tests: compiled, not executed.** `assembleAndroidTest` proves the - instrumentation sources compile; nothing ran them. The CI runner has no KVM access: the - default API-30 image did not boot within the runner's timeout, and the `aosp_atd` image - boots but is skipped as "Unknown API Level" by Gradle. Two consecutive attempts failed - for these reasons, so the emulator job was removed rather than left flaky. Instrumented - tests must be executed on a physical device or a KVM-enabled emulator before the PR is - marked verified; none of their results are claimed as executed here. -- **Physical-device smoke of the freeze workflow: pending.** The manual lifecycle matrix - in `docs/TESTING_PHYSICAL_DEVICE.md` (rows 14–22) covers the notification freeze, the - FROZEN/Resume cycle, duplicate taps, disabled notifications and stop-with-frozen-frame. - None of it has been run on a device. -- **MediaProjection lifecycle matrix remains NOT RUN** (rows 2–11). -- **On-device recognition accuracy: not measured.** The no-match threshold and the matcher - weights are uncalibrated; the visual-information heuristics are conservative starting - points, not MLBB-calibrated thresholds. Real-screen evaluation and calibration remain - pending; see `docs/sprints/SPRINT_1B_HERO_RECOGNITION.md` § Known limitations. -- **Catalog entries actually loaded** are measured at device sync time; not recorded here. -- Compose screens are not UI-tested (no compose-ui-test dependency exists in this project); - the interaction model is covered by the interaction state-machine tests. -- The service-level notification actions (`ACTION_PAUSE` / `ACTION_RESUME_LIVE` delivery through - `onStartCommand`) is exercised only through the JVM-level `CaptureFreeze`/`CaptureStateHolder` - paths and manual device verification; no instrumented test taps the system notification. - -## Honest failure behaviour - -The suites assert that failures are reported rather than hidden: invalid dimensions, payload -mismatches, oversized output, unsupported formats and render errors produce a -`ProcessFrameResult.Failure` with a reason code; unsupported/corrupt calibration documents -are reported and untouched; a failed cache download never replaces a good asset and never -serves a stale one under a changed URL; a failed persistence operation is reported, not -hidden behind a success state. The instrumentation suite covers the Android-only paths -(bitmap extraction, decode validation, manifest). +# Testing + +## Where tests run + +GitHub Actions is the build of record. `.github/workflows/build.yml` runs `test`, +`assembleAndroidTest`, `lint`, `assembleDebug`, uploads the APK and inspects its manifest on +every push and pull request. `main` is protected by the build check. + +The authoring host is aarch64 while the SDK's `aapt2` is x86_64-only, so resource-touching +Gradle tasks cannot run locally here. At the PR #5 verification head `4eb9948`, CI reported +**918 unit tests, 0 failures, 0 errors, 0 skipped**, strict lint passing, and instrumentation +compiled. + +## JVM unit tests + +The 47 JVM test files cover the pure logic. Grouped by area: + +- **Geometry and frames** — `vision/geometry/GeometryTest`, `vision/model/FrameOrientationTest`, + `vision/model/VisionFrameTest`, plus `TestFrames` fixtures. Rotation round trips, normalized + and pixel conversions, frame contracts. +- **Preprocessing** — `PreprocessingMathTest`, `PixelArrayFramePreprocessorTest`, + `SourceToProcessedMatrixTest`. Validation failures, sizing and matrix agreement. +- **Annotation and calibration** — `SlotAnnotationStoreTest`, `SlotAnnotationCodecTest`, + `CalibrationCodecCompatibilityTest`, `CalibrationProfileDataStoreTest`. Validation, strict + codecs, profile compatibility and persistence. +- **Canvas and gestures** — `AnnotationGeometryTest`, `AnnotationHitTestTest`, + `AnnotationInteractionTest`, `CanvasGeometryTest`, `OrientedBitmapLayoutTest`, + `SlotPlacementTest`, `VisionLabViewModelTest`. Hit testing, gesture state machine, fitted + layout and ViewModel flows. +- **Capture logic** — `FrameSamplerTest`, `DisplayChangePolicyTest`, `CaptureStartupExecutorTest`, + `CaptureStateHolderTest`, `GenerationCounterTest`, `PendingResizeTest`, `RgbaPlaneCopyTest`, + `CaptureDiagnosticsTest`, `CaptureFreezeTest`, `CaptureVisibilityPolicyTest`, + `VisibilityFreezeSequenceTest`, `PublishStateTest`, `CaptureVisionFrameSourceTest`, + `CaptureSessionStateTest`, `FrameSnapshotsTest`, `ProjectionPermissionTest`. +- **Recognition** — `HeroCatalogCodecTest`, `HeroCatalogSyncTest`, `FileHeroCatalogRepositoryTest`, + `HeroRedirectPolicyTest`, `WriteAtomicallyTest`, `SlotImageExtractorTest`, + `StructuralImageValidatorTest`, `VisualInformationTest`, `HeroMatcherTest`, + `HeroRecognitionTest`, `HeroRecognitionStoreTest`. Cache safety, extraction for all + rotations, visual-information rejection, ranking and confirmation persistence. +- **Overlay** — `OverlayPositionTest`, `OverlayEnablementTest`. Position math, edge snapping and + permission decision. + +## Instrumented tests + +Five files compile in CI but are **not executed**: `CaptureLifecycleInstrumentedTest`, +`FramePipelineInstrumentedTest`, `AndroidBitmapExtractionInstrumentedTest`, +`ReferenceImageValidatorInstrumentedTest` and `FloatingControlOverlayInstrumentedTest`. The +hosted runner has no KVM; the default emulator image does not boot within the timeout and the +`aosp_atd` image is skipped as an unknown API level. Run these on a physical device or a +KVM-enabled emulator. + +## What CI cannot prove + +CI does not exercise MediaProjection consent, overlay visibility on real hardware, rotation and +resize storms, notification shade behaviour, or recognition accuracy on actual MLBB screens. +Those items are in [TESTING_PHYSICAL_DEVICE.md](TESTING_PHYSICAL_DEVICE.md) and remain honestly +NOT RUN. Transport tests use fake responses, so the number of catalog entries actually loaded +is known only at device sync time. diff --git a/docs/decisions/ADR-001-android-native.md b/docs/decisions/ADR-001-android-native.md index e21c4d1..69497e0 100644 --- a/docs/decisions/ADR-001-android-native.md +++ b/docs/decisions/ADR-001-android-native.md @@ -1,29 +1,39 @@ -# ADR-001 — Native Android (Kotlin + Compose), single module for Sprint 0 +# ADR-001 — Native Android, single module -**Status:** Accepted (Sprint 0). +**Status:** Accepted. Confirmed by Sprints 1A, 1B and 1C. ## Context -DraftLens needs continuous screen capture (MediaProjection), a foreground service, and later a -floating overlay — all Android-platform capabilities. The brief explicitly requires native. + +DraftLens needs continuous screen capture through MediaProjection, a typed foreground +service, a persistent notification, and a `TYPE_APPLICATION_OVERLAY` control. These are +Android platform APIs. It also needs image handling, local persistence and a small amount +of deterministic matching. The team is one maintainer shipping increments in public. ## Decision -- Native Kotlin + Jetpack Compose, MVVM, Coroutines/Flow. Not Flutter/KMP: MediaProjection and - foreground-service-type APIs are Android-only and need no cross-platform abstraction today. -- **Single Gradle module** (`:app`). The brief's logical packages exist as Kotlin packages, - not modules. Rationale: at Sprint 0 scale, module boundaries cost build time and config - surface without buying test/deploy independence. -- Version pins: Kotlin 2.0.20, AGP 8.5.2, Gradle 8.7, compile/target 35, min 26, JDK 17 — - the newest stable line that a stock `ubuntu-latest` GitHub runner supports with zero - toolchain workarounds. - -## Split trigger -Split into modules (`:core`, `:feature-*`, …) when the first feature needs to (a) be tested -independently of the capture service, (b) deploy/ship separately, or (c) be reused by a second -app. The `capture/`, `detection/`, `database/`, `datastore/` packages are already seam-shaped -to make that move mechanical. + +Build a native Android app in Kotlin with Jetpack Compose (Material 3), Coroutines and Flow, +and DataStore Preferences. Keep a single Gradle module, `:app`, with logical layers expressed +as Kotlin packages. + +The build pins JDK 17, Gradle 8.7, AGP 8.6.1, Kotlin 2.0.20, compileSdk and targetSdk 35, +minSdk 26. Versions live in `gradle/libs.versions.toml`. CI uses GitHub Actions on +ubuntu-latest and is the build of record because the authoring host is aarch64 while the +Android SDK's `aapt2` is x86_64-only. + +## Reasons + +- MediaProjection, foreground-service types and overlays have no useful cross-platform + abstraction here; Flutter or KMP would add a bridge without removing platform work. +- One module keeps the build and CI short while the feature scope is still one app. A second + module becomes justified when a feature needs an independent test or release cycle. +- Pure Kotlin packages for geometry, codecs, policy and matching keep the risky logic + unit-testable on the JVM without an emulator. ## Consequences -- `DataStore` is used for preferences now; `Room` is deferred until the first persisted entity - to avoid an empty database and dead code. -- All detection interfaces return explicit `NotImplemented`; adding a real model later means - writing *one* new implementation per interface, not restructuring. + +- Android lifecycle and permission behaviour must be handled directly, including consent, + notification permission and the overlay permission requested only when the user opts in. +- Verification splits in two: CI proves compilation, pure logic and lint; a physical device + is still required for the MediaProjection and overlay rows in the testing checklist. +- A future split into modules should start with the pure vision/recognition layer, since it + already has no Android dependencies at its core. diff --git a/docs/sprints/SPRINT_1B_HERO_RECOGNITION.md b/docs/sprints/SPRINT_1B_HERO_RECOGNITION.md index f61a5f1..bd55dca 100644 --- a/docs/sprints/SPRINT_1B_HERO_RECOGNITION.md +++ b/docs/sprints/SPRINT_1B_HERO_RECOGNITION.md @@ -1,279 +1,38 @@ -# Sprint 1B — Hero Recognition - -**Branch:** `sprint-1b-hero-recognition` · **PR:** #3 (draft, **not merged**) - -Sprint 1B builds the first real, on-device hero-identification pipeline: a hero catalog from -public data, slot-image extraction from the annotation canvas, and a deterministic matching -baseline with a manual confirmation flow. It is about identifying heroes, not about draft -recommendations. - -**What is explicitly not here:** no recommendation or synergy scoring, no counter ranking, no -floating overlay for MLBB itself, no automatic draft-phase detection, no OCR of arbitrary -pixels beyond the slot crop, no cloud recognition API, no ML model, no OpenCV. Those arrive -after recognition has been measured on real device samples. - -## Review-fix pass (this revision) - -This branch now also contains the fixes for the PR #3 review findings: - -| Part | Finding | Resolution | -|---|---|---| -| A | No way to freeze the game frame while inside MLBB | Capture-side freeze via an explicit `ACTION_FREEZE` / `ACTION_RESUME_LIVE` command routed to the running service; notification gains a **Freeze current frame** action; an independently owned snapshot is pinned in the process-local holder; Vision Lab shows the pinned frame; a **Resume live capture** notification action and an in-app **Resume** both exist | -| B | 90°/270° pixel rotation used incorrect indexing | `PixelSlotImageExtractor` rewritten to the exact mappings established by `FrameTransform.sourceToOriented()`; tests now use asymmetric non-square images and hand-computed expectations | -| C | Android bitmap extractor: unverified Bitmap/rotation contract | Payload dimensions validated against `OriginalFrame.sourceWidth/sourceHeight`; rotation rewritten to pixel-exact `sourceToOriented` mapping; source bitmap never mutated/recycled; temporaries recycled in every path; instrumented tests added | -| D | Blank gray crop produced three hero candidates at ~0.73 | New pure-Kotlin `VisualInformationAnalyzer` gate before matching; new distinct `InsufficientVisualInformation` outcome and `INSUFFICIENT_INFO` UI state; blank crops never produce candidates | -| E | Manual correction hidden in some states | Manual search reachable from CANDIDATES, NO_MATCH, FAILED, INSUFFICIENT_INFO and CONFIRMED; **Change…** now opens correction instead of re-running Identify; confirmation can be corrected without clearing first; failed clears report failure instead of showing success | -| F | Cache freshness: non-empty file was always reused | Per-asset `*.meta` sidecar binds the cached image to URL, SHA-256, size and content type; serve-time validation (size, hash, magic bytes, container structure, content type, plus a real decode on Android); changed URLs force re-download; failed refreshes keep the previous asset and never expose it under a changed URL; HTTPS-only redirect policy re-validates every hop | -| G | `renameTo` failure fell back to a direct write, destroying snapshots | `writeAtomically()` is now temp-sibling + flush/sync + `ATOMIC_MOVE` only; failure preserves the previous target and cleans the temp file; stale `*.tmp` siblings swept on init | -| H | `loadReference` recycled the bitmap, then read its width/height | Dimensions are read before recycling; pixel reads complete before release; exceptional paths recycle too | -| I | Matcher honesty | Matcher stays labelled a prototype ranking baseline; threshold/weights marked uncalibrated; evaluation section added below | -| J | Scope | Sprint 1A behavior preserved (see the preservation section) | -| K | Reporting | See "Evaluation and verification" and `docs/IMPLEMENTATION_REPORT.md` | - -## Capturing Mobile Legends: the freeze workflow - -> **Superseded by Sprint 1C** (`docs/sprints/SPRINT_1C_APP_ONLY_CAPTURE.md`): prefer -> **Single app / app window → Mobile Legends** when offered; the freeze is now primarily -> the circular floating button (rows 33–43 of `docs/TESTING_PHYSICAL_DEVICE.md`). The -> full-display guidance below remains valid as the fallback. - -1. Open DraftLens → **Debug Capture**. When Android asks what to capture, choose - **Entire screen / full device display**. App-window capture shows the DraftLens screen, not - the game, and cannot support this workflow. (The app says this on screen too.) -2. Open Mobile Legends and wait for the draft screen you want to annotate. -3. Pull down the notification shade (without leaving the game) and tap - **Freeze current frame** on the ongoing DraftLens capture notification. -4. DraftLens pins an independently owned copy of the latest validated game frame in memory. - The MediaProjection session, VirtualDisplay and ImageReader stay running; images are still - acquired and closed, but no conversion or publication work is performed while frozen. -5. Return to DraftLens. Vision Lab shows the pinned MLBB frame, not newer frames showing - DraftLens itself. Draw slot rectangles and run **Identify hero** on that frame. -6. Tap **Resume** (Vision Lab) or **Resume live capture** (notification) to see live frames - again. No second projection or VirtualDisplay is ever created for this. - -Properties of the implementation: - -- The notification action is an explicit component with an immutable `PendingIntent` - (`PendingIntent.getService` → `CaptureSessionService`). No Activity is launched. -- The action never re-promotes the service, never restarts MediaProjection, never requests - consent again and never creates a VirtualDisplay. With no active capture session it is a - no-op (a stale command re-creating the service stops itself immediately). -- Only one frozen frame is kept; a new capture session drops it, and resuming replaces live - publication. Freezing is idempotent (a duplicate tap keeps the first pinned frame). -- The frozen snapshot is independent of the live pipeline's payloads: pixels are copied, the - source is never mutated or recycled, and the frozen frame is displayed while newer live - frames keep arriving (they never overwrite it). -- A projection stopped by Android releases the VirtualDisplay, ImageReader and projection, - but preserves the pinned frame in memory for editing while the process lives. It does not - survive process death. -- **Notifications:** the ongoing foreground-service notification stays visible with an - **Stop** action and, depending on state, **Freeze current frame** (LIVE) or - **Resume live capture** (FROZEN). Actions appear in the expanded notification. - If the user disabled notifications (API 33+), the capture can still run but the actions are - not reachable from the shade; the app checks and shows this on the capture screen and in - the Vision Lab guidance. -- Nothing is written to disk or uploaded by the freeze or the capture path. - -## New modules - -| Module | File(s) | What it does | -|---|---|---| -| Hero catalog model | `vision/heroes/HeroModels.kt`, `HeroIds.kt` | Provider-independent `HeroId` (slug from the English name), entries, assets, attribution, cross-check result | -| Catalog codec | `vision/heroes/HeroCatalogCodec.kt` | Strict v1 snapshot codec; duplicate-id protection; HTTPS-only asset URLs | -| HTTP transport | `vision/heroes/HeroCatalogTransport.kt`, `AndroidHeroCatalogTransport.kt` | HTTPS-only `HttpURLConnection` behind a small interface; size cap enforced while reading; one retry on I/O/5xx; 429 fails fast; manual redirects re-validated per hop (HTTPS only); no API key | -| Redirect policy | `vision/heroes/HeroRedirectPolicy.kt` | Every redirect hop resolved and constrained to HTTPS; relative redirects resolved; HTTP downgrades and non-HTTP schemes refused; unit tested | -| Catalog sync | `vision/heroes/HeroCatalogSync.kt` | Paged `GET /heroes` on Rone Arena (plus mirror fallback), MLBBDex for completeness cross-check and roles/lanes by slug | -| Catalog repository | `vision/heroes/FileHeroCatalogRepository.kt`, `HeroCatalogRepository.kt` | Temp-sibling + flush/sync + `ATOMIC_MOVE` snapshot write (no direct-write fallback); private image cache (`/hero-assets/`) with per-asset `*.meta` sidecars binding URL, SHA-256, size and content type; serve-time revalidation; offline browsing after sync | -| Image validation | `vision/heroes/ReferenceImageValidator.kt`, `AndroidReferenceImageValidator.kt` | Two-layer validation: a pure-Kotlin container structure check (signature, chunks, CRCs, trailers) used on all platforms, plus an on-device `BitmapFactory` decode check wired in `DraftLensApp`; JVM tests exercise the structure check, instrumentation tests exercise both | -| Slot extraction | `vision/heroes/SlotImageExtractor.kt`, `AndroidBitmapSlotImageExtractor.kt` | Axis-aligned crop of the source rect, upright for every rotation via the `FrameTransform.sourceToOriented` mapping (pixel-exact on both paths); never mutates/recycles the source; temporaries released exactly once; nothing written to disk | -| Visual information | `vision/heroes/VisualInformation.kt` | Deterministic, Android-free analysis of a crop (luma/color variance, edge density, meaningful differences, minimum dimensions); rejects blank/near-uniform/detail-free crops before matching | -| Matcher v0 | `vision/heroes/HeroMatcher.kt`, `HeroRecognition.kt` | dHash perceptual hash + normalized 3×3 mean-RGB descriptor; combined similarity in [0, 1]; ranked top-3; explicit no-match below threshold; explicit insufficient-information outcome | -| Reference loader | `vision/heroes/HeroReferenceImages.kt` | Decodes cached reference images; dimensions read before recycling; pixel reads complete before release; downsampled ≤256 px for cheap matching | -| Frame snapshots | `models/FrameSnapshots.kt` | Independently owned copies of frames for the freeze workflow (pixel copy, metadata preserved verbatim) | -| Freeze bridge | `capture/CaptureFreeze.kt`, `capture/FramePipeline.kt`, `capture/CaptureStateHolder.kt` | The freeze command path: service/in-app triggers → `CaptureFreeze` → pipeline publication gate → `CaptureStateHolder.frozenFrame` → Vision layer | -| Recognition store | `datastore/HeroRecognitionStore.kt` | Per-slot confirmed hero (id, name, timestamp) in its own preferences document; corrupt documents read as empty | -| UI | `feature/visionlab/HeroRecognitionCard.kt` | Sync button, Identify action, candidates, INSUFFICIENT_INFO / NO_MATCH / FAILED / CONFIRMED states, manual search everywhere | -| Permissions | `AndroidManifest.xml` | `INTERNET` (catalog) and `POST_NOTIFICATIONS` (freeze action reachability) declared; capture still requires explicit consent; `mediaProjection` foreground-service type unchanged | - -## Data sources and attribution - -| Provider | What is used | License / notice | -|---|---|---| -| **Rone Arena** (`arena.rone.dev`, mirror `arena.fastapicloud.dev/api`) | `GET /api/heroes` (paginated, English) for entries, `hero_id`, head/smallmap portrait URLs | BSD-3-Clause (community-maintained, unofficial). Not affiliated with or endorsed by Moonton. Game names and visual assets remain the property of their respective rights holders | -| **MLBBDex** (`mlbbdex.com`) | `GET /api/v1/heroes` for the completeness cross-check and roles/lanes/specialties by slug (`roles`, `lanes`, `specialties`) | Community wiki data under CC BY-SA (https://mobilelegends.fandom.com); written content MIT. Game visuals remain the property of Shanghai Moonton Technology Co., Ltd. | - -- No written analyses are copied into the app. No counter or build statistics are shown. -- Provider-independent identity is a slug derived from the English name; display names are - never an identity. -- `externalIds` records `roneArenaHeroId` and `mlbbdexSlug` per entry. -- Snapshot attribution and notices are kept in the cached document and shown in the app. -- The unofficial/non-affiliation notice is shown in the recognition card. - -### Re-verified contracts (2026-11-15) - -Rone Arena (primary): -- Base: `https://arena.fastapicloud.dev/api` (primary) / `https://arena.rone.dev/api` (mirror). - No key. Public endpoints only. -- `GET /heroes?size&index&lang` — `data.records[].{data.hero.data.{head,name,smallmap}, hero_id, relation}`, - `data.total`. Records ordered by `hero_id`. -- Error handling: a 503 means maintenance/high traffic (`alternative_endpoint` fallback); - a 429 `UPSTREAM_RATE_LIMITED` means back off, never parallel-hammer. - -MLBBDex (cross-check): -- `GET /api/v1/heroes` — `{data, total, source{name,url,license}}` with `slug`, `name`, - `roles`, `lanes`, `specialties`. Envelope source attribution is credited here and in the app. - -## Catalog behaviour - -- **Manual sync only.** Nothing downloads at startup; the user presses *Sync catalog*. -- **Pagination** with `size=100`, `index=1..`, capped at 20 pages; duplicates by - `hero_id` are collapsed and counted; records missing id/name/both images are skipped and counted. -- **Mirror fallback.** If the primary host fails, the request is retried once against the mirror. -- **Catalog entries actually loaded are reported**, never a hardcoded expected count. -- **Caching (freshness enforced).** Per reference image, a small sidecar - (`/.img.meta`) records the source URL, SHA-256 of the bytes, size, content - type and a save timestamp. An asset is reused only if everything matches: file present, - 4 MiB cap, sidecar URL equals the URL the current catalog advertises, size and SHA-256 - match, magic bytes valid, container parses (PNG CRCs / JPEG & GIF trailers / WebP RIFF - framing) and — on Android — the image actually decodes. Anything else is re-downloaded. -- **Changed URLs.** When the catalog starts advertising a different URL for the same hero - asset, the old file is never exposed as the new asset; the refresh re-downloads it, and - the old one is not served even if the refresh fails. -- **Failed refreshes.** A refresh failure never deletes the previous asset and never serves - it under a changed URL. With an unchanged URL and a valid file on disk, the cache is - simply kept; with a corrupt file on disk, that hero reports an `assetFailures` entry and - matching skips it. -- **Redirects.** The transport follows redirects manually and re-validates every hop: - HTTPS only, relative redirects resolved, non-HTTP schemes and over-long targets refused. - Downgrades are never accepted. -- **Snapshot atomicity.** `writeAtomically()` writes to a `*.tmp` sibling in the same - directory, flushes and syncs where supported, then performs an `ATOMIC_MOVE` into place. - If atomic replacement cannot be completed (e.g. `EISDIR`, cross-device), the previous - snapshot is preserved, the temp file is deleted and the write fails loudly. There is no - direct-write fallback. Abandoned `*.tmp` siblings older than an hour are swept on init. -- **Offline.** After a successful sync the catalog serves from cache with no network at all. -- **Asset failure handling.** If one image fails, the hero is still in the catalog; the - failure is reported by name in `assetFailures` and matching simply skips that hero. -- **Privacy.** The cache lives in the app's private dirs. No frames, crops or confirmation - data ever leave the device; no runtime permissions beyond INTERNET/POST_NOTIFICATIONS/ - the capture ones are used. - -## Slot extraction - -- The slot rectangle (normalized *source* coordinates) maps to whole source pixels - (`NormalizedRect.toPixelRect`, floor origin / ceil far edge / ≥1 px) — the same rounding - rule the annotation canvas uses — and the crop is cropped axis-aligned, then rotated - upright to match what the user sees on screen. -- The rotation mappings follow `FrameTransform.sourceToOriented()` exactly (oriented pixel - `(x, y)` ← source pixel at the inverse point), on both the JVM extractor and the Android - one, so Java and Android paths produce byte-identical outputs for the same crop. -- Aspect ratio is never changed; nothing is stretched. -- The original frame bitmap is never mutated or recycled. The Android path validates the - Bitmap dimensions against the frame's `sourceWidth/sourceHeight` before cropping. -- Ownership: the crop creates a fresh bitmap; the rotation creates at most one more; every - temporary is recycled exactly once; failures recycle their temporaries and return `null`/ - `Result.failure` instead of leaking them. -- Extracted crops live in memory only and are never written to disk. -- Tests cover portrait and landscape frames, all four rotations with non-square asymmetric - fixtures, crops touching every edge, minimum-size rectangles, below-one-pixel rectangles, - invalid coordinates, payload mismatch, recycled-bitmap refusal, source immutability, and - the Android path end-to-end on an instrumented device. - -## Recognition v0 — what it is and is not - -**This is a prototype ranking baseline, not production-grade hero recognition.** - -- Two deterministic descriptors: a **dHash** perceptual hash (9×8 grayscale, area-average - downsample, bit = left pixel brighter than right) and a normalized **3×3 mean-RGB** - descriptor (27 floats), combined as `0.6 × (1 − hamming/64) + 0.4 × (1 − colorDistance)`, - clamped to [0, 1]. -- **The combined number is a heuristic similarity, not a probability.** It must never be read - as a likelihood, a confidence, or a "73% chance". The no-match threshold (0.55) and the - ambiguities gap (0.05) are **uncalibrated starting defaults** chosen so the pipeline is - exercisable; they were **not** tuned on real MLBB screens. -- **Before matching** a crop goes through the visual-information gate (`VisualInformationAnalyzer`): - a near-uniform gray/black/white/solid-color crop, a detail-free crop, or one smaller than - 12×12 is reported as `InsufficientVisualInformation` and no candidate is generated. The - thresholds (luma variance < 2.0 and channel variance < 2.0 for "near-uniform"; < 8.0 with - under 1% edge density for "too little detail") are **conservative heuristics, not calibrated - MLBB thresholds**. Low-contrast-but-detailed crops are deliberately *not* rejected. - Real-screen calibration of these heuristics is pending device testing. -- `InsufficientVisualInformation` is deliberately distinct from `NoMatch`: the first says the - crop carries no usable signal, the second says the signal was present but unrecognizable. -- Recognition **never** assigns a hero by itself. The user confirms or corrects every outcome - (candidates, no-match, insufficient-information, failed, or already-confirmed) through the - manual name search; the confirmation flow is explicit everywhere, and a failed persistence - operation is reported, not hidden. -- dHash works best on near-uniform headshots; it does not understand hero identity, and it - can rank wrong heroes confidently. **Accuracy on real MLBB screens is not measured.** - -### Evaluation methodology (synthetic fixtures only) - -| What | How it is measured | Status | -|---|---|---| -| Algorithm correctness | Deterministic tests: identical images match, ranking order, threshold behaviour, ambiguous gap, corrupt references skipped, hamming/color math, round trips. | CI (see `docs/VISION_TESTING.md`) | -| Blank-image rejection | Deterministic fixtures: uniform gray/black/white, saturated solids, near-uniform with tiny noise, low-resolution crops. | CI; `VisualInformationTest` covers all of them and the `InsufficientVisualInformation` outcome | -| Top-1 / Top-3 accuracy | **Method when run:** manually identify heroes in slot crops on real device captures, run Identify, record the rank of the correct hero among returned candidates, aggregate. Requires consented captures with manually verified labels; none exist yet. | **Not run** | -| False-match rate | **Method when run:** feed known-non-matching hero crops (and blank UI regions) and count how often any candidate clears the threshold. Blank/low-information crops are now rejected before matching; whether *detailed but wrong* crops still false-match is unmeasured. | **Not run** | -| Top-1/Top-3 on synthetic fixtures | Identical fixtures match (similarity ≈ 1); the inverse-gradient fixture yields NoMatch. No synthetic benchmark suite beyond these deterministic cases exists; no synthetic numbers are claimed as accuracy. | CI only | -| Real-screen status | No MLBB device session has been run. | **Pending** | - -### Known limitations - -- The matcher does not understand hero identity; two similar portraits can both rank highly. -- The threshold and weights are uncalibrated; 0.55 may be far too low or too high on real - captures (compression, UI effects, partial crops, anti-aliasing). -- The visual-information heuristics may reject genuinely flat but correct crops, or admit - genuinely empty but noisy ones. They are not a substitute for calibration. -- Reference images are `HEAD` portraits; full-body or in-game appearance usually differs. -- Java and Android extraction paths are intended to be pixel-identical, on the strength of - the shared mapping and the fixture tests, but have not been compared on two devices. -- Nothing here is a measurement of recognition quality on MLBB content. - -## UI - -Vision Lab, under the slot list: - -- *Sync catalog* button; sync state shown (`Not synced`, progress while syncing, `Ready: N heroes`, or error with the cached count if offline). -- *Identify hero* on the selected slot; per-slot state (`idle → processing → candidates | no match | insufficient information | confirmed | failed`). -- Candidate cards: cached image, name, similarity, `ambiguous` marker when present; **Confirm** per candidate. -- **No match** and **Insufficient information** states both offer manual name search. -- **Confirmed** view with Clear and **Change…**; **Change…** opens the manual correction - search (it does not re-run Identify). Corrections persist per slot without clearing first. -- Manual name search is available in every recognition state (candidates, no-match, - insufficient information, failed, confirmed), plus the share note for saved confirmations. -- Attribution footer with the unofficial notice; images load from the local cache only. -- Capture guidance: choosing **Entire screen / full device display** when Android asks is - required for the notification freeze workflow; the Debug Capture screen and the Vision Lab - both say so. - -## Preservation of existing behavior (Part J) - -- Compact red X delete control at the top-right: unchanged. -- Long-press creation selects the slot without opening the property editor: unchanged. -- 450 ms long-press and early-drag cancellation: unchanged. -- Paste source/anchor semantics: unchanged. -- Context menu and selection synchronization: unchanged. -- Property-editor validation feedback: unchanged. -- Fullscreen editor without zoom/fit controls: unchanged. -- Gesture isolation from page scrolling: unchanged. -- Portrait/landscape rendering and the Vision Lab interaction model: unchanged. -- MediaProjection lifecycle: unchanged except the required freeze/resume integration; the - `mediaProjection` foreground-service type, `POST_NOTIFICATIONS` and capture - permissions are preserved; no overlay, accessibility, touch automation or memory reading. -- No hero counters, recommendations, synergy scoring, overlays, packet inspection or cloud - recognition were added. - -## Acceptance status - -- **CI green**: see the final CI section of `docs/VISION_TESTING.md` / - `docs/IMPLEMENTATION_REPORT.md` for the exact head SHA and run URL after this revision. -- Unit tests: see the CI section (totals vary as the suite grows; the required gate is - 0 failures / 0 errors / 0 skipped for both debug and release variants). -- Instrumentation tests: executed on an emulator in CI if the `instrumented` job is green; - otherwise compiled but not executed — see `docs/VISION_TESTING.md` for the honest status. -- Physical-device smoke: **pending** (full matrix in `docs/TESTING_PHYSICAL_DEVICE.md`). -- Catalog sync on a device: **pending** (verified only through fake transports in unit tests). -- On-device accuracy: **not measured**; real-screen threshold calibration is **pending**. -- Catalog entries actually loaded: measured at sync time on the device that syncs; not yet - recorded here. +# Sprint 1B — hero recognition (historical) + +**Status:** merged into `main` as PR #3 on 2026-11-16. + +Sprint 1B added the first on-device hero-identification pipeline. It is kept here as a +historical record; the current behaviour is described in +[Vision architecture](../VISION_ARCHITECTURE.md) and the +[implementation report](../IMPLEMENTATION_REPORT.md). + +## What shipped + +- **Catalog.** Rone Arena is the primary source (`GET /api/heroes`, paginated, HTTPS only); + MLBBDex supplies a completeness cross-check and roles/lanes by slug. Identity is a + name-derived slug. A file-based snapshot and private image cache give offline browsing; + sync happens only when the user asks. +- **Cache safety.** Per-asset sidecars bind URL, SHA-256, size and content type. Assets are + revalidated when served; changed URLs force a new download; a failed refresh keeps the + previous asset. Redirects are HTTPS-only. Snapshot writes are temp-sibling plus flush, + sync and atomic move, with no direct-write fallback. +- **Slot extraction.** The source rectangle is cropped upright for every rotation following + the `FrameTransform.sourceToOriented` convention. The source bitmap is never mutated; + short-lived temporaries are recycled by their owner. +- **Recognition v0.** dHash plus a normalized 3×3 mean-RGB descriptor form an explicitly + uncalibrated similarity, never a probability. Blank, near-uniform and detail-free crops are + rejected as insufficient visual information. Below threshold the result is an honest no + match. Manual confirm and manual correction are available in every state. +- **Capture.** Freeze and resume from the ongoing notification, an independently owned pinned + frame, and the LIVE/FROZEN presentation. The landscape capture fix re-derives size on + resize, numbers reader generations to drop stale frames, honours `image.cropRect` and padded + rows, and adds structured diagnostics without logging image contents. + +## Verification and caveats + +CI was green on the final head with 840 unit tests, strict lint and a debug APK. Instrumented +tests compiled but did not execute because the hosted runner has no KVM. Accuracy on real +MLBB screens was **not** measured, and the threshold and heuristics remain starting defaults. +The physical-device smoke of freeze and landscape capture remained pending at merge time; see +the [physical-device checklist](../TESTING_PHYSICAL_DEVICE.md). diff --git a/docs/sprints/SPRINT_1C_APP_ONLY_CAPTURE.md b/docs/sprints/SPRINT_1C_APP_ONLY_CAPTURE.md index 0b2ec72..bfcf704 100644 --- a/docs/sprints/SPRINT_1C_APP_ONLY_CAPTURE.md +++ b/docs/sprints/SPRINT_1C_APP_ONLY_CAPTURE.md @@ -1,184 +1,37 @@ -# Sprint 1C — App-Only Capture & Circular Floating Control +# Sprint 1C — app-only capture and floating control (historical) -**Branch:** `sprint-1c-app-only-capture` (stacked on `sprint-1b-hero-recognition`, PR #3 — **not merged**) · **PR:** #4 (base `sprint-1b-hero-recognition`) +**Status:** merged into `main` as PR #4 on 2026-11-16. It was originally stacked on Sprint 1B; +the base was retargeted to `main` before merge. -Sprint 1C makes the MLBB freeze workflow playable from inside the game: the user -captures only the MLBB window, and a single circular overlay button freezes/resumes -the capture — no need to open the notification shade and no in-app step at all. -No recommendation UI of any kind is added. - -**Dependency on Sprint 1B:** the freeze/snapshot machinery (`CaptureFreeze`, -`FrameSnapshots`), the Vision Lab pinned frame and the capture foreground service all -come from PR #3. Sprint 1C was built on top of it and cannot be reviewed meaningfully -until #3 lands. Do not merge this PR before #3 is merged and this PR is rebased onto -`main` if the maintainer prefers a clean stack. +This is a historical record. Current behaviour is described in +[Screen capture](../SCREEN_CAPTURE.md). ## What shipped -| Piece | Where | What it does | -|---|---|---| -| Visibility gate (pure) | `capture/CaptureVisibilityPolicy.kt` | `shouldPublish(active, generationCurrent, frozen)` plus visibility transitions; JVM-tested | -| Pipeline gate | `FramePipeline.kt` | While the captured-content window is hidden, images are acquired and closed but never converted/published; the publish-time synchronized block double-checks | -| Visibility callback | `CaptureSessionService.projectionCallback.onCapturedContentVisibilityChanged` | API 34+; forwards to the pipeline; ignored unless CAPTURING; resize path unchanged | -| Overlay math (pure) | `overlay/OverlayPosition.kt` | Normalized (0..1) position on the free axis; pixel round-trip; edge snapping; clamping; JVM-tested | -| Overlay enablement (pure) | `overlay/OverlayEnablement.kt` | `shouldAttachOverlay(preference, permission, session)` gate; JVM-tested | -| Pending-attach invalidation | `OverlayController.kt` | generation counter: a detach() invalidates any in-flight show(), so a teardown during the DataStore read can never orphan the button | -| Overlay position persistence | `AppPreferencesRepository` | `floating_control_enabled`, `floating_position_x/y` in the existing preferences store | -| Floating button | `overlay/FloatingControlView.kt` | 52dp circle, LIVE green / FROZEN orange, white glyph, content description, system font | -| Overlay window | `overlay/OverlayController.kt` | `TYPE_APPLICATION_OVERLAY` + `FLAG_NOT_FOCUSABLE`; tap = freeze/resume via the shared `CaptureFreeze` path; long-press (400 ms) drag; edge snap + persist on release; re-anchored on rotation via the display listener | -| Service wiring | `CaptureSessionService.kt` | Overlay attached only while capture is running (user enabled + permission granted), repainted on freeze/resume, removed on every teardown path; `ACTION_OVERLAY_UPDATE` re-syncs from the UI | -| Enable surface | Debug Capture screen | “Floating control” card: opt-in switch, permission explanation, open Android overlay settings, live LIVE/FROZEN label; capture works fully without it | -| Guidance | Debug Capture + Vision Lab screens | Two capture modes documented (single-app preferred; full-display fallback with honest limitation) | -| Manifest | `AndroidManifest.xml` | `SYSTEM_ALERT_WINDOW` declared; **requested only when the user enables the floating control** via `ACTION_MANAGE_OVERLAY_PERMISSION` | - -## Single-app capture (Android 14+) - -DraftLens never forces the mode. The system chooser decides what the user picks; the app -only guides: - -1. Tap **Start Capture**. -2. Prefer **Single app / app window** and choose **Mobile Legends: Bang Bang**. - The projection then carries only that window: live frames while MLBB is on screen, - and the pinned last-valid frame whenever it is hidden. -3. If the single-app option is not offered (older Android), choose **Entire screen / - full device display**. That fallback captures everything, including the notification - shade, system UI and other apps. **DraftLens cannot promise notification-free - capture on the full-display path.** The freeze workflow still works. - -Below API 34 the visibility callback does not exist, so a hidden app-only window is -undetectable — frames keep flowing exactly as before (full-display behavior). - -## Visibility lifecycle - -`onCapturedContentVisibilityChanged(isVisible)` (API 34+) is wired into the existing -CAPTURING-state guard alongside `onCapturedContentResize`: - -- **Hidden:** the pipeline acquires and closes images but performs no conversion and - publishes nothing. `latestFrame` (and, if frozen, the pinned snapshot) are untouched. - The shade/DraftLens/other apps can never replace the last valid game frame. -- **Visible again:** publication resumes automatically — unless a frame is pinned - (user intent LIVE vs FROZEN wins). A resume requested while hidden flips the intent - to LIVE and takes effect exactly when visibility returns; nothing black is ever shown - in between. -- **Freeze while hidden:** the pin stores a copy of the last valid frame; still no - publication; visibility callbacks change nothing until a resume. -- **Stop:** the overlay is detached first, then the normal termination path runs; the - pinned frame survives in process memory as documented (Sprint 1B contract). -- **Resize during freeze:** unchanged — the generation-switch resize path works while - frozen; freeze/resume never recreate the projection or the VirtualDisplay and never - create a second VirtualDisplay. - -## Overlay and capture isolation (critical, not settled by code) - -**Whether the DraftLens overlay appears inside a MediaProjection capture is a physical-device -question and is NOT verified in this sprint.** The overlay does not set any flag to hide -itself from capture, and no capture region is masked. Expectations to check on device -(rows 34/35 of `docs/TESTING_PHYSICAL_DEVICE.md`): - -- Single-app MLBB capture **should** carry only the MLBB window; on many devices the - overlay (a separate window belonging to DraftLens) will not be included. -- Full-display fallback capture will very likely include the overlay. If that happens, - the control sits in the top/bottom corner, snaps away from the central draft region, - and must not be read by the hero-slot cropping (annotation rects are user-placed on - the last valid frame, and the overlay is small — but this is not automatically safe). -- If the overlay contaminates hero slot crops on the fallback path, the documented - remedy is: use single-app capture, or disable the floating control and freeze from - the notification/in-app. **Do not** crop arbitrary regions to hide it. - -No claim is made that app-only capture excludes every overlay window; only on-device runs -can establish it per device/OEM. - -## Verification - -CI (GitHub Actions, ubuntu-latest) is the build of record — the authoring host has no -Android toolchain. - -- Implementation head `f992d45` (docs-only commits after it change no build inputs): - CI run 43784157138 — **886 unit tests, 0 failures / 0 errors / 0 skipped**; strict lint green; - `assembleAndroidTest` green; `assembleDebug` green; APK `app-debug.apk` - sha256 `1d1f50629e145d48e64ed56d0cf74228e7d48d4c6f94fde81eb6c22fc4795810`. -- Instrumentation tests: **compiled via `assembleAndroidTest`; not executed** (no KVM on the runner). -- Automated coverage gaps, stated honestly: **LIVE→FROZEN pinning a real frame** (an - ImageReader surface cannot be filled from a test Canvas, and the pin path is exercised - on device — rows 36/37) and **resize during freeze** (the gate is dimension-agnostic by - construction; policy truth-table pins it — on-device rows 25/30 pin the pipeline path). -- Physical-device rows 33–43: **NOT RUN** — see `docs/TESTING_PHYSICAL_DEVICE.md` - -## Explicitly not built (scope restrictions) - -No draft-phase detection, no slot detection, no hero models, no counter/synergy scoring, -no recommendation ranking or overlays, no game memory access, no automated interaction -with MLBB, no accessibility service, no panel/menu/toolbar from the overlay button. - -## Follow-up fixes (notification Pause, permissions-only UI, overlay independence) - -### Root cause analysis (overlay "freezing" during app-only capture) - -Traced end to end: `MediaProjection.Callback` → `onCapturedContentVisibilityChanged` → -`CaptureVisibilityPolicy` → `FramePipeline` → `CaptureFreeze` → overlay/notification state. - -What the trace showed: - -1. **The overlay was never lifecycle-coupled to capture visibility** — `FloatingControlView` - renders and handles touches with no frame input; `onCapturedContentVisibilityChanged` - gates only the `FramePipeline` (by design after the 1C review fixes). Window attachment - is independent of the ImageReader generation and of the sampler. -2. **Three real defects made the button misbehave though:** - - **Silent no-op tap**: tapping freeze before any frame had ever been published returned - `NoFrame` and produced no feedback (common in app-only mode, where nothing publishes - until MLBB is first on screen). The tap now surfaces a short toast ("No frame captured - yet"), and the button is no longer perceived as dead. - - **Glyph/notification could disagree with reality**: the overlay glyph and the - notification only followed `frozenFrame`. Any publication block that was not a pin - (e.g. notification Pause) left both surfaces showing LIVE/green while publication was - blocked. Both now derive from one shared `CapturePublishState` (LIVE/PAUSED/FROZEN, - pin wins over pause for presentation) — see `CapturePublishState.kt` — collected from a - combined `frozenFrame` + session-state flow. - - **Visual workaround risk ruled out**: nothing in the fix hides the button, masks - regions, or falls back to full-display capture. -3. **Unverifiable from CI**: whether a specific OEM/immersive game covers or excludes the - overlay window on the physical screen is a hardware finding — rows 44–53 of - `docs/TESTING_PHYSICAL_DEVICE.md` (all NOT RUN). No OEM workaround is shipped; if an - OEM covers overlays during immersive games, that is a documented limitation, not a code - defect, and the notification Stop/Pause actions remain available. - -### Notification: Freeze → Pause - -The ongoing notification now carries **Stop** and exactly one of **Pause** (while LIVE) / -**Resume** (while PAUSED or FROZEN) — never both, never a Freeze action. Pause maps onto the -existing shared `CaptureStatus.PAUSED` gate (`setPaused(true)` stops conversion/publication, -keeps the projection, VirtualDisplay and latest valid frame). Resume is a unified handler: -`CaptureFreeze.resumeLive()` (clear any pin) + `setPaused(false)` (clear any pause). The -in-app and overlay Freeze/Resume (pinning a snapshot) are unchanged. - -### Permissions-only UI - -Removed: the two-mode capture tutorial card, the floating-control tutorial card, the -in-game freeze help text, and the stale "Detection: not implemented in Sprint 0" line. -Kept, compacted in one "Permissions" card: the notification permission flow (status -Allowed/Not allowed + grant action), and the floating-overlay permission flow (status, -switch, open Android overlay settings, pending-enable on return from settings). The -in-app capture controls (Start/Stop/Pause), status readout, sampling interval and live -preview are unchanged. - -### Verification - -- Implementation head `130ce6c`: CI run 43922482183 — **904 unit tests, 0 failures / 0 errors / 0 skipped**; - lint + `assembleAndroidTest` + `assembleDebug` green; APK sha256 `48b510673919fe2caa9c906747d9dc766eb27bb170c589f67b58c9a175dd7b72`. -- Physical device: rows 44–53 remain **NOT RUN**. - -## Follow-up: single-app first-frame bug - -Root cause: Sprint 1C registered MediaProjection callbacks before the pipeline, -VirtualDisplay and session state were ready. Early visibility `false` and early -captured-content resize were lost/overwritten or treated as fatal; the boolean gate could -block the first frame forever. Diagnostics were added first (`CaptureDiagnostics`, callback -seq/readiness logs, per-stage drop reasons, no pixels); then the fix: service-owned explicit -UNKNOWN/VISIBLE/HIDDEN policy (UNKNOWN intentionally open, first frame promotes to VISIBLE), -service-owned `PendingResize` applied to both VirtualDisplay and ImageReader at startup, -no pipeline-side visibility reset, and holder delivery of frames published during STARTING. -CI run 43936524131 at head `3c7b58c`: **926 unit tests, 0 failures**; lint, -`assembleAndroidTest`, `assembleDebug` green; APK sha256 -`ade9ec10c5d6b9d4e372b325e6b521f87170d0fa5273c9b30e9ea9063a763070`. See -`docs/IMPLEMENTATION_REPORT.md` and the focused A–E device test in -`docs/TESTING_PHYSICAL_DEVICE.md` (**NOT RUN**). +- **App-only capture guidance.** Single app / app window → Mobile Legends is preferred on + Android 14+; whole-display capture remains a documented fallback that may include the shade + and other apps. Nothing is forced and no capture region is masked. +- **Visibility gate.** `onCapturedContentVisibilityChanged` feeds a pure + `CaptureVisibilityPolicy` that gates frame publication. When the captured window is hidden, + images are still acquired and closed but never converted or published, so the last valid + game frame and any pinned frame are preserved. The gate survives resize and resets only on a + fresh session. +- **One floating control.** A 52 dp circular `TYPE_APPLICATION_OVERLAY` view, attached by the + capture service only while a session runs. Tap toggles LIVE and paused/frozen through the + shared `CaptureFreeze` path; long-press-and-drag moves it with edge snapping; its normalized + position is persisted. The overlay permission is requested only when the user enables it, + and denial leaves capture and in-app controls working. +- **Shared publish state.** `CapturePublishState` derives LIVE, PAUSED and FROZEN from the + frozen frame and session state so the overlay glyph, its tap behaviour and the notification + action always agree. The notification offers Stop and exactly one state action. +- **Capture UI.** Tutorial and in-game help text were removed, leaving a compact permissions + card and the existing capture controls. + +## Verification and caveats + +CI was green on the final head with 926 unit tests, strict lint, compiled instrumentation and +a debug APK. Instrumented tests were not executed. Physical-device rows 33–43 and 44–53, +including OEM overlay isolation and app-only capture behaviour, were **NOT RUN** at merge +time. On API levels below 34 there is no visibility signal, so app-only capture cannot be +detected and whole-display semantics apply. See the +[physical-device checklist](../TESTING_PHYSICAL_DEVICE.md). diff --git a/docs/superpowers/plans/2026-11-06-sprint-1a-vision-foundation.md b/docs/superpowers/plans/2026-11-06-sprint-1a-vision-foundation.md index 187b053..9136328 100644 --- a/docs/superpowers/plans/2026-11-06-sprint-1a-vision-foundation.md +++ b/docs/superpowers/plans/2026-11-06-sprint-1a-vision-foundation.md @@ -1,843 +1,33 @@ -# Sprint 1A Vision Foundation Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build the image-bearing foundation for DraftLens vision work: non-null frame contracts, tested preprocessing and geometry, manual draft-slot annotation, Vision Lab, and versioned calibration profiles. - -**Architecture:** Single module `:app`, layered into pure Kotlin `vision/model`, `vision/geometry`, `preprocessing`, `annotation`, and `calibration`, with Android Bitmap and Compose at the edges. Capture exposes frames through a narrow `VisionFrameSource`; Vision never imports `CaptureSessionService` or `CaptureStateHolder`. - -**Tech Stack:** Kotlin 2.0.20, Android SDK 35, Compose Material 3, Coroutines, DataStore Preferences, `kotlinx-serialization-json`. - -**Spec:** `docs/superpowers/specs/2026-11-06-sprint-1a-vision-foundation-design.md` - -## Execution Checkpoint (2026-11-06) - -Branch: `sprint-1-vision-foundation`, based on `origin/main` at `ae92eaf`. - -| Task | Status | Commits | Review | -|------|--------|---------|--------| -| 1. Vision model and geometry value types | Done | `a2731d3` | Clean | -| 2. Geometry subsystem | Done | `357a40f` | Clean | -| 3. Capture non-null frames | Done | `8e1381a` | Clean | -| 4. Preprocessing engine | Done | `79bd211`, `41aa126`, `b81e740` | Clean after 2 fix rounds | -| 5. Slot annotation store and codec | Done | `1086b75`, `cab3d8d` | Clean after 1 fix round | -| 6. Calibration profiles | **In progress** | `08ca414` (WIP, unreviewed) | Not started | -| 7. Vision frame source and ViewModel | Not started | — | — | -| 8. Compose Vision Lab | Not started | — | — | -| 9. Documentation | Not started | — | — | -| 10. CI, PR, APK, report | Not started | — | — | - -Last verified state: `cab3d8d`, full JVM unit suite 127/127 passing (local aarch64 host). -Task 6 work exists but was not compiled or tested before the local Android SDK was cleared -from `/tmp/opencode/android-sdk`; treat `08ca414` as unverified. - -Resume point: - -1. Review the Task 6 WIP code against Task 6 of this plan and the design spec. The accepted - `CalibrationLoadResult` shape is `Success(profiles)`, `UnsupportedSchema(message)`, - `Corrupt(message)`. -2. Run `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.calibration.*'` and the full - `./gradlew test` once a working Android SDK is available; fix and commit Task 6. -3. Continue with Tasks 7–10. Push the branch and let CI be the build of record for lint, - `assembleAndroidTest`, and `assembleDebug`. - -## Global Constraints - -- Single Gradle module `:app`; logical packages only; no empty packages or new modules. -- Kotlin 2.0.20 · AGP 8.6.1 · Gradle 8.7 · compileSdk/targetSdk 35 · minSdk 26 · JDK 17. -- Vision must not depend directly on `CaptureSessionService` or `CaptureStateHolder`; the capture adapter lives in `capture/` and implements the Vision source seam. -- No ML model, OCR, hero recognition, automatic draft/ban/pick detection, or fabricated confidence/detection data. -- No network upload, no screenshot persistence, no game-process access, no automated interaction. -- Existing capture consent remains mandatory and unchanged. -- No `FLAG_SECURE`. -- No real game screenshots or captures in fixtures; use synthetic images. -- CI is the build of record; final verification runs `test`, `assembleAndroidTest`, `lint`, and `assembleDebug`. - -## Review Focus - -1. MediaProjection buffers are already display-oriented; a non-zero `displayRotation` must not rotate ordinary captures. Task 3 pins production wiring (`ROTATION_0` correction); Task 4 pins synthetic non-zero correction. -2. Cropped, rotated, scaled inverse mappings drift. Task 2 pins `FrameTransform` round trips for 90/180/270 with crop and downscale. -3. A process request can target a frame that is superseded before the worker finishes. Task 7 pins frozen-selection and stale-result discard. -4. A newer/corrupt DataStore profile document must not be silently replaced or exported. Task 6 pins unsupported-schema refusal and delete-as-escape-hatch behavior. -5. Manual rectangles can exceed bounds or duplicate indexes and appear valid. Task 5 pins invalid rect, duplicate `(type, team, index)`, and overlap validation. - ---- - -## File Structure - -### Create - -- `app/src/main/kotlin/com/draftlens/vision/model/FrameId.kt` — value class and `FrameId.random()`. -- `app/src/main/kotlin/com/draftlens/vision/model/FrameEnums.kt` — `PixelFormat`, `FrameRotation`, `OrientationCategory`. -- `app/src/main/kotlin/com/draftlens/vision/model/FramePayload.kt` — payload contract and `PixelArrayFramePayload`. -- `app/src/main/kotlin/com/draftlens/vision/model/VisionFrame.kt` — `OriginalFrame`, `ProcessedFrame`, `SlotImage`, marker interface. -- `app/src/main/kotlin/com/draftlens/vision/model/VisionFrameSource.kt` — source seam and `VisionSourceState`. -- `app/src/main/kotlin/com/draftlens/vision/model/android/BitmapFramePayload.kt` — Bitmap payload and ownership marker. -- `app/src/main/kotlin/com/draftlens/vision/geometry/NormalizedPoint.kt` -- `app/src/main/kotlin/com/draftlens/vision/geometry/NormalizedRect.kt` -- `app/src/main/kotlin/com/draftlens/vision/geometry/PixelPoint.kt` -- `app/src/main/kotlin/com/draftlens/vision/geometry/PixelRect.kt` -- `app/src/main/kotlin/com/draftlens/vision/geometry/FrameTransform.kt` -- `app/src/main/kotlin/com/draftlens/vision/preprocessing/PreprocessTypes.kt` -- `app/src/main/kotlin/com/draftlens/vision/preprocessing/PreprocessingMath.kt` -- `app/src/main/kotlin/com/draftlens/vision/preprocessing/PixelArrayFramePreprocessor.kt` -- `app/src/main/kotlin/com/draftlens/vision/preprocessing/AndroidFramePreprocessor.kt` -- `app/src/main/kotlin/com/draftlens/vision/annotation/DraftSlot.kt` -- `app/src/main/kotlin/com/draftlens/vision/annotation/SlotAnnotationStore.kt` -- `app/src/main/kotlin/com/draftlens/vision/annotation/SlotAnnotationCodec.kt` -- `app/src/main/kotlin/com/draftlens/vision/calibration/CalibrationProfile.kt` -- `app/src/main/kotlin/com/draftlens/vision/calibration/CalibrationCompatibility.kt` -- `app/src/main/kotlin/com/draftlens/vision/calibration/CalibrationProfileCodec.kt` -- `app/src/main/kotlin/com/draftlens/vision/calibration/CalibrationRepository.kt` — interface consumed by ViewModel. -- `app/src/main/kotlin/com/draftlens/datastore/CalibrationProfileDataStore.kt` — Android/DataStore implementation. -- `app/src/main/kotlin/com/draftlens/capture/CaptureVisionFrameSource.kt` — capture-to-Vision adapter. -- `app/src/main/kotlin/com/draftlens/feature/visionlab/VisionLabViewModel.kt` -- `app/src/main/kotlin/com/draftlens/feature/visionlab/VisionLabUiState.kt` -- `app/src/main/kotlin/com/draftlens/feature/visionlab/VisionLabScreen.kt` -- `app/src/main/kotlin/com/draftlens/feature/visionlab/AnnotationCanvas.kt` -- `app/src/main/kotlin/com/draftlens/feature/visionlab/SlotEditor.kt` -- `app/src/main/kotlin/com/draftlens/feature/visionlab/ProfileControls.kt` - -### Modify - -- `app/build.gradle.kts` — add `kotlinx-serialization-json` and the Kotlin serialization plugin. -- `gradle/libs.versions.toml` — pin `kotlinx-serialization-json` and plugin. -- `app/src/main/kotlin/com/draftlens/models/CapturedFrame.kt` — non-null frame payload/vision frame; no `bitmap: Bitmap?`. -- `app/src/main/kotlin/com/draftlens/capture/FramePipeline.kt` — return/publish non-null `CapturedFrame`; pass capture rotation. -- `app/src/main/kotlin/com/draftlens/capture/CaptureSessionService.kt` — read and pass display rotation on start/resize; preserve capture lifecycle. -- `app/src/main/kotlin/com/draftlens/capture/CaptureStateHolder.kt` — record/clear new `CapturedFrame` shape. -- `app/src/main/kotlin/com/draftlens/feature/home/DebugCaptureScreen.kt` — read bitmap through `BitmapFramePayload` only. -- `app/src/main/kotlin/com/draftlens/MainActivity.kt` — Material 3 `TabRow` with Debug Capture and Vision Lab. -- `app/src/main/kotlin/com/draftlens/DraftLensApp.kt` — expose calibration DataStore repository. -- `README.md`, `docs/ARCHITECTURE.md`, `docs/VISION_*.md`, `tasks/plan.md`, `tasks/todo.md` — status and recorded verification. - -### Test - -- `app/src/test/kotlin/com/draftlens/vision/model/VisionFrameTest.kt` -- `app/src/test/kotlin/com/draftlens/vision/geometry/GeometryTest.kt` -- `app/src/test/kotlin/com/draftlens/preprocessing/PreprocessingMathTest.kt` -- `app/src/test/kotlin/com/draftlens/preprocessing/PixelArrayFramePreprocessorTest.kt` -- `app/src/test/kotlin/com/draftlens/annotation/SlotAnnotationStoreTest.kt` -- `app/src/test/kotlin/com/draftlens/annotation/SlotAnnotationCodecTest.kt` -- `app/src/test/kotlin/com/draftlens/calibration/CalibrationCodecCompatibilityTest.kt` -- `app/src/test/kotlin/com/draftlens/capture/CaptureVisionFrameSourceTest.kt` -- `app/src/test/kotlin/com/draftlens/feature/visionlab/VisionLabViewModelTest.kt` -- Existing `CaptureStateHolderTest.kt`, fixtures updated for non-null frames. - ---- - -### Task 1: Vision model and geometry value types - -**Files:** -- Create: `app/src/main/kotlin/com/draftlens/vision/model/FrameId.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/model/FrameEnums.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/model/FramePayload.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/model/VisionFrame.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/model/android/BitmapFramePayload.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/geometry/NormalizedPoint.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/geometry/NormalizedRect.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/geometry/PixelPoint.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/geometry/PixelRect.kt` -- Test: `app/src/test/kotlin/com/draftlens/vision/model/VisionFrameTest.kt` - -**Interfaces:** -- Consumes: nothing. -- Produces: `FrameId`, `FrameRotation`, `OrientationCategory`, `PixelFormat`, `FramePayload`, `PixelArrayFramePayload`, `BitmapFramePayload`, `OriginalFrame`, `ProcessedFrame`, `SlotImage`, `NormalizedPoint`, `NormalizedRect`, `PixelPoint`, `PixelRect`. - -- [ ] **Step 1: Write the failing tests** - -```kotlin -@Test fun `pixel array payload copies input and estimates bytes`() { - val source = intArrayOf(1, 2, 3, 4) - val payload = PixelArrayFramePayload(width = 2, height = 2, pixels = source) - source[0] = 99 - assertEquals(1, payload.pixels()[0]) - assertEquals(16L, payload.byteCountEstimate) -} - -@Test fun `pixel array payload rejects empty or mismatched input`() { - assertThrows(IllegalArgumentException::class.java) { - PixelArrayFramePayload(width = 0, height = 1, pixels = IntArray(0)) - } - assertThrows(IllegalArgumentException::class.java) { - PixelArrayFramePayload(width = 2, height = 2, pixels = IntArray(3)) - } -} - -@Test fun `orientation category is explicit`() { - assertEquals(OrientationCategory.PORTRAIT, OrientationCategory.from(720, 1280)) - assertEquals(OrientationCategory.LANDSCAPE, OrientationCategory.from(1280, 720)) - assertEquals(OrientationCategory.SQUARE, OrientationCategory.from(100, 100)) -} - -@Test fun `invalid normalized and pixel rectangles throw`() { - assertThrows(IllegalArgumentException::class.java) { NormalizedRect(0.5f, 0f, 0.2f, 1f) } - assertThrows(IllegalArgumentException::class.java) { PixelRect(0, 0, 0, 10) } -} -``` - -- [ ] **Step 2: Run failing test** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.vision.model.VisionFrameTest' --stacktrace` -Expected: FAIL because model classes do not exist. - -- [ ] **Step 3: Implement contract** - -`FrameId` is `public value class FrameId(val value: String)` with `fun random()`. `PixelArrayFramePayload` defensively copies `pixels` in `init`, exposes `pixels(): IntArray` returning a copy, and uses `ARGB_8888 = 4 bytes/pixel` for `byteCountEstimate`. `BitmapFramePayload(width/height/format/byteCountEstimate)` reads directly from non-null `bitmap`; `bitmap.isRecycled` returns zero/invalid rather than a frame. `VisionFrame` is a marker interface. Data classes carry the exact fields from the spec: `displayRotation` and `rotation` are separate `FrameRotation` values; `rotation` is the pixel-orientation correction only. `SlotImage` carries `slotId`, `sourceRect: PixelRect`, and creates no pixels in this task. - -`NormalizedRect` validates finite non-NaN coordinates, `left < right`, `top < bottom`, and `[0,1]` bounds. `PixelRect` uses exclusive edges and validates non-negative dimensions. - -- [ ] **Step 4: Run tests** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.vision.model.VisionFrameTest' --stacktrace` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add app/src/main/kotlin/com/draftlens/vision app/src/test/kotlin/com/draftlens/vision/model/VisionFrameTest.kt -git commit -m "feat: add non-null vision frame contract" -``` - ---- - -### Task 2: Geometry subsystem with rotation and crop round trips - -**Files:** -- Create: `app/src/main/kotlin/com/draftlens/vision/geometry/FrameTransform.kt` -- Test: `app/src/test/kotlin/com/draftlens/vision/geometry/GeometryTest.kt` - -**Interfaces:** -- Consumes: `FrameRotation` from Task 1. -- Produces: `NormalizedPoint`, `NormalizedRect`, `PixelPoint`, `PixelRect`, `FrameTransform`, `FrameTransform.create(...)`, coordinate round-trip methods. - -- [ ] **Step 1: Write the failing geometry tests** - -Create tests with these exact cases: - -```kotlin -@Test fun `normalized rect rejects reversed or out of range bounds`() { - assertThrows(IllegalArgumentException::class.java) { NormalizedRect(0.5f, 0f, 0.2f, 1f) } - assertThrows(IllegalArgumentException::class.java) { NormalizedRect(-0.1f, 0f, 0.5f, 1f) } - assertThrows(IllegalArgumentException::class.java) { NormalizedRect(0f, 0f, 1.1f, 1f) } -} - -@Test fun `all rotations round trip source points`() { - for (rotation in FrameRotation.entries) { - val swapsAxes = rotation == FrameRotation.ROTATION_90 || rotation == FrameRotation.ROTATION_270 - val transform = FrameTransform.create( - sourceWidth = 4, sourceHeight = 2, - rotation = rotation, crop = null, - processedWidth = if (swapsAxes) 2 else 4, - processedHeight = if (swapsAxes) 4 else 2, - ) - val source = FloatPoint(1.25f, 0.75f) - assertNear(source, transform.processedToSource(transform.sourceToProcessed(source)), 0.01f) - } -} - -@Test fun `cropped downscaled 90 degree transform round trips`() { - val crop = NormalizedRect(0.1f, 0.2f, 0.9f, 0.8f) - val transform = FrameTransform.create( - sourceWidth = 1000, sourceHeight = 500, - rotation = FrameRotation.ROTATION_90, - crop = crop, processedWidth = 160, processedHeight = 240, - ) - val point = FloatPoint(123f, 210f) - assertNear(point, transform.processedToSource(transform.sourceToProcessed(point)), 1.0f) -} -``` - -Also cover portrait/landscape, crop 100% identity, rectangle mapping, NaN/infinity, invalid dimensions, crop outside range, and aspect mismatch. - -- [ ] **Step 2: Run failing tests** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.vision.geometry.GeometryTest' --stacktrace` -Expected: FAIL because geometry classes do not exist. - -- [ ] **Step 3: Implement types** - -`FrameTransform` stores source dimensions, pixel-orientation correction, oriented crop rect, oriented dimensions, crop-pixel rectangle, scale, and processed dimensions. - -`FrameTransform.create` validates positive dimensions, non-empty crop in oriented pixels with at least 1 px per edge, and output aspect ratio equal to the oriented crop aspect within one pixel of width or height. `sourceToProcessed` and `processedToSource` are continuous float mappings; integer `PixelRect` is used only for public pixel conversion. Rotation maps edges like the spec: `ROTATION_90` outputs `H × W`, `ROTATION_270` outputs `H × W`, and `ROTATION_180` preserves `W × H`. - -Add public helpers: - -```kotlin -fun NormalizedRect.toPixelRect(width: Int, height: Int): PixelRect -fun PixelRect.toNormalized(width: Int, height: Int): NormalizedRect -``` - -- [ ] **Step 4: Run tests** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.vision.geometry.GeometryTest' --stacktrace` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add app/src/main/kotlin/com/draftlens/vision/geometry app/src/test/kotlin/com/draftlens/vision/geometry -git commit -m "feat: add rotation-aware vision geometry" -``` - ---- - -### Task 3: Capture carries non-null frames and display rotation - -**Files:** -- Modify: `app/src/main/kotlin/com/draftlens/models/CapturedFrame.kt` -- Modify: `app/src/main/kotlin/com/draftlens/capture/FramePipeline.kt` -- Modify: `app/src/main/kotlin/com/draftlens/capture/CaptureSessionService.kt` -- Modify: `app/src/main/kotlin/com/draftlens/capture/CaptureStateHolder.kt` -- Modify: `app/src/main/kotlin/com/draftlens/feature/home/DebugCaptureScreen.kt` -- Modify: `app/src/test/kotlin/com/draftlens/capture/CaptureStateHolderTest.kt` -- Create: `app/src/test/kotlin/com/draftlens/vision/TestFrames.kt` - -**Interfaces:** -- Consumes: model contract from Task 1. -- Produces: `CapturedFrame.toVisionFrame()`, `CaptureStateHolder.latestFrame: StateFlow` with no nullable pixels; production `displayRotation` on the frame. - -- [ ] **Step 1: Update capture tests first** - -Replace `capturedFrameNullBitmap()` with a shared synthetic fixture: - -```kotlin -fun testCapturedFrame(): CapturedFrame = CapturedFrame( - frame = OriginalFrame( - id = FrameId("test-frame"), - payload = PixelArrayFramePayload(2, 2, intArrayOf(1, 2, 3, 4)), - sourceWidth = 2, - sourceHeight = 2, - displayRotation = FrameRotation.ROTATION_90, - rotation = FrameRotation.ROTATION_0, - capturedAtElapsedRealtimeMs = 10_000, - wallClockTimestampMs = 1_700_000_000_000, - captureLatencyMs = 4, - ), - processingLatencyMs = 12, -) -``` - -Update fixture assertions to use `CapturedFrame.width`, `.height`, `.wallClockTimestampMs`, and `.processingLatencyMs` forwarding properties or call sites to `frame.frame.*`. Keep CaptureStateHolder behavior unchanged. - -- [ ] **Step 2: Run capture tests to verify failure** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.capture.CaptureStateHolderTest' --stacktrace` -Expected: FAIL because `CapturedFrame` still exposes `bitmap`. - -- [ ] **Step 3: Implement non-null capture frame** - -`CapturedFrame` contains `frame: OriginalFrame`, exposes `width`, `height`, `displayRotation`, `wallClockTimestampMs`, `sampledAtElapsedRealtimeMs`, and `processingLatencyMs` via cheap forwards, and `fun toVisionFrame(): OriginalFrame = frame`. `FramePipeline.start(width, height, densityDpi, displayRotation: FrameRotation = FrameRotation.ROTATION_0)` passes rotation into every `OriginalFrame`. - -The pipeline’s `imageToBitmap` becomes `imageToPayload`: return a non-null `BitmapFramePayload` or drop the frame and log a warning. No nullable payload crosses into `CapturedFrame`. Stop paths still clear references without calling `recycle()`. - -`CaptureSessionService` reads `DisplayManager.DEFAULT_DISPLAY` rotation at start and display recreation. Map `Surface.ROTATION_0/90/180/270` to `FrameRotation`; if unavailable, use `ROTATION_0` and log. Pass `displayRotation` into `FramePipeline.start` and `prepareResize`. MediaProjection buffers are display-oriented, so pipeline sets `rotation = ROTATION_0` always; `displayRotation` records the device orientation without rotating the pixels. - -`DebugCaptureScreen` accepts only `BitmapFramePayload` for preview; synthetic/test payloads show the existing empty-state message. Remove `bitmap != null && !bitmap.isRecycled` checks and invalid-bitmap special cases. - -- [ ] **Step 4: Run capture tests** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.capture.*' --stacktrace` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add app/src/main/kotlin/com/draftlens/models/CapturedFrame.kt app/src/main/kotlin/com/draftlens/capture app/src/main/kotlin/com/draftlens/feature/home/DebugCaptureScreen.kt app/src/test -git commit -m "refactor: carry non-null vision frames from capture" -``` - ---- - -### Task 4: Preprocessing math and PixelArray renderer - -**Files:** -- Create: `app/src/main/kotlin/com/draftlens/vision/preprocessing/PreprocessTypes.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/preprocessing/PreprocessingMath.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/preprocessing/PixelArrayFramePreprocessor.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/preprocessing/AndroidFramePreprocessor.kt` -- Test: `app/src/test/kotlin/com/draftlens/preprocessing/PreprocessingMathTest.kt` -- Test: `app/src/test/kotlin/com/draftlens/preprocessing/PixelArrayFramePreprocessorTest.kt` - -**Interfaces:** -- Consumes: `OriginalFrame`, `PixelArrayFramePayload`, `FrameTransform`. -- Produces: `ProcessFrameRequest`, `ProcessFrameResult`, `PreprocessFailure`, `FramePreprocessor`, `AndroidFramePreprocessor`, `PixelArrayFramePreprocessor`. - -- [ ] **Step 1: Write failing validation and math tests** - -```kotlin -@Test fun `valid landscape frame is accepted and preserves aspect ratio`() { - val frame = syntheticOriginal(width = 1920, height = 1080) - val plan = PreprocessingMath.plan(frame, ProcessFrameRequest(maxProcessedDimension = 1280)) - assertTrue(plan.isSuccess) - assertEquals(1280, plan.getOrThrow().outputWidth) - assertEquals(720, plan.getOrThrow().outputHeight) -} - -@Test fun `invalid input reports honest failure`() { - val frame = syntheticOriginal(width = 0, height = 100) - val result = AndroidFramePreprocessor.process(frame, ProcessFrameRequest()) - assertEquals(PreprocessFailureCode.INVALID_DIMENSIONS, (result as ProcessFrameResult.Failure).reason.code) -} - -@Test fun `90 degree correction swaps output axes without stretching`() { - val frame = syntheticOriginal(width = 1920, height = 1080, rotation = FrameRotation.ROTATION_90) - val result = PixelArrayFramePreprocessor.process(frame, ProcessFrameRequest(maxProcessedDimension = 500)) - val success = result as ProcessFrameResult.Success - assertEquals(281, success.frame.payload.width) - assertEquals(500, success.frame.payload.height) -} -``` - -- [ ] **Step 2: Run tests to verify failure** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.preprocessing.*' --stacktrace` -Expected: FAIL because preprocessing classes do not exist. - -- [ ] **Step 3: Implement planning and rendering** - -`ProcessFrameRequest` contains `maxProcessedDimension: Int = 1280`, `maxPixels: Long = 8_000_000`, and `crop: NormalizedRect? = null`. `PreprocessFailure(code, message)` codes are `INVALID_DIMENSIONS`, `PAYLOAD_MISMATCH`, `TOO_MANY_PIXELS`, `INVALID_CROP`, `OUTPUT_TOO_LARGE`, `UNSUPPORTED_FORMAT`, `RENDER_FAILED`. `FramePreprocessor` is a `fun interface` with `process(input: OriginalFrame, request: ProcessFrameRequest): ProcessFrameResult`. - -`PreprocessingMath.plan` validates source dimensions, payload consistency, `maxProcessedDimension`, `maxPixels`, oriented dimensions, crop in oriented pixels, output dimensions, aspect ratio preservation within one pixel, and output pixel budget. The plan stores source/oriented/crop/output dimensions, rotation, and scale. - -`PixelArrayFramePreprocessor` performs rotation, crop, and downscale with nearest-neighbor sampling. It never upscales. If `rotation == ROTATION_0`, crop is null, and output equals source size, it returns the same `PixelArrayFramePayload` reference and an identity transform. For every other path it allocates exactly one output pixel array and returns a copied payload. Duration uses `System.nanoTime()`. - -`AndroidFramePreprocessor` validates through the same math and, for `BitmapFramePayload`, draws into exactly one destination bitmap with a Canvas/Matrix. It catches rendering exceptions and returns `RENDER_FAILED`; it never publishes a partial frame. For non-Bitmap payloads in Android code it returns `UNSUPPORTED_FORMAT`; JVM tests use the PixelArray implementation. - -- [ ] **Step 4: Run preprocessing tests** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.preprocessing.*' --stacktrace` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add app/src/main/kotlin/com/draftlens/vision/preprocessing app/src/test/kotlin/com/draftlens/preprocessing -git commit -m "feat: add validated frame preprocessing engine" -``` - ---- - -### Task 5: Manual slot annotation store and JSON codec - -**Files:** -- Create: `app/src/main/kotlin/com/draftlens/vision/annotation/DraftSlot.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/annotation/SlotAnnotationStore.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/annotation/SlotAnnotationCodec.kt` -- Modify: `app/src/main/kotlin/com/draftlens/vision/geometry/NormalizedRect.kt` -- Modify: `app/build.gradle.kts` -- Modify: `gradle/libs.versions.toml` -- Test: `app/src/test/kotlin/com/draftlens/annotation/SlotAnnotationStoreTest.kt` -- Test: `app/src/test/kotlin/com/draftlens/annotation/SlotAnnotationCodecTest.kt` - -**Interfaces:** -- Consumes: `NormalizedRect`, `FrameRotation` not required. -- Produces: `SlotType`, `SlotTeam`, `DraftSlot`, `SlotAnnotationStore`, `SlotAnnotationCodec`, `SlotIssue`. - -- [ ] **Step 1: Add failing annotation tests** - -```kotlin -@Test fun `duplicate index is rejected within type team scope`() { - val store = SlotAnnotationStore(idGenerator = sequenceOf("a", "b").iterator()::next) - assertTrue(store.add(SlotType.BAN, SlotTeam.ALLY, rect(0.1f), index = 1).isSuccess) - val duplicate = store.add(SlotType.BAN, SlotTeam.ALLY, rect(0.5f), index = 1) - assertEquals(SlotIssueCode.DUPLICATE_INDEX, duplicate.exceptionOrNull()?.issueCode) -} - -@Test fun `move clamps rectangle to normalized bounds`() { - val store = SlotAnnotationStore(idGenerator = { "slot" }) - val slot = store.add(SlotType.PICK, SlotTeam.ENEMY, NormalizedRect(0.8f, 0.1f, 0.95f, 0.4f)).getOrThrow() - store.move(slot.id, deltaX = 0.5f, deltaY = 0f) - assertEquals(1.0f, store.slots().single().rect.right, 0.0001f) -} - -@Test fun `codec round trip preserves ids and rect`() { ... } -@Test fun `codec rejects unsupported schema`() { ... } -``` - -Cover add/update/delete/reset, stable IDs, team/type update, duplicate index, overlap severity, bounds, serialization/deserialization, and unsupported versions. - -- [ ] **Step 2: Run red tests** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.annotation.*' --stacktrace` -Expected: FAIL because annotation classes do not exist. - -- [ ] **Step 3: Implement annotation model** - -`DraftSlot` is `@Serializable`, with `schemaVersion = 1`, `id: String`, `type: BAN|PICK`, `team: ALLY|ENEMY|GLOBAL`, `index: Int`, `rect: NormalizedRect`, and `label: String?`. Annotate `NormalizedRect` with `@Serializable`; enums are serialized as their names. - -`SlotAnnotationStore` exposes immutable snapshots: - -```kotlin -fun slots(): List -fun selectedSlotId(): String? -fun add(type, team, rect, index: Int? = null, label: String? = null): Result -fun move(id: String, deltaX: Float, deltaY: Float): Result -fun resize(id: String, rect: NormalizedRect): Result -fun updateTeam(id: String, team: SlotTeam): Result -fun updateType(id: String, type: SlotType): Result -fun updateIndex(id: String, index: Int): Result -fun updateLabel(id: String, label: String?): Result -fun select(id: String?) -fun delete(id: String): Boolean -fun reset() -fun validate(): List -``` - -Index uniqueness is `(type, team, index)`. Overlap across all rects is reported; same `(type, team)` overlap is `ERROR`, cross-scope overlap is `WARNING`. `resize` accepts a new valid `NormalizedRect`; Compose computes it from handles. Minimum rectangle size is `0.01` in both axes. IDs are injected and never reused. - -`SlotAnnotationCodec` uses `kotlinx.serialization-json` with `ignoreUnknownKeys = false`; `schemaVersion != 1` throws `SlotSchemaException`. Decode validates rectangles and duplicate indexes; invalid JSON throws a codec exception with a user-readable message. - -- [ ] **Step 4: Run tests** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.annotation.*' --stacktrace` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add app/build.gradle.kts gradle/libs.versions.toml app/src/main/kotlin/com/draftlens/vision/annotation app/src/test/kotlin/com/draftlens/annotation -git commit -m "feat: add manual draft slot annotations" -``` - ---- - -### Task 6: Versioned calibration profiles with rejection rules - -**Files:** -- Create: `app/src/main/kotlin/com/draftlens/vision/calibration/CalibrationProfile.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/calibration/CalibrationCompatibility.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/calibration/CalibrationProfileCodec.kt` -- Create: `app/src/main/kotlin/com/draftlens/vision/calibration/CalibrationRepository.kt` -- Create: `app/src/main/kotlin/com/draftlens/datastore/CalibrationProfileDataStore.kt` -- Test: `app/src/test/kotlin/com/draftlens/calibration/CalibrationCodecCompatibilityTest.kt` - -**Interfaces:** -- Consumes: `OrientationCategory`, slots from Task 5. -- Produces: `CalibrationProfile`, `CalibrationRepository`, `CalibrationLoadResult`, Android DataStore implementation. - -- [ ] **Step 1: Write failing compatibility and codec tests** - -```kotlin -@Test fun `equal ratio and orientation is compatible`() { - val profile = profile(targetAspectRatio = 16.0 / 9.0, expectedOrientation = OrientationCategory.LANDSCAPE) - assertEquals( - CalibrationCompatibility.Compatible, - CalibrationCompatibility.check(profile, orientedWidth = 2400, orientedHeight = 1080), - ) -} - -@Test fun `ratio mismatch returns expected and actual values`() { - val result = CalibrationCompatibility.check(profile(16.0 / 9.0, LANDSCAPE), 1080, 1920) - assertTrue(result is CalibrationCompatibility.Incompatible) -} - -@Test fun `future schema is rejected and delete remains possible`() { - val document = """{"schemaVersion":2,"profiles":[]}""" - assertThrows(UnsupportedCalibrationSchemaException::class.java) { - CalibrationProfileCodec.decodeDocument(document) - } -} -``` - -- [ ] **Step 2: Run failing tests** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.calibration.*' --stacktrace` -Expected: FAIL because calibration classes do not exist. - -- [ ] **Step 3: Implement model, codec, repository** - -`CalibrationProfile` fields: `schemaVersion = 1`, `profileId`, `label?`, `targetAspectRatio: Double`, `expectedOrientation: OrientationCategory`, `createdAtMs`, `updatedAtMs`, `slots: List`. `CalibrationDocument` is `schemaVersion = 1` and `profiles: List`. Target ratio is always `orientedWidth / orientedHeight`, where oriented means after pixel-orientation correction and before crop or scale. Compatibility tolerance is 1% and orientation must match exactly; report expected vs actual values for UI. - -`CalibrationRepository` interface: - -```kotlin -val profiles: Flow -suspend fun upsert(profile: CalibrationProfile): Result -suspend fun delete(profileId: String): Result -suspend fun clearDocument(): Result -``` - -`CalibrationProfileCodec` accepts only envelope/profile/slot schema versions equal to 1. Unknown future versions throw `UnsupportedCalibrationSchemaException`; malformed JSON throws `CalibrationDocumentException`; neither is converted to an empty list. - -`CalibrationProfileDataStore` stores one JSON string under `profiles_json` in `draftlens_calibration`. Every mutating operation reads and decodes the current document first; on unsupported/corrupt data it returns failure without writing. `clearDocument()` is the only operation available for a too-new/corrupt document and is used by the "Delete all calibration data" action. It never writes screenshots, bitmaps, transform IDs, or pixels. Preferences backup exclusion already covers DataStore; no manifest permission is added. - -- [ ] **Step 4: Run tests** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.calibration.*' --stacktrace` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add app/src/main/kotlin/com/draftlens/vision/calibration app/src/main/kotlin/com/draftlens/datastore app/src/test/kotlin/com/draftlens/calibration -git commit -m "feat: add versioned calibration profiles" -``` - ---- - -### Task 7: Vision frame source and explicit processing state - -**Files:** -- Create: `app/src/main/kotlin/com/draftlens/vision/model/VisionFrameSource.kt` -- Create: `app/src/main/kotlin/com/draftlens/capture/CaptureVisionFrameSource.kt` -- Create: `app/src/main/kotlin/com/draftlens/feature/visionlab/VisionLabUiState.kt` -- Create: `app/src/main/kotlin/com/draftlens/feature/visionlab/VisionLabViewModel.kt` -- Test: `app/src/test/kotlin/com/draftlens/capture/CaptureVisionFrameSourceTest.kt` -- Test: `app/src/test/kotlin/com/draftlens/feature/visionlab/VisionLabViewModelTest.kt` -- Modify: `app/src/main/kotlin/com/draftlens/DraftLensApp.kt` - -**Interfaces:** -- Consumes: `CaptureStateHolder`, `CapturedFrame`, `OriginalFrame`, `FramePreprocessor`, `CalibrationRepository`, annotation store. -- Produces: `VisionSourceState`, `VisionFrameSource`, `VisionLabViewModel`, freeze/process/annotation actions. - -- [ ] **Step 1: Write failing source and viewmodel tests** - -```kotlin -@Test fun `adapter maps paused capture to paused vision state`() = runTest { - val frames = MutableStateFlow(testCapturedFrame()) - val capture = MutableStateFlow(CaptureSessionState(status = CaptureStatus.PAUSED)) - val source = CaptureVisionFrameSource(frames, capture, backgroundScope) - assertEquals(VisionSourceState.PAUSED, source.state.first()) - assertEquals("test-frame", source.frames.first()?.id?.value) -} - -@Test fun `freeze pins current frame while live frames continue`() = runTest { ... } - -@Test fun `stale processing result is discarded after a newer request`() = runTest { - val preprocessor = BlockingFakePreprocessor() - val vm = VisionLabViewModel(source, preprocessor, fakeCalibrationRepository()) - vm.processFrame() - vm.processFrame() - preprocessor.releaseFirst() - assertEquals("second", vm.uiState.value.processed?.sourceId?.value) -} -``` - -- [ ] **Step 2: Run red tests** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.capture.CaptureVisionFrameSourceTest' --tests 'com.draftlens.feature.visionlab.VisionLabViewModelTest' --stacktrace` -Expected: FAIL because seam and ViewModel do not exist. - -- [ ] **Step 3: Implement source seam and viewmodel** - -`CaptureVisionFrameSource` is the only code importing both capture and vision. It maps ` CapturedFrame.toVisionFrame()` and maps `CaptureStatus` to `VisionSourceState`: `CAPTURING`, `PAUSED`, `IDLE/STOPPING→STOPPED`, `ERROR→ERROR`, `AWAITING_PERMISSION/STARTING→UNKNOWN`. It accepts injected flows and scope for tests; production defaults read the singleton holder. - -`VisionLabViewModel` holds: - -- `liveFrame` and `sourceState` from source -- `frozenFrame: OriginalFrame?` -- `displayFrame = frozenFrame ?: liveFrame` -- `processedFrame: ProcessedFrame?` -- `processStatus`, `processError`, `processingGeneration: Long` -- `SlotAnnotationStore` state -- `CalibrationLoadResult` and profile error/success messages -- selected `maxProcessedDimension` (default 1280) - -Freeze stores the displayed `OriginalFrame`; resume clears it; clear releases the reference. `processFrame()` selects frozen frame or latest live frame, captures it locally, and starts one job on `Dispatchers.Default`. A new request increments `processingGeneration` and cancels the prior job; only a result whose generation is still current is published. `onCleared` cancels the job. No automatic process call exists. - -`DraftLensApp` exposes `calibrationProfiles: CalibrationRepository` lazily. The ViewModel does not instantiate Android APIs itself. Profile create/update/delete actions call the injected repository and refresh `profiles`; a newer-than-supported envelope leaves delete-all available and save/update disabled. - -- [ ] **Step 4: Run tests** - -Run: `./gradlew :app:testDebugUnitTest --tests 'com.draftlens.capture.CaptureVisionFrameSourceTest' --tests 'com.draftlens.feature.visionlab.VisionLabViewModelTest' --stacktrace` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add app/src/main/kotlin/com/draftlens/vision/model/VisionFrameSource.kt app/src/main/kotlin/com/draftlens/capture/CaptureVisionFrameSource.kt app/src/main/kotlin/com/draftlens/feature/visionlab app/src/main/kotlin/com/draftlens/DraftLensApp.kt app/src/test -git commit -m "feat: wire explicit frame processing into vision lab state" -``` - ---- - -### Task 8: Compose Vision Lab with synchronized preview and overlays - -**Files:** -- Modify: `app/src/main/kotlin/com/draftlens/MainActivity.kt` -- Create: `app/src/main/kotlin/com/draftlens/feature/visionlab/VisionLabScreen.kt` -- Create: `app/src/main/kotlin/com/draftlens/feature/visionlab/AnnotationCanvas.kt` -- Create: `app/src/main/kotlin/com/draftlens/feature/visionlab/SlotEditor.kt` -- Create: `app/src/main/kotlin/com/draftlens/feature/visionlab/ProfileControls.kt` -- Modify: `app/src/main/kotlin/com/draftlens/feature/home/DebugCaptureScreen.kt` - -**Interfaces:** -- Consumes: `VisionLabViewModel` state and actions. -- Produces: user-visible tab UI, interactive canvas, profile controls. - -- [ ] **Step 1: Capture the current screenshot or layout reference** - -No product screenshot is committed. Use the existing debug UI and this plan/design as the visual source. Do not add copied game imagery. If a design mock is needed for local iteration, keep it under `/tmp/opencode` and out of git. - -- [ ] **Step 2: Implement MainActivity tabs** - -`MainActivity` owns `rememberSaveable { mutableIntStateOf(0) }` and a `TabRow` for Debug Capture (index 0) and Vision Lab (index 1). Each tab is a full-height screen. Capture remains alive regardless of selected tab. Use Material 3 color scheme already configured by MaterialTheme/Dark default; no new navigation dependency. - -- [ ] **Step 3: Implement VisionLabScreen sections** - -Build a scrollable column with these cards: - -1. **Frame Preview** — image or empty state; original dimensions; display rotation; processing correction; processed dimensions; duration; source state; `LIVE`/`FROZEN`; Freeze/Resume; Process Frame; Clear. -2. **Annotation Canvas** — interactive card described below. -3. **Slot Inspector** — list with ID, type, team, index, coordinates, validation. -4. **Processing Inspector** — source/output dims, rotation, crop, scale, duration, validation errors, live/frozen status. Show “Detection: not implemented” text; no confidence or hero fields. Include a "Max processed dimension" dropdown/text setting (default 1280) so processing resolution is configurable. -5. **Profiles** — label text field, save, list, apply, update, delete, delete-all when schema is unsupported, and mismatch message with expected/actual ratio and orientation. - -The preview image box uses the aspect ratio of the oriented frame and `ContentScale.Fit`. If only a `PixelArrayFramePayload` is present, show a neutral test-fixture placeholder instead of crashing; production emits Bitmap payloads. - -- [ ] **Step 4: Implement annotation canvas** - -`AnnotationCanvas` computes its fitted image rectangle from constraints and frame aspect ratio; all drawing and hit testing use that rectangle. It draws: - -- bitmap with unmodified pixels, -- translucent selected fill, outlined rectangles, 1 dp strokes, -- four corner handles on the selected rectangle, -- label `A-B1`, `E-P2`, `G-B1` from team and type/index. - -Gestures: tap select/topmost or deselect; drag inside moves (clamped); drag corner handle resizes; drag empty canvas creates a pending rect and commits above `0.01` normalized size; no source pixels are read/written beyond rendering. Pointer positions map display→oriented→source through `FrameTransform.create(..., crop = null, processedWidth = orientedWidth, processedHeight = orientedHeight)`. - -Overlay labels use uppercase team initial, `-`, type initial, index (`A-B2`). Canvas exposes selected slot ID to the parent. - -- [ ] **Step 5: Implement accessible editor and profile controls** - -`SlotEditor` displays each slot as a card with select/delete buttons, team/type/index controls (`FilterChip` rows), label text field, and editable normalized `left/top/right/bottom` OutlinedTextFields. Apply validates bounds, duplicate index, and overlap; errors shown inline. Include "Reset annotations" with a confirmation dialog. Slot metadata and geometry are fully editable without canvas gestures. - -`ProfileControls` disables save/apply for incompatible/corrupt states; apply shows exact compatibility result; delete requires confirmation; too-new schema shows only "Delete all calibration data". Use monospace text for rectangle values where useful. - -- [ ] **Step 6: Build and manually verify against existing debug workflow** - -Run: `./gradlew :app:compileDebugKotlin --stacktrace` -Expected: BUILD SUCCESSFUL. - -Run the app only if a device/emulator is actually available; do not claim device verification otherwise. - -- [ ] **Step 7: Commit** - -```bash -git add app/src/main/kotlin/com/draftlens/MainActivity.kt app/src/main/kotlin/com/draftlens/feature/visionlab app/src/main/kotlin/com/draftlens/feature/home/DebugCaptureScreen.kt -git commit -m "feat: add Vision Lab annotation interface" -``` - ---- - -### Task 9: Documentation and project notes - -**Files:** -- Create: `docs/VISION_ARCHITECTURE.md` -- Create: `docs/VISION_FRAME_CONTRACT.md` -- Create: `docs/VISION_COORDINATES.md` -- Create: `docs/VISION_LAB.md` -- Create: `docs/VISION_TESTING.md` -- Modify: `docs/ARCHITECTURE.md` -- Modify: `README.md` -- Modify: `tasks/plan.md` -- Modify: `tasks/todo.md` - -Before writing docs, record the wall-clock duration of the JVM test task and the output of the geometry/preprocessing test filters, for example: - -```bash -TIMEFORMAT='%R seconds' -time ./gradlew test --stacktrace -./gradlew :app:testDebugUnitTest --tests 'com.draftlens.vision.geometry.GeometryTest' --tests 'com.draftlens.preprocessing.*' --stacktrace -``` - -Use only measurements actually obtained; if a command fails, record the failure instead of a duration. - -**Interfaces:** -- Consumes: implementation results and CI run state. -- Produces: accurate docs, no fake verification claims. - -- [ ] **Step 1: Write new Vision documents** - -Each document states what exists, what does not, and what was actually run. Include: - -- data flow and package boundaries, -- payload ownership/lifetime and capture adapter, -- coordinate conventions and rotation correction vs display rotation, -- preprocessing limits and honest failure behavior, -- annotation model and profile schema/1% tolerance, -- freeze/resume, explicit process-only operation, -- test inventory, synthetic fixtures only, -- CI/APK status and limitations, -- deliberate exclusions: no recognition, no OCR, no screenshots persisted, no upload. - -Do not cite `docs/PRIVACY.md`; it was removed. Do not claim physical-device tests unless run. - -- [ ] **Step 2: Update README and architecture docs** - -README gets Sprint 1A header, Vision Lab status, tab workflow, and build commands. `docs/ARCHITECTURE.md` lists Vision packages and the capture→Vision adapter boundary. Both state Debug Capture remains available and no automatic detection exists. - -- [ ] **Step 3: Update task lists** - -Add Sprint 1A checklist with actual executed commands and leave device verification explicitly unchecked/pending unless run. Keep prior Sprint 0.1 history unchanged. - -- [ ] **Step 4: Commit docs** - -```bash -git status --short -git add README.md docs tasks -git commit -m "docs: document Sprint 1A vision foundation" -``` - ---- - -### Task 10: Full verification, PR, CI, APK, report - -**Files:** -- Modify: files from Task 9 only if verification reveals stale docs. -- No new feature files. - -- [ ] **Step 1: Run full JVM tests locally** - -Run: `./gradlew test --stacktrace` -Expected: all unit tests pass. If aarch64/SDK mismatch blocks an Android task, record the exact failure and rely on CI; do not change assertions to fit the host. - -- [ ] **Step 2: Run CI-equivalent assembly/lint locally only if host supports it** - -Run: `./gradlew lint assembleAndroidTest assembleDebug --stacktrace` -Expected: success on compatible host; if host fails due to x86_64 build-tools, record it and do not claim local build. - -- [ ] **Step 3: Inspect diff and assert no prohibited artifacts** - -```bash -git status --short -git diff --check -git diff --stat main...HEAD -``` - -Verify no screenshots, no image assets, no network client, no `FLAG_SECURE`, no generated build output. - -- [ ] **Step 4: Push branch and open PR** - -```bash -git push -u origin sprint-1-vision-foundation -gh pr create --base main --head sprint-1-vision-foundation --title "Sprint 1A: Vision foundation and Vision Lab" --body "" -``` - -Do not merge. If PR already exists, update it rather than create a duplicate. - -- [ ] **Step 5: Watch CI and capture artifact** - -Use `gh pr checks --watch` or the OpenChamber PR watch for the new PR. Record run URL. When green: - -```bash -gh run list --branch sprint-1-vision-foundation --limit 5 -gh run view --json url,conclusion,workflowName -``` - -Find `draftlens-debug` artifact URL from the run. If CI fails, fix the smallest cause, commit, push, and watch again; do not bypass lint or tests. - -- [ ] **Step 6: Update docs with actual CI result and commit** - -Replace any pending CI language in `docs/VISION_TESTING.md` or task lists with the run URL and outcome. Commit: - -```bash -git add README.md docs tasks -git commit -m "docs: record Sprint 1A CI verification" -git push -``` - -- [ ] **Step 7: Final report** - -Report: changed files, commits, tests actually run, CI run link, APK artifact link, merged status (not merged), limitations, and exact statement that instrumentation tests were compiled but not run unless an emulator/device was used. +# Sprint 1A implementation plan (archived) + +**Status:** completed and merged as PR #2. Kept as a record; current behaviour is documented +in [Architecture](../../ARCHITECTURE.md) and [Vision architecture](../../VISION_ARCHITECTURE.md). + +## Goal + +Give capture an image-bearing foundation without claiming automatic detection: non-null frame +contracts, tested preprocessing and geometry, manual draft-slot annotation, a Vision Lab and +versioned calibration profiles. + +## Shape of the work + +1. Pure Kotlin foundation — frame enums and ids, `FramePayload`, normalized and pixel value + types, geometry with rotation round trips. +2. Capture adaptation — non-null captured frames carrying display rotation; a narrow + `VisionFrameSource` seam so Vision never imports the capture service. +3. Preprocessing — a validator that returns failure codes instead of throwing, pure pixel-path + math and an Android renderer using the same matrix elements. +4. Annotation — `DraftSlot`, a validating store with stable ids, overlap and duplicate checks, + undo/redo, and a strict JSON codec that refuses unknown fields and unsupported schemas. +5. Calibration — versioned profiles with aspect-ratio tolerance, exact orientation match and + a delete-all escape hatch for documents this build cannot read. +6. UI — Vision Lab with frame preview, freeze, explicit processing, annotation canvas, slot + editor and profile controls. +7. Documentation and CI — package docs, test inventory, unit tests, strict lint and an APK + artifact. + +## Decisions retained + +Single `:app` module; pure logic in JVM-testable packages; latest-frame-only publication; no +bitmap recycling while Compose may render; metadata-only persistence with no pixels or +screenshots. These decisions still hold and are described in ADR-001 and the current docs. diff --git a/docs/superpowers/plans/2026-11-16-sprint-1c-app-only-capture.md b/docs/superpowers/plans/2026-11-16-sprint-1c-app-only-capture.md index 74f6ced..73a0998 100644 --- a/docs/superpowers/plans/2026-11-16-sprint-1c-app-only-capture.md +++ b/docs/superpowers/plans/2026-11-16-sprint-1c-app-only-capture.md @@ -1,149 +1,29 @@ -# Sprint 1C — App-Only Capture & Circular Floating Control Implementation Plan +# Sprint 1C implementation plan (archived) -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +**Status:** completed and merged as PR #4. Current behaviour is documented in +[Screen capture](../../SCREEN_CAPTURE.md). -**Goal:** A maintainer can freeze/resume MLBB capture from a small circular overlay button without leaving the game, with single-app capture preferred and the last valid game frame preserved when MLBB is hidden. +## Goal -**Architecture:** A pure-Kotlin visibility policy gates frame publication in the existing `FramePipeline` (replacing the implicit "only freeze gates" model); the capture service feeds it from `onCapturedContentVisibilityChanged`. A thin `TYPE_APPLICATION_OVERLAY` WindowManager view (attached only during capture sessions) calls the existing `CaptureFreeze` path on tap. Position/snap math is pure Kotlin + unit-tested. +Let a maintainer freeze and resume MLBB capture from a small circular overlay without leaving +the game, with single-app capture preferred and the last valid game frame preserved when MLBB +is hidden. No recommendation UI of any kind. -**Tech Stack:** Kotlin, Compose, WindowManager, MediaProjection (API 34+ callback), DataStore preferences, JUnit4 (JVM) + compiled-only instrumented tests. +## Shape of the work -**Spec:** `docs/superpowers/specs/2026-11-16-sprint-1c-app-only-capture-design.md` (+ the maintainer Sprint 1C brief, expanded in `docs/sprints/SPRINT_1C_APP_ONLY_CAPTURE.md`) +1. A pure `CaptureVisibilityPolicy` that decides whether a captured window's visibility + permits publication, with a JVM truth table. +2. Wire `onCapturedContentVisibilityChanged` (API 34+) into the capture guard and the frame + pipeline; keep acquiring and closing images while publication is blocked. +3. Normalized overlay position math with edge snapping, persisted in DataStore. +4. A one-button `WindowManager` overlay attached only during a session, with tap to toggle and + long-press drag to move; requested permission only on explicit enable. +5. Service attach/detach on every teardown path, rotation re-anchor, and capture-mode guidance. +6. Tests for the policy, position math, freeze/resume/visibility sequences and overlay + attachment; documentation and device-checklist rows. -## Global Constraints +## Decisions retained -- minSdk 26, compileSdk/targetSdk 35, JDK 17, AGP 8.6.1, Gradle 8.7 — unchanged. -- Never force single-app mode; both capture modes must keep working. -- No new third-party dependencies. -- The overlay is one circular button ≤56dp; no panel, cards, menu, toolbar, fullscreen overlay, accessibility service, or game touches. -- `SYSTEM_ALERT_WINDOW` requested only when the user enables the floating control. -- Anti-goals enforced: no masking of captured content, no screenshot cropping of shade content, no silent exclusion claims. -- Build of record: GitHub Actions `test` → `assembleAndroidTest` → `lint` → `assembleDebug`; do not merge; PR base is `sprint-1b-hero-recognition`. -- Keep Sprint 1B history intact; never modify `main`. - -## Review Focus - -1. Black/hidden-app frames replacing the pinned frame (single-app capture while MLBB is backgrounded) — pinned by Task 1 policy tests + Task 2 pipeline wiring + Task 7 docs matrix row. -2. Visibility callback racing a resize re-creation (both fire on rotation/visibility flips) — Task 2 keeps `recreateDisplay` behind the same CAPTURING guard; Task 1 tests the gate truth table. -3. Overlay persisting stale position after rotation or after density change — Task 3 normalized math + Task 6 reposition on display metrics; pinned by Task 3 tests. -4. Overlay permission denied leaving a "dead" enable toggle — Task 4 decision helper tests + Task 6 screen copy. -5. Service stopped while frozen leaving an orphaned overlay attached — Task 6 removal in the shared teardown path; pinned by Task 8 holder/freeze tests where JVM-reachable. - ---- - -### Task 1: Pure visibility/publication gate - -**Files:** -- Create: `app/src/main/kotlin/com/draftlens/capture/CaptureVisibilityPolicy.kt` -- Test: `app/src/test/kotlin/com/draftlens/capture/CaptureVisibilityPolicyTest.kt` - -**Interfaces:** -- Produces: `CaptureVisibilityPolicy` (class) — `fun isContentVisible(): Boolean` (initial true), `fun onContentVisibilityChanged(visible: Boolean): VisibilityTransition`, `fun isContentVisibleNow(): Boolean`, `fun shouldPublish(isActive: Boolean, generationCurrent: Boolean, frozen: Boolean): Boolean`; `data class VisibilityTransition(val wasVisible: Boolean, val isVisible: Boolean)` with `val changed: Boolean`. Used by Tasks 2 and 6. - -- [ ] **Step 1: Write failing tests** — truth table: initially visible; hidden→visible transitions; `shouldPublish` = active && generationCurrent && !frozen && visible; `shouldPublish` false when hidden even if not frozen; transitions report was/is. -- [ ] **Step 2: Run** `./gradlew testDebugUnitTest --tests '*CaptureVisibilityPolicyTest*'` → FAIL (class missing). (Local host has no JDK — delegate this run to CI if local Gradle is unavailable; record that.) -- [ ] **Step 3: Implement** the policy as a single class with a lock, no Android imports. -- [ ] **Step 4: Re-run** the same filter → PASS. -- [ ] **Step 5: Commit** `feat: capture visibility gate (pure policy) for app-only windows` - -### Task 2: Pipeline visibility gate + service callback wiring - -**Files:** -- Modify: `app/src/main/kotlin/com/draftlens/capture/FramePipeline.kt` — add `@Volatile`-backed policy `val visibilityPolicy`, `fun onContentVisibilityChanged(visible: Boolean): VisibilityTransition`, fold `shouldPublish` into the image-available listener (next to the `frozen` checks) and into the publish-time `synchronized(freezeGate)` block. -- Modify: `app/src/main/kotlin/com/draftlens/capture/CaptureSessionService.kt` — override `onCapturedContentVisibilityChanged(isVisible: Boolean)` in `projectionCallback` (API 34+); route to `pipeline.onContentVisibilityChanged`; log; call `overlayController?.onContentVisibilityChanged` if added in Task 6 (keep one call site); no release of any resource on hide (projection stays valid); ignore when not CAPTURING. -- Test: covered by Task 1 truth-table (pipeline is Android-bound) + instrumented compile in Task 7. - -**Key points:** images acquired & closed while hidden; no conversion; `_latestFrame` untouched; resize path unchanged and still allowed during freeze; the visibility gate adds a second, independent condition — never replace latest valid frame with an image acquired while hidden. - -- [ ] **Step 1:** Implement pipeline gate (one flag set, two check sites). -- [ ] **Step 2:** Implement `onCapturedContentVisibilityChanged` override (mirror `onCapturedContentResize` style; guard the CAPTURING state like the resize path does). -- [ ] **Step 3:** Commit `feat: gate frame publication on captured-content visibility (API 34+)` - -### Task 3: Overlay position math (pure) - -**Files:** -- Create: `app/src/main/kotlin/com/draftlens/overlay/OverlayPosition.kt` -- Test: `app/src/test/kotlin/com/draftlens/overlay/OverlayPositionTest.kt` - -**Interfaces:** -- Produces: `OverlayPosition` — `data class Normalized(val x: Float, val y: Float)` (0..1 fraction of free axis, edge-agnostic), `data class SnapResult(val edge: HorizontalEdge, val normalizedX: Float, val normalizedY: Float)`, `enum class HorizontalEdge { LEFT, RIGHT }`; functions: `pixelsToNormalized(pxLeft: Int, pxTop: Int, screenWidth: Int, screenHeight: Int, sizePx: Int): Normalized`, `normalizedToPixels(n: Normalized, screenWidth: Int, screenHeight: Int, sizePx: Int): Pair`, `clampToScreen(...)`, `snapToNearestEdge(leftPx: Int, topPx: Int, screenWidth: Int, sizePx: Int): Pair` (snaps x to `0` or `screenWidth - sizePx`, keeps y clamped), `snapNormalized(n: Normalized, screenWidth: Int, screenHeight: Int, sizePx: Int)`. All pure, no Android imports (density handled by callers in px). Used by Tasks 4 and 6. - -- [ ] **Step 1: Write failing tests** — round-trip px→normalized→px; clamp leaves in-screen positions untouched; out-of-screen clamps are pulled in; `snapToNearestEdge` returns LEFT below/at half and RIGHT above; y untouched by snap; landscape→portrait round-trip keeps the normalized position and changes only px; all functions handle zero/degenerate sizes without NaN. -- [ ] **Step 2:** Run filter → FAIL. -- [ ] **Step 3:** Implement. -- [ ] **Step 4:** Re-run → PASS. -- [ ] **Step 5:** Commit `feat: normalized overlay position with edge-snapping math` - -### Task 4: Overlay enablement decision + preferences - -**Files:** -- Create: `app/src/main/kotlin/com/draftlens/overlay/OverlayEnablement.kt` — pure `fun decideOverlayRequest(canDrawOverlays: Boolean, overlayEnabledPreference: Boolean): OverlayRequestOutcome` with `sealed interface` outcomes: `Show`, `RequestPermission`, `Disabled`. Used by the screen (Task 6). -- Modify: `app/src/main/kotlin/com/draftlens/datastore/AppPreferencesRepository.kt` — add keys `floating_control_enabled` (bool), `floating_position_x`/`floating_position_y` (float); flows `floatingControlEnabled: Flow`, `floatingPosition: Flow`; suspend `setFloatingControlEnabled(Boolean)`, `setFloatingPosition(OverlayPosition.Normalized)`, `clearFloatingPosition()` not needed (keep minimal). -- Test: `app/src/test/kotlin/com/draftlens/overlay/OverlayEnablementTest.kt` — grant/deny truth tables; denial never implies enablement. - -- [ ] **Step 1:** Write failing decision tests → FAIL. -- [ ] **Step 2:** Implement + wire preferences (no new DataStore instance; reuse `draftlens_preferences`). -- [ ] **Step 3:** Re-run → PASS. -- [ ] **Step 4:** Commit `feat: overlay enablement decision and persisted position keys` - -### Task 5: Floating control view + WindowManager shell - -**Files:** -- Create: `app/src/main/res/drawable/ic_floating_live.xml`, `app/src/main/res/drawable/ic_floating_frozen.xml` (simple white vector glyphs on transparent background). -- Create: `app/src/main/kotlin/com/draftlens/overlay/FloatingControlView.kt` — circular 52dp view built in code: solid circle background (LIVE green `#1DB954`-like blue/green; FROZEN orange `#F57C00`-like), white glyph `ImageView` centered, content description "Freeze game frame"/"Resume live capture", high-contrast stroke for visibility on bright and dark scenes. -- Create: `app/src/main/kotlin/com/draftlens/overlay/OverlayController.kt` — owns `WindowManager` handle; `attach()`, `detach()`, `setFrozen(Boolean)` (repaint), `repositionForDisplay(screenWidthPx, screenHeightPx, sizePx)` using Task-3 math + persisted normalized position (default: RIGHT edge, upper third), `toggleFreeze()` calling `CaptureFreeze.freezeCurrentFrame()` / `CaptureFreeze.resumeLive()`. Touch handling in the view: DOWN → 400 ms long-press timeout (Handler); MOVE beyond 12dp before timeout → cancel gesture (no drag, no toggle); after timeout → drag (update `LayoutParams.x/y`); UP → snap to nearest edge, persist normalized position, `setFrozen(CaptureStateHolder.frozenFrame.value != null)`. TAP (no cancel, short) → `toggleFreeze()`. -- No new foreground service; the controller is attached/removed by `CaptureSessionService` only. - -- [ ] **Step 1:** Implement view + vectors (no automated test; instrumented in Task 7). -- [ ] **Step 2:** Implement controller with gesture state machine; keep < 300 lines. -- [ ] **Step 3:** Commit `feat: circular floating freeze button (WindowManager overlay)` - -### Task 6: Service + screen + manifest wiring - -**Files:** -- Modify: `app/src/main/kotlin/com/draftlens/capture/CaptureSessionService.kt` — `OverlayController?` field created when (user preference enabled) && (SYSTEM_ALERT_WINDOW granted); attach after `CaptureStateHolder.beginCapture(...)`; remove in `releaseCaptureResources()`; re-attach/update on existing display-listener metrics change; observe `frozenFrame` to repaint (the existing `frozenJob` can call `overlayController?.setFrozen(...)`); gate actual attach on `Build.VERSION.SDK_INT` not needed (TYPE_APPLICATION_OVERLAY API 26+, minSdk 26 ✓). -- Modify: `app/src/main/kotlin/com/draftlens/capture/CaptureController.kt` — no change (enablement flows through the preferences/DataStore, not the service intent). -- Modify: `app/src/main/AndroidManifest.xml` — add ``. -- Modify: `app/src/main/kotlin/com/draftlens/feature/home/DebugCaptureScreen.kt` — replace the Entire-screen guidance with the two-mode guide (single-app preferred with MLBB selection steps; full-display fallback with honest limitation); add a **Floating control** card: switch ("Show the freeze button while capture runs"), current LIVE/FROZEN state when overlay attached, denial explanation + "Open Android settings to enable drawing over other apps" button (`ACTION_MANAGE_OVERLAY_PERMISSION`, package-scoped), and a note that capture works fully without it. -- Modify: `app/src/main/kotlin/com/draftlens/feature/visionlab/VisionLabScreen.kt` (only if the guidance card lives there too — check; update the same capture-mode wording if present). - -- [ ] **Step 1:** Manifest + service attach/detach (teardown removes overlay on every end path). -- [ ] **Step 2:** Debug screen rewrite of the guide + floating-control card; keep Sprint 1B content intact otherwise. -- [ ] **Step 3:** Commit `feat: enable floating freeze control during capture; two-mode capture guidance` - -### Task 7: Instrumented tests (compile-only) - -**Files:** -- Create: `app/src/androidTest/kotlin/com/draftlens/FloatingControlOverlayInstrumentedTest.kt` — attaches the view via `OverlayController` under an `ActivityScenario`/service context, asserts `WindowManager` view added then removed after `detach()`, asserts `setFrozen` toggles glyph resource, skips gracefully with `Assume.assumeTrue(Settings.canDrawOverlays(context))` when overlay permission is absent (always, on CI emulators). - -- [ ] **Step 1:** Write the test; mirror the header/skip style of `CaptureLifecycleInstrumentedTest`. -- [ ] **Step 2:** Commit `test: instrumented overlay attach/removal (compile-only)` - -### Task 8: Remaining JVM coverage - -**Files:** -- Test: `app/src/test/kotlin/com/draftlens/capture/ServiceStopAndFreezeTest.kt` (or extend `CaptureFreezeTest`) — repeated freeze commands (AlreadyFrozen keeps first pin), freeze with no frame (NoFrame), stale command after stop (`hasActivePipeline` false → NoPipeline), `terminateSession` while frozen preserves the pin, `reset` clears it, `resumeLive` while "hidden" (policy) then visible resumes publication (policy truth table via Task 1), resize during freeze does not clear the frozen flag (policy-level: `shouldPublish` stays false with frozen=true across dimension deltas — assert the policy takes no dimensions as input). -- Verify existing `CaptureFreezeTest`, `CaptureStateHolderTest`, `FrameSnapshotsTest` pass unchanged; add only what's missing. - -- [ ] **Step 1:** Add the missing cases; run the full JVM suite in CI (no local toolchain). -- [ ] **Step 2:** Commit `test: freeze/visibility/stop edge cases for Sprint 1C` - -### Task 9: Documentation - -**Files:** -- Create: `docs/sprints/SPRINT_1C_APP_ONLY_CAPTURE.md` — objective, branch/PR, what shipped, visibility lifecycle section, overlay section, honest overlay/capture isolation statement, verification matrix (CI run id, device rows marked NOT RUN), limitations. -- Modify: `docs/SCREEN_CAPTURE.md` — rewrite the single-app (Android 14+ chooser guidance) and visibility sections; remove "onCapturedContentVisibilityChanged is not handled"; document overlay isolation unknowns. -- Modify: `docs/TESTING_PHYSICAL_DEVICE.md` — add rows: overlay appearance in single-app capture, overlay appearance in full-display capture, tap-freeze in MLBB, edge snap, rotation, denial path, shade test. -- Modify: `README.md`, `tasks/todo.md`, `tasks/plan.md`, `docs/IMPLEMENTATION_REPORT.md` — Sprint 1C sections; CI run id, unit totals, APK digest after the green run (fill in during Task 10, second docs commit). - -- [ ] **Step 1:** Write all docs except CI numbers; commit `docs: Sprint 1C capture-visibility and overlay documentation` - -### Task 10: CI, PR, report - -**Files:** none (verification only). - -- [ ] **Step 1:** Push branch; note the CI run id from `gh run list --branch sprint-1c-app-only-capture`. -- [ ] **Step 2:** If red: diagnose systematically (superpowers:systematic-debugging), fix, re-push. -- [ ] **Step 3:** Harvest unit totals from the run, APK digest from the artifact (`gh run download` + `sha256sum`), instrumented compiled-vs-executed status; fill into docs; commit `docs: record Sprint 1C CI results` and push. -- [ ] **Step 4:** `gh pr create --base sprint-1b-hero-recognition --title "Sprint 1C: App-only capture & circular floating control"` with a body covering requirements 1–11 (unmerged dependency on #3 stated; device verification NOT RUN). -- [ ] **Step 5:** Link the session to the PR (openchamber), name the task; final report with SHA, run id, unit totals, digest, instrumentation status, device status, limitations. Do not merge. +The overlay gate touches only frame publication, never the overlay's own window or touch +handling. The visibility policy is pure Kotlin. The notification and overlay derive their +state from one shared publish-state flow. See the current capture documentation for details. diff --git a/docs/superpowers/specs/2026-11-06-sprint-1a-vision-foundation-design.md b/docs/superpowers/specs/2026-11-06-sprint-1a-vision-foundation-design.md index dd9f008..8c8338c 100644 --- a/docs/superpowers/specs/2026-11-06-sprint-1a-vision-foundation-design.md +++ b/docs/superpowers/specs/2026-11-06-sprint-1a-vision-foundation-design.md @@ -1,572 +1,35 @@ -# Sprint 1A — Vision Foundation, Frame Preprocessing & Vision Lab Design +# Sprint 1A vision foundation design (archived) -Date: 2026-11-06 -Repository: `xeriomy/DraftLens` -Branch: `sprint-1-vision-foundation` -Base: `main` after PR #1 (`ae92eaf`) +**Status:** implemented and merged as PR #2. Current contracts are documented in +[Vision frame contract](../../VISION_FRAME_CONTRACT.md), +[Vision coordinates](../../VISION_COORDINATES.md) and +[Vision architecture](../../VISION_ARCHITECTURE.md). ## Intent -Make captured frames usable by independent computer-vision components without claiming automatic detection. Sprint 1A delivers a real image-bearing frame contract, rotation-aware preprocessing, tested normalized-coordinate geometry, manual slot annotation, a Vision Lab debugging screen, and versioned calibration profiles. - -This sprint deliberately does **not** implement OCR, hero recognition, automatic draft detection, ML models, overlays, or any network behavior. Manual annotations are calibration data, not detections. - -## Goals - -1. Replace metadata-only detection hooks with a production `VisionFrame` contract carrying real pixels or an explicit test payload. -2. Keep capture working: latest-frame-only ownership, no recycling while Compose may read a bitmap, no disk persistence. -3. Add a pure, JVM-tested geometry subsystem for source, normalized, oriented, cropped, scaled, and processed coordinates. -4. Add a preprocessing engine that validates input, applies right-angle rotation and optional crop, preserves aspect ratio, and reports dimensions, scale, and duration honestly. -5. Add manual `BAN` / `PICK` slot annotations with stable IDs, normalized rectangles, team/type metadata, validation, and JSON serialization. -6. Add a dedicated Compose Vision Lab behind tabs that keep Debug Capture one tap away. -7. Persist calibration metadata only in DataStore; never persist screen images. -8. Run CI as the build of record and deliver a debug APK artifact. - -## Non-goals and anti-goals - -- No OCR, LiteRT, TensorFlow, OpenCV, model training, hero catalog, recognition, or automatic ban/pick detection. -- No hero IDs, confidence values, or detection results may be fabricated. -- No automatic full-resolution processing on every frame by default. -- No writing screenshots to disk automatically. -- No upload, packet inspection, game-process access, automated touches, or `FLAG_SECURE`. -- No Room unless a concrete persisted entity is introduced; DataStore is sufficient for the profile envelope. -- No separate Gradle module in this sprint. - -## Constraints - -- Single Gradle module `:app`; logical packages only. -- Kotlin 2.0.20, AGP 8.6.1, Gradle 8.7, compileSdk/targetSdk 35, minSdk 26, JDK 17. -- Vision must not depend on `CaptureSessionService` or `CaptureStateHolder`. Capture supplies frames through a narrow `VisionFrameSource` interface; an adapter in capture wiring maps live capture state. -- Capture behavior must remain operational and keep its consent flow. -- Local host is aarch64 with x86_64 Android SDK tools; local `assembleDebug` is not the build of record. GitHub Actions runs test, Android-test compilation, strict lint, and debug assembly. -- CI must remain green; instrumentation tests are compiled but not claimed as executed without a device or emulator. - -## Approaches considered - -### A. Layered single-module foundation (selected) - -Pure Kotlin packages `vision/model`, `vision/geometry`, `preprocessing`, `annotation`, and `calibration` carry domain logic and tests. Android Bitmap operations, Compose UI, and DataStore stay at the edges. `CapturedFrame` is refactored to a non-null payload. A capture-side adapter implements `VisionFrameSource` without exposing capture services to Vision. - -Trade-offs: keeps the sprint within one module and reuses Compose/DataStore conventions; tests cover the parts most likely to break. It does not physically isolate Vision into a separate artifact yet. - -### B. Separate `:vision` Gradle module now - -Cleaner dependency graph immediately, but ADR-001 and this sprint require one module until a second feature justifies independent build and deploy. It adds build configuration before a stable interface exists. - -### C. UI-first prototype with core extracted later - -Fastest way to see rectangles move, but the UI would be built against temporary shapes and likely fake state. It conflicts with the requirement for a production contract and JVM-tested geometry before UI claims alignment. - -**Decision:** A. - -## Repository audit findings that shape the design - -- `CapturedFrame.bitmap` is nullable solely to keep JVM tests simple; production should not accept null pixels. -- `FramePipeline` already performs all pixel work on `HandlerThread("draftlens-capture")`, keeps one latest bitmap, and drops stale frames. -- `CaptureStateHolder` is a process-local singleton; stopping clears frame references; Compose must never see a recycled bitmap. -- `DetectionInterfaces.FrameRef` is metadata-only and cannot feed preprocessing or annotation. -- Tests can use synthetic images if frame metadata and payloads are decoupled from Android Bitmap. -- DataStore is already used for preferences and is an appropriate store for a small number of JSON profile documents. -- Static screens may not emit new frames under `AUTO_MIRROR`; UI must show freshness/live/frozen state honestly. -- `AUTO_MIRROR` and partial screen capture mean the Vision Lab must tolerate crops, empty states, and changing dimensions. - -## Package map - -```text -com.draftlens - capture/ - ... existing capture code - CaptureVisionFrameSource.kt # adapter: CaptureStateHolder/CapturedFrame -> VisionFrameSource - datastore/ - AppPreferencesRepository.kt # unchanged - CalibrationProfileDataStore.kt # profile JSON envelope, schema gate - models/ - CapturedFrame.kt # payload becomes non-null - vision/ - model/ # pure Kotlin contracts and metadata - model/android/ # Android Bitmap payload implementation - geometry/ # pure Kotlin normalized/pixel/transform tests - preprocessing/ # validation, sizing, Android Bitmap renderer - annotation/ # pure Kotlin slot store, JSON codec - calibration/ # pure profile compatibility + DataStore adapter - processing/ # explicit-frame processor interface and generation gate - feature/ - home/ - DebugCaptureScreen.kt # existing capture UI retained - visionlab/ - VisionLabScreen.kt - VisionLabViewModel.kt - AnnotationCanvas.kt -``` - -No empty packages or new Gradle modules are introduced. `vision/processing` exists only if the implementation needs a narrow explicit processor interface; otherwise the interface stays in `vision/model`. - -## Vision Input Contract - -### Types - -```kotlin -@JvmInline value class FrameId(val value: String) - -enum class PixelFormat { ARGB_8888 } -enum class FrameRotation { ROTATION_0, ROTATION_90, ROTATION_180, ROTATION_270 } -enum class OrientationCategory { PORTRAIT, LANDSCAPE, SQUARE } - -sealed interface FramePayload { - val width: Int - val height: Int - val format: PixelFormat - val byteCountEstimate: Long -} - -class PixelArrayFramePayload( - override val width: Int, - override val height: Int, - pixels: IntArray, - override val format: PixelFormat = PixelFormat.ARGB_8888, -) : FramePayload -``` - -`PixelArrayFramePayload` defensively copies pixels and exists for JVM tests and synthetic fixtures. - -Android implementation: - -```kotlin -package com.draftlens.vision.model.android - -class BitmapFramePayload( - val bitmap: Bitmap, - val ownership: BitmapOwnership = BitmapOwnership.SHARED_READ_ONLY, -) : FramePayload -``` - -`SHARED_READ_ONLY` means consumers may inspect and draw the bitmap but must not mutate or recycle it, and must drop their reference on freeze/reset/stop. Outputs created by preprocessing are ordinary shared-read-only bitmaps; ownership is descriptive, not a synchronized lifetime system. No Compose or Vision path calls `recycle()`. - -### Original, processed, and slot images - -```kotlin -data class OriginalFrame( - val id: FrameId, - val payload: FramePayload, - val sourceWidth: Int, - val sourceHeight: Int, - /** Display rotation at acquisition, recorded by capture for inspection. */ - val displayRotation: FrameRotation, - /** Correction needed to make the pixels upright. MediaProjection buffers are usually already display-oriented. */ - val rotation: FrameRotation, - val capturedAtElapsedRealtimeMs: Long, - val wallClockTimestampMs: Long, - val captureLatencyMs: Long?, -) - -data class ProcessedFrame( - val id: FrameId, - val sourceId: FrameId, - val payload: FramePayload, - val sourceWidth: Int, - val sourceHeight: Int, - val displayRotation: FrameRotation, - val rotation: FrameRotation, - val capturedAtElapsedRealtimeMs: Long, - val wallClockTimestampMs: Long, - val transform: FrameTransform, - val processingDurationMs: Long, -) - -data class SlotImage( - val id: FrameId, - val sourceId: FrameId, - val slotId: String, - val payload: FramePayload, - val sourceRect: PixelRect, - val createdAtElapsedRealtimeMs: Long, -) -``` - -All three implement the common `VisionFrame` marker interface or sealed contract. Metadata is immutable. Pixel payloads are non-null at every production boundary. The previous nullable `CapturedFrame.bitmap` is removed; tests construct `PixelArrayFramePayload`. - -`CapturedFrame` remains the capture-layer model and embeds `OriginalFrame` or exposes `toVisionFrame()`. It carries no density-derived coordinates. - -### Capture rotation wiring - -`CaptureSessionService` reads `WindowManager.defaultDisplay.rotation` at capture start and display recreation, maps `Surface.ROTATION_*` to `FrameRotation`, passes it through `FramePipeline`, and stores it on `CapturedFrame` as `displayRotation`. - -MediaProjection receives an already display-oriented buffer, so production frames set `rotation = ROTATION_0` and keep `displayRotation` for inspection. Double-rotating an upright capture is never implicit. Synthetic fixtures and future authorized imports may set a non-zero `rotation`, and the preprocessing/geometry code applies that correction normally. This keeps display orientation visible without pretending MediaProjection pixels need another transform. - -### Frame source seam - -```kotlin -enum class VisionSourceState { IDLE, CAPTURING, PAUSED, ERROR, STOPPED, UNKNOWN } - -interface VisionFrameSource { - val frames: StateFlow - val state: StateFlow -} -``` - -`CaptureVisionFrameSource` in `capture/` maps `CaptureStateHolder.latestFrame` and `CaptureStateHolder.state` to this interface. Vision packages never import capture services. Tests use `StaticVisionFrameSource` with synthetic frames. - -## Coordinate conventions - -Normalized coordinates use source-image pixel axes as captured: `x` from left to right, `y` from top to bottom, both in `[0.0, 1.0]`. They are not display dp, not density-scaled, and not affected by crop origin. - -Rectangle rules: - -- `left < right` -- `top < bottom` -- all four coordinates in `[0.0, 1.0]` -- `NormalizedRect.EMPTY` is not valid for storage or input - -Rotation semantics: `rotation` is the pixel-orientation correction required to make the source upright. `displayRotation` records the device display rotation at acquisition and is not by itself a reason to transform already-upright MediaProjection pixels. The same 0/90/180/270 mappings apply to any non-zero correction. Source-to-oriented mappings are: - -| Rotation | Output dimensions | Source point `(x, y)` → oriented point `(x', y')` | -|---|---|---| -| 0° | `W × H` | `(x, y)` | -| 90° CW | `H × W` | `(H - 1 - y, x)` | -| 180° | `W × H` | `(W - 1 - x, H - 1 - y)` | -| 270° CW | `H × W` | `(y, W - 1 - x)` | - -Processing pipeline order is fixed: - -```text -validate → normalize orientation → optional crop on oriented image → resize without distortion -``` - -The crop rectangle is a normalized rectangle in **oriented** coordinates. The final `ProcessedFrame.transform` stores: - -- source dimensions, -- rotation, -- crop rectangle or identity crop, -- oriented dimensions, -- crop pixel rectangle, -- scale factor, -- processed dimensions. - -`FrameTransform` provides round trips: - -- `normalized ↔ sourcePixel` -- `sourcePixel ↔ orientedPixel` -- `orientedPixel ↔ cropPixel` -- `sourcePixel ↔ processedPixel` -- `sourceNormalizedRect ↔ processedPixelRect` -- inverse transforms with a documented floating-point tolerance - -Transformations are reversible when the output does not intentionally discard pixels. Crop and downscale are affine maps and remain reversible within tolerance. Anything below the configured processed resolution is not recovered by scaling up. - -## Preprocessing - -```kotlin -data class ProcessFrameRequest( - val maxProcessedDimension: Int = 1280, - val maxPixels: Long = 8_000_000, - val crop: NormalizedRect? = null, -) - -sealed interface ProcessFrameResult { - data class Success(val frame: ProcessedFrame) : ProcessFrameResult - data class Failure(val reason: PreprocessFailure) : ProcessFrameResult -} -``` - -Validation rejects: - -- width or height ≤ 0 -- pixel count > configured max -- source dimensions inconsistent with payload -- unsupported/non-ARGB format -- empty or out-of-range crop -- `left >= right` or `top >= bottom` -- output dimension ≤ 0 -- processed byte estimate above the configured limit - -`calculateOutputSize()` uses the oriented/cropped aspect ratio. The processed long side is bounded by `maxProcessedDimension`; smaller sources are never upscaled. Width and height are rounded to preserve aspect ratio within one pixel and to avoid stretching. - -Rendering: - -- Android path uses `Bitmap` with a destination sized to the final output and one canvas/matrix operation for rotation, crop, and scale. No intermediate full-size oriented bitmap is created. -- If rotation is 0°, crop is null, and scaling is not required, the result may reference the source payload instead of allocating a duplicate; the result still records an identity transform and honest duration. -- JVM path renders `PixelArrayFramePayload` using pure Kotlin for tests and synthetic fixtures. -- Processing runs on a dedicated `Dispatchers.Default` job, never on the UI thread, never on the capture HandlerThread, and only after explicit user request. -- One processing job may be active. A new request cancels or discards the older job; stale results cannot overwrite the current request. -- Duration uses monotonic nanoseconds (`System.nanoTime()`), not wall-clock time. -- Failure is returned as `ProcessFrameResult.Failure`; no partial bitmap is published as success. - -No strict memory bound is claimed. The docs record expected allocations and measured durations from synthetic fixtures, but do not claim stable 24/7 behavior without a physical-device soak test. - -## Draft Slot Annotation - -```kotlin -enum class SlotType { BAN, PICK } -enum class SlotTeam { ALLY, ENEMY, GLOBAL } - -data class DraftSlot( - val schemaVersion: Int = SlotSchema.CURRENT, - val id: String, - val type: SlotType, - val team: SlotTeam, - val index: Int, - val rect: NormalizedRect, - val label: String? = null, -) -``` - -`SlotAnnotationStore` is pure Kotlin and owns operations: - -- add with injected ID generator; default next index in the selected `(type, team)` scope -- move and resize with clamping to normalized bounds -- delete and select -- update team, type, index, label -- change index with duplicate validation in `(type, team)` scope -- overlap detection across all slots; same-scope overlap is an error, cross-scope overlap is a warning -- reset to empty -- JSON encode/decode with schema version - -Bounds are always validated. Rectangle creation requires a minimum size in normalized units (`0.01`), so accidental taps do not create slots. IDs are stable and are not recycled after deletion. - -Annotations store normalized source-frame coordinates. They are calibration metadata, never hardcoded device pixels and never presented as detected slots. The UI labels the card “Manual slots” and shows “Not detected” rather than a confidence value. - -## Calibration profiles - -### Schema - -```json -{ - "schemaVersion": 1, - "profileId": "uuid", - "label": "MLBB draft — 16:9 landscape", - "targetAspectRatio": 1.7777778, - "expectedOrientation": "LANDSCAPE", - "createdAtMs": 1762400000000, - "updatedAtMs": 1762400000000, - "slots": [ - { - "schemaVersion": 1, - "id": "uuid", - "type": "BAN", - "team": "ALLY", - "index": 1, - "rect": { "left": 0.05, "top": 0.10, "right": 0.14, "bottom": 0.28 }, - "label": null - } - ] -} -``` - -The enclosing DataStore document also has an envelope version. Profiles are stored in a dedicated `draftlens_calibration` preferences DataStore under one serialized JSON value. This is small metadata only; it never contains bitmaps, pixels, or screenshots. - -Rules: - -- Version `1` is accepted and written. -- `schemaVersion > CURRENT` is rejected with a clear error; the stored document is left untouched, `update/save` is refused, and deletion remains allowed as the escape hatch. There is no export action in this sprint. -- Corrupted JSON produces a visible error state, not an empty list that could overwrite user data. -- Target and candidate aspect ratios use **oriented dimensions**: source dimensions after `rotation` is applied, before crop and before scale. This describes the frame as the user sees and annotates it. -- Aspect ratio compatibility uses `abs(candidate - target) / target <= 0.01` (1%). -- Expected orientation must match the candidate’s `PORTRAIT` / `LANDSCAPE` / `SQUARE` classification exactly. -- Applying an incompatible profile is refused; the UI shows expected vs actual ratio/orientation and offers recalibration. -- Saving never applies silently. Apply, update, and delete are explicit actions. -- Created and updated timestamps use wall-clock milliseconds; frame transform IDs and bitmaps are not persisted. - -Required CRUD: - -- create profile from current annotations and selected reference frame metadata -- list profiles -- load/apply profile into the current annotation set after compatibility check -- update label/slots/timestamp -- delete profile - -## Vision Lab UI - -`MainActivity` uses a Material 3 `TabRow` with two destinations: `Debug Capture` and `Vision Lab`. Capture state lives in the existing service/holder, so switching tabs does not stop capture. Debug Capture remains visible and usable. No `FLAG_SECURE` is added. - -### A. Frame Preview - -Shows either the frozen frame or the latest live frame: - -- original width × height -- rotation -- processed width × height when available -- processing duration in milliseconds or `—` -- current source state (`CAPTURING`, `PAUSED`, `IDLE`, etc.) -- `LIVE` / `FROZEN` badge -- Freeze/Resume button; freeze pins the currently displayed frame in memory only -- Process Frame button; explicitly processes the frozen frame if any, otherwise the latest frame at press time -- Clear freeze control; releases the pinned reference - -Nothing is written to disk automatically. - -### B. Annotation Canvas - -The canvas draws the **oriented, uncropped** frame with `ContentScale.Fit` geometry inside a black rounded card. "Oriented" means `rotation` is applied to produce an upright view; crop is not part of the annotation viewport. Overlay coordinates are computed from the actual fitted rectangle, not from screen dp guesses. Pointer positions are converted through the frame transform to source-normalized annotation coordinates; if the underlying buffer is already display-oriented (`rotation = ROTATION_0`), the conversion is an affine fit of the source image. - -Interactions: - -- tap an existing rectangle to select it -- drag inside a rectangle to move it -- drag one of four corner handles to resize -- drag empty canvas to create a rectangle -- tap empty canvas to deselect -- delete via button or keyboard action -- overlay never redraws or mutates the source bitmap - -Each overlay shows a short label such as `A-B1`, `E-P2`, or `G-B1`. Selection is visible. The same annotation set is available below the canvas as text controls: select from list, edit normalized `left/top/right/bottom`, change type, team, index, label, and delete. This is the accessibility alternative; screen readers can edit all metadata without gesture-only resize. - -### C. Slot Inspector - -A list of manual slots with: - -- slot ID (short ID shown; full ID available) -- type and team -- index -- rectangle coordinates -- validation state and overlap/duplicate messages - -Editing metadata updates the in-memory annotation set immediately and invalidates any “profile saved” indicator. - -### D. Processing Inspector - -Displays actual values only: - -- source dimensions -- output dimensions -- rotation -- crop region (or `none`) -- scale factor -- processing duration -- validation errors or honest failure -- whether the displayed pixels are frozen or live - -No detection confidence, hero result, or draft phase is shown. The card explicitly says “Detection: not implemented”. - -### Profiles - -The screen includes profile controls: save current annotations, list profiles, apply compatible profile, update profile, delete profile, and show mismatch warnings. A profile action never silently changes slots. - -## State and ownership flow - -```text -MediaProjection → VirtualDisplay → ImageReader → capture HandlerThread - → Bitmap (latest only) → CaptureStateHolder → CaptureVisionFrameSource - → VisionLabViewModel (live or frozen reference) → VisionLabScreen - → explicit ProcessFrame request → background job → ProcessedFrame - → Compose overlay / inspectors -``` - -Freeze stores one extra bitmap reference. Resume releases it. Capture keeps running independently. Vision preprocessing does not share the capture HandlerThread or ImageReader. - -## Serialization and schema testing - -`kotlinx-serialization-json` is added for profile JSON. `SlotAnnotationCodec` and `CalibrationProfileCodec` are pure Kotlin and JVM-tested. Unknown future schema versions are rejected; malformed JSON is reported; valid v1 round-trips byte-for-byte at the object level; slot IDs survive round-trip; invalid rectangles are rejected after decode. - -## Testing - -### Geometry (JVM) - -- all four rotations -- portrait, landscape, and square classification -- source→normalized→source round trip -- source→processed→source round trip -- normalization after 90/180/270 rotation -- crop-aware mapping and inverse -- downscaled mapping and inverse -- invalid rectangles: reversed edges, out-of-range, NaN/infinity -- invalid image dimensions and inconsistent payload dimensions -- orientation category mismatch -- crop outside oriented image -- aspect-ratio mismatch calculation - -### Preprocessing (JVM via `PixelArrayFramePayload` + Bitmap compile checks) - -- valid synthetic input accepted -- zero/negative dimensions rejected -- inconsistent payload and metadata rejected -- aspect ratio preserved for portrait and landscape -- output respects long-edge and pixel-count limits -- no unnecessary upscale -- 0/90/180/270 metadata produces expected dimensions and pixels -- crop dimensions are correct and output aspect matches crop -- failure is returned honestly, without partial success -- duration is recorded and non-negative -- transform metadata matches output -- identity path does not create an unnecessary copy - -### Annotation and profiles (JVM) - -- add/update/delete/select -- stable IDs across edits -- duplicate type+team+index rejected -- bounds enforced for move/resize/create -- overlap validation -- JSON round-trip -- unsupported schema version rejected -- corrupt DataStore payload reported, not silently reset -- aspect-ratio/orientation compatibility accepts and rejects correctly - -### Capture regression - -- `CaptureStateHolderTest` fixtures now use `PixelArrayFramePayload`; null-bitmap tests are removed rather than preserved by special cases. -- Existing capture lifecycle and sampler tests remain green. -- Debug Capture remains available in the tab UI; no new capture permission flow. - -### CI - -CI continues to run: - -```sh -./gradlew test --stacktrace -./gradlew assembleAndroidTest --stacktrace -./gradlew lint --stacktrace -./gradlew assembleDebug --stacktrace -``` - -The APK artifact remains `draftlens-debug`. Instrumentation tests are not claimed as executed. - -## Documentation deliverables - -- `docs/VISION_ARCHITECTURE.md` — package boundaries, data flow, ownership, processing rules -- `docs/VISION_FRAME_CONTRACT.md` — contract, payload ownership, original/processed/slot images -- `docs/VISION_COORDINATES.md` — normalized coordinates, rotation tables, crop semantics, transforms -- `docs/VISION_LAB.md` — user-facing controls, freeze behavior, slot editing, profile rules -- `docs/VISION_TESTING.md` — test inventory, synthetic fixture policy, CI gates, physical-device limits -- `docs/ARCHITECTURE.md` — updated package map and Vision data flow -- `README.md` — Sprint 1A status and Vision Lab usage -- `tasks/plan.md` and `tasks/todo.md` — Sprint 1A checklist and verification state - -No real game screenshots are committed. Physical-device verification remains explicitly pending unless actually run. - -## Privacy and performance - -- Frames remain in memory; no automatic screenshot file, upload, or network client. -- No game-process access, packet inspection, taps, overlays outside the app, or hidden capture. -- Existing consent flow remains mandatory; session-scoped and per-session. -- Profiles persist only normalized rectangles and labels. -- No image is persisted for diagnostics without an explicit opt-in and review. -- Canvas overlay draws only current composition; it does not alter source pixels. -- Processing is explicit, background, latest/frozen only, and at most one job. -- Docs record measured synthetic processing times if obtained; no claim of 24/7 readiness or strict memory bounds. - -## Decided product and builder calls - -1. `TabRow` is used for the two current destinations instead of Navigation Compose. Add navigation only when a third top-level destination or independent back stack is required. -2. The calibration DataStore uses one serialized envelope to keep updates atomic and schema rejection all-or-nothing. -3. Aspect compatibility tolerance is 1%; it is displayed to the user and can be tightened later without schema change. -4. `SQUARE` is classified separately rather than squeezed into portrait or landscape. -5. The default processed long edge is 1280 px. It is configurable in advanced Vision Lab controls and recorded in test fixtures. -6. No automatic application of profiles, ever; compatibility failure blocks slot replacement. -7. Capture wires display rotation into frame metadata. MediaProjection buffers are already display-oriented, so production `rotation` correction is `ROTATION_0`; `displayRotation` records device orientation without double-rotating pixels. -8. The annotation canvas draws the oriented, uncropped frame. Crop belongs to processing/inspection and is not the annotation viewport. -9. Profile aspect ratio and expected orientation are derived from oriented dimensions, before crop and scale. -10. `SlotImage` is part of the production contract but has no UI or crop-export flow in this sprint. A `vision/processing` package is not created unless the processor needs it. -11. A DataStore document newer than supported schema is rejected. `update` and `save` are refused, deletion remains available, and there is no export action in this sprint. - -## Completion criteria - -Sprint 1A is complete when all of the following are true: - -- Vision code consumes real non-null payloads, not metadata placeholders. -- Capture still starts, stops, pauses, resumes, and displays frames from Debug Capture. -- Geometry and preprocessing JVM suites pass locally and in CI. -- Vision Lab can receive live/frozen frames and draw/select/move/resize/delete manual slots. -- Overlay remains aligned with preview aspect ratio for portrait and landscape images. -- Profiles save/load/reject incompatible inputs without screenshot persistence. -- CI is green and produces a debug APK artifact. -- Docs state exactly what ran, what did not, and what is not implemented. +Make captured frames usable by independent vision components without claiming automatic +detection. Deliver an image-bearing frame contract, rotation-aware preprocessing, tested +normalized-coordinate geometry, manual slot annotation, a Vision Lab and versioned calibration +profiles. Explicitly out of scope: OCR, recognition, draft detection, ML models, overlays and +network behaviour. Manual annotations are calibration data, never detections. + +## Key decisions + +- **Single `:app` module** with pure Kotlin packages for model, geometry, preprocessing, + annotation and calibration. Android Bitmap and Compose live at the edges. +- **Non-null payloads** at every frame boundary; the nullable capture bitmap is gone. +- **Two rotations** — `displayRotation` is a record, `rotation` is a correction applied by the + transform. MediaProjection buffers are already display-oriented, so production sets the + correction to `ROTATION_0`. +- **Validate before render.** Preprocessing returns failure codes for invalid dimensions, + payload mismatch, bad crops and oversized output instead of throwing into the UI. +- **Strict codecs.** Unknown fields and unsupported schema versions are refused, never silently + converted to empty data. A delete-all action is the escape hatch for unreadable documents. +- **Latest-frame-only publication.** Stale frames are worthless for draft inspection; bitmap + references are cleared on stop and reset rather than recycled while Compose may draw them. +- **Narrow seam.** Vision consumes `VisionFrameSource` only. + +## Consequences + +Pure logic is JVM-testable without an emulator, and the capture and vision layers can evolve +independently. Hardware behaviour of the capture path remains a separate verification concern. diff --git a/docs/superpowers/specs/2026-11-16-sprint-1c-app-only-capture-design.md b/docs/superpowers/specs/2026-11-16-sprint-1c-app-only-capture-design.md index 6ccc38e..e522544 100644 --- a/docs/superpowers/specs/2026-11-16-sprint-1c-app-only-capture-design.md +++ b/docs/superpowers/specs/2026-11-16-sprint-1c-app-only-capture-design.md @@ -1,59 +1,34 @@ -# Sprint 1C — App-Only Capture & Circular Floating Control (Design) +# Sprint 1C app-only capture design (archived) -**Date:** 2026-11-16 · **Branch:** `sprint-1c-app-only-capture` (stacked on `sprint-1b-hero-recognition`, PR #3) · **Spec:** maintainer Sprint 1C brief (in task tracker) +**Status:** implemented and merged as PR #4. Current behaviour is documented in +[Screen capture](../../SCREEN_CAPTURE.md). ## Intent -Let the user freeze/resume the live MLBB capture without leaving the game. One small -circular overlay button; the notification remains only a Stop affordance; no recommendation -UI of any kind. +Let the user freeze and resume the live MLBB capture without leaving the game. One small +circular overlay button; the notification keeps Stop and a single state action; no +recommendation UI of any kind. ## Key decisions -1. **Capture chooser is never forced.** `createScreenCaptureIntent()` is unchanged. The - debug screen guides the user to pick *Single app / app window → Mobile Legends* - (Android 14+ partial capture). The full-display path stays supported and documented as - the fallback, with no promise of notification-free frames. The old guidance text that - told users to choose Entire screen is corrected. -2. **Visibility is a gate, not a frame source.** API 34+ - `MediaProjection.Callback.onCapturedContentVisibilityChanged()` is overridden. While the - selected window is hidden, images are acquired and closed but never converted or - published; the last valid published frame is preserved untouched. While visible, live - publication resumes automatically *unless a frame is pinned* (user intent = FROZEN). - A resume requested while hidden is safely remembered: it flips the intent LIVE, clears - the pin, and frames flow again when visibility returns — publication while hidden stays - gated regardless of the pin state. -3. **Freeze semantics are unchanged.** `CaptureFreeze` + `FrameSnapshots` keep their - contracts (independently owned copy, projection stays running, idempotent). The overlay - button tap runs the same `freezeCurrentFrame()` / `resumeLive()` calls as everything else. -4. **The floating control is a button, not a panel.** One 48–56dp circular `WindowManager` - view with `TYPE_APPLICATION_OVERLAY`, attached by `CaptureSessionService` (the existing - foreground mediaProjection service, so it is alive exactly while capture is running). - `SYSTEM_ALERT_WINDOW` is requested (ACTION_MANAGE_OVERLAY_PERMISSION) only when the user - taps **Enable floating control**; denied → capture + in-app Freeze/Resume unaffected, - explanation shown; the enable is retried on enable tap or re-request, never forced. -5. **Overlay drives off pure Kotlin seams.** Position storage/snapping/clamping and the - visibility/publication gate are pure classes with JVM unit tests. The `WindowManager` - shell stays thin (attach/detach/touch). No DataStore on the hot path; position persists - as normalized coordinates on drag end. -6. **Overlay is attached only during capture sessions** and always removed on stop, - terminate, revocation and resize-failure teardown paths. Rotation re-derives pixel - position from the normalized coordinates. -7. **No masking of capture content.** The overlay sets no flags to hide itself from capture; - whether it appears in single-app vs full-display captures is a physical-device finding, - documented honestly, never "proven" by JVM tests. -8. **Stacked delivery.** PR base = `sprint-1b-hero-recognition`; 1B history untouched; no - merge; physical-device acceptance is maintainer-run and marked NOT RUN. - -## Non-goals - -Everything in the spec §10: no draft/phase detection, no slot detection, no recognition -models, no scoring, no recommendation overlays, no game interaction, no accessibility service. - -## Verification - -CI (GitHub Actions) is the build of record (host has no Android toolchain): `test`, -`lint`, `assembleAndroidTest`, `assembleDebug` + APK artifact. Instrumentation tests are -compiled, not executed (no KVM emulator). The physical-device matrix for overlay and -single-app capture is written into `docs/TESTING_PHYSICAL_DEVICE.md` and the sprint doc, -marked NOT RUN. +- **Pure visibility policy.** `CaptureVisibilityPolicy` decides whether a captured window is + visible enough to publish. It is JVM-tested and has no Android dependencies. +- **Publication gate only.** Visibility blocks conversion and publication in `FramePipeline`; + images continue to be acquired and closed so the reader stays healthy. The overlay's own + window, rendering and touch handling are independent of capture visibility. +- **One shared state.** `CapturePublishState` derives LIVE, PAUSED and FROZEN from the pinned + frame and session state. The overlay glyph, the overlay tap and the notification action all + use it, so they cannot disagree. +- **Opt-in overlay.** `SYSTEM_ALERT_WINDOW` is declared but requested only when the user + enables the control. Denial does not affect capture or in-app freeze/resume. +- **Normalized position.** The button stores a normalized position with edge snapping and + re-anchors on rotation. +- **No masking or automation.** The app never reads game memory, injects touches or masks the + capture region. On API levels below 34 there is no visibility signal and whole-display + semantics apply. + +## Verification boundary + +CI proves compilation, pure policy and position math, and lint. Overlay behaviour across OEMs +and single-app capture on a real device remain hardware findings recorded in the +[physical-device checklist](../../TESTING_PHYSICAL_DEVICE.md). diff --git a/tasks/plan.md b/tasks/plan.md index 94123fb..151f08a 100644 --- a/tasks/plan.md +++ b/tasks/plan.md @@ -1,242 +1,55 @@ -# Implementation Plan: DraftLens - -## Sprint 1A — Vision Foundation, Preprocessing & Vision Lab -**Objective:** Give the app a real image-bearing foundation — non-null frames, tested -geometry, validated preprocessing, manual slot annotation, a Vision Lab, and versioned -calibration profiles. No recognition, no recommendations, no overlays. - -### Work items in order -- [x] Task 1: Vision model and geometry value types (`a2731d3`) -- [x] Task 2: Rotation-aware geometry and round trips (`357a40f`) -- [x] Task 3: Non-null capture frames plus display rotation (`8e1381a`) -- [x] Task 4: Validated preprocessing engine (`79bd211`, `41aa126`, `b81e740`) -- [x] Task 5: Manual slot annotations and JSON codec (`1086b75`, `cab3d8d`) -- [x] Task 6: Versioned calibration profiles (`08ca414` reviewed, completed in `b84b69f`, `85f04c8`) -- [x] Task 7: Vision frame source and ViewModel (`fedaa4e`, `ec24a9a`, `0c7d286`, `b4c8c78`) -- [x] Task 8: Compose Vision Lab (`6ad2287`, `45e2f2c`, `691b9d2`) -- [x] Task 9: Vision documentation (`303b818`, `da35ca0`) -- [x] Task 10: CI, PR, APK artifact, implementation report -- [x] Final whole-branch review; findings fixed in `691b9d2` with a failing test each -- [x] Focused review fixes (`8f046c0`, `313775d`, `007621f`): display rotation updated - independently of the capture size, canvas letterbox rejected, Clear cancels in-flight - processing, shared slot validator for calibration documents, frozen-frame required for - profile save/update/apply, documentation and CI record updated -- [x] Canvas rendering and workspace fixes (`01603de`, `4c42513`): the bitmap is scaled - uniformly into the fitted rectangle so the pixels and the overlay share one set of - bounds, the canvas is sized from the oriented aspect ratio, `displayRotation` is - `@Volatile`, and the Vision Lab explains its controls in place -- [x] Sprint 1A.1 interaction redesign (`927e8e0`, `dffc12d`): an explicit gesture state - machine, a contextual slot property editor with copy/paste/duplicate on an internal - clipboard, a delete control with undo, a grouped slot list keyed by slot id, a - fullscreen editor, and team colours -- [x] Sprint 1A.1 corrections (`24a43af`, `593eeb4`): the delete control took over the - top-right resize handle, the long press opens the menu on its own timeout, the - fullscreen editor lost its zoom and fit controls, and the canvas no longer lets the - page scroll during annotation -- [x] Sprint 1A.2 bug fixes (`25307b8`): paste reads from the clipboard with the target - only anchoring the placement, the menu selects the slot it shows, an early drag - cancels the pending long press, one rotation snapshot per captured frame, and - validation failures surfaced in the property editor - -### Sprint 1A verification -CI (GitHub Actions, ubuntu-latest x86_64) is the build of record: `test`, -`assembleAndroidTest`, `lint`, `assembleDebug`, plus a `draftlens-debug` APK artifact. -The authoring host is aarch64 and AGP 8.6.1 ships only an x86_64 `aapt2`, so no Gradle -task that touches resources can run there. Instrumentation tests are compiled, not executed: -no device or emulator was available. Vision Lab interaction behavior is maintainer-reported -PASS on a physical device (device details not supplied); the MediaProjection lifecycle -matrix is **NOT RUN**. - -## Sprint 0.1 — Capture Hardening & Technical Cleanup -**Objective:** Harden the reported-working capture path before Vision Engine. No recognition, -recommendations, overlays, or architecture rewrites. - -### Work items in order -- [x] Task 1: AGP 8.5.2 → 8.6.1; keep Gradle 8.7, Kotlin 2.0.20; run test/lint/assembleAndroidTest -- [x] Task 2: Strict lint: remove silent `abortOnError=false` / release checks; fail CI on lint errors -- [x] Task 3: Frame memory ownership: no unsafe Bitmap recycling; latest-only ownership; stale refs cleared -- [x] Task 4: Frame timing: monotonic/Android elapsed-realtime sampler; wall-clock retained separately; latency semantics clarified; tests -- [x] Task 5: MediaProjection: one VirtualDisplay per projection, API-appropriate resize path, WindowMetrics/appropriate fallback, minimum display flags -- [x] Task 6: Physical-device lifecycle matrix document; mark results explicitly pending until executed -- [x] Task 7: Docs updated: IMPLEMENTATION_REPORT, SCREEN_CAPTURE, ARCHITECTURE -- [x] Task 8: CI green: test, lint, assembleAndroidTest, assembleDebug, APK manifest check - -### Sprint 0.1 verification -Gradle: `test`, `lint`, `assembleAndroidTest`, `assembleDebug`. Physical device matrix is pending and is not claimed here. -Manual smoke test already performed by maintainer and is recorded as such. - -## Overview -Single-module native Android app (Kotlin + Compose, MVVM, Coroutines/Flow) that validates -MediaProjection screen capture in a foreground service and shows a live debug preview. -No recognition models, no recommenders, no overlays this sprint. Foundation must be ready to -grow without re-architecture. - -## Architecture Decisions -- Kotlin 2.0.20, AGP 8.5.2, Gradle 8.7, compileSdk/targetSdk 35, minSdk 26 (MediaProjection - baseline), JDK 17. Rationale: stable, compatible with SDK 35 + Android 14/15 MediaProjection - changes; minSdk 26 avoids legacy permission-branch complexity while covering vast majority. -- Single Gradle module `:app`. Rationale: Sprint 0 has no justified module boundaries yet; ADR-001 - documents the split trigger (second feature needing independent test/deploy cycle). -- Foreground service type `mediaProjection`, notification with Stop action, consent re-requested - every session via `ActivityResultContracts.CreateScreenCaptureIntent()`. Android 14+: - service must start before `getMediaProjection()`; `onStop` + `onCapturedContentResize` - handled; `MediaProjection.Callback` releases resources. -- Frame pipeline: ImageReader (maxImages=3, backpressure-drop oldest), sampled at 2 FPS default - (configurable 500–2000 ms via DataStore, capped 3 FPS), frames held in memory as `Bitmap` - (LIFO single-slot — only latest kept to avoid OOM), no disk writes, no upload. -- State: `CaptureSessionState` sealed-ish data class + StateFlow; ViewModel exposes UI state. - Rotation/resize recreates VirtualDisplay via `onCapturedContentResize` + config-change. -- Domain seams (explicit NotImplemented, never fake data): ScreenCaptureProvider, - FrameProcessor, DraftPhaseDetector, HeroRecognizer, HeroSlotDetector, DetectionValidator. -- Data sources: Gradle dependencies for lifecycle/compose/coroutines/datastore. Room - ships with first persisted entity to avoid empty database. Preferences repository has - `samplingIntervalMs`. No network client in Sprint 0 — - DATA_SOURCES.md records audit; API sync deferred to later milestone per scope rules. - -## Task List -### Phase 1: Foundation -- [x] Task 1: Scaffold Gradle project, manifest, nav (single landing screen), CI workflow -- [x] Task 2: Core models + DataStore preferences repository (Room deferred) -- [x] Task 3: Capture state machine + unit tests (TDD, red first) - -### Phase 2: Capture pipeline -- [x] Task 4: CaptureSessionService (foreground, notification, stop action) + ScreenCaptureController -- [x] Task 5: FramePipeline (ImageReader → Bitmap → StateFlow) + FrameSampler -- [x] Task 6: MediaProjection permission launcher + ViewModel wiring - -### Phase 3: UI -- [x] Task 7: Debug screen (Compose) with status, preview, metrics, pause/resume, errors -- [x] Task 8: Detection stub interfaces (NotImplemented states) - -### Phase 4: Docs + verify -- [x] Task 9: README, ARCHITECTURE, SCREEN_CAPTURE, DATA_SOURCES (verified live), ADR-001 -- [x] Task 10: Tests pass, lint passes, debug APK built, manifest verified, report written - -## Risks and Mitigations -| Risk | Impact | Mitigation | -|------|--------|------------| -| MediaProjection revoked mid-session on Android 14+ | Service crash / leaked resources | Callback + try/catch → STOPPED state, full resource release in finally | -| ImageReader backpressure OOM on high-res | App kill | maxImages=3, single-latest-bitmap slot, drop-oldest, sampling throttle | -| Emulator lacks MediaProjection reliability | False "works" claim | Docs + report explicitly state emulator-only, physical-device test plan required | -| API data definitions differ (Rone vs MLBBDex) | Wrong recommendations later | No recommendation code in Sprint 0; DATA_SOURCES.md records definitions verbatim | - -## Open Questions -- App ID / name: using `com.draftlens` / "DraftLens" (rename freely). -- Kotlin/AGP/SDK pins above are builder calls, recorded in ADR-001. - -## Checkpoints -- After Phase 1: `./gradlew test` green, app assembles (empty landing OK). -- After Phase 2: unit tests for state machine + sampler green; lint + assemble green. -- Final: test + lint + assembleDebug all green; APK exists; manifest grep verified. - -### Sprint 1B — Hero recognition (first pipeline) -- Hero catalog sync (manual refresh, atomic snapshot, offline browsing) wired through a small transport seam. -- Slot extraction from the annotation canvas with rotation coverage. -- Recognition v0 with labeled similarity and explicit no-match; manual confirmation required. -- Draft PR #3; accuracy on real screens not measured; device smoke pending. - -## Sprint 1B — Review fixes and game-frame freeze (this revision) -Objective: fix the PR #3 review findings (freeze from the notification; rotation bugs; -blank-image rejection; manual correction everywhere; cache freshness; atomic persistence; -reference-loader ownership) and wire the freeze workflow through the capture architecture. -No recommendation features, no overlays, no automation, PR stays draft and unmerged. - -### Work items in order -- [x] Freeze a game frame from the notification (`CaptureFreeze`, `ACTION_FREEZE` (superseded: notification now issues Pause) / - `ACTION_RESUME_LIVE`, `FramePipeline` freeze gate, independently owned snapshot via - `FrameSnapshots`, FROZEN notification with Resume action, in-app Resume unchanged) -- [x] Full-device-capture guidance; POST_NOTIFICATIONS handling; freeze actions in the - expanded notification; no Activity launched by the command -- [x] Part B: pixel extractor rotation fixed to the `FrameTransform.sourceToOriented()` - convention; tests rewritten with asymmetric non-square fixtures and hand-computed - expected values -- [x] Landscape capture fix (MLBB portrait→landscape rotation bug): - - authoritative `onCapturedContentResize` signal feeds one idempotent resize path; - - `DisplayManager` fallback re-derives the capture size on any display change; - - VirtualDisplay/ImageReader re-created at the reported size; - - capture generations reject stale reader callbacks at listener and publish time; - - plane conversion honors `image.cropRect` and padded rows (`RgbaPlaneCopy`, JVM-tested - with asymmetric fixtures, letterboxed crops and stride variants); - - structured diagnostics in `CaptureSessionService`/`FramePipeline`/`VisionPreprocessor`; - - physical-device verification pending (rows 23–32 of `docs/TESTING_PHYSICAL_DEVICE.md`) -- [x] Part C: Android extractor validates Bitmap dimensions, pixel-exact rotation, source - bitmap preserved, every temporary released exactly once; instrumentation tests added -- [x] Part D: `VisualInformationAnalyzer` (variance, edge density, meaningful differences, - minimum dimensions) gates matching with a distinct `InsufficientVisualInformation` - outcome and `INSUFFICIENT_INFO` UI state; blank crops never produce candidates -- [x] Part E: manual correction reachable from CANDIDATES, NO_MATCH, FAILED, - INSUFFICIENT_INFO and CONFIRMED; Change… opens correction instead of Identify; - failed persistence reported, never shown as success -- [x] Part F: cache freshness — per-asset `*.meta` sidecar (URL, SHA-256, size, content - type), serve-time validation (hash, magic bytes, container structure, Android decode), - URL change forces re-download, failed refresh keeps previous asset, HTTP downgrades - refused everywhere -- [x] Part G: snapshot/asset writes are temp-sibling + flush/sync + `ATOMIC_MOVE` only; - no direct-write fallback; failures preserve the previous target; `*.tmp` swept -- [x] Part H: reference loader reads dimensions before recycling; pixels read before release; - exceptional paths recycle too -- [x] CI green on the final head (run 43746648882, head 1c7397b925dd2683b3d763e552d610d2e464ecb7; instrumentation compiled, not executed — no KVM on the runner) - (no KVM on the runner) — needs a device or KVM-enabled emulator -- [ ] Physical-device smoke of the freeze workflow; on-device accuracy - evaluation; real-screen threshold and heuristic calibration - -### Sprint 1B verification -See `docs/VISION_TESTING.md` for the exact CI runs, test totals and what was and was not -executed. The authoring host remains aarch64, so the CI is the build of record. -Physical-device verification of the freeze workflow is pending. - -## Sprint 1C — App-only capture & circular floating control -Objective: let the user freeze/resume the live MLBB capture without leaving the game — -prefer single-app (partial) capture, keep the last valid game frame when MLBB is hidden, -and add exactly one circular overlay button. No recommendation UI, no panels, no automation. -PR #4 stacked on PR #3 (`sprint-1b-hero-recognition`); nothing merges automatically; -Sprint 1B history intact. - -### Work items in order -- [x] Pure visibility policy + truth-table tests (hidden ⇒ nothing publishes; frozen wins over visible) -- [x] `FramePipeline` gate: images acquired/closed but never converted while hidden; - publish-time synchronized re-check; API-34 `onCapturedContentVisibilityChanged` in the - existing CAPTURING guard -- [x] Freeze-while-hidden / resume-while-hidden semantics: last valid frame preserved; - LIVE intent remembered across hidden periods; no second VirtualDisplay/projection ever -- [x] Overlay position math (normalized free-axis coords, edge snap) + JVM tests -- [x] Overlay enablement decision + DataStore persistence (position + enabled) -- [x] 52dp circular `TYPE_APPLICATION_OVERLAY` button: tap = Freeze/Resume (green/orange, - white glyph), 400 ms long-press drag, edge snap, rotation re-anchor, non-focusable -- [x] Service attach/detach only during capture, every teardown path removes it; - `ACTION_OVERLAY_UPDATE` re-sync; manifest declares `SYSTEM_ALERT_WINDOW`, requested - only from the enable toggle -- [x] Two-mode capture guidance in the debug screen + Vision Lab; denial path documented - (capture + in-app Freeze/Resume unaffected) -- [x] Instrumentation test for overlay attach/remove (compile-only in CI) -- [x] JVM sequence tests: hidden→freeze→resume→visible; explicit freeze blocks auto-resume -- [ ] CI green record, APK digest, device rows 33–43 NOT RUN (maintainer) -- [ ] PR #4 description + report; physical-device acceptance checklist is maintainer-run - -## Sprint 1C — Follow-up fixes (this revision) -Objective: make the floating control robust for app-only capture and slim the in-app UI to -permissions only; replace the notification Freeze action with Pause/Resume on the shared state. - -### Work items -- [x] Root-cause pass on the app-only overlay: no visibility↔overlay lifecycle coupling found; - fixes are the silent-tap feedback (toast on NoFrame) and a single `CapturePublishState` - derivation shared by notification and overlay (pin wins over pause for presentation) -- [x] Unified `CapturePublishState` (LIVE/PAUSED/FROZEN) with JVM-tested decision matrix -- [x] Notification actions: Stop + Pause (LIVE) / Stop + Resume (PAUSED/FROZEN); Pause maps to - the existing CaptureStatus.PAUSED gate; Resume clears pin + pause; combined - frozen-frame + session-state collector keeps notification and overlay in sync -- [x] UI slim-down: removed tutorial cards / in-game help texts; one compact Permissions card - (notification permission + overlay permission, status + grant/settings actions) -- [x] Tests: PublishStateMatrixTest, holder pause-preservation tests, sequence pause test, - instrumented overlay publication-flip attach test -- [ ] CI green record; physical-device rows 44–53 NOT RUN - -## Sprint 1C — single-app first-frame fix (this revision) -- [x] Diagnostics first: `CaptureDiagnostics` + per-stage callback/acquisition/gate logs -- [x] Service-owned `CaptureVisibilityPolicy` with UNKNOWN/VISIBLE/HIDDEN; early callbacks - preserved; UNKNOWN open; first frame promotes to VISIBLE; stale callbacks rejected -- [x] Service-owned `PendingResize`: early captured-content dimensions retained, applied to - both ImageReader and VirtualDisplay, idempotent; no session kill before VD creation -- [x] Pipeline no longer resets the gate; holder records frames published during STARTING -- [x] JVM tests: early callbacks, UNKNOWN semantics, promotions, pending-resize retention, - sequences; device A–E checklist added (NOT RUN) +# Roadmap + +DraftLens is built in small vertical increments. Each increment ends with a green CI run and +an APK artifact; hardware behaviour is tracked separately in the physical-device checklist. +This file describes the current shape of the work, not a commit-by-commit history. + +## Done + +### Sprint 0 and 0.1 — capture foundation +Native Android app, single `:app` module, MediaProjection consent flow, typed foreground +service with a persistent notification, frame sampling, capture state machine, debug preview, +permission handling, strict lint and CI. + +### Sprint 1A — vision foundation +Non-null frame contract with pixel payloads; rotation-aware geometry with round-trip tests; +validated preprocessing with honest failure codes; manual slot annotation with validation, +undo/redo and strict JSON codecs; versioned calibration profiles; Vision Lab tab. No +recognizer: slots are calibration data, never detections. + +### Sprint 1B — hero recognition +Public static hero catalog from Rone Arena with an MLBBDex completeness cross-check, private +image cache with content validation and HTTPS-only redirects, atomic snapshot writes, upright +slot extraction for every rotation, and a dHash plus mean-RGB similarity baseline. Blank or +detail-free crops are rejected and low scores are reported as no match. The user always +confirms or corrects manually; results are labelled as uncalibrated similarity, not +probability. Also includes notification freeze/resume and the landscape capture fix with +reader generations and `cropRect`-aware plane copying. + +### Sprint 1C — app-only capture and floating control +Captured-window visibility gates frame publication so the last valid game frame is kept when +MLBB is hidden. One 52 dp overlay button, attached only during capture and opt-in, toggles +LIVE and paused/frozen through the shared publish state. The notification offers Stop and +exactly one state action. The capture UI is reduced to permissions and status, and the +single-app first-frame ordering is fixed with explicit visibility and pending-resize states. + +## Current focus + +1. **Device verification.** CI proves compilation, pure logic and lint. It does not prove + MediaProjection behaviour, overlay behaviour across OEMs, or recognition quality. The + checklists and their honest NOT RUN status are in + `docs/TESTING_PHYSICAL_DEVICE.md`. +2. **Calibration.** Measure top-1/top-3 and rejection behaviour on real MLBB screens, then + replace the starting threshold and heuristic values in the recognition code. +3. **Instrumented tests.** Run them where KVM is available; the hosted runner compiles them + only. + +## Architecture constraints for future work + +- Keep pure logic in JVM-testable packages: geometry, codecs, policy and matching have no + Android imports at their core. +- Vision receives frames and freeze commands only through `VisionFrameSource`. +- Frames, slot crops and catalog references stay on-device; only public static catalog data + and reference images are downloaded. +- A second Gradle module becomes justified when a feature needs an independent test or release + cycle. Until then, logical packages are enough (see ADR-001). diff --git a/tasks/todo.md b/tasks/todo.md index 563ca8d..3338cb7 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -1,153 +1,46 @@ -# Sprint 0 Task List - -## Sprint 1A: Vision Foundation -- [x] Task 1: Vision model and geometry value types (`a2731d3`) -- [x] Task 2: Rotation-aware geometry and round trips (`357a40f`) -- [x] Task 3: Non-null capture frames plus display rotation (`8e1381a`) -- [x] Task 4: Validated preprocessing engine (`79bd211`, `41aa126`, `b81e740`) -- [x] Task 5: Manual slot annotations and JSON codec (`1086b75`, `cab3d8d`) -- [x] Task 6: Versioned calibration profiles — WIP `08ca414` reviewed and completed in - `b84b69f` + `85f04c8` (red CI `42905781209`, green CI `42906245805`) -- [x] Task 7: Vision frame source and ViewModel (`fedaa4e`, `ec24a9a`) -- [x] Task 8: Compose Vision Lab (`6ad2287`, `45e2f2c`, `691b9d2`) -- [x] Task 9: Vision documentation (`303b818`, `da35ca0`) -- [x] Task 10: CI, PR, APK artifact, implementation report -- [x] Final whole-branch review; findings fixed in `691b9d2` with a failing test each -- [x] Focused review fixes: display rotation updates, canvas letterbox, clear cancels - processing, calibration slot validation, frozen-frame profile policy, docs and CI - (`8f046c0`, `313775d`, `007621f`) -- [x] Canvas rendering and workspace fixes: bitmap scaled into the fitted rectangle, canvas - sized by the oriented aspect ratio, `@Volatile` display rotation, control explanations - (`01603de`, `4c42513`) -- [x] Sprint 1A.1 Vision Lab interaction redesign: explicit gesture state machine, contextual - slot property editor, copy/paste/duplicate on an internal clipboard, delete control - with undo, grouped slot list with stable keys, fullscreen editor, slot colours - (`927e8e0`, `dffc12d`) -- [x] Sprint 1A.1 corrections: the delete control replaced the top-right resize handle, the - long press opens the menu on its own timeout, the fullscreen editor lost zoom and fit - controls, and the page no longer scrolls during annotation (`24a43af`, `593eeb4`) -- [x] Sprint 1A.2 bug fixes: paste reads from the clipboard with the target only anchoring - placement, the property menu selects the same slot it shows, an early drag cancels - the pending long press, one rotation snapshot per captured frame, validation failures - surfaced in the property editor (`25307b8`) - -Verification for Sprint 1A runs in GitHub Actions (ubuntu-latest, x86_64) because the -authoring host is aarch64 and AGP's `aapt2` is an x86_64-only binary, so no Gradle task that -touches resources can run locally. Draft PR: https://github.com/xeriomy/DraftLens/pull/2 - -Executed commands (CI, `./gradlew`): -- `./gradlew test` — 269 JVM unit tests, 0 failures, 0 errors, 0 skipped (538 executions, both variants) -- `./gradlew assembleAndroidTest` — instrumentation tests compiled, **not executed** -- `./gradlew lint` — strict lint, `abortOnError = true` -- `./gradlew assembleDebug` — debug APK, uploaded as the `draftlens-debug` artifact - -Latest green run: https://github.com/xeriomy/DraftLens/actions/runs/43643640990 -(head SHA `5ffb0c1`). - -The frame sizing and rendering fix was confirmed by the user on a physical device before the -Sprint 1A.1 work started and was left unchanged. The Vision Lab interaction behavior in the -retained 1A.2 state was also smoke-tested on a physical device by the maintainer and confirmed -working: slot selection, movement and resizing; the long-press property menu; long-press-and- -drag creation; the original compact red X deletion control; copy, paste and duplicate; -fullscreen entry/exit and Android Back; annotation gestures isolated from page scrolling; and -portrait/landscape rendering. Device model, Android version, install source/APK artifact, test -date, logs and recordings were not supplied. The MediaProjection lifecycle matrix in -`docs/TESTING_PHYSICAL_DEVICE.md` remains **NOT RUN**. - -Whole-branch review against the plan and spec: eight findings fixed in `691b9d2`, each with a -test that failed first. The remaining minor findings are listed in the implementation report -and the review ledger. - -Physical-device verification of the Vision Lab interaction behavior is **maintainer-reported -PASS** (device details not supplied). The MediaProjection lifecycle matrix is **NOT RUN**. - -## Sprint 0.1 Capture Hardening -- [x] Task 1: AGP 8.6.1 (API 35 support); Gradle/Kotlin unchanged -- [x] Task 2: Strict lint gate in module and CI -- [x] Task 3: Frame memory ownership: safe bounded latest-frame model -- [x] Task 4: FrameSampler monotonic/Android elapsed time + wall-clock separation + tests -- [x] Task 5: WindowMetrics (API 30+) and deprecated fallback; minimize VIRTUAL_DISPLAY flags -- [x] Task 6: Physical-device lifecycle matrix checklist; results pending -- [x] Task 7: Update implementation, architecture and capture docs -- [x] Task 8: CI green; deliver hardened APK and report - -## Sprint 0 - -## Phase 1: Foundation -- [x] Task 1: Scaffold Gradle project, manifest, single-screen nav, CI workflow -- [x] Task 2: Core models + DataStore preferences repository -- [x] Task 3: Capture state machine + unit tests (TDD red first) - -## Phase 2: Capture pipeline -- [x] Task 4: CaptureSessionService (foreground, notification, stop action) + controller -- [x] Task 5: FramePipeline (ImageReader → Bitmap) + FrameSampler (TDD) -- [x] Task 6: Permission launcher + ViewModel wiring - -## Phase 3: UI -- [x] Task 7: Debug screen (Compose): status, preview, metrics, pause/resume, errors -- [x] Task 8: Detection stub interfaces with explicit NotImplemented states - -## Phase 4: Docs + verify -- [x] Task 9: README, ARCHITECTURE, SCREEN_CAPTURE, DATA_SOURCES, ADR-001 -- [x] Task 10: Tests + lint + assembleDebug green; APK + manifest verified; report written - -## Sprint 1B: Hero catalog, slot extraction and recognition -- [x] Hero catalog module (Rone Arena primary, MLBBDex cross-check, atomic cache, offline) -- [x] Slot image extraction (all rotations, edge cases) -- [x] Recognition v0: dHash + color descriptor, explicit no-match, manual confirm/correct -- [x] Vision Lab card: sync, Identify, candidates, confirmed state, manual correction -- [x] Unit coverage for catalog, cache, extraction, matching, VM flows -- [x] CI green: 323 tests (646 executions), strict lint, assembleDebug, assembleAndroidTest compiled (run 43732908605, head e58611c; code head 7d0c903 run 43731484922) - -## Sprint 1B review fixes + game-frame freeze (PR #3 follow-up) -- [x] Part A: freeze/resume from the notification through the capture architecture - (explicit action to the running service; independently owned snapshot; images stay - acquired+closed while frozen; one frozen frame; notification LIVE/FROZEN; in-app Resume; - full-device capture guidance; POST_NOTIFICATIONS handling) -- [x] Part B: pixel rotation fixed to the FrameTransform convention; asymmetric non-square - tests with independently calculated expectations -- [x] Part C: Android bitmap extractor validated and made pixel-exact; instrumentation tests - for all rotations, edge crops, mismatches, source validity and snapshot ownership -- [x] Part D: visual-information gate (uniform/near-uniform/detail-free crops rejected); - distinct InsufficientVisualInformation outcome and UI state -- [x] Part E: manual correction everywhere; failed clears reported honestly -- [x] Part F: cache freshness (URL-bound sidecar metadata; hash/structure/decode validation; - URL change forces re-download; failed refresh keeps previous asset; HTTPS-only redirects) -- [x] Part G: atomic snapshot/asset writes; failure preserves previous snapshot; temp files swept -- [x] Part H: reference Bitmap dims read before recycling -- [x] Landscape capture fix (MLBB): authoritative captured-content resize via - `onCapturedContentResize` + `DisplayManager` fallback to current window metrics; - VirtualDisplay/ImageReader re-created at the reported size; generation numbers reject - stale reader callbacks; plane conversion honors `image.cropRect` and padded rows; - structured diagnostics in `CaptureSessionService`/`FramePipeline`/`VisionPreprocessor` -- [x] CI green on the new head for this revision (run 43773398149, head 5252016cd15c824009368e831ed499f24209980d; 840 unit tests; instrumentation compiled, not executed — no KVM on the runner) -- [ ] Physical-device smoke of the freeze workflow; on-device accuracy evaluation; - threshold/heuristic calibration (pending) -- [ ] Physical-device verification of the landscape capture fix in MLBB (rows 23–32) (pending) - -## Sprint 1C: App-Only Capture & Circular Floating Control -- [x] Task 1: Pure visibility gate (`CaptureVisibilityPolicy`) + JVM truth table (`d68f7bc`) -- [x] Task 2: Pipeline visibility gate; `onCapturedContentVisibilityChanged` (API 34+) wired into the CAPTURING guard (`6eaceb9`) -- [x] Task 3: Overlay position math (normalized/free-axis, edge snap) + JVM tests (`ffc4f5e`) -- [x] Task 4: Overlay enablement decision + DataStore keys (`48e0e11`) -- [x] Task 5: 52dp circular overlay button (`FloatingControlView`/`OverlayController`), long-press drag, tap toggle (`f88cce4`) -- [x] Task 6: Service attach/detach on every teardown path, rotation re-anchor, enable card + two-mode capture guidance, manifest `SYSTEM_ALERT_WINDOW` (`494df62`) -- [x] Task 7: Instrumented overlay attach/remove test (compile-only) (`c6edb52`) -- [x] Task 8: Visibility/freeze/stop sequence tests (`94662b3`) -- [x] Task 9: Documentation (sprint doc, capture docs, device matrix rows 33–43) (this branch) -- [x] Task 10: CI green record (run 43784157138 at `f992d45`: 886 unit tests, 0 failures; APK sha256 `1d1f…`; stacked PR #4); review findings M1/M2 fixed -Verification runs in GitHub Actions because the authoring host has no Android toolchain (below). Physical-device rows 33–43 are **NOT RUN**. - -## Sprint 1C follow-up: notification Pause, permissions-only UI, overlay independence -- [x] Root-cause trace + CapturePublishState unification (`CapturePublishState.kt`) -- [x] Notification Freeze → Pause (LIVE → Pause, PAUSED/FROZEN → Resume); combined state collector -- [x] Overlay glyph/tap driven by the shared publish state; toast on NoFrame -- [x] Permissions-only UI (notification + overlay); tutorial and help text removed -- [x] Docs sync + device rows 44–53; tests added (JVM + compile-only instrumented) -- [x] Task 10: CI green record (run 43922482183 at `130ce6c`: 904 unit tests, 0 failures; APK sha256 `48b5…`); report - -## Sprint 1C single-app first-frame fix -- [x] Diagnostics before policy change (`CaptureDiagnostics`) -- [x] Explicit visibility state model + pending captured-content resize; startup ordering fixed -- [x] JVM regression tests + physical-device A–E checklist (NOT RUN) -- [x] Task 10: CI green record (run 43936524131 at `3c7b58c`: 926 unit tests, 0 failures; APK sha256 `ade9…`); report +# Task ledger + +## Delivered + +- **Sprint 0 / 0.1** — Android foundation, MediaProjection capture, foreground service, + sampling, permissions, capture state machine, strict lint and CI. +- **Sprint 1A** — image-bearing frame contract, rotation-aware geometry, validated + preprocessing, manual slots and strict codecs, calibration profiles, Vision Lab. +- **Sprint 1B** — hero catalog sync, private cached references, slot extraction, recognition + v0, manual confirm/correct, capture freeze from the notification, landscape capture fix. +- **Sprint 1C** — captured-window visibility gate, one circular overlay control, notification + pause/resume on a shared publish state, permissions-only capture UI, first-frame fix for + single-app capture. + +All four pull requests are merged into `main`. The old per-commit histories live in git; this +file tracks only what remains open. + +## Open work + +- [ ] Run the physical-device rows in `docs/TESTING_PHYSICAL_DEVICE.md` and record commit + SHA, APK run ID, device model and Android version for each result. Nothing may be marked + PASS without that evidence. +- [ ] Verify the freeze/pause workflow on a device (rows 14–22). +- [ ] Verify the landscape capture fix, rotation and resize storm handling (rows 23–32). +- [ ] Verify app-only visibility and the floating control on real MLBB and across OEMs + (rows 33–43 and 44–53). +- [ ] Measure identification accuracy on real MLBB screens; calibrate the no-match threshold + and the visual-information heuristics. The current values are starting defaults. +- [ ] Execute the instrumented tests on a device or a KVM-enabled emulator. CI compiles them + but the hosted runner has no KVM, so they are not executed. +- [ ] Decide whether a second Gradle module is warranted once a feature needs its own test or + release cycle; see `docs/decisions/ADR-001-android-native.md`. + +## Not planned + +Recommendation, counter/synergy/role scoring, automatic draft-phase detection, game-memory +reading, injected touches, cloud recognition, ML models, OpenCV, and any overlay beyond the +single capture-freeze control. + +## Verification records + +CI is the build of record: `test`, `assembleAndroidTest` (compile only), `lint`, +`assembleDebug`, APK upload and manifest inspection run on every push and pull request in +`.github/workflows/build.yml`. The latest verification record is PR #5 head `4eb9948`; see +`docs/IMPLEMENTATION_REPORT.md` for the summary and `docs/VISION_TESTING.md` for the test +inventory.