From c30f8ec05f1eff00ca5cab7d4cb6fcb956e8a8ef Mon Sep 17 00:00:00 2001 From: DoodleBear Date: Thu, 22 Oct 2026 15:13:44 +0900 Subject: [PATCH] docs(ax): retain iOS and Android QA evidence for releases Issue: #2569 Agent-Task: codex/local/01a16230-879c-7ea2-bdc1-9f65f57a031a --- AGENTS.md | 4 + apps/mobile/AGENTS.md | 1 + docs/agents/android-device-qa.md | 4 + docs/agents/changelog.md | 4 + docs/agents/device-qa.md | 4 + docs/agents/pr-ui-visual-evidence.md | 4 + docs/agents/qa-evidence.md | 107 +++++++++++++++++++++++++++ 7 files changed, 128 insertions(+) create mode 100644 docs/agents/qa-evidence.md diff --git a/AGENTS.md b/AGENTS.md index 45a3432171..4e2681d46d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -79,6 +79,10 @@ The dev VPS lives on the Human's Tailscale tailnet. Per-machine values (SSH host Native checks for `apps/mobile` share devices and QA Hosts among coding agents. Before booting an emulator/Simulator, building the app, starting a Host or touching a phone, follow [shared device QA](docs/agents/device-qa.md): choose the Mac/iOS or Windows/Android queue, submit to its owner-cut batch, use the dedicated QA Host, and report redacted results. Windows emulator and USB-phone checks use the shared `android-qa.ps1` lease and scrcpy capture workflow. +### QA evidence (AX) + +Before capturing development screenshots or recordings, handing off macOS/iOS or Windows/Android QA, or retaining release images, follow [local QA evidence and release archives](docs/agents/qa-evidence.md). Keep an indexed task folder in the machine's main checkout and preserve accepted evidence by release and SHA; platform queues still own devices and capture tooling. + ### Machine-local secrets (AX) Files that device QA and store builds need but git must never hold (Firebase `google-services.json`, FCM and Play keys) live in the gitignored `.secrets/` folder of each machine's main checkout; every worktree resolves to the same folder. Read [machine-local secrets](docs/agents/device-secrets.md) for the file list, how the app finds them, local push builds and clean-clone EAS builds. diff --git a/apps/mobile/AGENTS.md b/apps/mobile/AGENTS.md index fb6cbf46ac..ae81ced149 100644 --- a/apps/mobile/AGENTS.md +++ b/apps/mobile/AGENTS.md @@ -51,6 +51,7 @@ Screens follow [DESIGN.md](./DESIGN.md): the Host-as-workspace structure, the sc - `smoke --ios --no-start --no-reload --port --json`: one exit code for "the app is connected and threw nothing". `--no-start` keeps it from starting Metro, a Simulator or Expo Go. - Never run `agents:setup`, `new`, `install`, `deploy` or anything with `--eas`: they rewrite `AGENTS.md`/`CLAUDE.md`, change dependencies or spend EAS credits. - Native devices and QA Hosts are shared by every agent. Read [shared device QA](../../docs/agents/device-qa.md) before builds or device interaction; it routes Mac/iOS and Windows/Android to their own batch queues. Windows emulator/USB phone control and scrcpy evidence use `scripts/android-qa.ps1` from the repo root. +- Screenshots, recordings and release image retention follow [local QA evidence](../../docs/agents/qa-evidence.md): index each platform's actual source/build and results in the main checkout, and retain accepted evidence outside disposable worktrees. - Cloud sessions have no simulator. They use the web screenshots ("Screenshots without a phone") and reach a Mac through Remote Control for native checks. ## Profiling diff --git a/docs/agents/android-device-qa.md b/docs/agents/android-device-qa.md index 4724381259..2cd7154681 100644 --- a/docs/agents/android-device-qa.md +++ b/docs/agents/android-device-qa.md @@ -129,6 +129,10 @@ existing route state before changing a shared proxy. ## Phone control, screenshots and video +Index captures for local inspection and release retention using +[local QA evidence and release archives](qa-evidence.md). The helper's queue evidence directory +remains the capture source; link or copy selected evidence into the task/release index with hashes. + The Human connects/unlocks the phone, enables USB debugging and accepts the RSA authorization. Use `devices` to distinguish `device`, `unauthorized` and `offline`; the helper never chooses the first device automatically. Install only the development application ID, preserving the diff --git a/docs/agents/changelog.md b/docs/agents/changelog.md index f1d6afd890..19c7daa823 100644 --- a/docs/agents/changelog.md +++ b/docs/agents/changelog.md @@ -184,6 +184,10 @@ history immutable and describe a restoration, withdrawal, or fix in the next ver ## Preparing a GitHub Release draft +Retain accepted screenshots and recordings using [local QA evidence and release archives](qa-evidence.md). +The archive binds separate macOS/iOS and Windows/Android results to one acceptance SHA and each +artifact's build identity; only reviewed public derivatives are candidates for release attachments. + After archiving `Unreleased` under a dated version, generate a review-only payload from that exact canonical release section: diff --git a/docs/agents/device-qa.md b/docs/agents/device-qa.md index 31ccd90465..a253231ed1 100644 --- a/docs/agents/device-qa.md +++ b/docs/agents/device-qa.md @@ -204,6 +204,10 @@ Some steps only the Human can do. Ask once, with the exact command or action and ## Reporting results +Save and index captures using [local QA evidence and release archives](qa-evidence.md). +macOS/iOS and Windows/Android keep separate platform results and share the frozen acceptance SHA +for final release checks; existing runner output locations remain intact. + - **One comment per PR**, posted by the runner with `gh` from the Mac. It gives the PR head SHA and the batch branch commit it ran on. It gives pass, fail or untested for each step, with the reason for anything untested (for example "Android untested", "no second Host", or "needs a diff --git a/docs/agents/pr-ui-visual-evidence.md b/docs/agents/pr-ui-visual-evidence.md index 4ff246d301..ffcd4c3322 100644 --- a/docs/agents/pr-ui-visual-evidence.md +++ b/docs/agents/pr-ui-visual-evidence.md @@ -4,6 +4,10 @@ Use this guide when a PR changes a rendered screen or user interaction: layout, ## Capture +Keep local originals, redacted copies and their source/build records according to +[local QA evidence and release archives](qa-evidence.md). Local inspection paths supplement the +public, commit-pinned evidence required below. + 1. Run the base revision and the PR revision through the real UI. Capture the baseline before editing when possible; otherwise use a separate base checkout. Take a **Before** and **After** screenshot for each materially changed view or interaction state. Include each viewport whose behavior changes. 2. Match viewport dimensions, theme, locale, test data, and UI state across each pair. Use representative data and remove secrets or personal information before publishing. For a new screen with no direct predecessor, show the prior entry point or absence state as **Before** and label it clearly. 3. Check the images side by side. They should show the actual change without relying on a mockup or a code diff. For motion or a multi-step interaction, capture the key states and add a short recording or exact reproduction steps. diff --git a/docs/agents/qa-evidence.md b/docs/agents/qa-evidence.md new file mode 100644 index 0000000000..781c4f7bf2 --- /dev/null +++ b/docs/agents/qa-evidence.md @@ -0,0 +1,107 @@ +# Local QA evidence and release archives + +Use this convention when capturing development screenshots or recordings, handing native QA +between machines, or retaining images for a release. macOS owns iOS Simulator/iPhone checks; +Windows owns Android emulator/phone checks. Follow each machine's +[iOS queue](device-qa.md) or [Android queue](android-device-qa.md) before operating resources. +An archive does not allocate a device or authorize a release. + +## Save where the Human can find it + +Default to the **main checkout on the machine that ran the check**, outside disposable +worktrees. Resolve that checkout from the Git common directory and the local queue; independent +clones use the queue's configured main checkout. Use repository-relative paths in shared reports, +not a hard-coded macOS or Windows home directory. A Human-selected folder overrides this default; +record its location privately in the task index. + +```text +.humanlayer/tasks// + README.md # clickable local evidence index and current blockers + screenshots//// + raw/ # original PNGs; private, unchanged + redacted/ # reviewed copies suitable for the intended audience + recordings//// + manifest.json # source, build and per-step evidence provenance +``` + +Use `ios/simulator`, `ios/phone`, `android/emulator` or `android/phone` for platform/lane. +Use a short run ID containing the capture date and source SHA prefix. File names describe the +scene, before/after state, locale, theme and OS text size, for example +`composer-after-zh-dark-max.png`. Record the actual OS text-size setting in the manifest; +`max` alone cannot identify the setting. Reruns get new run IDs rather than overwriting evidence. +Keep feature-showcase images under a named scene in the same layout. + +Existing tools keep their output contracts. In particular, `android-qa.ps1` writes captures and +private serial manifests under the Windows queue's `evidence///`; verification +tools may use their own task directories. Add links from the task index to these originals, or +copy selected files and record their source and SHA-256. Do not move live runner output, alter +its manifests, or duplicate a whole build/cache just to follow the display layout. + +## Record what each image proves + +Before capture, freeze the source and build identity. Alongside each journey record: + +- Issue/PR, task and batch ID; full source SHA, base/candidate SHAs, and acceptance SHA when + assigned. Record every dependency head for an integration batch and any dirty-source diff. +- Platform, simulator/emulator versus phone, device model, OS version, app identity, version, + build number/profile, native input fingerprint and binary hash/build receipt where available. + A reused dev client records both binary source and loaded JavaScript source. +- Capture time with timezone, locale, theme, viewport and exact OS text-size setting; route + type and model identifier when relevant. Use synthetic display names and test content. +- Action, visible result, canonical Host readback when required, and `pass`, `fail` or + `untested` with a reason. Link each screenshot/recording and record its SHA-256. +- Original-to-redacted mapping, redaction/inspection status and any public evidence URLs. + +Keep serials, private paths and environment details in the private manifest only. Keep credentials, +pairing links, QR payloads, account secrets and private keys out of captures and manifests. +Separate automated results, Human QA, Simulator/emulator, physical device and real-network +results. A screenshot does not prove a gesture, notification delivery, persistence or a model +reply; retain the corresponding interaction/recording/readback. iOS evidence never qualifies +Android, and a development binary does not qualify a store binary. + +Before sharing, inspect every selected frame for status-bar details, real computer/account names, +paths, keyboard suggestions and unrelated personal content. Keep originals local and produce +separate redacted copies. For full-frame showcase images, prepare synthetic names/data and a +privacy-safe status bar before capture; preserve the original resolution and disclose any +redaction. Generated artwork is presentation material, not runtime QA evidence. + +## Preserve a release snapshot + +After the release coordinator freezes one acceptance SHA, both machines use that SHA for their +final source/build checks. Each machine retains its own platform evidence under: + +```text +.humanlayer/releases/// + README.md + manifest.json + ios/ # retained on macOS + android/ # retained on Windows +``` + +For example, a 1.5.0 run uses `release-1.5.0` as the task slug and `1.5.0` as the release folder. +The shared directory shape is not automatic synchronization: the coordinator's index records +which machine holds each platform and which sanitized report is accessible to reviewers. +Mobile app versions/build numbers remain independent of the npm version and are recorded. + +Copy the selected originals, redacted derivatives, necessary recordings and provenance before +cleaning up a task worktree. Verify copied file hashes and open the images from the archive index. +The release manifest maps the accepted commit to the npm tag/package and exact mobile builds, +with separate acceptance and publication statuses; leave unknown or pending fields explicit. +If a fix changes the acceptance SHA, create a new archive directory and identify which checks +were rerun or remain untested. Keep prior evidence as prior-SHA evidence. + +Release archives and unresolved QA evidence are retained material, not disposable caches. Keep +them until the Human explicitly authorizes deletion or a verified relocation. Do not copy Host +profiles, credentials, app sandboxes, dependencies or native build folders into the archive. + +## Publish selected evidence + +For PRs, use reviewed derivatives and the commit-pinned image format in +[UI/UX PR visual evidence](pr-ui-visual-evidence.md); the queue may publish its assigned evidence +branch. Record the public URL and read it back to confirm that it renders. A local path is a +local inspection link, not proof that a GitHub reviewer can open the image. + +For release notes or store material, select from the accepted archive, check the intended +audience and attach only the approved public derivatives. Image retention does not authorize +uploading private originals, publishing a GitHub Release, submitting a store build or notifying +testers. Keep local inspection links in the Human handoff and public evidence links in GitHub.