Skip to content

feat(screenshot): add fill, editable shapes, redaction hardening, and recording auto zoom - #486

Merged
ggbond268 merged 5 commits into
mactools-app:mainfrom
idev19:feat/screenshot-editing-recording-zoom
Oct 16, 2026
Merged

ggbond268 merged 5 commits into
mactools-app:mainfrom
idev19:feat/screenshot-editing-recording-zoom

Conversation

@idev19

@idev19 idev19 commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Problem and result

This improves the Screenshot plugin's editing, redaction, text recognition, scrolling, pinning, saving, and recording, built on the plugin's existing architecture (overlay pool, RecordingLifecycle, CaptureStreamSession, ScreenshotEnvironment, declarative settings). The existing toolbar, mode bar, and capture surfaces keep their layout; new controls appear only where a feature needs them.

Changes

Editing

  • Spotlight tool: keeps dragged rounded regions clear and dims the rest; overlapping regions share one dimming layer.
  • Rectangles, ellipses, and spotlight regions are selected after drawing and can be moved, resized, nudged with arrow keys, and undone. There is no separate selection tool; choosing the active tool again returns to editing, so the annotation toolbar gains only the Spotlight button.
  • Fill for rectangles and ellipses, in the existing style row (hidden for other tools). It applies before drawing or to the selected shape; color changes also apply to the selected shape. Fills are opaque and draw above mosaic and spotlight.

Redaction

  • Mosaic now uses coarse 14-pt blocks that are area-averaged, quantized, offset by per-capture random jitter, and always opaque. Exact block averages can be matched against rendered candidates (as Depix does); the jitter seed differs per capture but is stable between preview and export.
  • Mosaic, filled rectangles, and QR masks expand to whole device pixels. Before this change, a cover whose edge fell on a half pixel blended the source into its edge column (verified by a test that fails without the alignment).
  • Blur is removed, including its string, keywords, synonyms, and the long descriptions in all locales. Docs keep recommending an opaque fill for sensitive identifiers.

Recognition — TextLayout rebuilds indentation, table columns, paragraph gaps, and side-by-side panels from Vision fragment geometry instead of joining lines by vertical order.

  • Closing the result panel of text recognition chosen from the mode bar now ends the capture. Previously it revealed the screenshot annotation toolbar in text-recognition mode; recognition started from the screenshot toolbar still returns to annotation. Annotation tools also stay hidden while a text-recognition selection is adjusted.
  • The "N lines" status excludes the blank lines that preserve layout, the result panel width is clamped to the screen, and Vision warm-up includes Chinese text so the first Chinese recognition does not load its model on demand.

Scrolling capture — Besides the existing ≤ 6 absolute threshold, a frame matches when its best offset is clearly better than the median alignment score. With capture noise ≥ 20 gray levels the previous stitcher rejected every frame after the first (900 of 2,400 rows). The ambiguity and stationary-footer checks remain; the footer tolerance scales with the matched noise floor and stays at 6 for clean frames.

Pinned images — Hover shows Copy / Save As / Close using the overlay's BarButton and glass style; images too small to contain them show the controls in an adjacent child panel. Pinned scrolling screenshots close after they are copied or saved.

Saving — Folder saves never replace an existing file (numbered names, atomic move). Cancelling Save As returns to the same selection and annotations instead of ending the capture.

Recording auto zoom (macOS 15+)

  • On by default; controlled by an Auto Zoom checkbox on the record bar and a new Recording section in plugin settings, sharing one plugin-scoped preference.
  • During capture, AppKit mouse monitors record pointer movement, clicks, and scrolling inside the region; keyboard input is never observed and no Accessibility permission is requested. Events are aligned to the first encoded frame through an SCStreamOutput clock.
  • After capture, RecordingZoomPlan builds a smoothed camera path (1.6× zoom, dead-zone cursor following, overview after 2 s idle or a window/app switch), and RecordingZoomExporter renders it at 60 fps with unchanged dimensions.
  • The recording panel shows progress and Keep Original. Cancellation or failure keeps the original (… Original.mov) and reports its name; a clip without clicks or scrolling moves the original into place without re-encoding; the finished movie is published before the original is removed.
  • A processing recorder no longer blocks screenshots, recognition, or scrolling capture; only a new recording waits. Plugin disable/teardown cancels processing and keeps the original.
  • privacy.dataObserved adds pointer-events; docs describe the in-memory, per-recording retention.

Capture before macOS 26 — Stream configurations now set ignoreShadowsSingleWindow = false. Its default also strips menu and popover shadows from display captures, which affected screenshots, scrolling capture, and recording on macOS 14 and 15 (macOS 26+ already sets ignoreShadows = false on the screenshot API). Not yet checked visually on macOS 14/15.

Compatibility: no PluginKit or host API changes; minHostVersion is unchanged. New strings cover all 12 locales.

Verification

  • Screenshot test classes, run with -only-testing (69 tests, all passing): AnnotationRendererTests, ScreenshotAnnotationEditingTests, ScrollSessionTests, TextLayoutTests, PinViewTests, RecordingZoomTests, RecordingActivityTests, RecordingZoomExporterTests, RecorderTests, CapturePipelineTests, CaptureRegionTests, ScreenshotPluginTests, RecognitionSessionTests. A new editing test covers mode-bar text recognition and fails without the fix.
  • New coverage targets consequential boundaries: Vision cannot read text under mosaic or fill; half-pixel cover edges do not blend the source (fails without the fix); mosaic is opaque, stable within a capture, and varies between captures; fill editing and undo; noisy scrolling frames (fails before the stitcher change); exporter duration/size/orientation/audio, 60 fps with sparse source frames, unchanged bytes without zoom, and cancellation/collision safety.
  • make script-tests (changelog, generated website data, complete localization): passing.
  • git diff --check: clean. No new compiler warnings in Screenshot sources.
  • Overlay tests log "Conflicting constraints" for the mode bar's zero-width initial frame; this PR does not change the mode bar.

The MacTools Dev build was launched locally to confirm the settings section. Still to exercise manually on macOS 15+: auto-zoom recording end to end (including Keep Original), Save As cancel returning to the selection, and pin controls on a small image.

UI evidence

Captured on macOS 26 from the plugin's real overlay, pin, and status-panel windows over a synthetic document (so no desktop content is included); the settings section is from the running MacTools Dev build. Assets live in docs/pr-assets/screenshot-editing-recording/.

Annotation toolbar — mosaic over the account number, a filled rectangle (selected, Fill highlighted at the end of the style row), and Spotlight on the total. The only new toolbar button is Spotlight.

Light Dark
Annotation toolbar with fill and spotlight, light Annotation toolbar with fill and spotlight, dark

Record bar with the Auto Zoom checkbox:

Light Dark
Record bar with Auto Zoom, light Record bar with Auto Zoom, dark

Recording panel while rendering auto zoom, with Keep Original:

Light Dark
Auto-zoom processing panel, light Auto-zoom processing panel, dark

Pinned images — hover controls inside a large image; beside an image too small to contain them:

Light Dark
Pin hover controls, light Pin hover controls, dark
Pin controls beside a small image, light Pin controls beside a small image, dark

Settings — new Recording section:

Recording settings section with Auto Zoom

Checklist

  • Core behavior and relevant risks are verified using existing tests, focused additions, or manual checks; results or limitations are recorded.
  • UI evidence is attached where applicable; shared components, themes, localization, and accessibility are respected.
  • User-facing documentation and changelog fragments are updated when required.
  • Plugin metadata, runtime contracts, action policies, and minimum-host compatibility agree.
  • Licensing and third-party notices are complete; no credentials, local configuration, or unrelated generated files are included.

🤖 Generated with Claude Code

Jonah and others added 4 commits October 15, 2026 04:14
… recording auto zoom

- Spotlight, plus selectable, movable, resizable rectangles, ellipses, and spotlight regions.
- Opaque fill for rectangles and ellipses, applied before drawing or to a selected shape.
- Mosaic hardening: 14-pt area-averaged, quantized, per-capture jittered, opaque blocks;
  masks expand to whole device pixels so fractional edges cannot blend the source.
- Remove the blur tool and its metadata.
- Recording auto zoom: pointer-driven 60 fps camera rendered after capture, with progress,
  Keep Original cancellation, and a shared record-bar/settings preference.
- OCR keeps indentation, table columns, paragraph gaps, and side-by-side panels.
- Scrolling stitch accepts matches clearly better than the median alignment, so capture
  noise no longer stalls stitching.
- Pinned images gain hover Copy/Save As/Close controls; scrolling results close after use.
- Folder saves never replace files; cancelling Save As returns to the selection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closing the panel in text-recognition mode ends the capture instead of revealing
annotation tools, moving the selection keeps those tools hidden, line counts exclude
layout blank lines, the panel width stays on screen, and Vision warm-up includes
Chinese text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ggbond268

Copy link
Copy Markdown
Collaborator

Thanks for the improvements! I found two issues worth fixing:

  1. Keep the long-screenshot result open until saving succeeds.
    In Pin.swift, the result window closes as soon as saveAs returns true, but that only means a destination was selected—the file is still being written. If writing fails, the user cannot retry from that window. Please close it only after a successful write, and keep it open on cancellation or failure.

  2. Preserve recording completion messages when another screenshot is taken.
    Starting a screenshot during auto-zoom processing changes the shared generation, so the recording's completion callback skips its success, failure, or “original recording kept” message. I reproduced this with an isolated callback test; the original recording is still preserved. Please use a separate recording session token so new screenshots do not invalidate these messages, while still suppressing them when the plugin is disabled.

Both issues appear limited to the screenshot plugin.

…rding messages

- Save As reports whether the file was written. A pinned scrolling screenshot closes only
  after a successful write, and capture returns to the selection after a write failure as
  well as a cancellation.
- Recording completion messages follow a session token that only plugin cancellation
  changes, so starting a screenshot while auto zoom renders no longer suppresses them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@idev19

idev19 commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Both are fixed in 24f6db2:

  1. saveAs now reports whether the file was actually written. A pinned scrolling screenshot closes only after a successful write and stays open on cancellation or failure. The capture Save As flow follows the same rule and returns to the selection if writing fails.
  2. Recording completion messages now use a session token that only cancel() (plugin disable) changes, so starting a screenshot while auto zoom renders no longer suppresses the saved / original-kept / failure messages. Added ScreenshotCoordinatorTests for this.

@ggbond268
ggbond268 merged commit 62d3bb5 into mactools-app:main Oct 16, 2026
2 checks passed
@idev19
idev19 deleted the feat/screenshot-editing-recording-zoom branch October 16, 2026 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants