feat: simulate-mouse-ui clicks and long-presses reach UI Toolkit controls

.agents/skills/uloop-screenshot/references/annotated-elements.md

@@ -10,7 +10,7 @@ Device Simulator is supported for this flow: prefer `--capture-mode rendering` (
 
 - `Label`: Index label in JSON (`A` = first in that sort, `B` = next, ...). Treat it as a stable handle, not as proof of what is actually in front: when overlapping elements cross parents, pick the target by `Path` and confirm the result with `simulate-mouse-ui`'s `HitGameObjectName` — the handler that received the events, resolved up the hierarchy from the raycast hit (or from `--target-path` in bypass mode), so it can name an ancestor of the element you clicked. Screenshot labels also include the interaction hint, such as `A / CLICK` or `B / DRAG`.
 - `Name`: Element name
-- `Path`: Hierarchy path from the scene root, for example `Canvas/Panel/Button`. Use this as `simulate-mouse-ui --target-path` when bypassing raycast blockers. UI Toolkit entries use a different form that `--target-path` does not accept; see [UI Toolkit Elements](#ui-toolkit-elements).
+- `Path`: Hierarchy path from the scene root, for example `Canvas/Panel/Button`. Use this as `simulate-mouse-ui --target-path` when bypassing raycast blockers. UI Toolkit entries use a different form that `--target-path` does not accept; click them at `SimX`/`SimY` instead. See [UI Toolkit Elements](#ui-toolkit-elements).
 - `Type`: Element type (`Button`, `Toggle`, `Slider`, `Dropdown`, `InputField`, `Scrollbar`, `Draggable`, `DropTarget`, `Selectable`, `PhysicsCollider`)
 - `Interaction`: Derived interaction category (`Click`, `Drag`, `Drop`, `Text`) or `Raycast` for clustered physics collider entries. Use this to choose between `simulate-mouse-ui --action Click`, drag actions, or `simulate-mouse-input` (including `--dry-run`).
 - `UiSystem`: UI system the entry came from: `UGUI`, `UIToolkit`, or empty for `PhysicsCollider` entries.
@@ -53,9 +53,9 @@ Limits:
 
 Clicking:
 
-- `simulate-mouse-input --action Click --x <SimX> --y <SimY>` reaches UI Toolkit controls when the scene has an EventSystem with an Input System UI input module.
-- Without an EventSystem the click does not reach UI Toolkit controls.
-- `simulate-mouse-ui` does not reach UI Toolkit controls, and its `--target-path` does not accept UI Toolkit `Path` values.
+- `simulate-mouse-ui --action Click --x <SimX> --y <SimY>` (or `LongPress`) reaches UI Toolkit controls, with or without an EventSystem. Its `--target-path` does not accept UI Toolkit `Path` values.
+- `simulate-mouse-input --action Click --x <SimX> --y <SimY>` also reaches them when the scene has an EventSystem with an Input System UI input module.
+- Without an EventSystem the `simulate-mouse-input` click does not reach UI Toolkit controls.
 
 ### PhysicsCollider Entries Per Closed Region
 

.agents/skills/uloop-simulate-mouse-input/SKILL.md

@@ -67,7 +67,7 @@ All rows below assume the New Input System is installed.
 
 | Scenario | Tool |
 |----------|------|
-| Click a Unity UI Button (IPointerClickHandler) | `simulate-mouse-ui` |
+| Click a uGUI or UI Toolkit button | `simulate-mouse-ui` |
 | Runtime logic reads `Mouse.current.leftButton` | `simulate-mouse-input` |
 | Runtime logic reads right-click | `simulate-mouse-input --button Right` |
 | Drag a UI slider | `simulate-mouse-ui --action Drag` |
@@ -110,7 +110,7 @@ uloop simulate-mouse-input --action SmoothDelta --delta-x 300 --delta-y 0 --dura
 ## Coordinate System
 
 `--x`/`--y` use top-left Game View coordinates: pass `AnnotatedElements[].SimX/SimY`
-directly (UI Toolkit entries need an EventSystem in the scene), convert raw screenshot pixels with `ScreenshotToInputFormula`, and never flip Y
+directly (UI Toolkit entries need an EventSystem with an Input System UI module here; `simulate-mouse-ui` reaches them without one), convert raw screenshot pixels with `ScreenshotToInputFormula`, and never flip Y
 in the caller (the tool converts internally). Conversion formula, Device Simulator
 notes, and how the injected position reads back inside Unity are in
 `references/output-and-coordinates.md`.

.agents/skills/uloop-simulate-mouse-input/references/output-and-coordinates.md

@@ -30,7 +30,7 @@ Verify visual outcome with a follow-up screenshot.
 
 - `--x` / `--y` use **top-left Game View coordinates**.
 - Raw image pixels from `uloop screenshot --capture-mode rendering` must be converted with `ScreenshotToInputFormula`.
-- `AnnotatedElements[].SimX/SimY` can be passed directly to this tool. For UI Toolkit entries (`UiSystem` `UIToolkit`), a `Click` reaches the control only when the scene has an EventSystem with an Input System UI input module; without one, nothing receives the click.
+- `AnnotatedElements[].SimX/SimY` can be passed directly to this tool. For UI Toolkit entries (`UiSystem` `UIToolkit`), a `Click` reaches the control only when the scene has an EventSystem with an Input System UI input module; without one, nothing receives the click. `simulate-mouse-ui` reaches them without an EventSystem.
 - Do not flip Y in the caller. The tool converts internally for Unity Input System:
 
 ```text

.agents/skills/uloop-simulate-mouse-ui/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: uloop-simulate-mouse-ui
 toolName: simulate-mouse-ui
-description: "Simulate PlayMode EventSystem UI mouse actions using screen coordinates. Use for UI clicks, long-presses, or drags from annotated screenshots."
+description: "Simulate PlayMode UI mouse actions on uGUI and UI Toolkit using screen coordinates. Use for UI clicks, long-presses, or drags from annotated screenshots."
 ---
 
 # Task
@@ -64,7 +64,7 @@ uloop simulate-mouse-ui --action <action> --x <x> --y <y> [options]
 - Get coordinates from `AnnotatedElements` JSON (`SimX`/`SimY`) — do NOT look up GameObject positions
 - `--bypass-raycast` still uses coordinates for pointer event positions, but chooses the clicked, long-pressed, or dragged GameObject by `--target-path`
 - If `--target-path` or `--drop-target-path` matches multiple active GameObjects, the command fails instead of choosing an arbitrary duplicate
-- Not for UI Toolkit entries (`UiSystem` `UIToolkit`); see `references/output.md`.
+- UI Toolkit entries (`UiSystem` `UIToolkit`) can be clicked and long-pressed at `SimX`/`SimY`; `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObjects only. Details: `references/ui-toolkit.md`.
 - Device Simulator play view is supported. Prefer `uloop screenshot --capture-mode rendering --annotate-elements` for coordinates; they use the simulated device resolution (`Handles.GetMainGameViewSize()` / `Screen`), not the Simulator chrome scale.
 
 ## Pause Point Inspection (Standard for E2E)
@@ -86,9 +86,6 @@ uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300
 # Long-press a button for 3 seconds
 uloop simulate-mouse-ui --action LongPress --x 400 --y 300 --duration 3.0
 
-# One-shot drag (start to end in one call)
-uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300
-
 # Slow drag for visual inspection
 uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300 --drag-speed 200
 
@@ -102,14 +99,14 @@ uloop simulate-mouse-ui --action DragEnd --x 600 --y 300
 ## Prerequisites
 
 - Unity must be in **PlayMode**
-- Target scene must have an **EventSystem** GameObject
-- UI elements must have a **GraphicRaycaster** on their Canvas
+- uGUI targets need an **EventSystem** and a **GraphicRaycaster** on their Canvas; UI Toolkit targets need neither
 - If you need runtime mouse input rather than UI pointer events, `simulate-mouse-input` assumes the project uses the New Input System; otherwise prefer `execute-dynamic-code`
 
 ## Output
 
 The response reports `Success`, `Message`, `Action`, `HitGameObjectName`, the used
-coordinates, and `InterruptedByPausePoint`/`PausePointHits`. Two readings that matter:
+coordinates, and `InterruptedByPausePoint`/`PausePointHits`; `UiSystem` and `HitPath` say
+which UI system was hit and give its path in the annotation's form. Two readings that matter:
 `Click`/`LongPress` on empty space still return `Success = true` with
 `HitGameObjectName = null` (drag actions on empty space fail), and on a pause-point
 interruption `Message` states whether the pointer event fired before the pause.

.agents/skills/uloop-simulate-mouse-ui/references/output.md

@@ -5,7 +5,9 @@ Returns JSON with:
 - `Success`: Whether the operation succeeded
 - `Message`: Status message (e.g. "Hit element: ButtonStart" or "No UI element under (x, y)")
 - `Action`: Echoes which action was executed (`Click`, `Drag`, `DragStart`, `DragMove`, `DragEnd`, or `LongPress`)
-- `HitGameObjectName`: Name of the GameObject that received the pointer events (nullable string; null if nothing was hit). It is the event-handler target resolved up the hierarchy from the raycast hit (the first `IPointerDownHandler`, else the first `IPointerClickHandler`, for `Click`/`LongPress`; the first `IDragHandler` for drags), so it can be an ancestor of the topmost raycast hit — a Button's child Text reports the Button. With `--bypass-raycast` it is the handler resolved the same way from `--target-path` (also possibly an ancestor), not the raycast hit and not necessarily the raw path GameObject itself
+- `HitGameObjectName`: Name of the GameObject that received the pointer events (nullable string; null if nothing was hit). It is the event-handler target resolved up the hierarchy from the raycast hit (the first `IPointerDownHandler`, else the first `IPointerClickHandler`, for `Click`/`LongPress`; the first `IDragHandler` for drags), so it can be an ancestor of the topmost raycast hit — a Button's child Text reports the Button. With `--bypass-raycast` it is the handler resolved the same way from `--target-path` (also possibly an ancestor), not the raycast hit and not necessarily the raw path GameObject itself. For a UI Toolkit hit it is the control's `name`, the same as the annotation's `Name`
+- `UiSystem`: Which UI system received the press (nullable string; `Click`/`LongPress` only, null if nothing was hit): `UGUI` for a target under a Canvas, `UIToolkit` for a UI Toolkit control, and `""` for a non-UI target such as a physics object
+- `HitPath`: Path of the hit target in the form `AnnotatedElements[].Path` uses (nullable string; `Click`/`LongPress` only): a hierarchy path for GameObjects, `Document/Path#element` for UI Toolkit controls
 - `PositionX`: Target X coordinate that was used
 - `PositionY`: Target Y coordinate that was used
 - `EndPositionX`: Drag end X coordinate (nullable float; populated for drag actions only)
@@ -18,4 +20,4 @@ Note: Click and LongPress on empty space (no UI element) still return `Success =
 
 ## UI Toolkit Elements
 
-`simulate-mouse-ui` drives the uGUI EventSystem raycast path, so it does not reach UI Toolkit controls (`AnnotatedElements` entries with `UiSystem` `UIToolkit`): a click at their `SimX`/`SimY` lands on the panel's own GameObject instead. `--target-path` does not accept their `Path` either. Click them with `simulate-mouse-input` instead, which works when the scene has an EventSystem with an Input System UI input module.
+`Click` and `LongPress` reach UI Toolkit controls at their annotated `SimX`/`SimY`. How presses are delivered and what is not supported: `ui-toolkit.md`.

.agents/skills/uloop-simulate-mouse-ui/references/ui-toolkit.md

@@ -0,0 +1,46 @@
+# simulate-mouse-ui and UI Toolkit
+
+`Click` and `LongPress` reach UI Toolkit controls on screen-space panels — the
+`AnnotatedElements` entries with `UiSystem` `UIToolkit`. Pass the entry's `SimX`/`SimY` as
+`--x`/`--y`; the response reports `UiSystem` `UIToolkit`, the control's `Name` as
+`HitGameObjectName`, and its annotation `Path` as `HitPath`.
+
+## How a press is delivered
+
+- The point is mapped to each panel the same way the screenshot annotation maps controls to
+  `SimX`/`SimY`, so an annotated point lands on the annotated control. It does not depend on
+  which Editor view was drawn last.
+- Panels are searched front to back by their panel settings' `sortingOrder`; the first panel
+  with a control under the point gets the press. The reported control is the outermost one an
+  annotation would list, so a click on a scrollbar's thumb reports the scrollbar.
+- Where a uGUI element and a UI Toolkit panel are both under the point, the uGUI element wins
+  only when its Canvas sorts above the panel (raycaster sort priority greater than the panel's
+  `sortingOrder`), the same rule the EventSystem applies to real input. A uGUI image without
+  any pointer handler in front still blocks the panel, as real input would.
+- A full-screen container left at the default `pickingMode` (`Position`) counts as a hit, so
+  uGUI underneath it is not reached — real input behaves the same way.
+- UI Toolkit presses do not need an EventSystem. With neither an EventSystem nor a UI Toolkit
+  control under the point, the command fails and says so.
+- A disabled control receives the press but does not react, as with real input.
+
+## LongPress
+
+A UI Toolkit `Button` fires `clicked` once when the long press is released (UI Toolkit's
+`Clickable` behaviour); a uGUI long press sends no click. A `RepeatButton` fires repeatedly
+while held.
+
+## Not supported
+
+- `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObject paths only;
+  a UI Toolkit `Path` (it contains `#`) is reported as not found.
+- Drag actions go through the uGUI EventSystem only.
+- World-space panels (Unity 6), as in the annotation.
+- Two panels with the same `sortingOrder` overlapping at the point: Unity orders them by
+  creation order, which is not exposed, so either one may be chosen.
+
+## Notes
+
+- Move the real mouse cursor out of the Game View while simulating. Real pointer input over
+  the Game View reaches the same panels and can interleave with the simulated press.
+- Device Simulator has the same limits as the annotation: coordinates use the simulated device
+  resolution.

.claude/skills/uloop-screenshot/references/annotated-elements.md

@@ -10,7 +10,7 @@ Device Simulator is supported for this flow: prefer `--capture-mode rendering` (
 
 - `Label`: Index label in JSON (`A` = first in that sort, `B` = next, ...). Treat it as a stable handle, not as proof of what is actually in front: when overlapping elements cross parents, pick the target by `Path` and confirm the result with `simulate-mouse-ui`'s `HitGameObjectName` — the handler that received the events, resolved up the hierarchy from the raycast hit (or from `--target-path` in bypass mode), so it can name an ancestor of the element you clicked. Screenshot labels also include the interaction hint, such as `A / CLICK` or `B / DRAG`.
 - `Name`: Element name
-- `Path`: Hierarchy path from the scene root, for example `Canvas/Panel/Button`. Use this as `simulate-mouse-ui --target-path` when bypassing raycast blockers. UI Toolkit entries use a different form that `--target-path` does not accept; see [UI Toolkit Elements](#ui-toolkit-elements).
+- `Path`: Hierarchy path from the scene root, for example `Canvas/Panel/Button`. Use this as `simulate-mouse-ui --target-path` when bypassing raycast blockers. UI Toolkit entries use a different form that `--target-path` does not accept; click them at `SimX`/`SimY` instead. See [UI Toolkit Elements](#ui-toolkit-elements).
 - `Type`: Element type (`Button`, `Toggle`, `Slider`, `Dropdown`, `InputField`, `Scrollbar`, `Draggable`, `DropTarget`, `Selectable`, `PhysicsCollider`)
 - `Interaction`: Derived interaction category (`Click`, `Drag`, `Drop`, `Text`) or `Raycast` for clustered physics collider entries. Use this to choose between `simulate-mouse-ui --action Click`, drag actions, or `simulate-mouse-input` (including `--dry-run`).
 - `UiSystem`: UI system the entry came from: `UGUI`, `UIToolkit`, or empty for `PhysicsCollider` entries.
@@ -53,9 +53,9 @@ Limits:
 
 Clicking:
 
-- `simulate-mouse-input --action Click --x <SimX> --y <SimY>` reaches UI Toolkit controls when the scene has an EventSystem with an Input System UI input module.
-- Without an EventSystem the click does not reach UI Toolkit controls.
-- `simulate-mouse-ui` does not reach UI Toolkit controls, and its `--target-path` does not accept UI Toolkit `Path` values.
+- `simulate-mouse-ui --action Click --x <SimX> --y <SimY>` (or `LongPress`) reaches UI Toolkit controls, with or without an EventSystem. Its `--target-path` does not accept UI Toolkit `Path` values.
+- `simulate-mouse-input --action Click --x <SimX> --y <SimY>` also reaches them when the scene has an EventSystem with an Input System UI input module.
+- Without an EventSystem the `simulate-mouse-input` click does not reach UI Toolkit controls.
 
 ### PhysicsCollider Entries Per Closed Region
 

.claude/skills/uloop-simulate-mouse-input/SKILL.md

@@ -67,7 +67,7 @@ All rows below assume the New Input System is installed.
 
 | Scenario | Tool |
 |----------|------|
-| Click a Unity UI Button (IPointerClickHandler) | `simulate-mouse-ui` |
+| Click a uGUI or UI Toolkit button | `simulate-mouse-ui` |
 | Runtime logic reads `Mouse.current.leftButton` | `simulate-mouse-input` |
 | Runtime logic reads right-click | `simulate-mouse-input --button Right` |
 | Drag a UI slider | `simulate-mouse-ui --action Drag` |
@@ -110,7 +110,7 @@ uloop simulate-mouse-input --action SmoothDelta --delta-x 300 --delta-y 0 --dura
 ## Coordinate System
 
 `--x`/`--y` use top-left Game View coordinates: pass `AnnotatedElements[].SimX/SimY`
-directly (UI Toolkit entries need an EventSystem in the scene), convert raw screenshot pixels with `ScreenshotToInputFormula`, and never flip Y
+directly (UI Toolkit entries need an EventSystem with an Input System UI module here; `simulate-mouse-ui` reaches them without one), convert raw screenshot pixels with `ScreenshotToInputFormula`, and never flip Y
 in the caller (the tool converts internally). Conversion formula, Device Simulator
 notes, and how the injected position reads back inside Unity are in
 `references/output-and-coordinates.md`.

.claude/skills/uloop-simulate-mouse-input/references/output-and-coordinates.md

@@ -30,7 +30,7 @@ Verify visual outcome with a follow-up screenshot.
 
 - `--x` / `--y` use **top-left Game View coordinates**.
 - Raw image pixels from `uloop screenshot --capture-mode rendering` must be converted with `ScreenshotToInputFormula`.
-- `AnnotatedElements[].SimX/SimY` can be passed directly to this tool. For UI Toolkit entries (`UiSystem` `UIToolkit`), a `Click` reaches the control only when the scene has an EventSystem with an Input System UI input module; without one, nothing receives the click.
+- `AnnotatedElements[].SimX/SimY` can be passed directly to this tool. For UI Toolkit entries (`UiSystem` `UIToolkit`), a `Click` reaches the control only when the scene has an EventSystem with an Input System UI input module; without one, nothing receives the click. `simulate-mouse-ui` reaches them without an EventSystem.
 - Do not flip Y in the caller. The tool converts internally for Unity Input System:
 
 ```text