diff --git a/.agents/skills/uloop-screenshot/references/annotated-elements.md b/.agents/skills/uloop-screenshot/references/annotated-elements.md index 69466c0964..69272a2bb2 100644 --- a/.agents/skills/uloop-screenshot/references/annotated-elements.md +++ b/.agents/skills/uloop-screenshot/references/annotated-elements.md @@ -53,7 +53,7 @@ Limits: Clicking: -- `simulate-mouse-ui --action Click --x --y ` (or `LongPress`) reaches UI Toolkit controls, with or without an EventSystem. Its `--target-path` does not accept UI Toolkit `Path` values. +- `simulate-mouse-ui --action Click --x --y ` (or `LongPress`, or a drag from that point) reaches UI Toolkit controls, with or without an EventSystem. Its `--target-path` does not accept UI Toolkit `Path` values. - `simulate-mouse-input --action Click --x --y ` also reaches them when the scene has an EventSystem with an Input System UI input module. - Without an EventSystem the `simulate-mouse-input` click does not reach UI Toolkit controls. diff --git a/.agents/skills/uloop-simulate-mouse-ui/SKILL.md b/.agents/skills/uloop-simulate-mouse-ui/SKILL.md index 0be9c8c4fb..3d67862bbd 100644 --- a/.agents/skills/uloop-simulate-mouse-ui/SKILL.md +++ b/.agents/skills/uloop-simulate-mouse-ui/SKILL.md @@ -64,7 +64,7 @@ uloop simulate-mouse-ui --action --x --y [options] - Get coordinates from `AnnotatedElements` JSON (`SimX`/`SimY`) — do NOT look up GameObject positions - `--bypass-raycast` still uses coordinates for pointer event positions, but chooses the clicked, long-pressed, or dragged GameObject by `--target-path` - If `--target-path` or `--drop-target-path` matches multiple active GameObjects, the command fails instead of choosing an arbitrary duplicate -- UI Toolkit entries (`UiSystem` `UIToolkit`) can be clicked and long-pressed at `SimX`/`SimY`; `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObjects only. Details: `references/ui-toolkit.md`. +- UI Toolkit entries (`UiSystem` `UIToolkit`) can be clicked, long-pressed, and dragged at `SimX`/`SimY`; `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObjects only. Details: `references/ui-toolkit.md`. - Device Simulator play view is supported. Prefer `uloop screenshot --capture-mode rendering --annotate-elements` for coordinates; they use the simulated device resolution (`Handles.GetMainGameViewSize()` / `Screen`), not the Simulator chrome scale. ## Pause Point Inspection (Standard for E2E) @@ -86,8 +86,8 @@ uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300 # Long-press a button for 3 seconds uloop simulate-mouse-ui --action LongPress --x 400 --y 300 --duration 3.0 -# Slow drag for visual inspection -uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300 --drag-speed 200 +# One-shot drag (add --drag-speed 200 to slow it for visual inspection) +uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300 # Split drag with hold (for inspection between steps) uloop simulate-mouse-ui --action DragStart --x 400 --y 300 diff --git a/.agents/skills/uloop-simulate-mouse-ui/references/output.md b/.agents/skills/uloop-simulate-mouse-ui/references/output.md index af20d8c00b..03086035fa 100644 --- a/.agents/skills/uloop-simulate-mouse-ui/references/output.md +++ b/.agents/skills/uloop-simulate-mouse-ui/references/output.md @@ -5,9 +5,9 @@ Returns JSON with: - `Success`: Whether the operation succeeded - `Message`: Status message (e.g. "Hit element: ButtonStart" or "No UI element under (x, y)") - `Action`: Echoes which action was executed (`Click`, `Drag`, `DragStart`, `DragMove`, `DragEnd`, or `LongPress`) -- `HitGameObjectName`: Name of the GameObject that received the pointer events (nullable string; null if nothing was hit). It is the event-handler target resolved up the hierarchy from the raycast hit (the first `IPointerDownHandler`, else the first `IPointerClickHandler`, for `Click`/`LongPress`; the first `IDragHandler` for drags), so it can be an ancestor of the topmost raycast hit — a Button's child Text reports the Button. With `--bypass-raycast` it is the handler resolved the same way from `--target-path` (also possibly an ancestor), not the raycast hit and not necessarily the raw path GameObject itself. For a UI Toolkit hit it is the control's `name`, the same as the annotation's `Name` -- `UiSystem`: Which UI system received the press (nullable string; `Click`/`LongPress` only, null if nothing was hit): `UGUI` for a target under a Canvas, `UIToolkit` for a UI Toolkit control, and `""` for a non-UI target such as a physics object -- `HitPath`: Path of the hit target in the form `AnnotatedElements[].Path` uses (nullable string; `Click`/`LongPress` only): a hierarchy path for GameObjects, `Document/Path#element` for UI Toolkit controls +- `HitGameObjectName`: Name of the GameObject that received the pointer events (nullable string; null if nothing was hit). It is the event-handler target resolved up the hierarchy from the raycast hit (the first `IPointerDownHandler`, else the first `IPointerClickHandler`, for `Click`/`LongPress`; the first `IDragHandler` for drags), so it can be an ancestor of the topmost raycast hit — a Button's child Text reports the Button. With `--bypass-raycast` it is the handler resolved the same way from `--target-path` (also possibly an ancestor), not the raycast hit and not necessarily the raw path GameObject itself. For a UI Toolkit hit it is the name of the outermost control an annotation would list (the same as the annotation's `Name`); when the picked element is not inside such a control (a bare `Label` or `VisualElement`), it is that element's own `name`, or its type name when empty, and the element is not annotated. Unlike uGUI, which reports null when no handler is found, UI Toolkit reports the picked element whether or not it handles pointer events +- `UiSystem`: Which UI system received the press (nullable string; `Click`/`LongPress` and drags, null if nothing was hit): `UGUI` for a target under a Canvas, `UIToolkit` for a UI Toolkit control, and `""` for a non-UI target such as a physics object +- `HitPath`: Path of the hit target in the form `AnnotatedElements[].Path` uses (nullable string; `Click`/`LongPress` and drags): a hierarchy path for GameObjects, `Document/Path#element` for UI Toolkit controls. Null when nothing was hit, or when the hit element belongs to no document, such as the menu of an open dropdown - `PositionX`: Target X coordinate that was used - `PositionY`: Target Y coordinate that was used - `EndPositionX`: Drag end X coordinate (nullable float; populated for drag actions only) @@ -20,4 +20,4 @@ Note: Click and LongPress on empty space (no UI element) still return `Success = ## UI Toolkit Elements -`Click` and `LongPress` reach UI Toolkit controls at their annotated `SimX`/`SimY`. How presses are delivered and what is not supported: `ui-toolkit.md`. +`Click`, `LongPress`, and the drag actions reach UI Toolkit controls at their annotated `SimX`/`SimY` (`--from-x`/`--from-y` for `Drag`). How presses and drags are delivered and what is not supported: `ui-toolkit.md`. diff --git a/.agents/skills/uloop-simulate-mouse-ui/references/ui-toolkit.md b/.agents/skills/uloop-simulate-mouse-ui/references/ui-toolkit.md index ec3bb79d44..373a3deee1 100644 --- a/.agents/skills/uloop-simulate-mouse-ui/references/ui-toolkit.md +++ b/.agents/skills/uloop-simulate-mouse-ui/references/ui-toolkit.md @@ -1,9 +1,9 @@ # simulate-mouse-ui and UI Toolkit -`Click` and `LongPress` reach UI Toolkit controls on screen-space panels — the -`AnnotatedElements` entries with `UiSystem` `UIToolkit`. Pass the entry's `SimX`/`SimY` as -`--x`/`--y`; the response reports `UiSystem` `UIToolkit`, the control's `Name` as -`HitGameObjectName`, and its annotation `Path` as `HitPath`. +`Click`, `LongPress`, and the drag actions reach UI Toolkit controls on screen-space panels — +the `AnnotatedElements` entries with `UiSystem` `UIToolkit`. Pass the entry's `SimX`/`SimY` as +`--x`/`--y` (`--from-x`/`--from-y` for `Drag`); the response reports `UiSystem` `UIToolkit`, the +control's `Name` as `HitGameObjectName`, and its annotation `Path` as `HitPath`. ## How a press is delivered @@ -29,18 +29,33 @@ A UI Toolkit `Button` fires `clicked` once when the long press is released (UI T `Clickable` behaviour); a uGUI long press sends no click. A `RepeatButton` fires repeatedly while held. +## Drags + +- A drag whose start point resolves to a UI Toolkit control (by the rule above) presses it + with the left button, moves the pointer, and releases at the end, so a `Slider` or a + `Scroller` follows the drag. The start point is what counts: the drag stays with that panel + wherever it moves. +- A labelled slider's bounds include its label; start the drag on the track. +- `--drop-target-path` applies to uGUI drags only. A `Drag` with it fails before anything is + sent; a `DragEnd` with it fails and leaves the drag in progress, so call `DragEnd` again + without it. +- If the control's document is disabled or destroyed during a split drag, the next `DragMove` + or `DragEnd` fails and the drag ends. A one-shot `Drag` stops sending and still succeeds. +- If a UI Toolkit handler throws, the button is released and the exception ends the command. + A `DragMove` that throws leaves the drag in progress, as a cancelled `DragMove` does. + ## Not supported - `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObject paths only; a UI Toolkit `Path` (it contains `#`) is reported as not found. -- Drag actions go through the uGUI EventSystem only. - World-space panels (Unity 6), as in the annotation. - Two panels with the same `sortingOrder` overlapping at the point: Unity orders them by creation order, which is not exposed, so either one may be chosen. ## Notes -- Move the real mouse cursor out of the Game View while simulating. Real pointer input over - the Game View reaches the same panels and can interleave with the simulated press. +- Move the real mouse cursor out of the Game View while simulating, and keep it away between + `DragStart` and `DragEnd`. Real pointer input over the Game View reaches the same panels and + can interleave with the simulated press or move a held slider. - Device Simulator has the same limits as the annotation: coordinates use the simulated device resolution. diff --git a/.claude/skills/uloop-screenshot/references/annotated-elements.md b/.claude/skills/uloop-screenshot/references/annotated-elements.md index 69466c0964..69272a2bb2 100644 --- a/.claude/skills/uloop-screenshot/references/annotated-elements.md +++ b/.claude/skills/uloop-screenshot/references/annotated-elements.md @@ -53,7 +53,7 @@ Limits: Clicking: -- `simulate-mouse-ui --action Click --x --y ` (or `LongPress`) reaches UI Toolkit controls, with or without an EventSystem. Its `--target-path` does not accept UI Toolkit `Path` values. +- `simulate-mouse-ui --action Click --x --y ` (or `LongPress`, or a drag from that point) reaches UI Toolkit controls, with or without an EventSystem. Its `--target-path` does not accept UI Toolkit `Path` values. - `simulate-mouse-input --action Click --x --y ` also reaches them when the scene has an EventSystem with an Input System UI input module. - Without an EventSystem the `simulate-mouse-input` click does not reach UI Toolkit controls. diff --git a/.claude/skills/uloop-simulate-mouse-ui/SKILL.md b/.claude/skills/uloop-simulate-mouse-ui/SKILL.md index 0be9c8c4fb..3d67862bbd 100644 --- a/.claude/skills/uloop-simulate-mouse-ui/SKILL.md +++ b/.claude/skills/uloop-simulate-mouse-ui/SKILL.md @@ -64,7 +64,7 @@ uloop simulate-mouse-ui --action --x --y [options] - Get coordinates from `AnnotatedElements` JSON (`SimX`/`SimY`) — do NOT look up GameObject positions - `--bypass-raycast` still uses coordinates for pointer event positions, but chooses the clicked, long-pressed, or dragged GameObject by `--target-path` - If `--target-path` or `--drop-target-path` matches multiple active GameObjects, the command fails instead of choosing an arbitrary duplicate -- UI Toolkit entries (`UiSystem` `UIToolkit`) can be clicked and long-pressed at `SimX`/`SimY`; `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObjects only. Details: `references/ui-toolkit.md`. +- UI Toolkit entries (`UiSystem` `UIToolkit`) can be clicked, long-pressed, and dragged at `SimX`/`SimY`; `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObjects only. Details: `references/ui-toolkit.md`. - Device Simulator play view is supported. Prefer `uloop screenshot --capture-mode rendering --annotate-elements` for coordinates; they use the simulated device resolution (`Handles.GetMainGameViewSize()` / `Screen`), not the Simulator chrome scale. ## Pause Point Inspection (Standard for E2E) @@ -86,8 +86,8 @@ uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300 # Long-press a button for 3 seconds uloop simulate-mouse-ui --action LongPress --x 400 --y 300 --duration 3.0 -# Slow drag for visual inspection -uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300 --drag-speed 200 +# One-shot drag (add --drag-speed 200 to slow it for visual inspection) +uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300 # Split drag with hold (for inspection between steps) uloop simulate-mouse-ui --action DragStart --x 400 --y 300 diff --git a/.claude/skills/uloop-simulate-mouse-ui/references/output.md b/.claude/skills/uloop-simulate-mouse-ui/references/output.md index af20d8c00b..03086035fa 100644 --- a/.claude/skills/uloop-simulate-mouse-ui/references/output.md +++ b/.claude/skills/uloop-simulate-mouse-ui/references/output.md @@ -5,9 +5,9 @@ Returns JSON with: - `Success`: Whether the operation succeeded - `Message`: Status message (e.g. "Hit element: ButtonStart" or "No UI element under (x, y)") - `Action`: Echoes which action was executed (`Click`, `Drag`, `DragStart`, `DragMove`, `DragEnd`, or `LongPress`) -- `HitGameObjectName`: Name of the GameObject that received the pointer events (nullable string; null if nothing was hit). It is the event-handler target resolved up the hierarchy from the raycast hit (the first `IPointerDownHandler`, else the first `IPointerClickHandler`, for `Click`/`LongPress`; the first `IDragHandler` for drags), so it can be an ancestor of the topmost raycast hit — a Button's child Text reports the Button. With `--bypass-raycast` it is the handler resolved the same way from `--target-path` (also possibly an ancestor), not the raycast hit and not necessarily the raw path GameObject itself. For a UI Toolkit hit it is the control's `name`, the same as the annotation's `Name` -- `UiSystem`: Which UI system received the press (nullable string; `Click`/`LongPress` only, null if nothing was hit): `UGUI` for a target under a Canvas, `UIToolkit` for a UI Toolkit control, and `""` for a non-UI target such as a physics object -- `HitPath`: Path of the hit target in the form `AnnotatedElements[].Path` uses (nullable string; `Click`/`LongPress` only): a hierarchy path for GameObjects, `Document/Path#element` for UI Toolkit controls +- `HitGameObjectName`: Name of the GameObject that received the pointer events (nullable string; null if nothing was hit). It is the event-handler target resolved up the hierarchy from the raycast hit (the first `IPointerDownHandler`, else the first `IPointerClickHandler`, for `Click`/`LongPress`; the first `IDragHandler` for drags), so it can be an ancestor of the topmost raycast hit — a Button's child Text reports the Button. With `--bypass-raycast` it is the handler resolved the same way from `--target-path` (also possibly an ancestor), not the raycast hit and not necessarily the raw path GameObject itself. For a UI Toolkit hit it is the name of the outermost control an annotation would list (the same as the annotation's `Name`); when the picked element is not inside such a control (a bare `Label` or `VisualElement`), it is that element's own `name`, or its type name when empty, and the element is not annotated. Unlike uGUI, which reports null when no handler is found, UI Toolkit reports the picked element whether or not it handles pointer events +- `UiSystem`: Which UI system received the press (nullable string; `Click`/`LongPress` and drags, null if nothing was hit): `UGUI` for a target under a Canvas, `UIToolkit` for a UI Toolkit control, and `""` for a non-UI target such as a physics object +- `HitPath`: Path of the hit target in the form `AnnotatedElements[].Path` uses (nullable string; `Click`/`LongPress` and drags): a hierarchy path for GameObjects, `Document/Path#element` for UI Toolkit controls. Null when nothing was hit, or when the hit element belongs to no document, such as the menu of an open dropdown - `PositionX`: Target X coordinate that was used - `PositionY`: Target Y coordinate that was used - `EndPositionX`: Drag end X coordinate (nullable float; populated for drag actions only) @@ -20,4 +20,4 @@ Note: Click and LongPress on empty space (no UI element) still return `Success = ## UI Toolkit Elements -`Click` and `LongPress` reach UI Toolkit controls at their annotated `SimX`/`SimY`. How presses are delivered and what is not supported: `ui-toolkit.md`. +`Click`, `LongPress`, and the drag actions reach UI Toolkit controls at their annotated `SimX`/`SimY` (`--from-x`/`--from-y` for `Drag`). How presses and drags are delivered and what is not supported: `ui-toolkit.md`. diff --git a/.claude/skills/uloop-simulate-mouse-ui/references/ui-toolkit.md b/.claude/skills/uloop-simulate-mouse-ui/references/ui-toolkit.md index ec3bb79d44..373a3deee1 100644 --- a/.claude/skills/uloop-simulate-mouse-ui/references/ui-toolkit.md +++ b/.claude/skills/uloop-simulate-mouse-ui/references/ui-toolkit.md @@ -1,9 +1,9 @@ # simulate-mouse-ui and UI Toolkit -`Click` and `LongPress` reach UI Toolkit controls on screen-space panels — the -`AnnotatedElements` entries with `UiSystem` `UIToolkit`. Pass the entry's `SimX`/`SimY` as -`--x`/`--y`; the response reports `UiSystem` `UIToolkit`, the control's `Name` as -`HitGameObjectName`, and its annotation `Path` as `HitPath`. +`Click`, `LongPress`, and the drag actions reach UI Toolkit controls on screen-space panels — +the `AnnotatedElements` entries with `UiSystem` `UIToolkit`. Pass the entry's `SimX`/`SimY` as +`--x`/`--y` (`--from-x`/`--from-y` for `Drag`); the response reports `UiSystem` `UIToolkit`, the +control's `Name` as `HitGameObjectName`, and its annotation `Path` as `HitPath`. ## How a press is delivered @@ -29,18 +29,33 @@ A UI Toolkit `Button` fires `clicked` once when the long press is released (UI T `Clickable` behaviour); a uGUI long press sends no click. A `RepeatButton` fires repeatedly while held. +## Drags + +- A drag whose start point resolves to a UI Toolkit control (by the rule above) presses it + with the left button, moves the pointer, and releases at the end, so a `Slider` or a + `Scroller` follows the drag. The start point is what counts: the drag stays with that panel + wherever it moves. +- A labelled slider's bounds include its label; start the drag on the track. +- `--drop-target-path` applies to uGUI drags only. A `Drag` with it fails before anything is + sent; a `DragEnd` with it fails and leaves the drag in progress, so call `DragEnd` again + without it. +- If the control's document is disabled or destroyed during a split drag, the next `DragMove` + or `DragEnd` fails and the drag ends. A one-shot `Drag` stops sending and still succeeds. +- If a UI Toolkit handler throws, the button is released and the exception ends the command. + A `DragMove` that throws leaves the drag in progress, as a cancelled `DragMove` does. + ## Not supported - `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObject paths only; a UI Toolkit `Path` (it contains `#`) is reported as not found. -- Drag actions go through the uGUI EventSystem only. - World-space panels (Unity 6), as in the annotation. - Two panels with the same `sortingOrder` overlapping at the point: Unity orders them by creation order, which is not exposed, so either one may be chosen. ## Notes -- Move the real mouse cursor out of the Game View while simulating. Real pointer input over - the Game View reaches the same panels and can interleave with the simulated press. +- Move the real mouse cursor out of the Game View while simulating, and keep it away between + `DragStart` and `DragEnd`. Real pointer input over the Game View reaches the same panels and + can interleave with the simulated press or move a held slider. - Device Simulator has the same limits as the annotation: coordinates use the simulated device resolution. diff --git a/Assets/Tests/Demo/UIToolkitAnnotationDemo.cs b/Assets/Tests/Demo/UIToolkitAnnotationDemo.cs index 78a82557e3..097387a995 100644 --- a/Assets/Tests/Demo/UIToolkitAnnotationDemo.cs +++ b/Assets/Tests/Demo/UIToolkitAnnotationDemo.cs @@ -21,6 +21,13 @@ public int GetClickCount(string elementName) return _clickCounts.TryGetValue(elementName, out int count) ? count : 0; } + // NaN when the back panel has no slider by that name, so a missing control cannot pass as a value. + public float GetSliderValue(string elementName) + { + Slider? slider = backDocument.rootVisualElement?.Q(elementName); + return slider != null ? slider.value : float.NaN; + } + // Start runs after every Awake and OnEnable, so both documents have created their roots by now // regardless of component order. private void Start() @@ -33,7 +40,8 @@ private void BuildBackPanel(VisualElement root) { AddPlaced(root, CreateCountingButton("play"), 100f, 100f, 200f, 50f); AddPlaced(root, new Toggle("Sound") { name = "sound" }, 100f, 180f, 200f, 30f); - AddPlaced(root, new Slider("Volume", 0f, 1f) { name = "volume" }, 100f, 240f, 300f, 30f); + // No label, so the slider's bounds are its track and the E2E can drag between its edges. + AddPlaced(root, new Slider(0f, 1f) { name = "volume" }, 100f, 240f, 300f, 30f); AddPlaced(root, new TextField("Player") { name = "player-name" }, 100f, 300f, 300f, 40f); DropdownField quality = new DropdownField( "Quality", new List { "Low", "Medium", "High" }, 1) { name = "quality" }; diff --git a/Assets/Tests/Editor/MouseUiDragTargetResolverTests.cs b/Assets/Tests/Editor/MouseUiDragTargetResolverTests.cs index 104754fb03..53081d07fc 100644 --- a/Assets/Tests/Editor/MouseUiDragTargetResolverTests.cs +++ b/Assets/Tests/Editor/MouseUiDragTargetResolverTests.cs @@ -47,7 +47,7 @@ public void Resolve_WithBypassPathBelowDragHandler_ReturnsHandlerAndRawRaycast() GameObjectPathUtility.GetFullPath(rawTarget)); Vector2 position = new(10f, 20f); - (RaycastResult startRaycast, GameObject? target, SimulateMouseUiResponse? failureResponse) = + (MouseUiDragStartTarget start, SimulateMouseUiResponse? failureResponse) = MouseUiDragTargetResolver.Resolve( command, eventSystem, @@ -55,8 +55,8 @@ public void Resolve_WithBypassPathBelowDragHandler_ReturnsHandlerAndRawRaycast() position, position); - Assert.That(startRaycast.gameObject, Is.SameAs(rawTarget)); - Assert.That(target, Is.SameAs(expectedTarget)); + Assert.That(start.StartRaycast.gameObject, Is.SameAs(rawTarget)); + Assert.That(start.UguiTarget, Is.SameAs(expectedTarget)); Assert.That(failureResponse, Is.Null); } @@ -73,7 +73,7 @@ public void Resolve_WithBypassPathWithoutDragHandler_ReturnsRawRaycastWithoutFai GameObjectPathUtility.GetFullPath(rawTarget)); Vector2 position = new(30f, 40f); - (RaycastResult startRaycast, GameObject? target, SimulateMouseUiResponse? failureResponse) = + (MouseUiDragStartTarget start, SimulateMouseUiResponse? failureResponse) = MouseUiDragTargetResolver.Resolve( command, eventSystem, @@ -81,8 +81,8 @@ public void Resolve_WithBypassPathWithoutDragHandler_ReturnsRawRaycastWithoutFai position, position); - Assert.That(startRaycast.gameObject, Is.SameAs(rawTarget)); - Assert.That(target, Is.Null); + Assert.That(start.StartRaycast.gameObject, Is.SameAs(rawTarget)); + Assert.That(start.UguiTarget, Is.Null); Assert.That(failureResponse, Is.Null); } @@ -99,7 +99,7 @@ public void Resolve_WithMissingBypassPath_ReturnsPathFailure() targetPath); Vector2 position = new(50f, 60f); - (RaycastResult startRaycast, GameObject? target, SimulateMouseUiResponse? failureResponse) = + (MouseUiDragStartTarget start, SimulateMouseUiResponse? failureResponse) = MouseUiDragTargetResolver.Resolve( command, eventSystem, @@ -107,8 +107,8 @@ public void Resolve_WithMissingBypassPath_ReturnsPathFailure() position, position); - Assert.That(startRaycast.gameObject, Is.Null); - Assert.That(target, Is.Null); + Assert.That(start.StartRaycast.gameObject, Is.Null); + Assert.That(start.UguiTarget, Is.Null); Assert.That(failureResponse, Is.Not.Null); Assert.That(failureResponse!.Message, Is.EqualTo($"TargetPath '{targetPath}' was not found.")); Assert.That(failureResponse.Action, Is.EqualTo(MouseAction.DragStart.ToString())); diff --git a/Assets/Tests/Editor/MouseUiSimulationResponseFactoryTests.cs b/Assets/Tests/Editor/MouseUiSimulationResponseFactoryTests.cs index f09de75b6b..22f357e89f 100644 --- a/Assets/Tests/Editor/MouseUiSimulationResponseFactoryTests.cs +++ b/Assets/Tests/Editor/MouseUiSimulationResponseFactoryTests.cs @@ -216,12 +216,14 @@ public void CreateDragResult_WithBypassMode_MapsMessageAndFields( command, startPosition, endPosition, - "DragTarget"); + new MouseUiHitReport("DragTarget", "Canvas/DragTarget", "UGUI")); Assert.That(response.Success, Is.True); Assert.That(response.Message, Is.EqualTo(expectedMessage)); Assert.That(response.Action, Is.EqualTo(MouseAction.Drag.ToString())); Assert.That(response.HitGameObjectName, Is.EqualTo("DragTarget")); + Assert.That(response.UiSystem, Is.EqualTo("UGUI")); + Assert.That(response.HitPath, Is.EqualTo("Canvas/DragTarget")); Assert.That(response.PositionX, Is.EqualTo(startPosition.x)); Assert.That(response.PositionY, Is.EqualTo(startPosition.y)); Assert.That(response.EndPositionX, Is.EqualTo(endPosition.x)); @@ -244,18 +246,44 @@ public void CreateDragEndResult_WithFinalPosition_MapsMessageAndFields() SimulateMouseUiResponse response = MouseUiSimulationResponseFactory.CreateDragEndResult( command, endPosition, - "DragTarget"); + new MouseUiHitReport("DragTarget", "Canvas/DragTarget", "UGUI")); Assert.That(response.Success, Is.True); Assert.That( response.Message, Is.EqualTo("Drag ended on 'DragTarget' at (30.5, 40.3) at 125 px/s")); + Assert.That(response.UiSystem, Is.EqualTo("UGUI")); + Assert.That(response.HitPath, Is.EqualTo("Canvas/DragTarget")); Assert.That(response.Action, Is.EqualTo(MouseAction.DragEnd.ToString())); Assert.That(response.HitGameObjectName, Is.EqualTo("DragTarget")); Assert.That(response.PositionX, Is.EqualTo(endPosition.x)); Assert.That(response.PositionY, Is.EqualTo(endPosition.y)); } + /// + /// Verifies drag results name a UI Toolkit start element as such and carry its UI system and path. + /// + [Test] + public void CreateDragResults_WithUIToolkitHit_NameUIToolkitElement() + { + MouseUiHitReport hit = new MouseUiHitReport("volume", "Menu/Document#volume", "UIToolkit"); + Vector2 start = new(100f, 200f); + Vector2 end = new(300f, 200f); + + SimulateMouseUiResponse drag = MouseUiSimulationResponseFactory.CreateDragResult( + CreateCommand(new SimulateMouseUiSchema { Action = UnityCliLoopMouseUiAction.Drag, DragSpeed = 100f }), start, end, hit); + SimulateMouseUiResponse dragEnd = MouseUiSimulationResponseFactory.CreateDragEndResult( + CreateCommand(new SimulateMouseUiSchema { Action = UnityCliLoopMouseUiAction.DragEnd, DragSpeed = 100f }), end, hit); + + Assert.That(drag.Message, Is.EqualTo("Dragged UI Toolkit element 'volume' from (100.0, 200.0) to (300.0, 200.0) at 100 px/s")); + Assert.That(dragEnd.Message, Is.EqualTo("Drag ended on UI Toolkit element 'volume' at (300.0, 200.0) at 100 px/s")); + foreach (SimulateMouseUiResponse response in new[] { drag, dragEnd }) + { + Assert.That(response.UiSystem, Is.EqualTo("UIToolkit")); + Assert.That(response.HitPath, Is.EqualTo("Menu/Document#volume")); + } + } + private static MouseUiSimulationCommand CreateCommand(SimulateMouseUiSchema schema) { (MouseUiSimulationCommand? command, string? errorMessage) = diff --git a/Assets/Tests/PlayMode/SimulateMouseUiTests.cs b/Assets/Tests/PlayMode/SimulateMouseUiTests.cs index 2e4cb79f46..33ae6c3623 100644 --- a/Assets/Tests/PlayMode/SimulateMouseUiTests.cs +++ b/Assets/Tests/PlayMode/SimulateMouseUiTests.cs @@ -862,6 +862,42 @@ public IEnumerator DragSplit_Should_CompleteFullCycle() Assert.IsTrue(tracker.EndDragCalled, "EndDrag should be fired"); } + /// + /// Verifies a split drag whose uGUI target was destroyed counts as ended: DragMove reports no drag, and + /// neither a click nor a new DragStart is refused as overlapping an active drag. + /// + [UnityTest] + public IEnumerator SplitDrag_WhenUguiTargetDestroyed_TreatsDragAsEnded() + { + ClickTracker clickTracker = CreateClickableElement("ClickTarget", new Vector2(0, 150), new Vector2(200, 80)); + yield return StartDragOnNewElement(); + + Object.Destroy(lastDragTracker.gameObject); + yield return null; + + yield return RunTool(new JObject + { + ["action"] = MouseAction.DragMove.ToString(), + ["x"] = lastDragScreenPos.x + 50f, + ["y"] = lastDragScreenPos.y + }); + Assert.IsFalse(lastResponse.Success); + Assert.AreEqual("No drag in progress. Call DragStart first.", lastResponse.Message); + + Vector2 clickPos = ScreenToInput(GetScreenPosition(clickTracker.gameObject)); + yield return RunTool(new JObject + { + ["action"] = MouseAction.Click.ToString(), + ["x"] = clickPos.x, + ["y"] = clickPos.y + }); + Assert.IsTrue(lastResponse.Success, lastResponse.Message); + Assert.IsTrue(clickTracker.PointerClickCalled, "the click must reach its target"); + + yield return StartDragOnNewElement(); + Assert.AreEqual("DragTarget", lastResponse.HitGameObjectName); + } + [UnityTest] public IEnumerator DragStart_WhenAlreadyDragging_Should_ReturnFailure() { diff --git a/Assets/Tests/PlayMode/SimulateMouseUiToolkitDragTests.cs b/Assets/Tests/PlayMode/SimulateMouseUiToolkitDragTests.cs new file mode 100644 index 0000000000..bc7499dc7a --- /dev/null +++ b/Assets/Tests/PlayMode/SimulateMouseUiToolkitDragTests.cs @@ -0,0 +1,843 @@ +#nullable enable +using System.Collections; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; +using io.github.hatayama.UnityCliLoop.FirstPartyTools; +using io.github.hatayama.UnityCliLoop.Runtime; +using io.github.hatayama.UnityCliLoop.ToolContracts; +using Newtonsoft.Json.Linq; +using NUnit.Framework; +using UnityEditor; +using UnityEngine; +using UnityEngine.EventSystems; +using UnityEngine.TestTools; +using UnityEngine.UI; +using UnityEngine.UIElements; + +using Image = UnityEngine.UI.Image; +using Slider = UnityEngine.UIElements.Slider; +using UIToolkitButton = UnityEngine.UIElements.Button; + +namespace io.github.hatayama.UnityCliLoop.Tests.PlayMode +{ + /// + /// Verifies simulate-mouse-ui drags reach UI Toolkit controls in one shot and split into steps, work without + /// an EventSystem for UI Toolkit, refuse drop targets on UI Toolkit, and release the pointer on every exit. + /// The UI Toolkit control sits at the centre of the Game View. Each test decides whether an EventSystem + /// exists. + /// + public sealed class SimulateMouseUiToolkitDragTests + { + private const string RuntimeThemePath = "Assets/UI Toolkit/UnityThemes/UnityDefaultRuntimeTheme.tss"; + private const string DocumentName = "SimulateMouseUiToolkitDragTests_Document"; + private const float SliderWidth = 300f; + private const float SliderHeight = 40f; + private const float EdgeInset = 5f; + private const string NoEventSystemMessage = "No EventSystem found in the scene. Ensure an EventSystem GameObject exists."; + private const string NoDragMessage = "No drag in progress. Call DragStart first."; + + private readonly List _createdObjects = new List(); + private ExistingEventSystemDisableScope _eventSystemDisableScope = null!; + private GameObject _canvasObject = null!; + private SimulateMouseUiTool _tool = null!; + private SimulateMouseUiResponse _lastResponse = null!; + private DragTracker _lastDragTracker = null!; + private Slider _lastSlider = null!; + + [UnitySetUp] + public IEnumerator SetUp() + { + _eventSystemDisableScope = new ExistingEventSystemDisableScope(); + + _canvasObject = new GameObject("TestCanvas"); + _createdObjects.Add(_canvasObject); + Canvas canvas = _canvasObject.AddComponent(); + canvas.renderMode = RenderMode.ScreenSpaceOverlay; + _canvasObject.AddComponent(); + _canvasObject.AddComponent(); + + _tool = new SimulateMouseUiTool(); + yield return null; + } + + [UnityTearDown] + public IEnumerator TearDown() + { + MouseDragState.Clear(); + foreach (Object createdObject in _createdObjects) + { + if (createdObject != null) + { + Object.Destroy(createdObject); + } + } + + _createdObjects.Clear(); + yield return null; + + _eventSystemDisableScope.Restore(); + } + + /// + /// Verifies a one-shot drag across a UI Toolkit slider moves its value to the end, reports the slider, + /// and leaves no pointer capture (D1). + /// + [UnityTest] + public IEnumerator Drag_OnUIToolkitSlider_MovesValue() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider slider); + yield return WaitForLayout(); + + yield return RunTool(DragParameters(SliderLeft(), SliderRight(), 0f)); + + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.GreaterOrEqual(slider.value, 0.9f); + Assert.AreEqual("volume", _lastResponse.HitGameObjectName); + Assert.AreEqual("UIToolkit", _lastResponse.UiSystem); + Assert.AreEqual(DocumentName + "#volume", _lastResponse.HitPath); + Assert.IsNull(CapturingElement(document)); + } + + /// + /// Verifies a one-shot drag on a UI Toolkit slider works without an EventSystem (D2). + /// + [UnityTest] + public IEnumerator Drag_OnUIToolkitSliderWithoutEventSystem_MovesValue() + { + CreateDocumentWithSlider(out Slider slider); + yield return WaitForLayout(); + + yield return RunTool(DragParameters(SliderLeft(), SliderRight(), 0f)); + + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.GreaterOrEqual(slider.value, 0.9f); + } + + /// + /// Verifies a drag with neither an EventSystem nor a UI Toolkit control at the start fails naming both + /// (D5). + /// + [UnityTest] + public IEnumerator Drag_WithoutEventSystemOnEmptyPoint_Fails() + { + yield return null; + Vector2 start = new Vector2(10f, 10f); + + yield return RunTool(DragParameters(start, new Vector2(60f, 10f), 0f)); + + Assert.IsFalse(_lastResponse.Success); + Assert.AreEqual( + "No EventSystem found in the scene, and no UI Toolkit element is at (10.0, 10.0). Ensure an EventSystem GameObject exists for uGUI.", + _lastResponse.Message); + } + + /// + /// Verifies a drag on UI Toolkit with a drop target that exists fails before anything is sent (D6). + /// + [UnityTest] + public IEnumerator Drag_OnUIToolkitWithDropTargetPath_FailsBeforeDispatch() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider slider); + GameObject dropZone = CreateCanvasElement("DropZone", new Vector2(0f, 300f)); + yield return WaitForLayout(); + JObject parameters = DragParameters(SliderLeft(), SliderRight(), 0f); + parameters["dropTargetPath"] = GameObjectPathUtility.GetFullPath(dropZone); + + yield return RunTool(parameters); + + Assert.IsFalse(_lastResponse.Success); + Assert.AreEqual( + "DropTargetPath applies to uGUI drags only; this drag starts on UI Toolkit element 'volume'.", + _lastResponse.Message); + Assert.AreEqual(0f, slider.value); + Assert.IsNull(CapturingElement(document)); + } + + /// + /// Verifies a drag on UI Toolkit with a drop target that does not exist reports the UI Toolkit + /// restriction rather than the missing path (D20). + /// + [UnityTest] + public IEnumerator Drag_OnUIToolkitWithMissingDropTargetPath_ReportsUIToolkitRestriction() + { + CreateEventSystem(); + CreateDocumentWithSlider(out Slider slider); + yield return WaitForLayout(); + JObject parameters = DragParameters(SliderLeft(), SliderRight(), 0f); + parameters["dropTargetPath"] = "Missing/DropZone"; + + yield return RunTool(parameters); + + Assert.IsFalse(_lastResponse.Success); + Assert.AreEqual( + "DropTargetPath applies to uGUI drags only; this drag starts on UI Toolkit element 'volume'.", + _lastResponse.Message); + Assert.AreEqual(0f, slider.value); + } + + /// + /// Verifies a split drag on a UI Toolkit slider moves the value at each step and ends with no drag in + /// progress and no pointer capture (D7). + /// + [UnityTest] + public IEnumerator SplitDrag_OnUIToolkitSlider_MovesStepwiseAndClears() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider slider); + yield return WaitForLayout(); + + yield return RunSplitDrag(slider); + + Assert.LessOrEqual(slider.value, 0.1f); + Assert.IsFalse(MouseDragState.IsDragging); + Assert.IsNull(CapturingElement(document)); + } + + /// + /// Verifies a split drag on UI Toolkit runs every step without an EventSystem (D8). + /// + [UnityTest] + public IEnumerator SplitDrag_OnUIToolkitWithoutEventSystem_Completes() + { + UIDocument document = CreateDocumentWithSlider(out Slider slider); + yield return WaitForLayout(); + + yield return RunSplitDrag(slider); + + Assert.LessOrEqual(slider.value, 0.1f); + Assert.IsFalse(MouseDragState.IsDragging); + Assert.IsNull(CapturingElement(document)); + } + + /// + /// Verifies a uGUI split drag whose EventSystem goes away fails DragMove as before and keeps the drag + /// (D9). + /// + [UnityTest] + public IEnumerator SplitDrag_OnUguiWhenEventSystemRemoved_FailsAsBefore() + { + GameObject eventSystem = CreateEventSystem(); + GameObject dragTarget = CreateCanvasElement("UguiDragTarget", Vector2.zero); + dragTarget.AddComponent(); + dragTarget.AddComponent(); + yield return null; + Vector2 center = GameViewCenter(); + yield return RunTool(SplitParameters(MouseAction.DragStart, center)); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + + eventSystem.SetActive(false); + yield return RunTool(SplitParameters(MouseAction.DragMove, center + new Vector2(50f, 0f))); + + Assert.IsFalse(_lastResponse.Success); + Assert.AreEqual(NoEventSystemMessage, _lastResponse.Message); + Assert.IsTrue(MouseDragState.IsDragging, "the drag must stay in progress"); + } + + /// + /// Verifies a drag starting where a uGUI drag target's Canvas sorts above the panel drags the uGUI target + /// and leaves the slider alone. + /// + [UnityTest] + public IEnumerator Drag_WhereUguiCanvasSortsAbovePanel_DragsUgui() + { + yield return DragOverlap(canvasSortingOrder: 11, panelSortingOrder: 10f); + + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.AreEqual("UGUI", _lastResponse.UiSystem); + Assert.AreEqual("UguiDragTarget", _lastResponse.HitGameObjectName); + Assert.IsTrue(_lastDragTracker.BeginDragCalled, "the uGUI target must be dragged"); + Assert.AreEqual(0f, _lastSlider.value, "the slider must not move"); + } + + /// + /// Verifies a drag starting where the panel sorts above a uGUI drag target's Canvas drags the slider and + /// leaves the uGUI target alone. + /// + [UnityTest] + public IEnumerator Drag_WherePanelSortsAboveUguiCanvas_DragsUIToolkit() + { + yield return DragOverlap(canvasSortingOrder: 9, panelSortingOrder: 10f); + + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.AreEqual("UIToolkit", _lastResponse.UiSystem); + Assert.GreaterOrEqual(_lastSlider.value, 0.9f); + Assert.IsFalse(_lastDragTracker.BeginDragCalled, "the uGUI target must not be dragged"); + Assert.AreEqual(0, _lastDragTracker.DragCallCount, "the uGUI target must not be dragged"); + } + + /// + /// Verifies a bypass drag on an existing uGUI drag target fails without an EventSystem, for both a + /// one-shot Drag and a DragStart, and sends it nothing. + /// + [UnityTest] + public IEnumerator BypassDrag_WithoutEventSystem_FailsAsBefore() + { + Assert.IsNull(EventSystem.current, "Setup: no EventSystem must be active."); + GameObject dragTarget = CreateCanvasElement("UguiDragTarget", Vector2.zero); + dragTarget.AddComponent(); + DragTracker tracker = dragTarget.AddComponent(); + yield return null; + Vector2 center = GameViewCenter(); + + JObject drag = DragParameters(center, center + new Vector2(50f, 0f), 0f); + drag["bypassRaycast"] = true; + drag["targetPath"] = "TestCanvas/UguiDragTarget"; + yield return RunTool(drag); + + Assert.IsFalse(_lastResponse.Success); + Assert.AreEqual(NoEventSystemMessage, _lastResponse.Message); + + JObject dragStart = SplitParameters(MouseAction.DragStart, center); + dragStart["bypassRaycast"] = true; + dragStart["targetPath"] = "TestCanvas/UguiDragTarget"; + yield return RunTool(dragStart); + + Assert.IsFalse(_lastResponse.Success); + Assert.AreEqual(NoEventSystemMessage, _lastResponse.Message); + Assert.IsFalse(MouseDragState.IsDragging); + Assert.IsFalse(tracker.BeginDragCalled, "the uGUI target must receive nothing"); + Assert.AreEqual(0, tracker.DragCallCount, "the uGUI target must receive nothing"); + } + + /// + /// Verifies a split drag whose UI Toolkit document was disabled fails the next step and ends the drag + /// (D10). + /// + [UnityTest] + public IEnumerator SplitDrag_WhenUIToolkitDocumentDisabled_FailsAndClears() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider _); + yield return WaitForLayout(); + yield return RunTool(SplitParameters(MouseAction.DragStart, SliderLeft())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + + document.enabled = false; + yield return RunTool(SplitParameters(MouseAction.DragMove, GameViewCenter())); + + Assert.IsFalse(_lastResponse.Success); + Assert.AreEqual( + "Drag target UI Toolkit element is no longer available (its document was disabled or destroyed).", + _lastResponse.Message); + Assert.IsFalse(MouseDragState.IsDragging); + } + + /// + /// Verifies a click and a one-shot drag are refused while a UI Toolkit split drag is in progress (D11). + /// + [UnityTest] + public IEnumerator Click_WhileUIToolkitSplitDragActive_IsRejected() + { + CreateEventSystem(); + CreateDocumentWithSlider(out Slider _); + yield return WaitForLayout(); + yield return RunTool(SplitParameters(MouseAction.DragStart, SliderLeft())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + + yield return RunTool(SplitParameters(MouseAction.Click, GameViewCenter())); + Assert.AreEqual("Cannot Click while a split drag is active. Call DragEnd first.", _lastResponse.Message); + + yield return RunTool(DragParameters(SliderLeft(), SliderRight(), 0f)); + Assert.AreEqual("Cannot Drag while a split drag is active. Call DragEnd first.", _lastResponse.Message); + } + + /// + /// Verifies DragEnd with a drop target on a UI Toolkit drag fails without ending the drag, and DragEnd + /// without it then succeeds (D12). + /// + [UnityTest] + public IEnumerator DragEnd_OnUIToolkitWithDropTargetPath_FailsAndKeepsDrag() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider _); + GameObject dropZone = CreateCanvasElement("DropZone", new Vector2(0f, 300f)); + yield return WaitForLayout(); + yield return RunTool(SplitParameters(MouseAction.DragStart, SliderLeft())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + + JObject dragEnd = SplitParameters(MouseAction.DragEnd, SliderRight()); + dragEnd["dropTargetPath"] = GameObjectPathUtility.GetFullPath(dropZone); + yield return RunTool(dragEnd); + + Assert.IsFalse(_lastResponse.Success); + Assert.AreEqual( + "DropTargetPath applies to uGUI drags only; this drag starts on UI Toolkit element 'volume'.", + _lastResponse.Message); + Assert.IsTrue(MouseDragState.IsDragging, "the drag must stay in progress"); + + yield return RunTool(SplitParameters(MouseAction.DragEnd, SliderRight())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.IsNull(CapturingElement(document)); + } + + /// + /// Verifies a split drag whose UI Toolkit document was destroyed counts as ended, as a destroyed uGUI + /// target does (D17). + /// + [UnityTest] + public IEnumerator SplitDrag_WhenUIToolkitDocumentDestroyed_TreatsDragAsEnded() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider _); + yield return WaitForLayout(); + yield return RunTool(SplitParameters(MouseAction.DragStart, SliderLeft())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + + Object.Destroy(document.gameObject); + yield return null; + + yield return RunTool(SplitParameters(MouseAction.DragMove, GameViewCenter())); + Assert.IsFalse(_lastResponse.Success); + Assert.AreEqual(NoDragMessage, _lastResponse.Message); + + yield return RunTool(SplitParameters(MouseAction.Click, GameViewCenter())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + } + + /// + /// Verifies a drag from a UI Toolkit button to a point outside it sends the press and release without + /// clicking, as real input would (D18). + /// + [UnityTest] + public IEnumerator Drag_FromUIToolkitButtonToOutside_DoesNotClick() + { + CreateEventSystem(); + UIToolkitButton button = CreateDocumentWithButton(out int[] clicks); + yield return WaitForLayout(); + Vector2 center = GameViewCenter(); + + yield return RunTool(DragParameters(center, center + new Vector2(0f, 200f), 0f)); + + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.AreEqual(0, clicks[0]); + Assert.AreEqual(DocumentName + "#" + button.name, _lastResponse.HitPath); + } + + /// + /// Verifies DragMove without an EventSystem and without a drag reports that no drag is in progress (D19). + /// + [UnityTest] + public IEnumerator DragMove_WithoutEventSystemAndNoDrag_ReportsNoDrag() + { + yield return null; + + yield return RunTool(SplitParameters(MouseAction.DragMove, GameViewCenter())); + + Assert.IsFalse(_lastResponse.Success); + Assert.AreEqual(NoDragMessage, _lastResponse.Message); + } + + /// + /// Verifies cancelling a one-shot UI Toolkit drag while it moves releases the pointer capture (D21). + /// + [UnityTest] + public IEnumerator Drag_OnUIToolkitSlider_WhenCancelledMidway_ReleasesCapture() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider slider); + yield return WaitForLayout(); + using CancellationTokenSource cancellation = new CancellationTokenSource(); + Task task = _tool.ExecuteAsync(DragParameters(SliderLeft(), SliderRight(), 50f), cancellation.Token); + yield return WaitUntilSliderMoves(slider); + + cancellation.Cancel(); + yield return WaitForCancelled(task); + + Assert.IsNull(CapturingElement(document)); + } + + /// + /// Verifies cancelling a UI Toolkit DragEnd while it moves releases the pointer and ends the drag (D22). + /// + [UnityTest] + public IEnumerator DragEnd_OnUIToolkit_WhenCancelledMidway_ReleasesAndClears() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider slider); + yield return WaitForLayout(); + yield return RunTool(SplitParameters(MouseAction.DragStart, SliderLeft())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + using CancellationTokenSource cancellation = new CancellationTokenSource(); + Task task = _tool.ExecuteAsync(SlowSplitParameters(MouseAction.DragEnd, SliderRight()), cancellation.Token); + yield return WaitUntilSliderMoves(slider); + + cancellation.Cancel(); + yield return WaitForCancelled(task); + + Assert.IsNull(CapturingElement(document)); + Assert.IsFalse(MouseDragState.IsDragging); + } + + /// + /// Verifies cancelling a UI Toolkit DragMove keeps the button held and the drag in progress, and a later + /// DragEnd releases it (D23). + /// + [UnityTest] + public IEnumerator DragMove_OnUIToolkit_WhenCancelled_KeepsDrag() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider slider); + yield return WaitForLayout(); + yield return RunTool(SplitParameters(MouseAction.DragStart, SliderLeft())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + using CancellationTokenSource cancellation = new CancellationTokenSource(); + Task task = _tool.ExecuteAsync(SlowSplitParameters(MouseAction.DragMove, SliderRight()), cancellation.Token); + yield return WaitUntilSliderMoves(slider); + + cancellation.Cancel(); + yield return WaitForCancelled(task); + + Assert.IsTrue(MouseDragState.IsDragging, "the drag must stay in progress"); + Assert.IsNotNull(CapturingElement(document), "the button must stay held"); + + yield return RunTool(SplitParameters(MouseAction.DragEnd, SliderRight())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.IsNull(CapturingElement(document)); + } + + /// + /// Verifies a one-shot UI Toolkit drag whose document is disabled while it moves completes without error + /// and reports the element it started on (D24). + /// + [UnityTest] + public IEnumerator Drag_OnUIToolkitSlider_WhenDocumentDisabledMidway_CompletesWithoutError() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider slider); + yield return WaitForLayout(); + // Only logs during the drag count: creating panel settings warns before the test sets their theme. + List errors = new List(); + UnityEngine.Application.LogCallback recordErrors = (condition, stackTrace, type) => + { + if (type != LogType.Log && type != LogType.Warning) + { + errors.Add($"{type}: {condition}"); + } + }; + UnityEngine.Application.logMessageReceived += recordErrors; + Task task = _tool.ExecuteAsync(DragParameters(SliderLeft(), SliderRight(), 200f), CancellationToken.None); + yield return WaitUntilSliderMoves(slider); + + document.enabled = false; + yield return WaitForTask(task); + UnityEngine.Application.logMessageReceived -= recordErrors; + + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.AreEqual(DocumentName + "#volume", _lastResponse.HitPath); + Assert.IsEmpty(errors); + } + + /// + /// Verifies cancelling a UI Toolkit DragStart during its start animation releases the pointer and leaves + /// no drag in progress (D25). + /// + [UnityTest] + public IEnumerator DragStart_OnUIToolkitSlider_WhenCancelledDuringExpand_ReleasesAndClears() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider _); + yield return WaitForLayout(); + using CancellationTokenSource cancellation = new CancellationTokenSource(); + Task task = _tool.ExecuteAsync(SplitParameters(MouseAction.DragStart, SliderLeft()), cancellation.Token); + + yield return null; + Assert.IsFalse(task.IsCompleted, "Setup: DragStart must still be animating when cancelled."); + Assert.IsNotNull(CapturingElement(document), "Setup: the press must have arrived before cancelling."); + cancellation.Cancel(); + yield return WaitForCancelled(task); + + Assert.IsNull(CapturingElement(document)); + Assert.IsFalse(MouseDragState.IsDragging); + } + + /// + /// Verifies a DragStart whose UI Toolkit press handler throws releases the pointer, leaves no drag in + /// progress, and ends with the exception. + /// + [UnityTest] + public IEnumerator DragStart_WhenUIToolkitPressHandlerThrows_ReleasesAndLeavesNoDrag() + { + CreateEventSystem(); + CreateDocumentWithSlider(out Slider slider); + int pointerUps = 0; + slider.RegisterCallback(_ => throw new System.InvalidOperationException("press failed"), TrickleDown.TrickleDown); + slider.RegisterCallback(_ => pointerUps++, TrickleDown.TrickleDown); + yield return WaitForLayout(); + + yield return WaitForFaulted(_tool.ExecuteAsync(SplitParameters(MouseAction.DragStart, SliderLeft()), CancellationToken.None)); + + Assert.AreEqual(1, pointerUps, "the release must still be sent"); + Assert.IsFalse(MouseDragState.IsDragging); + } + + /// + /// Verifies a DragEnd whose UI Toolkit release handler throws still ends the drag and ends with the + /// exception. + /// + [UnityTest] + public IEnumerator DragEnd_WhenUIToolkitReleaseHandlerThrows_StillEndsDrag() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider _); + yield return WaitForLayout(); + yield return RunTool(SplitParameters(MouseAction.DragStart, SliderLeft())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + CapturingControl(document).RegisterCallback(_ => throw new System.InvalidOperationException("release failed")); + + yield return WaitForFaulted(_tool.ExecuteAsync(SplitParameters(MouseAction.DragEnd, SliderRight()), CancellationToken.None)); + + Assert.IsFalse(MouseDragState.IsDragging); + } + + /// + /// Verifies a DragMove whose UI Toolkit move handler throws keeps the drag in progress and the button + /// held, as a cancelled DragMove does, so a later DragEnd releases it. + /// + [UnityTest] + public IEnumerator DragMove_WhenUIToolkitMoveHandlerThrows_KeepsDrag() + { + CreateEventSystem(); + UIDocument document = CreateDocumentWithSlider(out Slider _); + yield return WaitForLayout(); + yield return RunTool(SplitParameters(MouseAction.DragStart, SliderLeft())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + bool throwOnMove = false; + CapturingControl(document).RegisterCallback(_ => + { + if (throwOnMove) + { + throw new System.InvalidOperationException("move failed"); + } + }); + + throwOnMove = true; + yield return WaitForFaulted(_tool.ExecuteAsync(SplitParameters(MouseAction.DragMove, GameViewCenter()), CancellationToken.None)); + + Assert.IsTrue(MouseDragState.IsDragging, "the drag must stay in progress"); + Assert.IsNotNull(CapturingElement(document), "the button must stay held"); + throwOnMove = false; + yield return RunTool(SplitParameters(MouseAction.DragEnd, SliderRight())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.IsNull(CapturingElement(document)); + } + + // Steps right to the centre, then back to the left edge, checking the value after each step. + private IEnumerator RunSplitDrag(Slider slider) + { + yield return RunTool(SplitParameters(MouseAction.DragStart, SliderLeft())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.AreEqual("UIToolkit", _lastResponse.UiSystem); + + yield return RunTool(SplitParameters(MouseAction.DragMove, GameViewCenter())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.That(slider.value, Is.InRange(0.3f, 0.7f)); + + yield return RunTool(SplitParameters(MouseAction.DragEnd, SliderLeft())); + Assert.IsTrue(_lastResponse.Success, _lastResponse.Message); + Assert.AreEqual("UIToolkit", _lastResponse.UiSystem); + } + + private GameObject CreateEventSystem() + { + GameObject eventSystemObject = new GameObject("TestEventSystem"); + _createdObjects.Add(eventSystemObject); + eventSystemObject.AddComponent(); + eventSystemObject.AddComponent(); + return eventSystemObject; + } + + // A uGUI drag target laid exactly over the slider, so the drag start point hits both. + private IEnumerator DragOverlap(int canvasSortingOrder, float panelSortingOrder) + { + CreateEventSystem(); + _canvasObject.GetComponent().sortingOrder = canvasSortingOrder; + CreateDocumentWithSlider(out Slider slider, panelSortingOrder); + GameObject dragTarget = CreateCanvasElement("UguiDragTarget", Vector2.zero); + dragTarget.AddComponent(); + _lastDragTracker = dragTarget.AddComponent(); + _lastSlider = slider; + yield return WaitForLayout(); + + yield return RunTool(DragParameters(SliderLeft(), SliderRight(), 0f)); + } + + // Offset from the Canvas centre, which is the Game View centre in PlayMode; Canvas y grows upwards. + private GameObject CreateCanvasElement(string name, Vector2 offset) + { + GameObject element = new GameObject(name); + element.transform.SetParent(_canvasObject.transform, false); + RectTransform rect = element.AddComponent(); + rect.anchorMin = new Vector2(0.5f, 0.5f); + rect.anchorMax = new Vector2(0.5f, 0.5f); + rect.anchoredPosition = offset; + rect.sizeDelta = new Vector2(SliderWidth, SliderHeight); + return element; + } + + private UIDocument CreateDocumentWithSlider(out Slider slider, float panelSortingOrder = 0f) + { + slider = new Slider(0f, 1f) { name = "volume" }; + UIDocument document = CreateDocument(panelSortingOrder); + PlaceAtCenter(document, slider); + return document; + } + + // The counter is an array so the click callback can update it after this method returns. + private UIToolkitButton CreateDocumentWithButton(out int[] clicks) + { + int[] counter = new int[1]; + UIToolkitButton button = new UIToolkitButton { name = "play" }; + button.clicked += () => counter[0]++; + PlaceAtCenter(CreateDocument(), button); + clicks = counter; + return button; + } + + // The object starts inactive so the document sees its panel settings when it first enables. + private UIDocument CreateDocument(float panelSortingOrder = 0f) + { + PanelSettings settings = ScriptableObject.CreateInstance(); + settings.scaleMode = PanelScaleMode.ConstantPixelSize; + settings.scale = 1f; + settings.sortingOrder = panelSortingOrder; + settings.themeStyleSheet = AssetDatabase.LoadAssetAtPath(RuntimeThemePath); + Assert.IsNotNull(settings.themeStyleSheet, "the runtime theme must exist at " + RuntimeThemePath); + _createdObjects.Add(settings); + + GameObject documentObject = new GameObject(DocumentName); + documentObject.SetActive(false); + _createdObjects.Add(documentObject); + UIDocument document = documentObject.AddComponent(); + document.panelSettings = settings; + documentObject.SetActive(true); + return document; + } + + private static void PlaceAtCenter(UIDocument document, VisualElement element) + { + Vector2 center = GameViewCenter(); + element.style.position = Position.Absolute; + element.style.left = center.x - SliderWidth / 2f; + element.style.top = center.y - SliderHeight / 2f; + element.style.width = SliderWidth; + element.style.height = SliderHeight; + element.style.marginLeft = 0f; + element.style.marginTop = 0f; + element.style.marginRight = 0f; + element.style.marginBottom = 0f; + document.rootVisualElement.Add(element); + } + + // While the pointer is captured, its events go to the capturing element only, so handlers that must see + // them are registered there. + private static VisualElement CapturingControl(UIDocument document) + { + VisualElement? capturing = CapturingElement(document) as VisualElement; + Assert.IsNotNull(capturing, "Setup: the drag must have captured the pointer."); + return capturing!; + } + + private static IEventHandler? CapturingElement(UIDocument document) + { + return document.rootVisualElement.panel.GetCapturingElement(PointerId.mousePointerId); + } + + private static Vector2 GameViewCenter() + { + Vector2 size = Handles.GetMainGameViewSize(); + return new Vector2(Mathf.Round(size.x / 2f), Mathf.Round(size.y / 2f)); + } + + private static Vector2 SliderLeft() + { + return GameViewCenter() + new Vector2(-SliderWidth / 2f + EdgeInset, 0f); + } + + private static Vector2 SliderRight() + { + return GameViewCenter() + new Vector2(SliderWidth / 2f - EdgeInset, 0f); + } + + private static JObject DragParameters(Vector2 from, Vector2 to, float dragSpeed) + { + return new JObject + { + ["action"] = MouseAction.Drag.ToString(), + ["fromX"] = from.x, + ["fromY"] = from.y, + ["x"] = to.x, + ["y"] = to.y, + ["dragSpeed"] = dragSpeed + }; + } + + private static JObject SplitParameters(MouseAction action, Vector2 point) + { + return new JObject + { + ["action"] = action.ToString(), + ["x"] = point.x, + ["y"] = point.y, + ["dragSpeed"] = 0f + }; + } + + private static JObject SlowSplitParameters(MouseAction action, Vector2 point) + { + JObject parameters = SplitParameters(action, point); + parameters["dragSpeed"] = 50f; + return parameters; + } + + private IEnumerator RunTool(JObject parameters) + { + yield return WaitForTask(_tool.ExecuteAsync(parameters, CancellationToken.None)); + } + + private IEnumerator WaitForTask(Task task) + { + float timeoutAt = Time.realtimeSinceStartup + 5f; + yield return new WaitUntil(() => task.IsCompleted || Time.realtimeSinceStartup >= timeoutAt); + Assert.IsTrue(task.IsCompleted, "Tool execution timed out."); + Assert.AreEqual(TaskStatus.RanToCompletion, task.Status, $"Tool execution should complete: {task.Exception}"); + _lastResponse = (SimulateMouseUiResponse)task.Result; + } + + private static IEnumerator WaitForFaulted(Task task) + { + float timeoutAt = Time.realtimeSinceStartup + 5f; + yield return new WaitUntil(() => task.IsCompleted || Time.realtimeSinceStartup >= timeoutAt); + Assert.IsTrue(task.IsCompleted, "Tool execution timed out."); + Assert.IsTrue(task.IsFaulted, $"the handler's exception must end the command, got {task.Status}"); + Assert.IsInstanceOf(task.Exception!.InnerException); + } + + private static IEnumerator WaitForCancelled(Task task) + { + float timeoutAt = Time.realtimeSinceStartup + 5f; + yield return new WaitUntil(() => task.IsCompleted || Time.realtimeSinceStartup >= timeoutAt); + yield return null; + Assert.IsTrue(task.IsCompleted, "the drag must end after cancelling"); + Assert.IsTrue(task.IsCanceled || task.Exception?.InnerException is System.OperationCanceledException, + $"the drag must end by cancellation, got {task.Status}: {task.Exception}"); + } + + // A press near the left edge leaves the value near 0, so reaching 0.2 means the interpolation is running. + private static IEnumerator WaitUntilSliderMoves(Slider slider) + { + float timeoutAt = Time.realtimeSinceStartup + 5f; + yield return new WaitUntil(() => slider.value >= 0.2f || Time.realtimeSinceStartup >= timeoutAt); + Assert.GreaterOrEqual(slider.value, 0.2f, "Setup: the drag must be moving before the step."); + } + + // The root's size is set by the runtime panel update in the player loop, so frames must pass. + private static IEnumerator WaitForLayout() + { + yield return null; + yield return null; + } + } +} diff --git a/Assets/Tests/PlayMode/SimulateMouseUiToolkitDragTests.cs.meta b/Assets/Tests/PlayMode/SimulateMouseUiToolkitDragTests.cs.meta new file mode 100644 index 0000000000..e07eaa1c5a --- /dev/null +++ b/Assets/Tests/PlayMode/SimulateMouseUiToolkitDragTests.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: ec8c2e4a0427848269f4d61bd4ddfbec +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Assets/Tests/PlayMode/SimulateMouseUiToolkitTests.cs b/Assets/Tests/PlayMode/SimulateMouseUiToolkitTests.cs index c27f38a486..d5336b7318 100644 --- a/Assets/Tests/PlayMode/SimulateMouseUiToolkitTests.cs +++ b/Assets/Tests/PlayMode/SimulateMouseUiToolkitTests.cs @@ -251,29 +251,6 @@ public IEnumerator Drag_WithNegativeDragSpeedAndNoEventSystem_ReportsDragSpeedEr Assert.AreEqual("DragSpeed must be non-negative, got: -1", _lastResponse.Message); } - /// - /// Verifies a drag with no EventSystem still fails with the existing message. - /// - [UnityTest] - public IEnumerator Drag_WithoutEventSystem_FailsAsBefore() - { - CreateDocumentWithCenteredButton(0f, out ClickCounter _); - yield return WaitForLayout(); - Vector2 point = GameViewCenter(); - - yield return RunTool(new JObject - { - ["action"] = MouseAction.Drag.ToString(), - ["fromX"] = point.x, - ["fromY"] = point.y, - ["x"] = point.x + 10f, - ["y"] = point.y - }); - - Assert.IsFalse(_lastResponse.Success); - Assert.AreEqual(NoEventSystemMessage, _lastResponse.Message); - } - /// /// Verifies a right click reaches the UI Toolkit button as button 1 and does not click it (#20). /// diff --git a/Packages/src/Documentation~/tools.md b/Packages/src/Documentation~/tools.md index 172653a7aa..ede904bff7 100644 --- a/Packages/src/Documentation~/tools.md +++ b/Packages/src/Documentation~/tools.md @@ -158,7 +158,7 @@ When enabled, dynamic code execution runs with full Unity Editor process permiss ### 14. simulate-mouse-ui - Simulate Mouse Input on PlayMode UI Simulate mouse click, long-press, and drag on PlayMode UI elements. Uses EventSystem and ExecuteEvents to dispatch pointer events directly — works independently of both old and new Input System. For game logic that reads Input System (e.g. `Mouse.current.leftButton.wasPressedThisFrame`), use `simulate-mouse-input` instead. -Supports 6 actions: Click, LongPress, Drag (one-shot), DragStart/DragMove/DragEnd (split drag). Click and LongPress also reach UI Toolkit controls at their annotated coordinates, without an EventSystem. +Supports 6 actions: Click, LongPress, Drag (one-shot), DragStart/DragMove/DragEnd (split drag). Click, LongPress, and drags also reach UI Toolkit controls at their annotated coordinates, without an EventSystem. ```text → screenshot (CaptureMode: rendering, AnnotateElements: true) diff --git a/Packages/src/Documentation~/tools_ja.md b/Packages/src/Documentation~/tools_ja.md index 411720f305..9987e31ba9 100644 --- a/Packages/src/Documentation~/tools_ja.md +++ b/Packages/src/Documentation~/tools_ja.md @@ -158,7 +158,7 @@ Unity Editor内で動的にC#コードを実行します。 ### 14. simulate-mouse-ui - PlayMode UI要素のマウス操作シミュレーション PlayMode中のUI要素に対してマウスクリック・長押し・ドラッグをシミュレーションします。EventSystemとExecuteEventsを使ってポインタイベントを直接ディスパッチするため、旧Input System・新Input Systemの両方に依存せず動作します。ゲームロジックがInput Systemを直接読み取る場合（例：`Mouse.current.leftButton.wasPressedThisFrame`）は、`simulate-mouse-input` を使用してください。 -6つのアクションに対応: Click、LongPress、Drag（ワンショット）、DragStart/DragMove/DragEnd（分割ドラッグ）。Click と LongPress は、注釈の座標で UI Toolkit のコントロールにも届きます（EventSystem は不要）。 +6つのアクションに対応: Click、LongPress、Drag（ワンショット）、DragStart/DragMove/DragEnd（分割ドラッグ）。Click・LongPress・ドラッグは、注釈の座標で UI Toolkit のコントロールにも届きます（EventSystem は不要）。 ```text → screenshot (CaptureMode: rendering, AnnotateElements: true) diff --git a/Packages/src/Editor/FirstPartyTools/Common/UIToolkit/UIToolkitPointerTarget.cs b/Packages/src/Editor/FirstPartyTools/Common/UIToolkit/UIToolkitPointerTarget.cs index 0a4fb7fda4..853b280006 100644 --- a/Packages/src/Editor/FirstPartyTools/Common/UIToolkit/UIToolkitPointerTarget.cs +++ b/Packages/src/Editor/FirstPartyTools/Common/UIToolkit/UIToolkitPointerTarget.cs @@ -49,6 +49,9 @@ internal UIToolkitPointerTarget(IPanel panel, UIDocument document, string name, public bool IsAvailable => _document != null && _document.isActiveAndEnabled && _document.rootVisualElement?.panel == _panel; + /// True once the document that owns the panel has been destroyed, not merely disabled. + public bool IsDocumentDestroyed => _document == null; + /// Presses a mouse button (0 left, 1 right, 2 middle) at a Game View point. /// Does nothing once the target is no longer available. public void PressDown(Vector2 gameViewTopLeft, int button) diff --git a/Packages/src/Editor/FirstPartyTools/Screenshot/Skill/references/annotated-elements.md b/Packages/src/Editor/FirstPartyTools/Screenshot/Skill/references/annotated-elements.md index 69466c0964..69272a2bb2 100644 --- a/Packages/src/Editor/FirstPartyTools/Screenshot/Skill/references/annotated-elements.md +++ b/Packages/src/Editor/FirstPartyTools/Screenshot/Skill/references/annotated-elements.md @@ -53,7 +53,7 @@ Limits: Clicking: -- `simulate-mouse-ui --action Click --x --y ` (or `LongPress`) reaches UI Toolkit controls, with or without an EventSystem. Its `--target-path` does not accept UI Toolkit `Path` values. +- `simulate-mouse-ui --action Click --x --y ` (or `LongPress`, or a drag from that point) reaches UI Toolkit controls, with or without an EventSystem. Its `--target-path` does not accept UI Toolkit `Path` values. - `simulate-mouse-input --action Click --x --y ` also reaches them when the scene has an EventSystem with an Input System UI input module. - Without an EventSystem the `simulate-mouse-input` click does not reach UI Toolkit controls. diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Application/MouseDragState.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Application/MouseDragState.cs index 0b012fac0a..c8e1c8011b 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Application/MouseDragState.cs +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Application/MouseDragState.cs @@ -1,7 +1,5 @@ #nullable enable using UnityEditor; -using UnityEngine; -using UnityEngine.EventSystems; using io.github.hatayama.UnityCliLoop.Runtime; @@ -12,9 +10,9 @@ namespace io.github.hatayama.UnityCliLoop.FirstPartyTools /// internal sealed class MouseDragStateService { - internal bool IsDragging => Target != null && PointerData != null; - internal GameObject? Target { get; set; } - internal PointerEventData? PointerData { get; set; } + // A destroyed target ends the drag without a command, as Unity's null check on it already did. + internal bool IsDragging => Session != null && !Session.IsTargetDestroyed; + internal IMouseUiDragSession? Session { get; set; } internal void RegisterPlayModeCallbacks() { @@ -25,8 +23,7 @@ internal void RegisterPlayModeCallbacks() internal void Clear() { - Target = null; - PointerData = null; + Session = null; } private void OnPlayModeStateChanged(PlayModeStateChange state) @@ -53,16 +50,10 @@ internal static void InitializeForEditorStartup() internal static bool IsDragging => ServiceValue.IsDragging; - internal static GameObject? Target + internal static IMouseUiDragSession? Session { - get => ServiceValue.Target; - set => ServiceValue.Target = value; - } - - internal static PointerEventData? PointerData - { - get => ServiceValue.PointerData; - set => ServiceValue.PointerData = value; + get => ServiceValue.Session; + set => ServiceValue.Session = value; } internal static void Clear() diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/IMouseUiDragSession.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/IMouseUiDragSession.cs new file mode 100644 index 0000000000..7d3307313b --- /dev/null +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/IMouseUiDragSession.cs @@ -0,0 +1,35 @@ +#nullable enable +using UnityEngine; + +namespace io.github.hatayama.UnityCliLoop.FirstPartyTools +{ + /// + /// A drag in progress on one target: one-shot drags hold it for a single command, split drags keep it + /// between commands. Positions are Game View input coordinates with a top-left origin. + /// + internal interface IMouseUiDragSession + { + /// What the drag started on. Its name is never null. + MouseUiHitReport Report { get; } + + /// Where the pointer is now. + Vector2 CurrentInputPosition { get; } + + /// True once the target is destroyed; the drag then counts as no longer in progress. + bool IsTargetDestroyed { get; } + + /// + /// True when the drag goes through the EventSystem, which then must exist and alone handles drop targets. + /// + bool UsesEventSystem { get; } + + /// Returns why the drag cannot continue, or null when it can. + string? CheckStillActive(); + + /// Moves the pointer one step while the button stays held. + void MoveTo(Vector2 inputPosition); + + /// Releases the button, dropping onto the explicit target when one is given. + void Finish(GameObject? explicitDropTarget); + } +} diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/IMouseUiDragSession.cs.meta b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/IMouseUiDragSession.cs.meta new file mode 100644 index 0000000000..e5208b6b34 --- /dev/null +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/IMouseUiDragSession.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: 5e628bb72cf9f4df480ff925cca9378b +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragEventExecutor.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragEventExecutor.cs index ef9760cf70..05cd36921a 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragEventExecutor.cs +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragEventExecutor.cs @@ -81,17 +81,19 @@ private static void UpdatePointerRaycast(PointerEventData pointerData) pointerData.pointerCurrentRaycast = hit ?? new RaycastResult(); } + // Interpolates in top-left input coordinates. The flip to screen space is affine and the Game View + // height cannot change within one command, so the screen points a uGUI drag passes match interpolating + // in screen space. internal static async Task InterpolateDragPosition( - PointerEventData pointerData, - GameObject target, - Vector2 endPos, + IMouseUiDragSession session, + Vector2 endInput, float dragSpeed, CancellationToken ct) { Debug.Assert(dragSpeed >= 0f, "dragSpeed must be non-negative"); - Vector2 startPos = pointerData.position; - float distance = Vector2.Distance(startPos, endPos); + Vector2 startInput = session.CurrentInputPosition; + float distance = Vector2.Distance(startInput, endInput); float duration = dragSpeed > 0f ? distance / dragSpeed : 0f; if (duration <= 0f) @@ -103,13 +105,8 @@ internal static async Task InterpolateDragPosition( } await MainThreadSwitcher.SwitchToMainThread(ct); - Vector2 previousPosition = pointerData.position; - pointerData.position = endPos; - pointerData.delta = endPos - previousPosition; - MouseUiInputSystemSync.SyncMousePosition(endPos); - ExecuteEvents.Execute(target, pointerData, ExecuteEvents.dragHandler); - - SimulateMouseUiOverlayState.UpdatePosition(MouseUiCoordinateConverter.ScreenToInput(endPos)); + session.MoveTo(endInput); + SimulateMouseUiOverlayState.UpdatePosition(endInput); return MouseUiFrameWaitOutcome.Completed; } @@ -127,16 +124,10 @@ internal static async Task InterpolateDragPosition( float elapsed = Time.realtimeSinceStartup - startTime; t = Mathf.Clamp01(elapsed / duration); - Vector2 previousPosition = pointerData.position; - Vector2 currentPosition = Vector2.Lerp(startPos, endPos, t); - - pointerData.position = currentPosition; - pointerData.delta = currentPosition - previousPosition; - - MouseUiInputSystemSync.SyncMousePosition(currentPosition); - ExecuteEvents.Execute(target, pointerData, ExecuteEvents.dragHandler); + Vector2 currentInput = Vector2.Lerp(startInput, endInput, t); - SimulateMouseUiOverlayState.UpdatePosition(MouseUiCoordinateConverter.ScreenToInput(currentPosition)); + session.MoveTo(currentInput); + SimulateMouseUiOverlayState.UpdatePosition(currentInput); } while (t < 1.0f); diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragStartTarget.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragStartTarget.cs new file mode 100644 index 0000000000..76103bcfa5 --- /dev/null +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragStartTarget.cs @@ -0,0 +1,70 @@ +#nullable enable +using System; +using UnityEngine; +using UnityEngine.EventSystems; + +namespace io.github.hatayama.UnityCliLoop.FirstPartyTools +{ + /// + /// What a drag will start on, resolved before anything is sent so that refusals leave no button held. + /// + internal sealed class MouseUiDragStartTarget + { + private readonly EventSystem? _eventSystem; + private readonly RaycastResult _startRaycast; + private readonly GameObject? _uguiTarget; + private readonly UIToolkitPointerTarget? _uiToolkitTarget; + + private MouseUiDragStartTarget( + EventSystem? eventSystem, + RaycastResult startRaycast, + GameObject? uguiTarget, + UIToolkitPointerTarget? uiToolkitTarget) + { + _eventSystem = eventSystem; + _startRaycast = startRaycast; + _uguiTarget = uguiTarget; + _uiToolkitTarget = uiToolkitTarget; + } + + internal static MouseUiDragStartTarget None { get; } = new MouseUiDragStartTarget(null, new RaycastResult(), null, null); + + /// A uGUI drag; a null target means the raycast hit has no drag handler. + internal static MouseUiDragStartTarget Ugui(EventSystem eventSystem, RaycastResult startRaycast, GameObject? target) + { + return new MouseUiDragStartTarget(eventSystem, startRaycast, target, null); + } + + internal static MouseUiDragStartTarget UIToolkit(UIToolkitPointerTarget target) + { + return new MouseUiDragStartTarget(null, new RaycastResult(), null, target); + } + + internal bool HasTarget => _uiToolkitTarget != null || _uguiTarget != null; + + /// The raycast a uGUI drag presses on; empty for UI Toolkit and for no target. + internal RaycastResult StartRaycast => _startRaycast; + + /// The uGUI drag handler, or null when the drag is not on uGUI or the hit has no handler. + internal GameObject? UguiTarget => _uguiTarget; + + /// The UI Toolkit control's name, or null when the drag is not on UI Toolkit. + internal string? UIToolkitElementName => _uiToolkitTarget?.Name; + + /// Presses on the target and returns the drag in progress. + internal IMouseUiDragSession Begin(Vector2 inputStart, Vector2 screenStart) + { + if (_uiToolkitTarget != null) + { + return UIToolkitDragSession.Begin(_uiToolkitTarget, inputStart); + } + + if (_uguiTarget == null) + { + throw new InvalidOperationException("A drag cannot begin without a target; check HasTarget first."); + } + + return UguiDragSession.Begin(_eventSystem!, screenStart, _startRaycast, _uguiTarget); + } + } +} diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragStartTarget.cs.meta b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragStartTarget.cs.meta new file mode 100644 index 0000000000..c1852ec5c1 --- /dev/null +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragStartTarget.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: a0e6b014039bd4fffa364700f86c448b +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragTargetResolver.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragTargetResolver.cs index 316e68d608..a0e067b3a7 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragTargetResolver.cs +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiDragTargetResolver.cs @@ -7,54 +7,93 @@ namespace io.github.hatayama.UnityCliLoop.FirstPartyTools { /// - /// Resolves the draggable target and starting raycast for mouse UI drag actions. + /// Decides what a Drag or DragStart starts on: a uGUI drag handler, a UI Toolkit control, or nothing. The + /// choice between uGUI and UI Toolkit follows the same rule as clicks. /// internal static class MouseUiDragTargetResolver { - internal static ( - RaycastResult StartRaycast, - GameObject? Target, - SimulateMouseUiResponse? FailureResponse) Resolve( - MouseUiSimulationCommand parameters, - EventSystem eventSystem, - MouseAction action, - Vector2 inputPosition, - Vector2 screenPosition) + internal static (MouseUiDragStartTarget Start, SimulateMouseUiResponse? FailureResponse) Resolve( + MouseUiSimulationCommand parameters, + EventSystem? eventSystem, + MouseAction action, + Vector2 inputPosition, + Vector2 screenPosition) { - RaycastResult? hit = parameters.BypassRaycast - ? null - : UiRaycastHelper.RaycastUI(screenPosition, eventSystem); - RaycastResult startRaycast = new(); - GameObject? rawTarget = null; - if (parameters.BypassRaycast) { - (GameObject? Target, SimulateMouseUiResponse? FailureResponse) resolution = - MouseUiPointerTargetResolver.ResolveGameObjectPath( - parameters.TargetPath, - "TargetPath", - action, - inputPosition); - if (resolution.FailureResponse != null) - { - return (startRaycast, null, resolution.FailureResponse); - } - - rawTarget = resolution.Target; - startRaycast = MouseUiPointerTargetResolver.CreateDirectRaycastResult(rawTarget!); + return ResolveBypass(parameters, eventSystem, action, inputPosition); + } + + // As with clicks, UI Toolkit is looked up at the requested point itself and the uGUI side leaves out + // the bridge's panel hits, which depend on Screen. + UIToolkitPointerTarget? toolkit = UIToolkitScreenElements.FindPointerTarget(inputPosition); + RaycastResult? hit = eventSystem != null + ? UiRaycastHelper.RaycastUISkippingUIToolkitPanels(screenPosition, eventSystem) + : null; + if (toolkit == null && hit == null) + { + return eventSystem == null + ? (MouseUiDragStartTarget.None, MouseUiSimulationResponseFactory.CreateFailure( + parameters, + MouseUiPressTargetResolver.NoEventSystemNorUIToolkitMessage(inputPosition))) + : (MouseUiDragStartTarget.None, null); } - else if (hit != null) + + if (toolkit != null && + (hit == null || !MouseUiHitPriority.IsUguiInFront(hit.Value.module.sortOrderPriority, toolkit.SortingOrder))) { - rawTarget = hit.Value.gameObject; - startRaycast = hit.Value; + return (MouseUiDragStartTarget.UIToolkit(toolkit), null); } // Execute dispatches only to the exact target; resolve the actual drag handler up the hierarchy. - GameObject? target = rawTarget != null - ? ExecuteEvents.GetEventHandler(rawTarget) - : null; + GameObject? target = ExecuteEvents.GetEventHandler(hit!.Value.gameObject); + return (MouseUiDragStartTarget.Ugui(eventSystem!, hit.Value, target), null); + } + + /// + /// Refuses a drop target for a drag on a UI Toolkit control, which has no EventSystem drop handling. + /// + internal static SimulateMouseUiResponse? RefuseDropTargetOnUIToolkit( + MouseUiSimulationCommand parameters, + string? uiToolkitElementName) + { + if (uiToolkitElementName == null || string.IsNullOrEmpty(parameters.DropTargetPath)) + { + return null; + } + + return MouseUiSimulationResponseFactory.CreateFailure( + parameters, + $"DropTargetPath applies to uGUI drags only; this drag starts on UI Toolkit element '{uiToolkitElementName}'."); + } + + private static (MouseUiDragStartTarget Start, SimulateMouseUiResponse? FailureResponse) ResolveBypass( + MouseUiSimulationCommand parameters, + EventSystem? eventSystem, + MouseAction action, + Vector2 inputPosition) + { + if (eventSystem == null) + { + return (MouseUiDragStartTarget.None, MouseUiSimulationResponseFactory.CreateFailure( + parameters, MouseUiPressTargetResolver.NoEventSystemMessage)); + } + + (GameObject? Target, SimulateMouseUiResponse? FailureResponse) resolution = + MouseUiPointerTargetResolver.ResolveGameObjectPath( + parameters.TargetPath, + "TargetPath", + action, + inputPosition); + if (resolution.FailureResponse != null) + { + return (MouseUiDragStartTarget.None, resolution.FailureResponse); + } - return (startRaycast, target, null); + GameObject rawTarget = resolution.Target!; + RaycastResult startRaycast = MouseUiPointerTargetResolver.CreateDirectRaycastResult(rawTarget); + GameObject? target = ExecuteEvents.GetEventHandler(rawTarget); + return (MouseUiDragStartTarget.Ugui(eventSystem, startRaycast, target), null); } } } diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiGameObjectHitReport.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiGameObjectHitReport.cs new file mode 100644 index 0000000000..b88102b10c --- /dev/null +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiGameObjectHitReport.cs @@ -0,0 +1,26 @@ +#nullable enable +using UnityEngine; + +namespace io.github.hatayama.UnityCliLoop.FirstPartyTools +{ + /// + /// Builds the hit report for a GameObject target reached through the EventSystem. + /// + internal static class MouseUiGameObjectHitReport + { + // Decided by whether the handler sits under a Canvas rather than by raycaster type, because a bypass + // press has no raycaster and both paths must report alike. + internal static MouseUiHitReport Create(GameObject? target) + { + if (target == null) + { + return MouseUiHitReport.None; + } + + string uiSystem = target.GetComponentInParent() != null + ? MouseUiHitSystem.UGUI + : MouseUiHitSystem.NonUi; + return new MouseUiHitReport(target.name, GameObjectPathUtility.GetFullPath(target), uiSystem); + } + } +} diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiGameObjectHitReport.cs.meta b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiGameObjectHitReport.cs.meta new file mode 100644 index 0000000000..8e31b5df43 --- /dev/null +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiGameObjectHitReport.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: 8817fdc880f5740a8a6d71c005803f85 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiIncrementalDragExecutor.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiIncrementalDragExecutor.cs index 02832f459a..b8061262a9 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiIncrementalDragExecutor.cs +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiIncrementalDragExecutor.cs @@ -17,7 +17,7 @@ internal static class MouseUiIncrementalDragExecutor { internal static async Task ExecuteDragStart( MouseUiSimulationCommand parameters, - EventSystem eventSystem, + EventSystem? eventSystem, MouseUiMainThreadCleanupScheduler cleanupScheduler, CancellationToken ct) { @@ -35,7 +35,7 @@ internal static async Task ExecuteDragStart( Vector2 inputPos = new(parameters.X, parameters.Y); Vector2 screenPos = MouseUiCoordinateConverter.InputToScreen(inputPos); - (RaycastResult startRaycast, GameObject? target, SimulateMouseUiResponse? targetFailureResponse) = + (MouseUiDragStartTarget start, SimulateMouseUiResponse? targetFailureResponse) = MouseUiDragTargetResolver.Resolve( parameters, eventSystem, @@ -47,7 +47,7 @@ internal static async Task ExecuteDragStart( return targetFailureResponse; } - if (target == null) + if (!start.HasTarget) { SimulateMouseUiOverlayState.Update( MouseAction.DragStart, inputPos, null, Handles.GetMainGameViewSize()); @@ -93,14 +93,10 @@ internal static async Task ExecuteDragStart( }; } - PointerEventData pointerData = MouseUiDragEventExecutor.InitiateDrag(eventSystem, screenPos, startRaycast, target, PointerEventData.InputButton.Left); - ExecuteEvents.Execute(target, pointerData, ExecuteEvents.beginDragHandler); - pointerData.dragging = true; + IMouseUiDragSession session = start.Begin(inputPos, screenPos); + MouseDragState.Session = session; - MouseDragState.Target = target; - MouseDragState.PointerData = pointerData; - - string targetName = target.name; + string targetName = session.Report.Name!; SimulateMouseUiOverlayState.Update( MouseAction.DragStart, inputPos, inputPos, Handles.GetMainGameViewSize()); @@ -130,8 +126,15 @@ internal static async Task ExecuteDragStart( { cleanupScheduler.ExecuteCleanupOnMainThread(() => { - MouseUiDragEventExecutor.FinalizeDrag(pointerData, target, null); - MouseDragState.Clear(); + // Cleared even when the release throws, so no drag is left in progress. + try + { + session.Finish(null); + } + finally + { + MouseDragState.Clear(); + } }); } } @@ -139,9 +142,11 @@ internal static async Task ExecuteDragStart( return new SimulateMouseUiResponse { Success = true, - Message = $"Drag started on '{targetName}' at ({inputPos.x:F1}, {inputPos.y:F1})", + Message = $"Drag started on {MouseUiSimulationResponseFactory.DescribeHit(session.Report)} at ({inputPos.x:F1}, {inputPos.y:F1})", Action = MouseAction.DragStart.ToString(), HitGameObjectName = targetName, + UiSystem = session.Report.UiSystem, + HitPath = session.Report.Path, PositionX = inputPos.x, PositionY = inputPos.y }; @@ -164,30 +169,27 @@ internal static async Task ExecuteDragMove( }; } - Debug.Assert(MouseDragState.Target != null, "Target must not be null when IsDragging is true"); - Debug.Assert(MouseDragState.PointerData != null, "PointerData must not be null when IsDragging is true"); + Debug.Assert(MouseDragState.Session != null, "Session must not be null when IsDragging is true"); - SimulateMouseUiResponse? invalidResponse = ValidateDragStillActive(parameters.Action); + SimulateMouseUiResponse? invalidResponse = ValidateDragStillActive(parameters); if (invalidResponse != null) { return invalidResponse; } Vector2 inputEnd = new(parameters.X, parameters.Y); - Vector2 screenEnd = MouseUiCoordinateConverter.InputToScreen(inputEnd); - PointerEventData pointerData = MouseDragState.PointerData!; - GameObject target = MouseDragState.Target!; - string targetName = target.name; + IMouseUiDragSession session = MouseDragState.Session!; + string targetName = session.Report.Name!; SimulateMouseUiOverlayState.Update( MouseAction.DragMove, - MouseUiCoordinateConverter.ScreenToInput(pointerData.position), + session.CurrentInputPosition, SimulateMouseUiOverlayState.DragStartPosition, Handles.GetMainGameViewSize()); // Cancellation leaves drag state intact so the user can continue with DragMove/DragEnd MouseUiFrameWaitOutcome dragOutcome = await MouseUiDragEventExecutor.InterpolateDragPosition( - pointerData, target, screenEnd, + session, inputEnd, parameters.DragSpeed, ct).ConfigureAwait(false); if (dragOutcome == MouseUiFrameWaitOutcome.TimedOut) { @@ -208,9 +210,11 @@ internal static async Task ExecuteDragMove( return new SimulateMouseUiResponse { Success = true, - Message = $"Drag moved on '{targetName}' to ({inputEnd.x:F1}, {inputEnd.y:F1}) at {parameters.DragSpeed:F0} px/s", + Message = $"Drag moved on {MouseUiSimulationResponseFactory.DescribeHit(session.Report)} to ({inputEnd.x:F1}, {inputEnd.y:F1}) at {parameters.DragSpeed:F0} px/s", Action = MouseAction.DragMove.ToString(), HitGameObjectName = targetName, + UiSystem = session.Report.UiSystem, + HitPath = session.Report.Path, PositionX = inputEnd.x, PositionY = inputEnd.y }; @@ -233,20 +237,26 @@ internal static async Task ExecuteDragEnd( }; } - Debug.Assert(MouseDragState.Target != null, "Target must not be null when IsDragging is true"); - Debug.Assert(MouseDragState.PointerData != null, "PointerData must not be null when IsDragging is true"); + Debug.Assert(MouseDragState.Session != null, "Session must not be null when IsDragging is true"); - SimulateMouseUiResponse? invalidResponse = ValidateDragStillActive(parameters.Action); + SimulateMouseUiResponse? invalidResponse = ValidateDragStillActive(parameters); if (invalidResponse != null) { return invalidResponse; } Vector2 inputEnd = new(parameters.X, parameters.Y); - Vector2 screenEnd = MouseUiCoordinateConverter.InputToScreen(inputEnd); - PointerEventData pointerData = MouseDragState.PointerData!; - GameObject target = MouseDragState.Target!; - string targetName = target.name; + IMouseUiDragSession session = MouseDragState.Session!; + string targetName = session.Report.Name!; + // The drag stays in progress so DragEnd can be retried without the option. + SimulateMouseUiResponse? dropRefusal = MouseUiDragTargetResolver.RefuseDropTargetOnUIToolkit( + parameters, + session.UsesEventSystem ? null : targetName); + if (dropRefusal != null) + { + return dropRefusal; + } + (GameObject? explicitDropTarget, SimulateMouseUiResponse? dropFailureResponse) = MouseUiPointerTargetResolver.ResolveDropTargetPath( parameters, @@ -259,7 +269,7 @@ internal static async Task ExecuteDragEnd( SimulateMouseUiOverlayState.Update( MouseAction.DragEnd, - MouseUiCoordinateConverter.ScreenToInput(pointerData.position), + session.CurrentInputPosition, SimulateMouseUiOverlayState.DragStartPosition, Handles.GetMainGameViewSize()); @@ -271,7 +281,7 @@ internal static async Task ExecuteDragEnd( try { MouseUiFrameWaitOutcome dragOutcome = await MouseUiDragEventExecutor.InterpolateDragPosition( - pointerData, target, screenEnd, + session, inputEnd, parameters.DragSpeed, ct).ConfigureAwait(false); if (dragOutcome == MouseUiFrameWaitOutcome.TimedOut) { @@ -304,8 +314,15 @@ internal static async Task ExecuteDragEnd( { cleanupScheduler.ExecuteCleanupOnMainThread(() => { - MouseUiDragEventExecutor.FinalizeDrag(pointerData, target, explicitDropTarget); - MouseDragState.Clear(); + // Cleared even when the release throws, so no drag is left in progress. + try + { + session.Finish(explicitDropTarget); + } + finally + { + MouseDragState.Clear(); + } }); } @@ -316,7 +333,7 @@ internal static async Task ExecuteDragEnd( if (dissipateOutcome == MouseUiFrameWaitOutcome.TimedOut) { cleanupScheduler.QueueOverlayClear(); - return MouseUiSimulationResponseFactory.CreateDragEndResult(parameters, inputEnd, targetName); + return MouseUiSimulationResponseFactory.CreateDragEndResult(parameters, inputEnd, session.Report); } if (dissipateOutcome == MouseUiFrameWaitOutcome.Paused) { @@ -327,36 +344,31 @@ internal static async Task ExecuteDragEnd( } await MainThreadSwitcher.SwitchToMainThread(ct); - return MouseUiSimulationResponseFactory.CreateDragEndResult(parameters, inputEnd, targetName); + return MouseUiSimulationResponseFactory.CreateDragEndResult(parameters, inputEnd, session.Report); } - // User input during a CLI drag can cause Unity's StandaloneInputModule to - // release or reassign the drag, leaving MouseDragState stale. - private static SimulateMouseUiResponse? ValidateDragStillActive(MouseAction action) + // A drag that can no longer continue is cleared; a missing EventSystem keeps the drag in progress so it + // can continue once one exists. UI Toolkit drags never use the EventSystem. + private static SimulateMouseUiResponse? ValidateDragStillActive(MouseUiSimulationCommand parameters) { - if (!MouseDragState.Target!.activeInHierarchy) + IMouseUiDragSession session = MouseDragState.Session!; + string? reason = session.CheckStillActive(); + if (reason != null) { MouseDragState.Clear(); SimulateMouseUiOverlayState.Clear(); return new SimulateMouseUiResponse { Success = false, - Message = "Drag target was destroyed or deactivated during drag.", - Action = action.ToString() + Message = reason, + Action = parameters.Action.ToString() }; } - if (!MouseDragState.PointerData!.dragging || - MouseDragState.PointerData.pointerDrag != MouseDragState.Target) + if (session.UsesEventSystem && EventSystem.current == null) { - MouseDragState.Clear(); - SimulateMouseUiOverlayState.Clear(); - return new SimulateMouseUiResponse - { - Success = false, - Message = "Drag was interrupted by user input or system event.", - Action = action.ToString() - }; + return MouseUiSimulationResponseFactory.CreateFailure( + parameters, MouseUiPressTargetResolver.NoEventSystemMessage); } return null; diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiOneShotDragExecutor.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiOneShotDragExecutor.cs index 33694a84f3..ee9ee6fb99 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiOneShotDragExecutor.cs +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiOneShotDragExecutor.cs @@ -17,15 +17,14 @@ internal static class MouseUiOneShotDragExecutor { internal static async Task ExecuteDragOneShot( MouseUiSimulationCommand parameters, - EventSystem eventSystem, + EventSystem? eventSystem, MouseUiMainThreadCleanupScheduler cleanupScheduler, CancellationToken ct) { Vector2 inputStart = new(parameters.FromX, parameters.FromY); Vector2 inputEnd = new(parameters.X, parameters.Y); Vector2 screenStart = MouseUiCoordinateConverter.InputToScreen(inputStart); - Vector2 screenEnd = MouseUiCoordinateConverter.InputToScreen(inputEnd); - (RaycastResult startRaycast, GameObject? target, SimulateMouseUiResponse? targetFailureResponse) = + (MouseUiDragStartTarget start, SimulateMouseUiResponse? targetFailureResponse) = MouseUiDragTargetResolver.Resolve( parameters, eventSystem, @@ -37,6 +36,14 @@ internal static async Task ExecuteDragOneShot( return targetFailureResponse; } + // Ahead of the drop target lookup, so a missing drop target does not hide why the option is refused. + SimulateMouseUiResponse? dropRefusal = + MouseUiDragTargetResolver.RefuseDropTargetOnUIToolkit(parameters, start.UIToolkitElementName); + if (dropRefusal != null) + { + return dropRefusal; + } + (GameObject? explicitDropTarget, SimulateMouseUiResponse? dropFailureResponse) = MouseUiPointerTargetResolver.ResolveDropTargetPath( parameters, @@ -47,18 +54,15 @@ internal static async Task ExecuteDragOneShot( return dropFailureResponse; } - if (target == null) + if (!start.HasTarget) { return await ExecuteNoTargetDragAsync( parameters, inputStart, inputEnd, cleanupScheduler, ct).ConfigureAwait(false); } - // uGUI drag controls (ScrollRect, Slider) only respond to left-button drags - PointerEventData pointerData = MouseUiDragEventExecutor.InitiateDrag(eventSystem, screenStart, startRaycast, target, PointerEventData.InputButton.Left); - ExecuteEvents.Execute(target, pointerData, ExecuteEvents.beginDragHandler); - pointerData.dragging = true; + IMouseUiDragSession session = start.Begin(inputStart, screenStart); - string targetName = target.name; + string targetName = session.Report.Name!; SimulateMouseUiOverlayState.Update( MouseAction.Drag, inputStart, inputStart, Handles.GetMainGameViewSize()); @@ -72,11 +76,9 @@ internal static async Task ExecuteDragOneShot( { SimulateMouseUiResponse? motionAbort = await AnimateDragMotionOrAbortAsync( parameters, - pointerData, - target, + session, inputStart, inputEnd, - screenEnd, targetName, DragInterruptedDuringMotionMessage, cleanupScheduler, @@ -88,7 +90,7 @@ internal static async Task ExecuteDragOneShot( } finally { - cleanupScheduler.ExecuteCleanupOnMainThread(() => MouseUiDragEventExecutor.FinalizeDrag(pointerData, target, explicitDropTarget)); + cleanupScheduler.ExecuteCleanupOnMainThread(() => session.Finish(explicitDropTarget)); } // Why switch after the helper: AnimateDragMotionOrAbortAsync ends on the main @@ -103,7 +105,7 @@ internal static async Task ExecuteDragOneShot( if (dissipateOutcome == MouseUiFrameWaitOutcome.TimedOut) { cleanupScheduler.QueueOverlayClear(); - return MouseUiSimulationResponseFactory.CreateDragResult(parameters, inputStart, inputEnd, targetName); + return MouseUiSimulationResponseFactory.CreateDragResult(parameters, inputStart, inputEnd, session.Report); } if (dissipateOutcome == MouseUiFrameWaitOutcome.Paused) { @@ -114,7 +116,7 @@ internal static async Task ExecuteDragOneShot( } await MainThreadSwitcher.SwitchToMainThread(ct); - return MouseUiSimulationResponseFactory.CreateDragResult(parameters, inputStart, inputEnd, targetName); + return MouseUiSimulationResponseFactory.CreateDragResult(parameters, inputStart, inputEnd, session.Report); } private static async Task ExecuteNoTargetDragAsync( @@ -172,11 +174,9 @@ await MouseUiOverlayAnimator.PlayDissipateAnimation(ct).ConfigureAwait(false), private static async Task AnimateDragMotionOrAbortAsync( MouseUiSimulationCommand parameters, - PointerEventData pointerData, - GameObject target, + IMouseUiDragSession session, Vector2 inputStart, Vector2 inputEnd, - Vector2 screenEnd, string targetName, string interruptedMessage, MouseUiMainThreadCleanupScheduler cleanupScheduler, @@ -197,7 +197,7 @@ await MouseUiOverlayAnimator.PlayExpandAnimation(ct).ConfigureAwait(false), await MainThreadSwitcher.SwitchToMainThread(ct); SimulateMouseUiResponse? dragAbort = AbortDragOnOverlayOutcome( - await MouseUiDragEventExecutor.InterpolateDragPosition(pointerData, target, screenEnd, parameters.DragSpeed, ct) + await MouseUiDragEventExecutor.InterpolateDragPosition(session, inputEnd, parameters.DragSpeed, ct) .ConfigureAwait(false), inputStart, inputEnd, diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiPressTargetResolver.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiPressTargetResolver.cs index 718bab3b9a..16f6b6bd60 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiPressTargetResolver.cs +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiPressTargetResolver.cs @@ -12,7 +12,7 @@ namespace io.github.hatayama.UnityCliLoop.FirstPartyTools /// internal static class MouseUiPressTargetResolver { - private const string NoEventSystemMessage = "No EventSystem found in the scene. Ensure an EventSystem GameObject exists."; + internal const string NoEventSystemMessage = "No EventSystem found in the scene. Ensure an EventSystem GameObject exists."; private const string UIToolkitPathHint = " UI Toolkit element paths are not accepted here; click the element at its SimX/SimY without --bypass-raycast."; @@ -48,7 +48,7 @@ internal static (IMouseUiPressTarget? Target, SimulateMouseUiResponse? Failure) { return (null, MouseUiSimulationResponseFactory.CreateFailure( parameters, - $"No EventSystem found in the scene, and no UI Toolkit element is at ({inputPos.x:F1}, {inputPos.y:F1}). Ensure an EventSystem GameObject exists for uGUI.")); + NoEventSystemNorUIToolkitMessage(inputPos))); } return (new UguiPressTarget(ResolvedPointerTargets.Empty, uguiPointerData), null); @@ -92,6 +92,11 @@ private static (IMouseUiPressTarget? Target, SimulateMouseUiResponse? Failure) R return (null, failure); } + internal static string NoEventSystemNorUIToolkitMessage(Vector2 inputPos) + { + return $"No EventSystem found in the scene, and no UI Toolkit element is at ({inputPos.x:F1}, {inputPos.y:F1}). Ensure an EventSystem GameObject exists for uGUI."; + } + // The press data carries the uGUI raycast hit, whose raycaster decides the order against the panel. private static bool IsUguiHitInFront(PointerEventData uguiPointerData, UIToolkitPointerTarget toolkit) { diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiSimulationResponseFactory.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiSimulationResponseFactory.cs index 4eb4db1682..d35ff3695e 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiSimulationResponseFactory.cs +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/MouseUiSimulationResponseFactory.cs @@ -109,7 +109,7 @@ internal static SimulateMouseUiResponse CreateLongPressResult( }; } - private static string DescribeHit(MouseUiHitReport hit) + internal static string DescribeHit(MouseUiHitReport hit) { return hit.UiSystem == MouseUiHitSystem.UIToolkit ? $"UI Toolkit element '{hit.Name}'" @@ -120,16 +120,18 @@ internal static SimulateMouseUiResponse CreateDragResult( MouseUiSimulationCommand parameters, Vector2 inputStart, Vector2 inputEnd, - string targetName) + MouseUiHitReport hit) { return new SimulateMouseUiResponse { Success = true, Message = parameters.BypassRaycast - ? $"Bypass-dragged '{targetName}' from ({inputStart.x:F1}, {inputStart.y:F1}) to ({inputEnd.x:F1}, {inputEnd.y:F1}) via '{parameters.TargetPath}' at {parameters.DragSpeed:F0} px/s" - : $"Dragged '{targetName}' from ({inputStart.x:F1}, {inputStart.y:F1}) to ({inputEnd.x:F1}, {inputEnd.y:F1}) at {parameters.DragSpeed:F0} px/s", + ? $"Bypass-dragged '{hit.Name}' from ({inputStart.x:F1}, {inputStart.y:F1}) to ({inputEnd.x:F1}, {inputEnd.y:F1}) via '{parameters.TargetPath}' at {parameters.DragSpeed:F0} px/s" + : $"Dragged {DescribeHit(hit)} from ({inputStart.x:F1}, {inputStart.y:F1}) to ({inputEnd.x:F1}, {inputEnd.y:F1}) at {parameters.DragSpeed:F0} px/s", Action = MouseAction.Drag.ToString(), - HitGameObjectName = targetName, + HitGameObjectName = hit.Name, + UiSystem = hit.UiSystem, + HitPath = hit.Path, PositionX = inputStart.x, PositionY = inputStart.y, EndPositionX = inputEnd.x, @@ -140,14 +142,16 @@ internal static SimulateMouseUiResponse CreateDragResult( internal static SimulateMouseUiResponse CreateDragEndResult( MouseUiSimulationCommand parameters, Vector2 inputEnd, - string targetName) + MouseUiHitReport hit) { return new SimulateMouseUiResponse { Success = true, - Message = $"Drag ended on '{targetName}' at ({inputEnd.x:F1}, {inputEnd.y:F1}) at {parameters.DragSpeed:F0} px/s", + Message = $"Drag ended on {DescribeHit(hit)} at ({inputEnd.x:F1}, {inputEnd.y:F1}) at {parameters.DragSpeed:F0} px/s", Action = MouseAction.DragEnd.ToString(), - HitGameObjectName = targetName, + HitGameObjectName = hit.Name, + UiSystem = hit.UiSystem, + HitPath = hit.Path, PositionX = inputEnd.x, PositionY = inputEnd.y }; diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/SimulateMouseUiResponse.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/SimulateMouseUiResponse.cs index a4127095cd..c94fdbc777 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/SimulateMouseUiResponse.cs +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/SimulateMouseUiResponse.cs @@ -14,8 +14,9 @@ public class SimulateMouseUiResponse : UnityCliLoopToolResponse public string Message { get; set; } = ""; public string Action { get; set; } = ""; /// - /// Name of what received the press: the uGUI or physics GameObject that handles it, or the UI Toolkit - /// control under the point (its name, or its type name when it has none). Null when nothing was hit. + /// Name of what received the press or drag: the uGUI or physics GameObject that handles it, or the UI + /// Toolkit control under the point (its name, or its type name when it has none). Null when nothing was + /// hit. /// public string? HitGameObjectName { get; set; } diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/SimulateMouseUiUseCase.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/SimulateMouseUiUseCase.cs index 948a558427..b4f2532848 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/SimulateMouseUiUseCase.cs +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/SimulateMouseUiUseCase.cs @@ -56,15 +56,8 @@ public async Task ExecuteAsync( return validationFailure; } - // Clicks and long presses can reach UI Toolkit without an EventSystem and check for one when they - // resolve their target. DragMove and DragEnd executors never receive it, so drags are checked here. - if (MouseUiSimulationValidator.IsDragAction(parameters.Action) && eventSystem == null) - { - return MouseUiSimulationResponseFactory.CreateFailure( - parameters, - "No EventSystem found in the scene. Ensure an EventSystem GameObject exists."); - } - + // UI Toolkit targets need no EventSystem, so each action checks for one only once it knows it is + // going through uGUI. MouseUiSimulationActivityLogger.LogSimulationStart(parameters, correlationId); OverlayCanvasFactory.EnsureExists(); @@ -97,18 +90,16 @@ private async Task ExecuteMouseAction( ct).ConfigureAwait(false); case MouseAction.Drag: - Debug.Assert(eventSystem != null, "Drags without an EventSystem must be rejected before dispatch."); return await MouseUiOneShotDragExecutor.ExecuteDragOneShot( parameters, - eventSystem!, + eventSystem, _mainThreadCleanupScheduler, ct).ConfigureAwait(false); case MouseAction.DragStart: - Debug.Assert(eventSystem != null, "Drags without an EventSystem must be rejected before dispatch."); return await MouseUiIncrementalDragExecutor.ExecuteDragStart( parameters, - eventSystem!, + eventSystem, _mainThreadCleanupScheduler, ct).ConfigureAwait(false); diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/SKILL.md b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/SKILL.md index 0be9c8c4fb..3d67862bbd 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/SKILL.md +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/SKILL.md @@ -64,7 +64,7 @@ uloop simulate-mouse-ui --action --x --y [options] - Get coordinates from `AnnotatedElements` JSON (`SimX`/`SimY`) — do NOT look up GameObject positions - `--bypass-raycast` still uses coordinates for pointer event positions, but chooses the clicked, long-pressed, or dragged GameObject by `--target-path` - If `--target-path` or `--drop-target-path` matches multiple active GameObjects, the command fails instead of choosing an arbitrary duplicate -- UI Toolkit entries (`UiSystem` `UIToolkit`) can be clicked and long-pressed at `SimX`/`SimY`; `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObjects only. Details: `references/ui-toolkit.md`. +- UI Toolkit entries (`UiSystem` `UIToolkit`) can be clicked, long-pressed, and dragged at `SimX`/`SimY`; `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObjects only. Details: `references/ui-toolkit.md`. - Device Simulator play view is supported. Prefer `uloop screenshot --capture-mode rendering --annotate-elements` for coordinates; they use the simulated device resolution (`Handles.GetMainGameViewSize()` / `Screen`), not the Simulator chrome scale. ## Pause Point Inspection (Standard for E2E) @@ -86,8 +86,8 @@ uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300 # Long-press a button for 3 seconds uloop simulate-mouse-ui --action LongPress --x 400 --y 300 --duration 3.0 -# Slow drag for visual inspection -uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300 --drag-speed 200 +# One-shot drag (add --drag-speed 200 to slow it for visual inspection) +uloop simulate-mouse-ui --action Drag --from-x 400 --from-y 300 --x 600 --y 300 # Split drag with hold (for inspection between steps) uloop simulate-mouse-ui --action DragStart --x 400 --y 300 diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/references/output.md b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/references/output.md index af20d8c00b..03086035fa 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/references/output.md +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/references/output.md @@ -5,9 +5,9 @@ Returns JSON with: - `Success`: Whether the operation succeeded - `Message`: Status message (e.g. "Hit element: ButtonStart" or "No UI element under (x, y)") - `Action`: Echoes which action was executed (`Click`, `Drag`, `DragStart`, `DragMove`, `DragEnd`, or `LongPress`) -- `HitGameObjectName`: Name of the GameObject that received the pointer events (nullable string; null if nothing was hit). It is the event-handler target resolved up the hierarchy from the raycast hit (the first `IPointerDownHandler`, else the first `IPointerClickHandler`, for `Click`/`LongPress`; the first `IDragHandler` for drags), so it can be an ancestor of the topmost raycast hit — a Button's child Text reports the Button. With `--bypass-raycast` it is the handler resolved the same way from `--target-path` (also possibly an ancestor), not the raycast hit and not necessarily the raw path GameObject itself. For a UI Toolkit hit it is the control's `name`, the same as the annotation's `Name` -- `UiSystem`: Which UI system received the press (nullable string; `Click`/`LongPress` only, null if nothing was hit): `UGUI` for a target under a Canvas, `UIToolkit` for a UI Toolkit control, and `""` for a non-UI target such as a physics object -- `HitPath`: Path of the hit target in the form `AnnotatedElements[].Path` uses (nullable string; `Click`/`LongPress` only): a hierarchy path for GameObjects, `Document/Path#element` for UI Toolkit controls +- `HitGameObjectName`: Name of the GameObject that received the pointer events (nullable string; null if nothing was hit). It is the event-handler target resolved up the hierarchy from the raycast hit (the first `IPointerDownHandler`, else the first `IPointerClickHandler`, for `Click`/`LongPress`; the first `IDragHandler` for drags), so it can be an ancestor of the topmost raycast hit — a Button's child Text reports the Button. With `--bypass-raycast` it is the handler resolved the same way from `--target-path` (also possibly an ancestor), not the raycast hit and not necessarily the raw path GameObject itself. For a UI Toolkit hit it is the name of the outermost control an annotation would list (the same as the annotation's `Name`); when the picked element is not inside such a control (a bare `Label` or `VisualElement`), it is that element's own `name`, or its type name when empty, and the element is not annotated. Unlike uGUI, which reports null when no handler is found, UI Toolkit reports the picked element whether or not it handles pointer events +- `UiSystem`: Which UI system received the press (nullable string; `Click`/`LongPress` and drags, null if nothing was hit): `UGUI` for a target under a Canvas, `UIToolkit` for a UI Toolkit control, and `""` for a non-UI target such as a physics object +- `HitPath`: Path of the hit target in the form `AnnotatedElements[].Path` uses (nullable string; `Click`/`LongPress` and drags): a hierarchy path for GameObjects, `Document/Path#element` for UI Toolkit controls. Null when nothing was hit, or when the hit element belongs to no document, such as the menu of an open dropdown - `PositionX`: Target X coordinate that was used - `PositionY`: Target Y coordinate that was used - `EndPositionX`: Drag end X coordinate (nullable float; populated for drag actions only) @@ -20,4 +20,4 @@ Note: Click and LongPress on empty space (no UI element) still return `Success = ## UI Toolkit Elements -`Click` and `LongPress` reach UI Toolkit controls at their annotated `SimX`/`SimY`. How presses are delivered and what is not supported: `ui-toolkit.md`. +`Click`, `LongPress`, and the drag actions reach UI Toolkit controls at their annotated `SimX`/`SimY` (`--from-x`/`--from-y` for `Drag`). How presses and drags are delivered and what is not supported: `ui-toolkit.md`. diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/references/ui-toolkit.md b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/references/ui-toolkit.md index ec3bb79d44..373a3deee1 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/references/ui-toolkit.md +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/Skill/references/ui-toolkit.md @@ -1,9 +1,9 @@ # simulate-mouse-ui and UI Toolkit -`Click` and `LongPress` reach UI Toolkit controls on screen-space panels — the -`AnnotatedElements` entries with `UiSystem` `UIToolkit`. Pass the entry's `SimX`/`SimY` as -`--x`/`--y`; the response reports `UiSystem` `UIToolkit`, the control's `Name` as -`HitGameObjectName`, and its annotation `Path` as `HitPath`. +`Click`, `LongPress`, and the drag actions reach UI Toolkit controls on screen-space panels — +the `AnnotatedElements` entries with `UiSystem` `UIToolkit`. Pass the entry's `SimX`/`SimY` as +`--x`/`--y` (`--from-x`/`--from-y` for `Drag`); the response reports `UiSystem` `UIToolkit`, the +control's `Name` as `HitGameObjectName`, and its annotation `Path` as `HitPath`. ## How a press is delivered @@ -29,18 +29,33 @@ A UI Toolkit `Button` fires `clicked` once when the long press is released (UI T `Clickable` behaviour); a uGUI long press sends no click. A `RepeatButton` fires repeatedly while held. +## Drags + +- A drag whose start point resolves to a UI Toolkit control (by the rule above) presses it + with the left button, moves the pointer, and releases at the end, so a `Slider` or a + `Scroller` follows the drag. The start point is what counts: the drag stays with that panel + wherever it moves. +- A labelled slider's bounds include its label; start the drag on the track. +- `--drop-target-path` applies to uGUI drags only. A `Drag` with it fails before anything is + sent; a `DragEnd` with it fails and leaves the drag in progress, so call `DragEnd` again + without it. +- If the control's document is disabled or destroyed during a split drag, the next `DragMove` + or `DragEnd` fails and the drag ends. A one-shot `Drag` stops sending and still succeeds. +- If a UI Toolkit handler throws, the button is released and the exception ends the command. + A `DragMove` that throws leaves the drag in progress, as a cancelled `DragMove` does. + ## Not supported - `--bypass-raycast`, `--target-path`, and `--drop-target-path` take GameObject paths only; a UI Toolkit `Path` (it contains `#`) is reported as not found. -- Drag actions go through the uGUI EventSystem only. - World-space panels (Unity 6), as in the annotation. - Two panels with the same `sortingOrder` overlapping at the point: Unity orders them by creation order, which is not exposed, so either one may be chosen. ## Notes -- Move the real mouse cursor out of the Game View while simulating. Real pointer input over - the Game View reaches the same panels and can interleave with the simulated press. +- Move the real mouse cursor out of the Game View while simulating, and keep it away between + `DragStart` and `DragEnd`. Real pointer input over the Game View reaches the same panels and + can interleave with the simulated press or move a held slider. - Device Simulator has the same limits as the annotation: coordinates use the simulated device resolution. diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UIToolkitDragSession.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UIToolkitDragSession.cs new file mode 100644 index 0000000000..19c93776fb --- /dev/null +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UIToolkitDragSession.cs @@ -0,0 +1,78 @@ +#nullable enable +using UnityEngine; + +namespace io.github.hatayama.UnityCliLoop.FirstPartyTools +{ + /// + /// A drag on a UI Toolkit control: the left button is pressed on the control, moved, and released through + /// its panel, so controls that capture the pointer (Slider, Scroller) follow it as with real input. + /// + internal sealed class UIToolkitDragSession : IMouseUiDragSession + { + private const int LeftButton = 0; + + private readonly UIToolkitPointerTarget _target; + private Vector2 _currentInputPosition; + + private UIToolkitDragSession(UIToolkitPointerTarget target, Vector2 inputStart) + { + _target = target; + _currentInputPosition = inputStart; + Report = new MouseUiHitReport(target.Name, target.Path, MouseUiHitSystem.UIToolkit); + } + + /// + /// Presses the left button on the control at the start point. + /// + // UI Toolkit passes handler exceptions on to the sender; a press that throws is released before the + // exception leaves, since no session exists yet to release it later. + internal static UIToolkitDragSession Begin(UIToolkitPointerTarget target, Vector2 inputStart) + { + bool pressed = false; + try + { + target.PressDown(inputStart, LeftButton); + pressed = true; + } + finally + { + if (!pressed) + { + target.Release(inputStart, LeftButton); + } + } + + return new UIToolkitDragSession(target, inputStart); + } + + public MouseUiHitReport Report { get; } + + // Stored as an input position: UI Toolkit maps Game View points itself and never goes through Screen. + public Vector2 CurrentInputPosition => _currentInputPosition; + + public bool IsTargetDestroyed => _target.IsDocumentDestroyed; + + public bool UsesEventSystem => false; + + // No "interrupted" check as uGUI has: some controls never capture the pointer, so the lack of a capture + // cannot tell an interrupted drag from a normal one. + public string? CheckStillActive() + { + return _target.IsAvailable + ? null + : "Drag target UI Toolkit element is no longer available (its document was disabled or destroyed)."; + } + + public void MoveTo(Vector2 inputPosition) + { + _target.MoveTo(inputPosition); + _currentInputPosition = inputPosition; + } + + // Callers refuse a drop target before a UI Toolkit drag starts or ends, so none reaches here. + public void Finish(GameObject? explicitDropTarget) + { + _target.Release(_currentInputPosition, LeftButton); + } + } +} diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UIToolkitDragSession.cs.meta b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UIToolkitDragSession.cs.meta new file mode 100644 index 0000000000..e57a5b28da --- /dev/null +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UIToolkitDragSession.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: b24745134b0d44ab6ad2ed49346d0edd +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UguiDragSession.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UguiDragSession.cs new file mode 100644 index 0000000000..4f05787a27 --- /dev/null +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UguiDragSession.cs @@ -0,0 +1,81 @@ +#nullable enable +using UnityEngine; +using UnityEngine.EventSystems; + +namespace io.github.hatayama.UnityCliLoop.FirstPartyTools +{ + /// + /// A drag on a uGUI or physics IDragHandler, driven through the EventSystem's pointer events. + /// + internal sealed class UguiDragSession : IMouseUiDragSession + { + private readonly PointerEventData _pointerData; + private readonly GameObject _target; + + private UguiDragSession(PointerEventData pointerData, GameObject target) + { + _pointerData = pointerData; + _target = target; + Report = MouseUiGameObjectHitReport.Create(target); + } + + /// + /// Presses on the start raycast and sends beginDrag to the drag handler. + /// + internal static UguiDragSession Begin( + EventSystem eventSystem, + Vector2 screenPos, + RaycastResult startRaycast, + GameObject target) + { + // uGUI drag controls (ScrollRect, Slider) only respond to left-button drags + PointerEventData pointerData = MouseUiDragEventExecutor.InitiateDrag( + eventSystem, screenPos, startRaycast, target, PointerEventData.InputButton.Left); + ExecuteEvents.Execute(target, pointerData, ExecuteEvents.beginDragHandler); + pointerData.dragging = true; + return new UguiDragSession(pointerData, target); + } + + public MouseUiHitReport Report { get; } + + // Derived from the stored screen position with the current Game View height rather than stored as an + // input position, so a split drag continues from the same screen point if the Game View is resized. + public Vector2 CurrentInputPosition => MouseUiCoordinateConverter.ScreenToInput(_pointerData.position); + + public bool IsTargetDestroyed => _target == null; + + public bool UsesEventSystem => true; + + // User input during a CLI drag can cause Unity's StandaloneInputModule to + // release or reassign the drag, leaving the stored session stale. + public string? CheckStillActive() + { + if (!_target.activeInHierarchy) + { + return "Drag target was destroyed or deactivated during drag."; + } + + if (!_pointerData.dragging || _pointerData.pointerDrag != _target) + { + return "Drag was interrupted by user input or system event."; + } + + return null; + } + + public void MoveTo(Vector2 inputPosition) + { + Vector2 screenPosition = MouseUiCoordinateConverter.InputToScreen(inputPosition); + Vector2 previousPosition = _pointerData.position; + _pointerData.position = screenPosition; + _pointerData.delta = screenPosition - previousPosition; + MouseUiInputSystemSync.SyncMousePosition(screenPosition); + ExecuteEvents.Execute(_target, _pointerData, ExecuteEvents.dragHandler); + } + + public void Finish(GameObject? explicitDropTarget) + { + MouseUiDragEventExecutor.FinalizeDrag(_pointerData, _target, explicitDropTarget); + } + } +} diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UguiDragSession.cs.meta b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UguiDragSession.cs.meta new file mode 100644 index 0000000000..400210f5d3 --- /dev/null +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UguiDragSession.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: 148da36fe4aef42229f20a15a164c074 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UguiPressTarget.cs b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UguiPressTarget.cs index ada219b6d9..808b9c7b5d 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UguiPressTarget.cs +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseUi/UguiPressTarget.cs @@ -23,26 +23,11 @@ internal UguiPressTarget(ResolvedPointerTargets resolvedTargets, PointerEventDat // UnityEngine.Object targets for null is not safe. bool shouldReleasePointer = resolvedTargets.RawTarget != null && resolvedTargets.Target != null; _shouldReleasePointer = shouldReleasePointer; - Report = CreateReport(resolvedTargets.Target); + Report = MouseUiGameObjectHitReport.Create(resolvedTargets.Target); } public MouseUiHitReport Report { get; } - // Decided by whether the handler sits under a Canvas rather than by raycaster type, because a bypass - // press has no raycaster and both paths must report alike. - private static MouseUiHitReport CreateReport(GameObject? target) - { - if (target == null) - { - return MouseUiHitReport.None; - } - - string uiSystem = target.GetComponentInParent() != null - ? MouseUiHitSystem.UGUI - : MouseUiHitSystem.NonUi; - return new MouseUiHitReport(target.name, GameObjectPathUtility.GetFullPath(target), uiSystem); - } - public void Click() { if (_resolvedTargets.RawTarget == null) diff --git a/scripts/test-screenshot-uitoolkit-annotation.ps1 b/scripts/test-screenshot-uitoolkit-annotation.ps1 index a5371d6d4d..1795660cce 100644 --- a/scripts/test-screenshot-uitoolkit-annotation.ps1 +++ b/scripts/test-screenshot-uitoolkit-annotation.ps1 @@ -1,6 +1,6 @@ <# E2E verification that screenshot --annotate-elements lists UI Toolkit controls and that simulate-mouse-ui and -simulate-mouse-input press them at the annotated points, using the uloop CLI against UIToolkitAnnotationDemoScene. +simulate-mouse-input press and drag them at the annotated points, using the uloop CLI against UIToolkitAnnotationDemoScene. Safe to run repeatedly: it stops PlayMode and restores the Game View size on exit, and never saves the scene, so the listener and EventSystem changes made in PlayMode are discarded. @@ -583,6 +583,63 @@ return GameObject.Find("ugui-button").GetComponentInChildren().text; Assert-EqualText -Actual (Invoke-DynamicCodeText -Code $labelCode -Context "Read ugui-button label") -Expected "clicked 1" -Context "ugui-button label" } +function Get-SliderValue { + [string]$code = @' +UIToolkitAnnotationDemo demo = UnityEngine.Object.FindObjectOfType(); +if (demo == null) { return "ERROR: UIToolkitAnnotationDemo not found"; } +return demo.GetSliderValue("volume").ToString(System.Globalization.CultureInfo.InvariantCulture); +'@ + [string]$value = Invoke-DynamicCodeText -Code $code -Context "Read volume value" + if ($value.StartsWith("ERROR:") -or $value -eq "NaN") { + throw "Read volume value failed: $value" + } + + return [double]::Parse($value, [Globalization.CultureInfo]::InvariantCulture) +} + +function Assert-InRange { + param( + [double]$Value, + [double]$Min, + [double]$Max, + [string]$Context + ) + + Assert-Condition -Condition ($Value -ge $Min -and $Value -le $Max) -Message "${Context}: expected $Min..$Max, got $Value" +} + +function Test-OneShotDrag { + param( + [pscustomobject]$ElementsOnly + ) + + [pscustomobject]$volume = Get-AnnotatedElement -Response $ElementsOnly -UiSystem "UIToolkit" -Name "volume" + [pscustomobject]$response = Invoke-SimulateUi -Context "simulate-mouse-ui drag on volume" -Arguments @( + "--action", "Drag", + "--from-x", (Format-Coordinate ($volume.BoundsMinX + 5)), "--from-y", (Format-Coordinate $volume.SimY), + "--x", (Format-Coordinate ($volume.BoundsMaxX - 5)), "--y", (Format-Coordinate $volume.SimY)) + Assert-EqualText -Actual $response.UiSystem -Expected "UIToolkit" -Context "simulate-mouse-ui drag on volume UiSystem" + Assert-InRange -Value (Get-SliderValue) -Min 0.9 -Max 1 -Context "volume after a drag to its right edge" +} + +function Test-SplitDrag { + param( + [pscustomobject]$ElementsOnly + ) + + [pscustomobject]$volume = Get-AnnotatedElement -Response $ElementsOnly -UiSystem "UIToolkit" -Name "volume" + [string]$y = Format-Coordinate $volume.SimY + [double]$left = $volume.BoundsMinX + 5 + [double]$right = $volume.BoundsMaxX - 5 + [string]$center = Format-Coordinate (($left + $right) / 2) + + Invoke-SimulateUi -Context "simulate-mouse-ui DragStart on volume" -Arguments @("--action", "DragStart", "--x", (Format-Coordinate $left), "--y", $y) | Out-Null + Invoke-SimulateUi -Context "simulate-mouse-ui DragMove on volume" -Arguments @("--action", "DragMove", "--x", $center, "--y", $y) | Out-Null + Assert-InRange -Value (Get-SliderValue) -Min 0.3 -Max 0.7 -Context "volume after DragMove to its centre" + Invoke-SimulateUi -Context "simulate-mouse-ui DragEnd on volume" -Arguments @("--action", "DragEnd", "--x", (Format-Coordinate $left), "--y", $y) | Out-Null + Assert-InRange -Value (Get-SliderValue) -Min 0 -Max 0.1 -Context "volume after DragEnd at its left edge" +} + # Runs last: the EventSystem stays disabled until PlayMode stops on exit. function Test-ClickWithoutEventSystem { param( @@ -618,42 +675,48 @@ Wait-UnityReady [pscustomobject]$originalGameViewSize = $null try { - Write-Host "[1/12] Loading UI Toolkit annotation demo scene..." + Write-Host "[1/14] Loading UI Toolkit annotation demo scene..." Initialize-DemoScene - Write-Host "[2/12] Selecting Full HD Game View resolution..." + Write-Host "[2/14] Selecting Full HD Game View resolution..." $originalGameViewSize = Select-FullHdGameView - Write-Host "[3/12] Starting PlayMode and waiting for UI Toolkit controls..." + Write-Host "[3/14] Starting PlayMode and waiting for UI Toolkit controls..." Invoke-UloopJson -CommandArguments @("control-play-mode", "--action", "Play") | Out-Null Wait-PlayMode Wait-UiToolkitPlay - Write-Host "[4/12] Checking the annotated capture..." + Write-Host "[4/14] Checking the annotated capture..." [pscustomobject]$annotated = Test-AnnotatedCapture - Write-Host "[5/12] Checking the elements-only capture matches..." + Write-Host "[5/14] Checking the elements-only capture matches..." [pscustomobject]$elementsOnly = Test-ElementsOnlyCapture -AnnotatedResponse $annotated - Write-Host "[6/12] Clicking UI Toolkit controls with simulate-mouse-ui..." + Write-Host "[6/14] Clicking UI Toolkit controls with simulate-mouse-ui..." Test-UiClick -ElementsOnly $elementsOnly - Write-Host "[7/12] Clicking a UI Toolkit button with simulate-mouse-input..." + Write-Host "[7/14] Clicking a UI Toolkit button with simulate-mouse-input..." Test-InputClick -ElementsOnly $elementsOnly - Write-Host "[8/12] Clicking a partly covered UI Toolkit button..." + Write-Host "[8/14] Clicking a partly covered UI Toolkit button..." Test-CoveredClick -ElementsOnly $elementsOnly - Write-Host "[9/12] Clicking a disabled UI Toolkit button..." + Write-Host "[9/14] Clicking a disabled UI Toolkit button..." Test-DisabledClick - Write-Host "[10/12] Long-pressing a UI Toolkit button..." + Write-Host "[10/14] Long-pressing a UI Toolkit button..." Test-LongPress -ElementsOnly $elementsOnly - Write-Host "[11/12] Clicking the uGUI button with simulate-mouse-ui..." + Write-Host "[11/14] Clicking the uGUI button with simulate-mouse-ui..." Test-UguiClick -ElementsOnly $elementsOnly - Write-Host "[12/12] Clicking a UI Toolkit button without an EventSystem..." + Write-Host "[12/14] Dragging a UI Toolkit slider in one shot..." + Test-OneShotDrag -ElementsOnly $elementsOnly + + Write-Host "[13/14] Dragging a UI Toolkit slider in steps..." + Test-SplitDrag -ElementsOnly $elementsOnly + + Write-Host "[14/14] Clicking a UI Toolkit button without an EventSystem..." Test-ClickWithoutEventSystem -ElementsOnly $elementsOnly Write-Host "" diff --git a/scripts/test-screenshot-uitoolkit-annotation.sh b/scripts/test-screenshot-uitoolkit-annotation.sh index 6c9359e017..e198563289 100755 --- a/scripts/test-screenshot-uitoolkit-annotation.sh +++ b/scripts/test-screenshot-uitoolkit-annotation.sh @@ -1,6 +1,6 @@ #!/bin/sh # E2E verification that screenshot --annotate-elements lists UI Toolkit controls and that simulate-mouse-ui and -# simulate-mouse-input press them at the annotated points, using the uloop CLI against +# simulate-mouse-input press and drag them at the annotated points, using the uloop CLI against # UIToolkitAnnotationDemoScene. Safe to run repeatedly: it stops PlayMode and restores the Game View size on # exit, and never saves the scene, so the listener and EventSystem changes made in PlayMode are discarded. @@ -419,6 +419,62 @@ return GameObject.Find("ugui-button").GetComponentInChildren().text; assert_text_equals "$(printf '%s\n' "$json" | jq -r '.Result')" "clicked 1" "ugui-button label" } +get_slider_value() { + json=$(run_uloop_json execute-dynamic-code --code ' +UIToolkitAnnotationDemo demo = UnityEngine.Object.FindObjectOfType(); +if (demo == null) { return "ERROR: UIToolkitAnnotationDemo not found"; } +return demo.GetSliderValue("volume").ToString(System.Globalization.CultureInfo.InvariantCulture); +') + assert_json_success "$json" "Read volume value" + value=$(printf '%s\n' "$json" | jq -r '.Result') + case "$value" in + ERROR:*|NaN) + fail "Read volume value failed: $value" + ;; + esac + printf '%s\n' "$value" +} + +# Fails unless the number is within [min, max]. +assert_in_range() { + value=$1 + min=$2 + max=$3 + context=$4 + jq -e -n --argjson v "$value" --argjson min "$min" --argjson max "$max" '$v >= $min and $v <= $max' >/dev/null \ + || fail "$context: expected $min..$max, got $value" +} + +VOLUME_SELECTOR='.UiSystem == "UIToolkit" and .Name == "volume"' + +# Points 5 pixels inside the slider's left and right edges, on its annotated row. +volume_left_x() { + jq -n --argjson min "$(annotated_field "$VOLUME_SELECTOR" BoundsMinX)" '$min + 5' +} + +volume_right_x() { + jq -n --argjson max "$(annotated_field "$VOLUME_SELECTOR" BoundsMaxX)" '$max - 5' +} + +verify_one_shot_drag() { + y=$(annotated_field "$VOLUME_SELECTOR" SimY) + json=$(simulate_ui "simulate-mouse-ui drag on volume" --action Drag \ + --from-x "$(volume_left_x)" --from-y "$y" --x "$(volume_right_x)" --y "$y") + assert_response_field "$json" UiSystem "UIToolkit" "simulate-mouse-ui drag on volume" + assert_in_range "$(get_slider_value)" 0.9 1 "volume after a drag to its right edge" +} + +verify_split_drag() { + y=$(annotated_field "$VOLUME_SELECTOR" SimY) + center_x=$(jq -n --argjson l "$(volume_left_x)" --argjson r "$(volume_right_x)" '($l + $r) / 2') + + simulate_ui "simulate-mouse-ui DragStart on volume" --action DragStart --x "$(volume_left_x)" --y "$y" >/dev/null + simulate_ui "simulate-mouse-ui DragMove on volume" --action DragMove --x "$center_x" --y "$y" >/dev/null + assert_in_range "$(get_slider_value)" 0.3 0.7 "volume after DragMove to its centre" + simulate_ui "simulate-mouse-ui DragEnd on volume" --action DragEnd --x "$(volume_left_x)" --y "$y" >/dev/null + assert_in_range "$(get_slider_value)" 0 0.1 "volume after DragEnd at its left edge" +} + # Runs last: the EventSystem stays disabled until PlayMode stops on exit. verify_click_without_event_system() { json=$(run_uloop_json execute-dynamic-code --code ' @@ -448,42 +504,48 @@ printf '=========================================\n' wait_unity_ready -printf '[1/12] Loading UI Toolkit annotation demo scene...\n' +printf '[1/14] Loading UI Toolkit annotation demo scene...\n' initialize_demo_scene -printf '[2/12] Selecting Full HD Game View resolution...\n' +printf '[2/14] Selecting Full HD Game View resolution...\n' select_full_hd_game_view -printf '[3/12] Starting PlayMode and waiting for UI Toolkit controls...\n' +printf '[3/14] Starting PlayMode and waiting for UI Toolkit controls...\n' run_uloop_json control-play-mode --action Play >/dev/null wait_play_mode wait_for_uitoolkit_play -printf '[4/12] Checking the annotated capture...\n' +printf '[4/14] Checking the annotated capture...\n' verify_annotated_capture -printf '[5/12] Checking the elements-only capture matches...\n' +printf '[5/14] Checking the elements-only capture matches...\n' verify_elements_only_capture -printf '[6/12] Clicking UI Toolkit controls with simulate-mouse-ui...\n' +printf '[6/14] Clicking UI Toolkit controls with simulate-mouse-ui...\n' verify_ui_click -printf '[7/12] Clicking a UI Toolkit button with simulate-mouse-input...\n' +printf '[7/14] Clicking a UI Toolkit button with simulate-mouse-input...\n' verify_input_click -printf '[8/12] Clicking a partly covered UI Toolkit button...\n' +printf '[8/14] Clicking a partly covered UI Toolkit button...\n' verify_covered_click -printf '[9/12] Clicking a disabled UI Toolkit button...\n' +printf '[9/14] Clicking a disabled UI Toolkit button...\n' verify_disabled_click -printf '[10/12] Long-pressing a UI Toolkit button...\n' +printf '[10/14] Long-pressing a UI Toolkit button...\n' verify_long_press -printf '[11/12] Clicking the uGUI button with simulate-mouse-ui...\n' +printf '[11/14] Clicking the uGUI button with simulate-mouse-ui...\n' verify_ugui_click -printf '[12/12] Clicking a UI Toolkit button without an EventSystem...\n' +printf '[12/14] Dragging a UI Toolkit slider in one shot...\n' +verify_one_shot_drag + +printf '[13/14] Dragging a UI Toolkit slider in steps...\n' +verify_split_drag + +printf '[14/14] Clicking a UI Toolkit button without an EventSystem...\n' verify_click_without_event_system printf '\n'