From 9b68129a981780775fdcb5760e11f00d331e16a6 Mon Sep 17 00:00:00 2001 From: DoodleBear Date: Thu, 22 Oct 2026 15:46:20 +0900 Subject: [PATCH] feat(mobile): use native iOS composer actions and dictation gestures Issue: #2575 Agent-Task: codex/local/01a1621c-a81b-7670-86be-1ddaf404c2e2 --- .../features/phone-composer.md | 6 +- apps/mobile/DESIGN.md | 5 +- .../src/ui/chat/composer-action.ios.tsx | 17 ++ apps/mobile/src/ui/chat/composer-action.tsx | 57 +++++ .../src/ui/chat/composer-chrome.ios.tsx | 43 ++-- apps/mobile/src/ui/chat/composer-chrome.tsx | 2 + apps/mobile/src/ui/chat/composer-symbol.ts | 1 + apps/mobile/src/ui/chat/composer.tsx | 163 ++----------- apps/mobile/src/ui/chat/jump-to-latest.tsx | 27 ++- .../ui/chat/native-composer-button.ios.tsx | 80 ++++++ apps/mobile/src/ui/chat/voice-button.ios.tsx | 141 +++++++++++ apps/mobile/src/ui/chat/voice-button.tsx | 92 +++++++ apps/mobile/test/composer-chrome-ios.test.tsx | 87 ++++++- apps/mobile/test/voice-button-ios.test.tsx | 227 ++++++++++++++++++ changes/2575-ios-native-composer-controls.md | 11 + 15 files changed, 773 insertions(+), 186 deletions(-) create mode 100644 apps/mobile/src/ui/chat/composer-action.ios.tsx create mode 100644 apps/mobile/src/ui/chat/composer-action.tsx create mode 100644 apps/mobile/src/ui/chat/composer-symbol.ts create mode 100644 apps/mobile/src/ui/chat/native-composer-button.ios.tsx create mode 100644 apps/mobile/src/ui/chat/voice-button.ios.tsx create mode 100644 apps/mobile/src/ui/chat/voice-button.tsx create mode 100644 apps/mobile/test/voice-button-ios.test.tsx create mode 100644 changes/2575-ios-native-composer-controls.md diff --git a/.agents/skills/verify-deepseekbot/features/phone-composer.md b/.agents/skills/verify-deepseekbot/features/phone-composer.md index 9c75669015..4493b13057 100644 --- a/.agents/skills/verify-deepseekbot/features/phone-composer.md +++ b/.agents/skills/verify-deepseekbot/features/phone-composer.md @@ -10,13 +10,15 @@ Open a paired computer, then a Human/Bot DM or a group conversation. Use the mes ## Driving it with verify-app -Use `docs/agents/device-qa.md` and its designated runner, qualified native client and isolated QA Host. Drive the actual native app by accessible labels or `composer-input`, `composer-attach` and `composer-send`. Source tests in `apps/mobile/test/dictation.test.ts` and the style/system-text guards do not prove native geometry. +Use `docs/agents/device-qa.md` and its designated runner, qualified native client and isolated QA Host. Drive the actual native app by accessible labels or `composer-input`, `composer-attach`, `composer-send`, `composer-expand`, `composer-dictation`, `composer-dictation-cancel` and `jump-to-latest`. Source tests in `apps/mobile/test/dictation.test.ts` and the style/system-text guards do not prove native geometry. 1. Capture matched base/candidate empty and short-draft states: iOS has one continuous rounded surface around +, field and trailing control; default compact height is 56pt and the controls are at least 44pt. Check opaque Reduce Transparency fallback too. 2. Open the real soft keyboard, type several lines and move the caret. Open/close the attachment tray, pick and remove an owned fixture, expand/return from editing and preserve the draft and selection. Required controls and the current editing line remain reachable. 3. Repeat light/dark at default, intermediate and maximum system text. Change default → maximum → default live with a draft open; retain the editable field, draft and focus. Record CJK IME, VoiceOver and orientation separately if unavailable. 4. Send a uniquely marked message and read the QA Host Channel timeline: exact text, attachments and mention metadata agree. A Bot reply requires a usable model. Exercise dictation start, cancel and stop with microphone permission; audio/transcription availability is a separate prerequisite. -5. Check the unaffected Android separate-button path when its queue/device is available; never infer that result from an iOS capture. +5. For iOS native controls, tap the entire 44pt target (including its edge), open/close attachments, send exactly once and expand/return without losing input focus or draft. Scroll to history, receive unseen messages, verify the count badge, then tap the native return button and confirm it follows the latest message. Repeat with Reduce Transparency. +6. With working microphone/recognition, tap the mic then Stop; hold ≥300ms and release; hold and drag left ≥80pt to cancel or up ≥60pt to lock; interrupt an unlocked hold or add a second finger. Confirm recording cleanup, exact draft preservation/discard semantics and no message sent automatically. VoiceOver activation is a separate physical/assistive check, not proven by a mocked callback. +7. Check the unaffected Android separate-button path when its queue/device is available; never infer that result from an iOS capture. ## Observable proof diff --git a/apps/mobile/DESIGN.md b/apps/mobile/DESIGN.md index 9fab70b548..0638d9de35 100644 --- a/apps/mobile/DESIGN.md +++ b/apps/mobile/DESIGN.md @@ -187,7 +187,7 @@ The shell is set by [ADR-0211](../../docs/adr/0211-the-mobile-app-has-two-tabs-w - Activity rows show Bot DMs, posts elsewhere, memory commits, Bot creation, departures and stopped turns as one left-aligned meta line with a kind icon and tail ellipsis. They use the same `bubbleRow(true)` inset and single `space.x4` (12pt) top gap as a new message group, with no bottom padding or vertically centred minimum-height box: bubble → activity, activity → activity and activity → new message group keep the same boundary gap. Icons and avatars match the meta line height. In groups, the actor avatar occupies the existing speaker gutter, followed by the icon at the bubble's left edge and the description. In a Human DM, omit the current Bot's avatar and redundant name; Bot-DM actions show the other Bot's avatar after the icon. Missing avatars use the person icon. - Activity rows do not navigate on tap. Long-press (or the accessibility details action) opens the message sheet with the complete plain-text content in a scrollable, selectable block capped at `messageSheet.textHeightRatio` of the window height, then Copy/Share and an explicit Open conversation, Open memory or Open new Bot action when a destination exists; there are no reactions. Consecutive posts by the same Bot into the same Channel on the same day fold into a counted row whose menu opens the latest post. Memory opens the Bot's memory files, not a commit diff. Bot-to-Bot DMs remain read-only. Keep full actor/action/detail text in the accessibility label and menu even when the compact memory line uses only the icon and commit subject. - A top-to-bottom list (not inverted), oldest first, on one code path for iOS and Android. It opens hidden and is shown once it has been scrolled to the end and its content has settled for a frame, so it never flashes the oldest messages first. Short conversations sit at the bottom, above the composer. Older history loads when the Human nears the top (`onStartReached`), and `maintainVisibleContentPosition` keeps the messages on screen where they were while it is prepended. Messages from the same author within 5 minutes form a group, and a time separator appears after a 30-minute gap. - - At the newest message the list follows a streaming reply: content growth pins it to the end with a non-animated scroll, at most once per frame. Once the Human begins a drag or scrolls up into history, nothing moves the list under them: the visual playback cursor holds while received text continues to queue and persist. Only finishing a return gesture near the bottom or choosing the jump button resumes playback; a layout-driven scroll callback can neither rearm nor release following. Only a drag, the status-bar scroll to top or a jump to a linked message leaves the newest message, so the inset adjustment, content-size reports and `maintainVisibleContentPosition` corrections while the list first fills cannot strand it at the oldest message. A round jump button with a ⌄ then floats at the trailing edge `space.x2` above the composer, aligned with its trailing edge, like Grok (a 36pt Liquid Glass circle like **+** on iOS 26 with Reduce Transparency off, otherwise `raised`), with an accent badge counting the Bot messages that arrived since they left (99+ at most), like the Web Client's "Return to latest messages". Tapping it, sending a message, or scrolling back down to the newest message follows again and hides the button. + - At the newest message the list follows a streaming reply: content growth pins it to the end with a non-animated scroll, at most once per frame. Once the Human begins a drag or scrolls up into history, nothing moves the list under them: the visual playback cursor holds while received text continues to queue and persist. Only finishing a return gesture near the bottom or choosing the jump button resumes playback; a layout-driven scroll callback can neither rearm nor release following. Only a drag, the status-bar scroll to top or a jump to a linked message leaves the newest message, so the inset adjustment, content-size reports and `maintainVisibleContentPosition` corrections while the list first fills cannot strand it at the oldest message. A round jump button with a ⌄ then floats at the trailing edge `space.x2` above the composer, aligned with its trailing edge, like Grok (a native SwiftUI glass button on iOS 26 with Reduce Transparency off, an opaque bordered native button on older iOS or Reduce Transparency, otherwise `raised`), with an accent badge counting the Bot messages that arrived since they left (99+ at most), like the Web Client's "Return to latest messages". Tapping it, sending a message, or scrolling back down to the newest message follows again and hides the button. - **Pull up to type (iOS).** At the newest message, pulling the list further up past its end (the rubber-band bounce) and letting go focuses the composer and raises the keyboard, like pull-to-refresh in reverse. A drag counts only when it begins at the newest message (within the follow slop), so reading history never triggers it. A capsule with a keyboard symbol fades in above the composer as the pull grows, reading "Pull up to type" and switching to "Release to type" with one light haptic once the pull passes `pullToCompose.threshold` (40pt of offset, about 95pt of finger travel against the rubber band); letting go follows the newest message again first, so the keyboard does not cover it; easing back below it before release cancels. It is off where there is no composer (a Bot-to-Bot DM) or the Host is not connected. Android's overscroll stretches without moving the offset, so there is nothing to measure while the finger pulls: on Android an upward flick at the newest message (finger speed at least `pullToCompose.flick`, 0.3 dp/ms, so a calm upward swipe counts as well as a quick flick, with the list still at the newest message on release) focuses the composer with the same haptic, and no hint is drawn. Android's list sends no drag at all when it cannot scroll, so on Android the list's content is kept one hairline taller than the viewport; a short DM whose messages do not fill the screen then takes the same flick. - Settings → **Message display** offers **Streaming** and **Complete bubbles**, saved on this device. Streaming uses a local playback queue with a short prebuffer and reserve, releasing graphemes on a bounded clock. Rate changes follow queue depth gradually; clauses, sentences and paragraphs add progressively longer pauses. Android fades the active prose tail on the UI thread (180ms fade, up to 72ms stagger, at most 48 animated spans, with up to eight new spans per update); earlier text stays visible. Source commitment preserves the displayed cursor and row while the rest drains, then enables message actions. Historical messages open directly. Reduce Motion skips the fade. Code/table/native-only Markdown keep their established renderer; their content still follows the playback cursor. Complete bubbles keeps the working indicator and reveals each committed Channel Message independently. Switching modes changes presentation only; received drafts, committed messages and interruption semantics remain owned by Conversation and the Host. A draft-only append updates its row subscription while the timeline snapshot and historical message references stay stable. The bottom fade belongs to the keyboard-sticky composer. - Android Markdown uses synchronous native text/layout for prose, headings, lists, quotes, code and tables. Completed source blocks retain their keys and avoid rerendering when only the tail changes. Images and native-only extensions retain the enriched renderer. Drafts and committed messages keep the same renderer to avoid a settlement-only layout switch; iOS keeps its existing enriched implementation. @@ -233,6 +233,7 @@ The shell is set by [ADR-0211](../../docs/adr/0211-the-mobile-app-has-two-tabs-w - iOS follows iOS 26 Messages: on iOS 26 with Reduce Transparency off, **+** is its own Liquid Glass circle and the field its own Liquid Glass capsule, both 40pt tall (`touch − space.x2`, like Grok's compose bar), and **+** keeps a 44pt hit area. Both use `glassEffect`, regular, tinted with the opaque `surface`, so they read as near-opaque white glass like Grok's compose bar and messages passing under the field never compete with the draft. They float over the messages with no bar behind them. Older iOS and Reduce Transparency keep the same layout on an opaque `surface` fill with a `borderStrong` hairline. Pressing **+** tints it like the header buttons. The trailing button is always a filled `fg` circle with a `base` glyph, like Grok's compose bar, not the Bot's accent: the mic at rest and while recording, Send once there is text. Android keeps the plain mic and the accent fill for Send. Every text field opens the keyboard in the app's chosen appearance, so a light app on a dark system gets the light keyboard. - Android follows Material 3 (Google Messages): **+** is a filled round icon button on the same `raised` fill as the field (like Grok's compose bar), with a bounded ripple, and the field is a filled capsule on the `raised` fill with no edge and no glass. The mic, Send and expand buttons ripple too. - **+** sits inside the iOS island without its own glass or opaque disc; on Android it stays outside the capsule, bottom-aligned as the field grows. While dictating it becomes **×** (cancel), and while the mic is held, a red dot. With one trailing button (the mic or Send) the capsule is one row: the field and the button. When the text no longer fits that one row, or with more (attached files, or the mic next to Send), the capsule becomes a rounded rectangle with the same corner radius and stacks like the web composer: the field on top at full width, and a row below with **+** (iOS) and expand on the left and the buttons on the right. The iOS compact island is 56pt at default text size, with at least 44pt hit targets; it grows for system text and multiline drafts without remounting the input. + - On iOS, **+**/cancel, Send, Expand and return-to-latest use SwiftUI buttons with system press feedback and SF Symbols. Embedded actions use plain button styling on the shared island; Send retains the monochrome primary fill. The input stays UIKit-backed through the stable React Native TextInput adapter. Android retains its existing controls. - **+** opens Photos, Camera and File below the composer. Picked files show as removable tiles inside the capsule, up to 10 per message. Send appears once there is text or a file. - While the field is focused and the clipboard holds an image, a panel above the composer, styled like the @ suggestions (Liquid Glass on iOS 26, otherwise the `sheet` layer), reads "Image on the clipboard" with a paste control. On iOS it is the system Paste button (`UIPasteControl`, through `expo-clipboard`'s `ClipboardPasteButton`, image only, icon only at accessibility text sizes), so no paste-permission prompt appears. On Android it is a text `Button` "Paste image" that reads the clipboard; Android only shows its own brief "pasted from your clipboard" notice. The image attaches as a JPEG tile and the row hides until something new is copied. The @ suggestions take that slot first. A text field cannot take an image paste on its own in React Native, so this is the paste path. - With a hardware keyboard, **⌘↩** sends on iOS and iPadOS (listed in the ⌘-hold shortcut overlay as "Send"), and on Android **Enter** sends while **Shift+Enter** adds a line. **Ctrl+Enter** also sends on Android. In the full-screen draft editor plain Enter adds a line, and only ⌘↩ or Ctrl+Enter sends. The on-screen keyboard never sends; its return key adds a line as before. Input methods that take a hardware Enter and send it again as their own key, such as WeType, still send. The local `hardware-send` module listens on the view around the field, so it only acts while the field is focused and the draft can be sent. @@ -243,7 +244,7 @@ The shell is set by [ADR-0211](../../docs/adr/0211-the-mobile-app-has-two-tabs-w 1. **Folders**: each granted folder as a row (name, then "Can edit" or "Read only" and its path) with a × that removes access, then **Add folder**. Add folder turns the sheet into a folder browser of the computer: folders registered on the computer, the path as tappable crumbs, and the subfolders; **Authorize this folder** resolves the current folder and asks for a scoped confirmation ("Confirm authorization for {Bot}" with the resolved path) before granting it. 2. **Access**: a **Full access** switch. Turning it on first shows the danger confirmation in place, in an inset box with a red "I understand, enable" button and Cancel; once on, a warning callout and a **Scope of full access** group (Chat and new Assignments, or New Assignments only) follow. 3. **Auto approval template**: Off, Cautious and Get the work done (plus any other template on the Host), each with its one-line hint, the current one checked. - - The mic dictates with the phone's speech recognizer, and the words fill the field so the Human edits them before sending. Like Telegram's voice button: hold the mic to talk and release to stop; while holding, a lock appears above the mic: slide left to cancel or up to lock. A tap works like a lock: it keeps listening until **■** stops it or the **×** that replaces **+** cancels it and clears the text. Dictation stays in the same row, and its status shows as the field's placeholder. A refused microphone or speech permission shows a notice and leaves the mic in place. + - The mic dictates with the phone's speech recognizer, and the words fill the field so the Human edits them before sending. Like Telegram's voice button: hold the mic to talk and release to stop; while holding, a lock appears above the mic: slide left to cancel or up to lock. A tap works like a lock: it keeps listening until **■** stops it or the **×** that replaces **+** cancels it and clears the text. Dictation stays in the same row, and its status shows as the field's placeholder. A refused microphone or speech permission shows a notice and leaves the mic in place. On iOS the microphone/stop uses SF Symbols and Gesture Handler native touch recognition: movement and cancel/lock thresholds run on the UI thread, and only gesture boundaries cross to the speech controller. A system interruption or second finger cancels an unlocked hold; an explicitly locked recording stays active until Stop/Cancel. VoiceOver activation can start/stop without dragging. This is speech-to-text, not an audio-message sender; native recording capability and permission are still owned by the existing speech adapter. - A group Channel opens the same chat screen, with these differences: - The header is the same native header. Its title holds the group's photo (or a round `#` mark), the Channel name and its member count; it opens **Group info**. The full client button is the trailing item. - Like Slack and the Web Client, each Bot message sits in a row with a 32 pt avatar column on the left: a rounded square (`radius.sm`, no state ring). The first bubble of each Bot's group, and each Bot's streaming draft, carry the Bot's name above it in `meta`. The avatar rests beside the group's last bubble, its bottom edge level with that bubble's bottom edge, never with its reaction or delivery row. That last bubble's bottom-left corner uses the avatar's `radius.sm`, so the two bottom edges meet with matching corners. While the group scrolls past, the avatar sticks to the list's visible bottom edge (above the floating composer and its fade) and follows the group up until it reaches the group's first bubble. This runs on the UI thread from the list's scroll offset (Reanimated `useScrollOffset` and `measure`). A Bot missing from the roster gets a `surface` rounded square with its initial. A different Bot starts a new group even within 5 minutes. diff --git a/apps/mobile/src/ui/chat/composer-action.ios.tsx b/apps/mobile/src/ui/chat/composer-action.ios.tsx new file mode 100644 index 0000000000..63f88927a8 --- /dev/null +++ b/apps/mobile/src/ui/chat/composer-action.ios.tsx @@ -0,0 +1,17 @@ +import type { ReactNode } from 'react'; +import { useDesignTheme } from '../design/theme'; +import { composerControl } from './composer-chrome'; +import { NativeComposerButton } from './native-composer-button.ios'; +import type { ComposerSymbol } from './composer-symbol'; + +export function ComposerActionButton(props: { + label: string; + filled: boolean; + symbol: ComposerSymbol; + onPress(): void; + testID?: string; + children: ReactNode; +}) { + const theme = useDesignTheme(); + return ; +} diff --git a/apps/mobile/src/ui/chat/composer-action.tsx b/apps/mobile/src/ui/chat/composer-action.tsx new file mode 100644 index 0000000000..144f3821bd --- /dev/null +++ b/apps/mobile/src/ui/chat/composer-action.tsx @@ -0,0 +1,57 @@ +import type { ReactNode } from 'react'; +import { Pressable } from 'react-native'; +import { pressTint, useDesignTheme } from '../design/theme'; +import { radius, space } from '../design/tokens'; +import { composerControl, composerPrimary, rippleFeedback } from './composer-chrome'; +import type { ComposerSymbol } from './composer-symbol'; + +export function ComposerActionButton({ + label, + filled, + onPress, + testID, + children, +}: { + label: string; + filled: boolean; + onPress(): void; + testID?: string; + symbol: ComposerSymbol; + children: ReactNode; +}) { + const theme = useDesignTheme(); + const size = composerControl(theme); + return ( + ({ + width: size, + height: size, + borderRadius: radius.round, + alignItems: 'center', + justifyContent: 'center', + backgroundColor: filled + ? composerPrimary(theme).fill + : pressed && !rippleFeedback + ? pressTint(theme.color.fg) + : undefined, + opacity: filled && pressed && !rippleFeedback ? 0.7 : 1, + })} + > + {children} + + ); +} diff --git a/apps/mobile/src/ui/chat/composer-chrome.ios.tsx b/apps/mobile/src/ui/chat/composer-chrome.ios.tsx index 3d4a8218db..e1940532ad 100644 --- a/apps/mobile/src/ui/chat/composer-chrome.ios.tsx +++ b/apps/mobile/src/ui/chat/composer-chrome.ios.tsx @@ -1,10 +1,12 @@ import { Circle, Host, RoundedRectangle } from '@expo/ui/swift-ui'; import { accessibilityHidden, foregroundStyle, glassEffect } from '@expo/ui/swift-ui/modifiers'; +import { NativeComposerButton } from './native-composer-button.ios'; +import type { ComposerSymbol } from './composer-symbol'; import type { ReactNode } from 'react'; -import { Pressable, StyleSheet, View } from 'react-native'; +import { StyleSheet, View } from 'react-native'; -import { pressTint, useDesignTheme, type DesignTheme } from '../design/theme'; -import { edge, radius, space } from '../design/tokens'; +import { useDesignTheme, type DesignTheme } from '../design/theme'; +import { edge, space } from '../design/tokens'; import { useLiquidGlass } from '../primitives/drawer-material'; export function composerPrimary(theme: DesignTheme): { fill: string; ink: string } { @@ -67,42 +69,25 @@ export function ComposerCircleButton({ size, embedded = false, onPress, - children, + symbol, }: { label: string; testID?: string; size: number; embedded?: boolean; onPress(): void; + symbol: ComposerSymbol; children: ReactNode; }) { - const theme = useDesignTheme(); - const glass = useLiquidGlass(); return ( - ({ - width: size, - height: size, - borderRadius: radius.round, - alignItems: 'center', - justifyContent: 'center', - ...(embedded || glass - ? { backgroundColor: pressed ? pressTint(theme.color.fg) : 'transparent' } - : { - borderWidth: edge.hairline, - borderColor: theme.color.borderStrong, - backgroundColor: pressed ? theme.layer.raised.pressed : theme.color.surface, - }), - })} - > - {glass && !embedded ? : null} - {children} - + symbol={symbol} + /> ); } diff --git a/apps/mobile/src/ui/chat/composer-chrome.tsx b/apps/mobile/src/ui/chat/composer-chrome.tsx index f0888be160..242a0b71f3 100644 --- a/apps/mobile/src/ui/chat/composer-chrome.tsx +++ b/apps/mobile/src/ui/chat/composer-chrome.tsx @@ -1,3 +1,4 @@ +import type { ComposerSymbol } from './composer-symbol'; import type { ReactNode } from 'react'; import { Pressable, StyleSheet, View } from 'react-native'; @@ -32,6 +33,7 @@ export function ComposerCircleButton({ size: number; embedded?: boolean; onPress(): void; + symbol: ComposerSymbol; children: ReactNode; }) { const theme = useDesignTheme(); diff --git a/apps/mobile/src/ui/chat/composer-symbol.ts b/apps/mobile/src/ui/chat/composer-symbol.ts new file mode 100644 index 0000000000..e1b5bb887b --- /dev/null +++ b/apps/mobile/src/ui/chat/composer-symbol.ts @@ -0,0 +1 @@ +export type ComposerSymbol = 'plus' | 'xmark' | 'arrow.up' | 'chevron.up' | 'chevron.down'; diff --git a/apps/mobile/src/ui/chat/composer.tsx b/apps/mobile/src/ui/chat/composer.tsx index 62c77f9727..1f69ee4deb 100644 --- a/apps/mobile/src/ui/chat/composer.tsx +++ b/apps/mobile/src/ui/chat/composer.tsx @@ -3,7 +3,6 @@ import { useRef, useState, type ComponentRef, type ReactNode, type Ref } from 'r import { Image, Keyboard, - PanResponder, Platform, Pressable, ScrollView, @@ -21,7 +20,7 @@ import { useSafeAreaInsets } from 'react-native-safe-area-context'; import { usePreventRemove } from 'expo-router'; import type { PickSource, PickedFile } from '../../host/pick-files'; -import { pressTint, useDesignTheme } from '../design/theme'; +import { useDesignTheme } from '../design/theme'; import { edge, motion, radius, space } from '../design/tokens'; import { HardwareSend } from '../primitives/hardware-send'; import { Text } from '../primitives/text'; @@ -32,10 +31,8 @@ import { CloseIcon, FileIcon, LockIcon, - MicIcon, PhotoIcon, PlusIcon, - RecordingStopIcon, } from '../shell/icons'; import { ComposerCircleButton, @@ -43,9 +40,9 @@ import { composerBar, composerControl, composerPrimary, - micAlwaysFilled, - rippleFeedback, } from './composer-chrome'; +import { ComposerActionButton } from './composer-action'; +import { VoiceButton } from './voice-button'; import { DraftEditor } from './draft-editor'; import { haptic } from '../haptics'; import { t } from '../strings'; @@ -54,8 +51,6 @@ import { composerFieldHeight, composerStacked, dictating, - voiceDrag, - voiceRelease, type DictationStatus, } from './dictation'; @@ -65,56 +60,6 @@ const SOURCES: Array<{ source: PickSource; label: string; Icon: typeof PhotoIcon { source: 'file', label: t.attachFile, Icon: FileIcon }, ]; -function RoundButton({ - label, - filled, - onPress, - testID, - children, -}: { - label: string; - filled: boolean; - onPress(): void; - testID?: string; - children: ReactNode; -}) { - const theme = useDesignTheme(); - const size = composerControl(theme); - return ( - ({ - width: size, - height: size, - borderRadius: radius.round, - alignItems: 'center', - justifyContent: 'center', - backgroundColor: filled - ? composerPrimary(theme).fill - : pressed && !rippleFeedback - ? pressTint(theme.color.fg) - : undefined, - opacity: filled && pressed && !rippleFeedback ? 0.7 : 1, - })} - > - {children} - - ); -} - function AttachmentTile({ file, onRemove }: { file: PickedFile; onRemove(): void }) { const theme = useDesignTheme(); const tile = theme.touch + space.x5; @@ -177,90 +122,6 @@ function AttachmentTile({ file, onRemove }: { file: PickedFile; onRemove(): void ); } -function VoiceButton({ - active, - onStart, - onStop, - onCancel, - onHeld, -}: { - active: boolean; - onStart(): void; - onStop(): void; - onCancel(): void; - onHeld(held: boolean): void; -}) { - const theme = useDesignTheme(); - const [held, setHeld] = useState(false); - const size = composerControl(theme); - const latest = useRef({ active, onStart, onStop, onCancel, onHeld }); - latest.current = { active, onStart, onStop, onCancel, onHeld }; - const gesture = useRef({ at: 0, startedHere: false, done: false }); - const hold = (next: boolean) => { - setHeld(next); - latest.current.onHeld(next); - }; - const responder = useRef( - PanResponder.create({ - onStartShouldSetPanResponder: () => true, - onPanResponderTerminationRequest: () => false, - onPanResponderGrant: () => { - const startedHere = !latest.current.active; - gesture.current = { at: Date.now(), startedHere, done: false }; - if (!startedHere) return; - latest.current.onStart(); - hold(true); - }, - onPanResponderMove: (_event, { dx, dy }) => { - const current = gesture.current; - if (!current.startedHere || current.done) return; - const drag = voiceDrag(dx, dy); - if (drag === undefined) return; - current.done = true; - hold(false); - if (drag === 'cancel') latest.current.onCancel(); - }, - onPanResponderRelease: () => { - const current = gesture.current; - hold(false); - if (current.done) return; - if (!current.startedHere) latest.current.onStop(); - else if (voiceRelease(Date.now() - current.at) === 'stop') latest.current.onStop(); - }, - onPanResponderTerminate: () => hold(false), - }), - ).current; - const filled = micAlwaysFilled || active || held; - return ( - (latest.current.active ? onStop() : onStart())} - hitSlop={space.x2} - style={{ - width: size, - height: size, - borderRadius: radius.round, - alignItems: 'center', - justifyContent: 'center', - backgroundColor: filled ? composerPrimary(theme).fill : undefined, - }} - > - {active && !held ? ( - - ) : ( - - )} - - ); -} - function HoldDot({ size }: { size: number }) { const theme = useDesignTheme(); return ( @@ -452,6 +313,8 @@ export function Composer({ action === 'send' ? ( - { @@ -536,7 +401,7 @@ export function Composer({ }} > - + ) : ( setEditing(true)}> + setEditing(true)} + > - + ) : null; return ( ) : null; - if (glass) + if (Platform.OS === 'ios' || glass) return ( - - - {count} - + + + + + + {count} + + ); return ( + + + ); +} diff --git a/apps/mobile/src/ui/chat/voice-button.ios.tsx b/apps/mobile/src/ui/chat/voice-button.ios.tsx new file mode 100644 index 0000000000..6114c745a4 --- /dev/null +++ b/apps/mobile/src/ui/chat/voice-button.ios.tsx @@ -0,0 +1,141 @@ +import { Host } from '@expo/ui'; +import { Image } from '@expo/ui/swift-ui'; +import { useCallback, useMemo, useRef, useState } from 'react'; +import { View } from 'react-native'; +import { Gesture, GestureDetector } from 'react-native-gesture-handler'; +import { useAnimatedReaction, useSharedValue } from 'react-native-reanimated'; +import { scheduleOnRN } from 'react-native-worklets'; +import { useDesignTheme } from '../design/theme'; +import { radius } from '../design/tokens'; +import { t } from '../strings'; +import { composerControl, composerPrimary } from './composer-chrome'; +import { voiceGesture, voiceRelease } from './dictation'; + +export function VoiceButton(props: { + active: boolean; + onStart(): void; + onStop(): void; + onCancel(): void; + onHeld(held: boolean): void; +}) { + const theme = useDesignTheme(); + const size = composerControl(theme); + const [held, setHeld] = useState(false); + const latest = useRef(props); + latest.current = props; + const session = useRef({ startedHere: false, done: true }); + const origin = useSharedValue({ x: 0, y: 0, at: 0 }); + const decision = useSharedValue<'none' | 'cancel' | 'lock'>('none'); + const hold = useCallback((value: boolean) => { + setHeld(value); + latest.current.onHeld(value); + }, []); + const begin = useCallback(() => { + const startedHere = !latest.current.active; + session.current = { startedHere, done: false }; + if (startedHere) { + latest.current.onStart(); + hold(true); + } + }, [hold]); + const commitDrag = useCallback( + (choice: 'none' | 'cancel' | 'lock') => { + if (choice === 'none' || !session.current.startedHere || session.current.done) return; + session.current.done = true; + hold(false); + if (choice === 'cancel') latest.current.onCancel(); + }, + [hold], + ); + const finish = useCallback( + (success: boolean, choice: 'none' | 'cancel' | 'lock', duration: number) => { + commitDrag(choice); + hold(false); + if (session.current.done) return; + session.current.done = true; + if (!success) { + if (session.current.startedHere) latest.current.onCancel(); + } else if (!session.current.startedHere || voiceRelease(duration) === 'stop') { + latest.current.onStop(); + } + }, + [commitDrag, hold], + ); + useAnimatedReaction( + () => decision.get(), + (next, previous) => { + if (next !== 'none' && next !== previous) scheduleOnRN(commitDrag, next); + }, + [commitDrag], + ); + const gesture = useMemo( + () => + Gesture.Manual() + .onTouchesDown((event, manager) => { + if (event.numberOfTouches !== 1) { + manager.fail(); + return; + } + const touch = event.allTouches[0]; + if (!touch) return; + origin.set({ x: touch.absoluteX, y: touch.absoluteY, at: Date.now() }); + decision.set('none'); + manager.activate(); + }) + .onStart(() => scheduleOnRN(begin)) + .onTouchesMove((event) => { + if (decision.get() !== 'none') return; + const touch = event.allTouches[0]; + if (!touch) return; + const start = origin.get(); + if (touch.absoluteX - start.x <= -voiceGesture.cancelDrag) decision.set('cancel'); + else if (touch.absoluteY - start.y <= -voiceGesture.lockDrag) decision.set('lock'); + }) + .onTouchesUp((_event, manager) => manager.end()) + .onTouchesCancelled((_event, manager) => manager.fail()) + .onFinalize((_event, success) => { + scheduleOnRN(finish, success, decision.get(), Date.now() - origin.get().at); + }), + [begin, decision, finish, origin], + ); + const activate = () => + latest.current.active ? latest.current.onStop() : latest.current.onStart(); + return ( + + { + if (event.nativeEvent.actionName === 'activate') activate(); + }} + onAccessibilityTap={activate} + testID="composer-dictation" + style={{ + width: size, + height: size, + borderRadius: radius.round, + alignItems: 'center', + justifyContent: 'center', + backgroundColor: composerPrimary(theme).fill, + }} + > + + + + + + ); +} diff --git a/apps/mobile/src/ui/chat/voice-button.tsx b/apps/mobile/src/ui/chat/voice-button.tsx new file mode 100644 index 0000000000..1fee081b96 --- /dev/null +++ b/apps/mobile/src/ui/chat/voice-button.tsx @@ -0,0 +1,92 @@ +import { useRef, useState } from 'react'; +import { PanResponder, View } from 'react-native'; +import { useDesignTheme } from '../design/theme'; +import { radius, space } from '../design/tokens'; +import { MicIcon, RecordingStopIcon } from '../shell/icons'; +import { t } from '../strings'; +import { composerControl, composerPrimary, micAlwaysFilled } from './composer-chrome'; +import { voiceDrag, voiceRelease } from './dictation'; + +export function VoiceButton({ + active, + onStart, + onStop, + onCancel, + onHeld, +}: { + active: boolean; + onStart(): void; + onStop(): void; + onCancel(): void; + onHeld(held: boolean): void; +}) { + const theme = useDesignTheme(); + const [held, setHeld] = useState(false); + const size = composerControl(theme); + const latest = useRef({ active, onStart, onStop, onCancel, onHeld }); + latest.current = { active, onStart, onStop, onCancel, onHeld }; + const gesture = useRef({ at: 0, startedHere: false, done: false }); + const hold = (next: boolean) => { + setHeld(next); + latest.current.onHeld(next); + }; + const responder = useRef( + PanResponder.create({ + onStartShouldSetPanResponder: () => true, + onPanResponderTerminationRequest: () => false, + onPanResponderGrant: () => { + const startedHere = !latest.current.active; + gesture.current = { at: Date.now(), startedHere, done: false }; + if (!startedHere) return; + latest.current.onStart(); + hold(true); + }, + onPanResponderMove: (_event, { dx, dy }) => { + const current = gesture.current; + if (!current.startedHere || current.done) return; + const drag = voiceDrag(dx, dy); + if (drag === undefined) return; + current.done = true; + hold(false); + if (drag === 'cancel') latest.current.onCancel(); + }, + onPanResponderRelease: () => { + const current = gesture.current; + hold(false); + if (current.done) return; + if (!current.startedHere) latest.current.onStop(); + else if (voiceRelease(Date.now() - current.at) === 'stop') latest.current.onStop(); + }, + onPanResponderTerminate: () => hold(false), + }), + ).current; + const filled = micAlwaysFilled || active || held; + return ( + (latest.current.active ? onStop() : onStart())} + hitSlop={space.x2} + style={{ + width: size, + height: size, + borderRadius: radius.round, + alignItems: 'center', + justifyContent: 'center', + backgroundColor: filled ? composerPrimary(theme).fill : undefined, + }} + > + {active && !held ? ( + + ) : ( + + )} + + ); +} diff --git a/apps/mobile/test/composer-chrome-ios.test.tsx b/apps/mobile/test/composer-chrome-ios.test.tsx index a01c0b5e66..53802e6c29 100644 --- a/apps/mobile/test/composer-chrome-ios.test.tsx +++ b/apps/mobile/test/composer-chrome-ios.test.tsx @@ -8,13 +8,46 @@ vi.mock('react-native', async () => import('./react-native-shim')); vi.mock('../src/ui/primitives/drawer-material', () => ({ useLiquidGlass: () => native.glass })); vi.mock('@expo/ui/swift-ui/modifiers', () => ({ foregroundStyle: () => ({}), + accessibilityLabel: (label: string) => ({ label }), + buttonStyle: (style: string) => ({ buttonStyle: style }), + buttonBorderShape: () => ({}), + contentShape: () => ({}), + background: () => ({}), + frame: () => ({}), + shapes: { circle: () => ({}), rectangle: () => ({}) }, accessibilityHidden: () => ({}), glassEffect: (params: unknown) => { native.effects.push(params); return {}; }, })); +vi.mock('@expo/ui', () => ({ + Host: ({ children }: { children: ReactNode }) => createElement('div', null, children), +})); vi.mock('@expo/ui/swift-ui', () => ({ + Button: ({ + children, + onPress, + testID, + modifiers, + }: { + children: ReactNode; + onPress(): void; + testID?: string; + modifiers: Array<{ label?: string; buttonStyle?: string }>; + }) => + createElement( + 'button', + { + 'aria-label': modifiers.find((m) => m.label)?.label, + 'data-testid': testID, + 'data-button-style': modifiers.find((m) => m.buttonStyle)?.buttonStyle, + onClick: onPress, + }, + children, + ), + Image: ({ systemName }: { systemName: string }) => + createElement('span', { 'data-symbol': systemName }), Host: ({ children }: { children: ReactNode }) => createElement('div', null, children), Circle: () => createElement('div', { 'data-material': 'circle' }), RoundedRectangle: ({ cornerRadius }: { cornerRadius: number }) => @@ -40,7 +73,13 @@ afterEach(() => act(() => root.unmount())); function Bar() { return ( <> - undefined}> + undefined} + > + @@ -71,3 +110,49 @@ it('keeps the same layout on opaque fills without glass', async () => { expect(container.querySelector('[data-material]')).toBeNull(); expect(native.effects).toEqual([]); }); + +const { ComposerActionButton } = await import('../src/ui/chat/composer-action.ios'); + +it('dispatches a native Send action once with its accessible label and stable identifier', async () => { + const send = vi.fn(); + await act(async () => + root.render( + + fallback + , + ), + ); + const button = container.querySelector('[data-testid="composer-send"]')!; + expect(button.getAttribute('aria-label')).toBe('Send'); + expect(button.dataset.buttonStyle).toBe('plain'); + expect(button.querySelector('[data-symbol="arrow.up"]')).not.toBeNull(); + await act(async () => button.click()); + expect(send).toHaveBeenCalledOnce(); +}); + +it.each([true, false])('keeps the return action native with glass available=%s', async (glass) => { + native.glass = glass; + const jump = vi.fn(); + await act(async () => + root.render( + + fallback + , + ), + ); + const button = container.querySelector('button')!; + expect(button.dataset.buttonStyle).toBe(glass ? 'glass' : 'plain'); + await act(async () => button.click()); + expect(jump).toHaveBeenCalledOnce(); +}); diff --git a/apps/mobile/test/voice-button-ios.test.tsx b/apps/mobile/test/voice-button-ios.test.tsx new file mode 100644 index 0000000000..ef7cfd5b14 --- /dev/null +++ b/apps/mobile/test/voice-button-ios.test.tsx @@ -0,0 +1,227 @@ +// @vitest-environment jsdom +import { act, createElement, useState, type ReactNode } from 'react'; +import { createRoot } from 'react-dom/client'; +import { afterEach, beforeEach, expect, it, vi } from 'vitest'; + +const native = vi.hoisted(() => ({ + callbacks: {} as Record void>, + reactions: new Set<() => void>(), +})); +vi.mock('react-native', async () => { + const shim = await import('./react-native-shim'); + return { + ...shim, + View: (props: Record) => + props['onAccessibilityTap'] + ? createElement( + 'button', + { + 'aria-label': props['accessibilityLabel'], + onClick: props['onAccessibilityTap'] as () => void, + }, + props['children'] as ReactNode, + ) + : shim.View(props), + }; +}); +vi.mock('@expo/ui', () => ({ Host: ({ children }: { children: ReactNode }) => children })); +vi.mock('@expo/ui/swift-ui', () => ({ + Image: ({ systemName }: { systemName: string }) => + createElement('span', { 'data-symbol': systemName }), +})); +vi.mock('../src/ui/chat/composer-chrome', () => ({ + composerControl: () => 44, + composerPrimary: () => ({ fill: 'black', ink: 'white' }), +})); +vi.mock('react-native-worklets', () => ({ + scheduleOnRN: (fn: (...args: any[]) => void, ...args: any[]) => fn(...args), +})); +vi.mock('react-native-reanimated', async () => { + const { useEffect, useState } = await import('react'); + return { + useSharedValue: (initial: unknown) => + useState(() => { + let current = initial; + return { + get: () => current, + set: (next: unknown) => { + current = next; + native.reactions.forEach((fn) => fn()); + }, + }; + })[0], + useAnimatedReaction: (read: () => unknown, react: (next: unknown, previous: unknown) => void) => + useEffect(() => { + let previous = read(); + const reaction = () => { + const next = read(); + const old = previous; + previous = next; + react(next, old); + }; + native.reactions.add(reaction); + return () => { + native.reactions.delete(reaction); + }; + }, [read, react]), + }; +}); +vi.mock('react-native-gesture-handler', () => ({ + GestureDetector: ({ children }: { children: ReactNode }) => children, + Gesture: { + Manual: () => { + const chain: Record void) => unknown> = {}; + for (const name of [ + 'onTouchesDown', + 'onStart', + 'onTouchesMove', + 'onTouchesUp', + 'onTouchesCancelled', + 'onFinalize', + ]) { + chain[name] = (fn) => { + native.callbacks[name] = fn; + return chain; + }; + } + return chain; + }, + }, +})); +const { VoiceButton } = await import('../src/ui/chat/voice-button.ios'); +const { t } = await import('../src/ui/strings'); +Object.assign(globalThis, { IS_REACT_ACT_ENVIRONMENT: true }); + +const start = vi.fn(); +const stop = vi.fn(); +const cancel = vi.fn(); +const held = vi.fn(); +let host: HTMLElement; +let root: ReturnType; +let now = 1000; +function Harness() { + const [active, setActive] = useState(false); + return ( + { + start(); + setActive(true); + }} + onStop={() => { + stop(); + setActive(false); + }} + onCancel={() => { + cancel(); + setActive(false); + }} + onHeld={held} + /> + ); +} +const manager = { + activate: () => native.callbacks['onStart']!(), + end: () => native.callbacks['onFinalize']!({}, true), + fail: () => native.callbacks['onFinalize']!({}, false), +}; +const touch = (x: number, y: number, numberOfTouches = 1) => ({ + numberOfTouches, + allTouches: [{ absoluteX: x, absoluteY: y }], +}); +const down = async () => + act(async () => native.callbacks['onTouchesDown']!(touch(200, 400), manager)); +const move = async (x: number, y: number) => + act(async () => native.callbacks['onTouchesMove']!(touch(x, y), manager)); +const up = async (duration: number) => { + now += duration; + await act(async () => native.callbacks['onTouchesUp']!({}, manager)); +}; + +beforeEach(async () => { + vi.clearAllMocks(); + native.reactions.clear(); + now = 1000; + vi.spyOn(Date, 'now').mockImplementation(() => now); + host = document.createElement('div'); + root = createRoot(host); + await act(async () => root.render()); +}); +afterEach(() => { + act(() => root.unmount()); + vi.restoreAllMocks(); +}); + +it('quick tap locks listening and the next tap stops once', async () => { + await down(); + await up(100); + expect(start).toHaveBeenCalledOnce(); + expect(stop).not.toHaveBeenCalled(); + expect(host.querySelector('[data-symbol="stop.fill"]')).not.toBeNull(); + await down(); + await up(100); + expect(stop).toHaveBeenCalledOnce(); +}); +it('holding and releasing stops once without discarding speech', async () => { + await down(); + await up(300); + expect(start).toHaveBeenCalledOnce(); + expect(stop).toHaveBeenCalledOnce(); + expect(cancel).not.toHaveBeenCalled(); + expect(held.mock.calls).toEqual([[true], [false]]); +}); +it('left drag cancels immediately and release cannot stop or start again', async () => { + await down(); + await move(120, 400); + await move(100, 300); + await up(900); + expect(cancel).toHaveBeenCalledOnce(); + expect(stop).not.toHaveBeenCalled(); + expect(start).toHaveBeenCalledOnce(); +}); +it('upward drag locks immediately and release keeps listening', async () => { + await down(); + await move(200, 340); + await move(100, 340); + await up(900); + expect(cancel).not.toHaveBeenCalled(); + expect(stop).not.toHaveBeenCalled(); + expect(host.querySelector('[data-symbol="stop.fill"]')).not.toBeNull(); +}); +it('movement below the cancel and lock thresholds keeps the held gesture', async () => { + await down(); + await move(121, 341); + expect(held.mock.calls).toEqual([[true]]); + await up(600); + expect(stop).toHaveBeenCalledOnce(); +}); +it('system cancellation cancels an unlocked hold exactly once', async () => { + await down(); + await act(async () => native.callbacks['onTouchesCancelled']!({}, manager)); + await up(900); + expect(cancel).toHaveBeenCalledOnce(); + expect(stop).not.toHaveBeenCalled(); +}); +it('a second finger interrupts an unlocked hold rather than restarting it', async () => { + await down(); + await act(async () => native.callbacks['onTouchesDown']!(touch(220, 400, 2), manager)); + expect(start).toHaveBeenCalledOnce(); + expect(cancel).toHaveBeenCalledOnce(); +}); +it('system cancellation after explicit lock leaves the recording active', async () => { + await down(); + await move(200, 340); + await act(async () => native.callbacks['onTouchesCancelled']!({}, manager)); + expect(cancel).not.toHaveBeenCalled(); + expect(stop).not.toHaveBeenCalled(); +}); +it('accessibility activation starts and stops without a drag', async () => { + await act(async () => + host.querySelector(`[aria-label="${t.dictate}"]`)!.click(), + ); + expect(start).toHaveBeenCalledOnce(); + await act(async () => + host.querySelector(`[aria-label="${t.stopDictation}"]`)!.click(), + ); + expect(stop).toHaveBeenCalledOnce(); +}); diff --git a/changes/2575-ios-native-composer-controls.md b/changes/2575-ios-native-composer-controls.md new file mode 100644 index 0000000000..52958b8146 --- /dev/null +++ b/changes/2575-ios-native-composer-controls.md @@ -0,0 +1,11 @@ +--- +section: Changed +--- + +## en + +- iOS chat actions now use native buttons and system symbols; dictation keeps its tap, hold, cancel and lock gestures with native touch recognition and cancels interrupted holds ([#2575](https://github.com/BotHarness/DeepSeekBot/issues/2575)). + +## zh + +- iOS 聊天操作改用原生按钮和系统图标；听写保留点按、长按、取消和锁定操作，使用原生触摸识别，并在未锁定的长按被打断时取消听写（[#2575](https://github.com/BotHarness/DeepSeekBot/issues/2575)）。