Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ option(BUILD_PLUGIN_OAK_CAMERA "Build OAK camera plugin (requires vcpkg for Dept
option(BUILD_PLUGIN_NOITOM_MOCAP "Build Noitom mocap plugin (downloads MocapApi SDK)" OFF)
option(BUILD_PLUGIN_OGLO "Build OGLO tactile glove plugin (BLE, Linux only; fetches SimpleBLE + nlohmann/json)" OFF)
option(BUILD_PLUGIN_WUJI_GLOVE "Build Wuji glove plugin (requires the wuji_sdk C SDK)" OFF)
option(BUILD_PLUGIN_OMNIGLOVE "Build AgiLink OmniGlove plugin (fetches a pinned SDK)" OFF)
option(BUILD_PLUGIN_SHARPA_AVATAR "Build Sharpa AvatarExo Glove plugin (Linux; skips if the Avatar SDK is missing)" OFF)
option(BUILD_EXAMPLES "Build examples" ON)
option(BUILD_EXAMPLE_TELEOP_ROS2 "Build only the teleop_ros2 ROS 2 reference integration (e.g. for Docker)" OFF)
Expand Down
20 changes: 20 additions & 0 deletions docs/source/_data/devices.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,26 @@ devices:
url: https://wuji.tech/en/
- email: sales@wuji.tech

- id: agilink-omniglove
name: AgiLink OmniGlove
url: https://www.agilink-ai.com
group: peripheral
modes: High-fidelity hand tracking
since: main
details:
setup:
- label: OmniGlove docs
doc: /device/hand_tracking/omniglove
- label: OmniGlove Plugin
url: https://github.com/NVIDIA/IsaacCapture/tree/main/src/plugins/omniglove
requirements:
- Linux, x86_64 or aarch64
- AgiLink OmniGlove SDK
- USB glove connection and host udev rule
acquire:
- label: AgiLink OmniGlove SDK
url: https://github.com/AgibotTech/agilink_omniglove_sdk

- id: noitom
name: Noitom Robotics Motion Capture
url: https://www.noitomrobotics.com
Expand Down
1 change: 1 addition & 0 deletions docs/source/device/hand_tracking/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ and hand exoskeletons. See :doc:`/device/trackers/index` for the common tracking
:titlesonly:

manus
omniglove
wuji_glove
sharpa_avatar
haptikos
189 changes: 189 additions & 0 deletions docs/source/device/hand_tracking/omniglove.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
.. SPDX-FileCopyrightText: Copyright (c) 2026 Agibot Co., Ltd.
.. SPDX-License-Identifier: Apache-2.0

AgiLink OmniGlove
=================

OmniGlove is a USB glove that measures the operator's finger skeleton. Its
Linux C++ plugin wraps the AgiLink OmniGlove SDK and publishes standard OpenXR
hand tracking for Isaac Capture. Robot-specific joint mapping runs in the
host's hand retargeter.

.. contents:: On this page
:local:
:depth: 2

Hardware and prerequisites
--------------------------

- Linux x86_64 or aarch64 with the system libusb runtime (``libusb-1.0-0``
on Ubuntu); the current hardware validation used x86_64.
- The Isaac Capture source-build dependencies and the
`AgiLink OmniGlove SDK <https://github.com/AgibotTech/agilink_omniglove_sdk>`_.
- One left and one right OmniGlove for two-hand CTS, connected over USB.
USB identifiers are ``cafe:4001`` (left) and ``cafe:4002`` (right).
- Host USB permission, CloudXR and a supported XR headset.
- Optical wrist tracking, or controllers rigidly mounted to the gloves with
separately measured left/right controller-aim-to-wrist transforms.

Data flow
---------

The SDK supplies ``OMNIGLOVE_STREAM_POST_DATA`` in DIRECT mode at a nominal
125 Hz: a wrist-relative skeleton with 21 landmarks. The plugin converts SDK
joint axes, combines the skeleton with a world-space wrist pose from
``plugin_utils::WristPoseSource``, and pushes each hand through its own
``plugin_utils::HandInjector``. The host reads the injected hands using the
standard ``HandsSource``.

.. code-block:: text

OmniGlove SDK (finger skeleton) ─┐
├─► OmniGlove plugin ─► OpenXR ─► HandsSource
Optical hand / controller pose ─┘

Installation
------------

Install the USB rule on the host:

.. code-block:: bash

./src/plugins/omniglove/install_udev_rules.sh

CMake retrieves the pinned SDK into the build directory and validates its
release checksums. The target architecture selects the SDK package. The SDK
runtime library and its license notices are installed with the plugin. libusb
is supplied by the operating system.

.. code-block:: bash

cmake -S . -B build -DBUILD_PLUGIN_OMNIGLOVE=ON
cmake --build build
cmake --install build --component omniglove

Local SDK checkouts use CMake's standard
``-DFETCHCONTENT_SOURCE_DIR_OMNIGLOVE_SDK=/path/to/agilink_omniglove_sdk`` override.

Running
-------

Start CloudXR and connect the headset:

.. code-block:: bash

python -m isaaccapture.cloudxr.service start

Source the runtime environment, select the wrist source
as described below, and launch the installed plugin:

.. code-block:: bash

source ~/.cloudxr/run/cloudxr.env
./install/plugins/omniglove/omniglove_plugin

Wrist source and calibration
----------------------------

``OMNIGLOVE_WRIST_SOURCE`` selects ``hand_tracking`` (default), ``controller``
or ``auto``. Optical mode needs the headset to track the wrist while the glove
is worn. Controller mode uses controller aim poses and per-hand installation
offsets. Auto mode allows both sources; validate each source and the transition
before using it.

Rigidly attach each controller using its final mount. From simultaneous
controller aim and trusted wrist poses, calculate:

.. code-block:: text

T_aim_to_wrist = inverse(T_world_aim) * T_world_wrist
T_world_joint = T_world_aim * T_aim_to_wrist * T_wrist_joint

Use multiple poses to estimate the fixed transform, then verify it with new
positions and wrist rotations. Keep the controller mounted rigidly; changes
to its installation require recalibration. Each transform is a translation
in metres followed by a unit quaternion in ``(x, y, z, w)`` order:

.. code-block:: bash

export OMNIGLOVE_WRIST_SOURCE=controller
export OMNIGLOVE_AIM_TO_WRIST_LEFT="px,py,pz,qx,qy,qz,qw"
export OMNIGLOVE_AIM_TO_WRIST_RIGHT="px,py,pz,qx,qy,qz,qw"

These are format placeholders; replace them with measured values for the
current mount. Controller and Auto mode reject missing transforms. Unknown
modes and invalid offsets are errors. Optical mode does not read the offsets.

Published collections and joint contract
----------------------------------------

``plugin.yaml`` declares ``/hand/left`` and ``/hand/right``, both with
``type: hand_tracker``. The plugin publishes ``XR_EXT_hand_tracking`` joints;
the host records them as ``core.HandPoseRecord`` from ``hand.fbs``.

- Each frame has ``XrHandJointLocationEXT[26]`` slots in ``XrHandJointEXT``
order. The wrist and four joints per finger are valid: 21 required joints.
- Palm and the four non-thumb metacarpals remain invalid because the SDK
does not supply them. No neighbouring joint is copied into an absent slot.
- Valid joints have both ``POSITION_VALID`` and ``ORIENTATION_VALID`` set.
Tracking bits additionally follow ``wrist.tracked``. A valid but untracked
wrist is still published with the tracking bits cleared.
- Positions are metres and orientations are unit ``(x, y, z, w)``
quaternions, in the base space passed to ``HandInjector``.
- OpenXR joint ``+Z`` points away from the fingertip and ``+Y`` toward the
back of each digit. SDK digit orientations receive a local Y half-turn,
followed for the thumb by a local Z rotation of -90 degrees on the left
and +90 degrees on the right to align the flexion hinge with local +X.
The right wrist basis is converted separately. Wrist and finger poses
are composed with the same world-space wrist transform.
- SDK monotonic sample times are converted into runtime ``XrTime`` through
``XrTimeConverter``; the wrist is queried at that same time.
- Unmapped, unconverged IK, invalid bone data, non-finite poses, zero
quaternions and stale frames are withheld. A sample is pushed only once.
Each hand is reported inactive when fresh, usable glove data or a valid
wrist pose is unavailable. Tracking resumes when both are available.

The plugin publishes the operator's skeleton. It emits no robot joint
commands or hand ``JointStateOutput`` data and defines no vendor hand schema.

Device status and CTS
---------------------

The shared device status publisher reports left and right hand availability
under the configured plugin root. A glove that stops streaming reports a
disconnected hand. A glove still streaming without a usable skeleton or wrist
reports a degraded hand. A transient rejected sample retains the last usable
pose until it becomes stale. Stream configuration or startup failures and
incompatible SDK sample contracts stop the plugin with an error.

Follow :doc:`/conformance/setup`, :doc:`/conformance/hand` and
:doc:`/conformance/submit` for recording, review and submission.

Troubleshooting
---------------

- **No USB glove:** install the udev rule on the host, reconnect the device
and check user access to ``cafe:4001`` / ``cafe:4002``.
- **No world-space hand:** connect the headset, load ``cloudxr.env`` and
check that the selected wrist source is available.
- **Misplaced or rotated hand:** verify the final controller mount and its
aim-to-wrist transform, multiplication order, units and quaternion order.
- **Inactive hand:** the plugin logs ``hand inactive`` when SDK data or the
wrist becomes invalid. On recovery, ``hand active`` includes the SDK
connection generation and source sequence. Do not expect a frozen pose.
- **Persistent IK rejection:** check glove fit and SDK/firmware state;
invalid skeleton frames are intentionally withheld.
- **Header/runtime ABI mismatch:** rebuild against the selected SDK and
verify that the installed executable loads its accompanying runtime library.

License
-------

The plugin source is Apache-2.0. The AgiLink OmniGlove SDK is licensed under
MulanPSL-2.0; its shared library is redistributed with the plugin, together with
``LICENSE.omniglove-sdk`` and ``THIRD_PARTY_NOTICES.omniglove-sdk.md``.

The SDK depends on libusb, licensed under LGPL-2.1-or-later. This plugin uses
the operating system's libusb and does not redistribute the SDK's bundled
libusb binary. The SDK notices describe the vendor's complete release;
only the SDK runtime library is copied into this plugin's install tree.
4 changes: 3 additions & 1 deletion docs/source/device/hand_tracking/wuji_glove.rst
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,9 @@ controller offsets when needed:
export WUJI_GLOVE_AIM_TO_WRIST_RIGHT="px,py,pz,qx,qy,qz,qw"

Position values are in meters and quaternion values use ``(x, y, z, w)``.
The built-in offsets assume mirrored left/right mounts.
The built-in offsets assume mirrored left/right mounts. Unknown wrist-source
modes and malformed explicit offsets stop the plugin with an error; unset
offsets use the built-in values.

Troubleshooting
---------------
Expand Down
3 changes: 3 additions & 0 deletions src/plugins/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ endif()
if(BUILD_PLUGIN_WUJI_GLOVE)
add_subdirectory(wuji_glove)
endif()
if(BUILD_PLUGIN_OMNIGLOVE)
add_subdirectory(omniglove)
endif()
# Gated: Linux + Avatar SDK under /opt/avatar-sdk. Missing SDK skips the plugin.
if(BUILD_PLUGIN_SHARPA_AVATAR)
add_subdirectory(sharpa_avatar)
Expand Down
6 changes: 6 additions & 0 deletions src/plugins/omniglove/60-omniglove-usb.rules
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 Agibot Co., Ltd.
# SPDX-License-Identifier: Apache-2.0

# Grant access to USB devices with the listed runtime product IDs.
ACTION=="add|change", SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ATTR{idVendor}=="cafe", ATTR{idProduct}=="4001", GROUP="plugdev", MODE="0660", TAG+="uaccess"
ACTION=="add|change", SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", ATTR{idVendor}=="cafe", ATTR{idProduct}=="4002", GROUP="plugdev", MODE="0660", TAG+="uaccess"
42 changes: 42 additions & 0 deletions src/plugins/omniglove/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 Agibot Co., Ltd.
# SPDX-License-Identifier: Apache-2.0

include(GNUInstallDirs)
include("${CMAKE_CURRENT_LIST_DIR}/sdk.cmake")

add_subdirectory(core)
add_subdirectory(app)

# Install the verified target artifact; SDK linker aliases may not be in its manifest.
get_target_property(_omniglove_sdk_soname OmnigloveSDK::api IMPORTED_SONAME)
install(FILES "${OMNIGLOVE_SDK_LIBRARY}"
DESTINATION plugins/omniglove
RENAME "${_omniglove_sdk_soname}"
COMPONENT omniglove
)

install(FILES "${OMNIGLOVE_SDK_RELEASE_ROOT}/LICENSE"
DESTINATION plugins/omniglove
RENAME LICENSE.omniglove-sdk
COMPONENT omniglove
)
install(FILES "${OMNIGLOVE_SDK_RELEASE_ROOT}/THIRD_PARTY_NOTICES.md"
DESTINATION plugins/omniglove
RENAME THIRD_PARTY_NOTICES.omniglove-sdk.md
COMPONENT omniglove
)
install(FILES
"${OMNIGLOVE_SDK_RELEASE_ROOT}/licenses/fmt-LICENSE"
"${OMNIGLOVE_SDK_RELEASE_ROOT}/licenses/wjwwood-serial-LICENSE"
DESTINATION plugins/omniglove/licenses
COMPONENT omniglove
)

install(FILES plugin.yaml README.md 60-omniglove-usb.rules
DESTINATION plugins/omniglove
COMPONENT omniglove
)
install(PROGRAMS install_udev_rules.sh
DESTINATION plugins/omniglove
COMPONENT omniglove
)
12 changes: 12 additions & 0 deletions src/plugins/omniglove/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!--
SPDX-FileCopyrightText: Copyright (c) 2026 Agibot Co., Ltd.
SPDX-License-Identifier: Apache-2.0
-->

# AgiLink OmniGlove

The Linux plugin publishes OmniGlove finger tracking as OpenXR hands for
Isaac Capture.

Installation, wrist calibration, running, licensing and troubleshooting:
[AgiLink OmniGlove documentation](https://nvidia.github.io/IsaacCapture/main/device/hand_tracking/omniglove.html).
15 changes: 15 additions & 0 deletions src/plugins/omniglove/app/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 Agibot Co., Ltd.
# SPDX-License-Identifier: Apache-2.0

add_executable(omniglove_plugin main.cpp)

target_link_libraries(omniglove_plugin PRIVATE isaac::omniglove_plugin)

set_target_properties(omniglove_plugin PROPERTIES
INSTALL_RPATH "$ORIGIN"
)

install(TARGETS omniglove_plugin
RUNTIME DESTINATION plugins/omniglove
COMPONENT omniglove
)
66 changes: 66 additions & 0 deletions src/plugins/omniglove/app/main.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 Agibot Co., Ltd.
// SPDX-License-Identifier: Apache-2.0

#include <log_bridge/logger.hpp>
#include <omniglove_plugin/omniglove_plugin.hpp>

#include <atomic>
#include <chrono>
#include <csignal>
#include <exception>
#include <string>
#include <thread>

namespace
{

static_assert(ATOMIC_BOOL_LOCK_FREE == 2, "lock-free atomic bool is required for signal safety");

std::atomic<bool> stop_requested{ false };

void signal_handler(int signal)
{
if (signal == SIGINT || signal == SIGTERM)
{
stop_requested.store(true, std::memory_order_relaxed);
}
}

} // namespace

int main(int argc, char** argv)
try
{
std::string plugin_root_id = "omniglove";
for (int index = 1; index < argc; ++index)
{
const std::string argument = argv[index];
constexpr const char* prefix = "--plugin-root-id=";
if (argument.rfind(prefix, 0) == 0)
{
plugin_root_id = argument.substr(std::char_traits<char>::length(prefix));
}
}

std::signal(SIGINT, signal_handler);
std::signal(SIGTERM, signal_handler);

plugins::omniglove::OmniglovePlugin plugin(plugin_root_id);
while (!stop_requested.load(std::memory_order_relaxed))
{
plugin.update();
constexpr auto kFramePeriod = plugins::omniglove::OmniglovePlugin::kSourceInterval;
std::this_thread::sleep_for(kFramePeriod);
}
return 0;
}
catch (const std::exception& error)
{
isaaccapture::Logger::get("isaaccapture.plugins.omniglove.main")->error("{}: {}", argv[0], error.what());
return 1;
}
catch (...)
{
isaaccapture::Logger::get("isaaccapture.plugins.omniglove.main")->error("{}: unknown error", argv[0]);
return 1;
}
Loading