---
title: Teleop Dataset Recording
description: Record demonstration datasets by teleoperating a simulated robot with a physical leader arm.
---

SO101-Nexus ships with a Gradio-based teleop recorder.
You drive a simulated robot with a physical SO-100 or SO-101 leader arm,
and demonstrations are saved as LeRobot v3 datasets ready to push to the
Hugging Face Hub.

## Quickstart with uvx

If you have [uv](https://docs.astral.sh/uv/) installed, you can launch the recorder without setting up a venv. `uvx` resolves the package and the `teleop` extra into an ephemeral environment:

```bash
uvx --from "so101-nexus[teleop]" so101-nexus teleop \
    --leader-port /dev/ttyACM0
```

This is the fastest way to try teleop end to end. The rest of this guide assumes you have run a regular install for repeat use, but the `uvx` command above is enough for a one-off session.

## Installation

Teleop deps (`lerobot[feetech]`, `gradio`, `plotly`, `opencv-python`) live
in the `teleop` extra on `so101-nexus`. Install them with:

```bash
uv sync --extra teleop
```

With pip:

```bash
pip install "so101-nexus[teleop]"
```

## Hardware

- A physical SO-100 or SO-101 leader arm connected via USB.
- A Linux host with access to the leader arm's serial port.

## Finding the serial port

Make the device writable if needed, then use the LeRobot port discovery
tool:

```bash
sudo chmod 666 /dev/ttyACM0
lerobot-find-port
```

The Configure step shows a leader-port status banner up front: it reports
whether the device exists and is readable and writable before you fill in
the rest of the form. Click `Recheck Port` after fixing permissions to
refresh it, and `Initialize Session` is blocked with the same guidance while
the port is not ready. If the recorder still cannot open the arm at connect
time, the Initialize step stays open and shows the connection error, with
exact recovery commands when the port is missing or blocked by permissions.
Fix the port in another terminal, then click `Retry Initialization` in the
same browser session.

## Launching the recorder

The `teleop` subcommand launches the recorder:

```bash
uv run so101-nexus teleop --leader-port /dev/ttyACM0
```

A Gradio UI opens in your browser. Most settings live in the UI; the CLI
only takes connection-level options.

If initialization fails because `/dev/ttyACM*` is missing or not writable,
teleop now keeps the session alive and prints targeted recovery guidance
instead of forcing a browser restart.

### CLI arguments

| Argument | Type | Default | Description |
| --- | --- | --- | --- |
| `--leader-port` | `str` | `/dev/ttyACM0` | Serial port of the leader arm |
| `--leader-id` | `str` | `so101_leader` | Device identifier for the leader arm |
| `--wrist-roll-offset-deg` | `float` | `-90.0` | Wrist roll offset in degrees |
| `--env-config-profile` | `str \| None` | `None` | JSON or TOML profile with environment config overrides |
| `--env-config-factory` | `str \| None` | `None` | Python `module:function` that returns a config or `gym.make` kwargs |
| `--env-module` | repeatable `str` | `[]` | Import a module that registers custom Gymnasium environments |
| `--extra-env-id` | repeatable `str` | `[]` | Add a custom environment ID to the UI dropdown |

## Cameras and recorded fields

The recorder captures **both** the wrist camera and the overhead camera on
every step, so you can decide after the fact which views end up in your
dataset without re-recording.

Under **Advanced Settings, Dataset fields**, a checkbox group selects which
fields are persisted:

- `observation.state` always saved (disabled checkbox)
- `action` always saved (disabled checkbox)
- `task` always saved by LeRobot v3 metadata
- `reward` always saved from the same environment step as the frame's action
- `success` always saved (episode success flag)
- `done` always saved (episode termination flag)
- `observation.environment_state` optional, on by default (privileged low-dim state: TCP/object pose, grasp, and offsets)
- `observation.images.wrist` optional, on by default
- `observation.images.overhead` optional, on by default

Deselecting an optional field removes it from both the declared dataset
feature schema and every recorded frame. `observation.state`, `action`, `reward`,
`success`, `done`, and `task` stay forced on because LeRobot datasets require them for policy
training and task indexing.

### Camera resolution

Separate width and height sliders for the wrist and overhead cameras
control the recording resolution. Both cameras are wired into the
environment's observation config automatically, even when the task's
default config only declares one.

## Environment customization

Teleop customization uses the same typed config objects as the regular
Gymnasium API. The recorder starts from the selected environment's default
config, applies UI/profile/factory overrides, then wires the wrist and
overhead camera observations required for recording.

The **Advanced Settings** section exposes common options:

- `Apply Environment Customization`: enables the UI environment overrides
  below. Leave it off to use the environment's default config unchanged.
- `Pick Object Pool`: built-in cube and YCB objects for pick tasks
- `Pick Distractors`: number of distractors to place in pick scenes
- `Ground Colors` and `Robot Colors`: sampled per reset when multiple
  colors are selected
- `Spawn Min Radius`, `Spawn Max Radius`, and `Spawn Angle Half Range (deg)`:
  spawn-region controls shared by supported tasks
- `Reset Settle Frames`: no-op frames advanced after reset before the first
  teleop observation and recorded frame. The default is 5.
- `Pick-and-Place Cube Colors` and `Pick-and-Place Target Colors`:
  sampled cube/target colors for pick-and-place tasks
- `Max Steps`: overrides the episode's step cap (default 1024)
- `Success Hold (s)`: seconds to keep recording after the env reports success before auto-stopping (default 0.5 s, 0 = stop immediately)

Options that do not apply to the selected environment are ignored after
customization is enabled. For example, pick-object settings apply to
`PickConfig` environments, while pick-and-place color settings apply to
`PickAndPlaceConfig` environments.

`Reset Settle Frames` is applied to the environment config before camera
observations are wired, so the first saved demonstration frame is captured
after reset settling.

### Config profiles

Use a profile when you want repeatable recording setups. Profiles may be
JSON or TOML and are merged in this order:

1. flat top-level override keys
2. `common`
3. matching task section, either `pick` or `pick_and_place`
4. `envs.<env_id>`

Later sections override earlier sections.

```toml
[common]
ground_colors = ["gray", "white"]
robot_colors = ["yellow"]
spawn_min_radius = 0.15
spawn_max_radius = 0.25
reset_settle_frames = 5

[pick]
n_distractors = 1
objects = [
  { type = "cube", color = "green" },
  { type = "ycb", model_id = "011_banana" },
  { type = "ycb", model_id = "058_golf_ball" },
]

[pick_and_place]
cube_colors = ["red", "green"]
target_colors = ["blue"]

[envs.MuJoCoPickLift-v1]
spawn_angle_half_range_deg = 60.0
```

Launch with:

```bash
uv run so101-nexus teleop \
    --leader-port /dev/ttyACM0 \
    --env-config-profile teleop-profile.toml
```

The profile object schema matches the object classes documented in
[Scene Objects](/docs/api/objects). String specs such as `cube:blue` and
`ycb:011_banana` are supported for built-in UI/CLI object choices. Mesh
objects use mapping syntax because file paths may contain separator
characters:

```toml
[[pick.objects]]
type = "mesh"
name = "custom widget"
collision_mesh_path = "/path/to/collision.stl"
visual_mesh_path = "/path/to/visual.obj"
mass = 0.02
scale = 1.0
```

`MeshObject` is MuJoCo-only.

### Config factories

For advanced setups, provide a Python factory:

```bash
uv run so101-nexus teleop \
    --leader-port /dev/ttyACM0 \
    --env-config-factory my_project.teleop_configs:build_config
```

The factory receives `(env_id, base_config)`. It may return a config object:

```python
from so101_nexus import CubeObject, PickConfig, YCBObject


def build_config(env_id, base_config):
    return PickConfig(
        objects=[CubeObject(color="green"), YCBObject("011_banana")],
        n_distractors=1,
    )
```

It may also return a `dict` of `gym.make` kwargs. If the dict includes
`"config"`, teleop still wires the recording cameras after the factory
runs.

### Custom environments

Custom environments must be registered with Gymnasium before teleop builds
the dropdown. Import the registration module and list the environment ID:

```bash
uv run so101-nexus teleop \
    --leader-port /dev/ttyACM0 \
    --env-module my_project.envs \
    --extra-env-id CustomPick-v1
```

Custom environments should accept SO-100/SO-101 six-joint actions in the
standard joint order, expose current qpos through `_get_current_qpos()`, and
return camera observations compatible with the selected recording fields.
If the unwrapped environment exposes `task_description`, teleop records it
as the required LeRobot `task` field.

## Session flow

1. **Configure.** Pick the env, robot type, episode count, FPS, camera
   resolutions, action space, countdown, dataset fields, Max Steps, and
   Success Hold. A leader-port
   status banner reports up front whether the configured serial port is
   reachable, with a `Recheck Port` button to refresh after fixing it.
2. **Initialize.** The app connects to the leader arm and creates the
   dataset. If the port is unavailable or blocked by permissions, fix it
   and retry from the same page.
3. **Record.** A countdown plays, then the simulation mirrors the leader
   arm. The live wrist feed renders in the UI.
4. **Review.** After each episode, the UI shows a video replay and a
   joint-state plot.
5. **Approve or discard.** Accept to write the episode into the dataset,
   or discard and re-record.
6. **Push.** When all episodes are done, push to the Hugging Face Hub from
   the UI.

## Recorded dataset format

Datasets use the LeRobot v3 SO follower schema. Depending on your selection,
frames include:

- `observation.state`: follower readback (6D), always present
- `action`: commanded absolute joint positions (6D) by default, always present
- `reward`: scalar transition reward from the same environment step as `action`, always present
- `success`: episode success flag, always present
- `done`: episode termination flag, always present
- `observation.images.wrist`: wrist camera frames, optional
- `observation.images.overhead`: overhead camera frames, optional
- `observation.environment_state`: privileged low-dim state (TCP/object pose, grasp, and offsets), optional
- `task`: language description, always present

Body joints are stored in LeRobot degree units and the gripper is stored in
`RANGE_0_100` percent, matching `sim_so_follower` and SO100/SO101 MolmoAct2
normalization. `observation.state` comes from simulated follower readback, not
the leader echo.

On finalize (and before any Hub push), the app writes the environment id and
customization overrides to `meta/so101_nexus_env.json` inside the dataset. The
file is itself a valid [config profile](#config-profiles), so you can reproduce
a recording's environment by passing it back as
`--env-config-profile meta/so101_nexus_env.json`. It captures UI/CLI overrides;
contributions from a separate `--env-config-profile` used at record time are not
merged in.


On first recording, the app creates a simulator-only calibration file at
`$HF_LEROBOT_CALIBRATION/robots/sim_so_follower/teleop_sim.json` (or the
LeRobot default calibration root when `HF_LEROBOT_CALIBRATION` is unset). No
physical follower calibration is required for simulator teleop.

## Example rollout datasets

Recorded MuJoCo teleoperation rollouts are available on Hugging Face:

| Task | Dataset | Episode viewer |
| --- | --- | --- |
| PickLift | [johnsutor/MuJoCoPickLift](https://huggingface.co/datasets/johnsutor/MuJoCoPickLift) | [episode 0](https://huggingface.co/spaces/lerobot/visualize_dataset?path=%2Fjohnsutor%2FMuJoCoPickLift%2Fepisode_0) |
| PickAndPlace | [johnsutor/MuJoCoPickAndPlace](https://huggingface.co/datasets/johnsutor/MuJoCoPickAndPlace) | [episode 0](https://huggingface.co/spaces/lerobot/visualize_dataset?path=%2Fjohnsutor%2FMuJoCoPickAndPlace%2Fepisode_0) |

## Troubleshooting

For Ubuntu viewer/runtime issues such as `libdecor-gtk` warnings, see
[Teleoperation Troubleshooting](/docs/teleoperation/troubleshooting).
