Metadata-Version: 2.4
Name: videosdk-teleop
Version: 0.1.0
Summary: Robot teleoperation over WebRTC: a lerobot transport with frame-accurate observation/action correlation and dataset recording
Author: VideoSDK
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/zujonow/videosdk-teleops
Project-URL: Source, https://github.com/zujonow/videosdk-teleops
Project-URL: Issues, https://github.com/zujonow/videosdk-teleops/issues
Keywords: robotics,teleoperation,webrtc,lerobot,so-101,imitation-learning,ros2
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Networking
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS :: MacOS X
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: videosdk>=0.3.8
Requires-Dist: vsaiortc>=0.0.17
Requires-Dist: numpy
Requires-Dist: av
Requires-Dist: pillow
Provides-Extra: lerobot
Requires-Dist: lerobot[feetech]==0.6.1; extra == "lerobot"
Provides-Extra: ros2
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Dynamic: license-file

# videosdk-teleop

Robot teleoperation over WebRTC. A **transport for lerobot**, with a
first-class link between **what the operator saw** and **what the robot did**
— and a recording that keeps it, all the way to a LeRobot dataset.

```
remote operator ─► video observations ─► observation/action correlation
      ─► WebRTC teleoperation ─► safety-controlled robot
      ─► demonstration recording ─► LeRobot dataset ─► policy
```

Your arm stays an ordinary lerobot `Robot`; your leader stays an ordinary
lerobot `Teleoperator`. `SO101FollowerAdapter` / `SO101LeaderAdapter` wrap one
of each and own the network session in its place — joining the meeting,
publishing joints and video, applying incoming actions, clamping them first —
so any arm lerobot already supports is a few lines from teleoperated over
WebRTC. The host owns the control loop (`tick()` / `run()`), not lerobot's own
script, which is what lets a ROS 2 timer or a bare Python `for`-loop drive the
same class just as well.

```python
# on the machine holding the arm
class MyArm(Follower):
    def descriptor(self):    return single_group_descriptor(...)
    def read_joints(self):   return {...}
    def write_joints(self, joints): ...   # RAISE on failure

MyArm("meeting-id", safety=SafetyConfig(slew=12.0)).run(hz=50)
```

---

## Install

```bash
pip install videosdk-teleop               # transport + cameras, ready to run
pip install "videosdk-teleop[lerobot]"    # + the lerobot adapters
pip install "videosdk-teleop[ros2]"       # + the ROS 2 nodes
pip install -e .                          # from a checkout
```

That first line is enough to run a session: it brings the WebRTC transport
and the camera stack with it. The extras add a hardware bus or dataset export.

Imports stay lazy regardless, so the protocol, safety, correlation and
recording layers still work on a machine where the WebRTC stack cannot load —
`videosdk` fails to import on Python 3.14, and the transport reports itself
unavailable rather than taking the rest of the SDK down with it.

## Watching a session

Either side can serve a live page: camera feeds on top, stats below, one half
each. The stats half is two labelled bands — `network` (latency, jitter, packet
loss, bandwidth in/out, data in/out) and then this machine's role: **leader**
adds actions sent, **follower** adds actions applied, observations and packets
dropped. Glass-to-glass latency is on both. Each tile carries a
120-second line chart with axes. One screen, no scrolling.

```bash
python examples/dashboard_demo.py            # both ends, loopback, no hardware
python examples/follower.py --dashboard      # on the robot
python examples/leader.py   --dashboard      # on the operator's machine
```

For the whole architecture over a **real meeting with no robot** — the only
setup where the network tiles and glass-to-glass populate — run the
hardware-free pair. Two USB cameras on one side, nothing on the other:

```bash
export VIDEOSDK_MEETING_ID=xxxx-xxxx-xxxx VIDEOSDK_TOKEN=...
python examples/demo_follower.py             # the machine with the cameras
python examples/demo_leader.py               # anywhere
```

```
                    ── the operator's page ──
┌──────────────────────────────────────────────────────────────┐
│ teleop  leader  active                peer 0.3s         live │
├──────────────────────────────┬───────────────────────────────┤
│            wrist             │            front              │
│  wrist 30 fps                │  front 30 fps                 │   half
├──────────────────────────────┴───────────────────────────────┤
│ NETWORK                                                      │
│ latency   jitter   packet loss   bandwidth      data         │
│                                    in / out     in / out     │
│   34ms      6ms       0.4%       1840/210kbps  412/88kB/s    │   half
├──────────────────────────────────────────────────────────────┤
│ LEADER                                                       │
│ glass-to-glass                actions sent                   │
│     118ms                        50/s                        │
└──────────────────────────────────────────────────────────────┘

  the robot's page: same NETWORK band, then FOLLOWER —
  glass-to-glass · actions applied · observations · packets dropped
```

Off unless asked for — it binds a socket and costs CPU, which is the operator's
call. `--dashboard-host 0.0.0.0` reaches it on a headless robot host, and
`ros2 run ... -p dashboard_port:=8080` is the ROS 2 equivalent.

Every number on a page describes the machine serving it, so the band label is
the provenance. Everything else the SDK measures — per-joint tracking error,
obs→applied p95, correlation, tick jitter, recorder queue depth, the video-sync
anchor — is still served in full at `/snapshot.json`.

Read [DASHBOARD.md](DASHBOARD.md) before trusting the latency figures: `—`
means no measurement rather than zero, and glass-to-glass crosses two
unsynchronised clocks.

---

## Quick start, no hardware

Runnable examples live in a separate repo,
[videosdk-teleops-examples](https://github.com/videosdk-live/videosdk-teleops-examples):

```bash
pip install -e .
git clone https://github.com/videosdk-live/videosdk-teleops-examples
cd videosdk-teleops-examples

python quickstart/protocol_demo.py --gates          # the wire format, live
python quickstart/remote_teleop.py --ticks 100      # full chain, loopback
python quickstart/my_arm.py                         # port your own arm, no hardware
python ros2/ros2_arm.py                              # same lesson, over ROS topics
python tools/inspect_session.py sessions/*  --trace 3
python lerobot-so101/export_lerobot.py sessions/*  --dry-run
```

`remote_teleop.py` runs a real follower against a real leader through an
in-process transport: real gates, real clamps, real observation log. Only the
network is swapped out. `my_arm.py` is the extension point itself: subclass
`Follower`, implement `descriptor()` / `read_joints()` / `write_joints()`, and
everything else — lease negotiation, the clamp, the watchdog, the e-stop
latch, correlation, recording — is inherited.

## On the rig

```bash
python -m videosdk_teleop.devices                 # stable by-path handles
python videosdk-teleops-examples/lerobot-so101/camera_test.py --wrist cam_a --front cam_b
python videosdk-teleops-examples/lerobot-so101/camera_test.py --servo-probe /dev/ttyACM0

export VIDEOSDK_TOKEN=... MEETING_ID=xxxx-xxxx-xxxx
cd videosdk-teleops-examples/lerobot-so101
python follower.py      # on the machine with the arm
python leader.py        # on the machine with the leader arm
python record.py        # anywhere: watches only, cannot move the arm
```

Nothing moves until an operator claims control, and the clamp inside `tick()`
holds the arm the moment they stop sending. There is no wrapper to forget and
no way around it — the clamp sits directly above the only call to
`write_joints()`.

`follower.py` / `leader.py` / `record.py` wrap the SO-101 with
`SO101FollowerAdapter` / `SO101LeaderAdapter`
(`videosdk_teleop.adapters.lerobot`), each a `Follower` / `Leader` subclass.
`videosdk-teleops-examples/quickstart/my_arm.py` is the same three-method
extension point for hardware lerobot has never heard of.

## On ROS 2

One node, `teleop_bridge_node`, one role per host. It never owns a servo bus
itself — it subscribes to your driver's joint-state topic and publishes
commands to your driver's joint-command topic, and carries the gap between
the two hosts over the meeting. That means the ROS topic hop between this
node and your driver is part of the actuation path, not just an
observability surface — full detail, including a real two-driver runbook, in
[ROS2.md](ROS2.md).

Install into the interpreter ROS 2 itself runs on, not whichever one your shell
points at — on Ubuntu 24.04 that needs `--break-system-packages` (it still only
writes to `~/.local`):

```bash
/usr/bin/python3 -m pip install --user --break-system-packages \
    -e '.[lerobot]'
```

Then build the ament package once:

```bash
ln -s "$(pwd)/ros2/videosdk_teleop_ros" ~/ros2_ws/src/
cd ~/ros2_ws && colcon build --symlink-install && source install/setup.bash
ros2 pkg executables videosdk_teleop_ros     # expect teleop_bridge_node
```

Now **two terminals**, each already running your own leader/follower driver
nodes and each starting from a fresh shell that has none of the environment:

```bash
# run these in EVERY terminal
source /opt/ros/jazzy/setup.bash
source ~/ros2_ws/install/setup.bash
export VIDEOSDK_MEETING_ID=abcd-efgh-ijkl    # both ends MUST join the same room
export VIDEOSDK_TOKEN=...                    # keep it here, not in the YAML
```

```bash
# 1 — follower host, relaying to your follower driver
ros2 launch videosdk_teleop_ros bridge.launch.py \
    role:=follower_side robot_id:=my_follower_arm \
    calibration_profile:=lerobot_so101_rad \
    cameras:=cam_a,cam_b labels:=wrist,front

# 2 — leader host, relaying to your leader driver
ros2 launch videosdk_teleop_ros bridge.launch.py \
    role:=leader_side robot_id:=my_leader_arm \
    calibration_profile:=lerobot_so101_rad

# 3 — look before you arm, then take control
ros2 topic echo /videosdk_bridge/remote/follower_states --once   # far arm, local units
ros2 service call /videosdk_bridge/enable std_srvs/srv/SetBool "{data: true}"
```

The leader logs the peer's shape as it is accepted or refused at arm time —
that is the check that catches the two ends disagreeing about units, and
reading one log line is cheaper than watching an arm move wrongly.

Nothing moves until that `enable` call. Release it with `{data: false}`, or stop
the arm outright:

```bash
ros2 service call /videosdk_bridge/estop std_srvs/srv/Trigger        # latched
ros2 service call /videosdk_bridge/clear_estop std_srvs/srv/Trigger  # only way out, on the follower host
```

Watch what it is doing:

```bash
ros2 topic hz   /follower/joint_commands           # what this node is sending your driver
ros2 topic echo /videosdk_bridge/remote/follower_states
ros2 bag record /follower/joint_commands /videosdk_bridge/remote/follower_states
```

---

## The API

Two abstract base classes, one per machine — subclass the one your machine
needs. The class name says which machine; three methods
(`descriptor()` / `read_joints()` / `write_joints()`) are the whole extension
point, and everything else — lease negotiation, the clamp, the watchdog, the
e-stop latch, correlation, recording — is inherited.

```python
# ---- follower host: the machine holding the arm ----------------------------
from videosdk_teleop.adapters.lerobot import SO101FollowerAdapter

arm = SO101Follower(SO101FollowerConfig(port="/dev/ttyACM0", cameras={...}))
follower = SO101FollowerAdapter(
    "robot-lab-1", robot=arm, token=TOKEN,
    safety=SafetyConfig(slew=12.0, watchdog_timeout_s=0.5))

follower.run(hz=50)          # start(), loop, stop() -- ctrl-c is handled
```

```python
# ---- leader host: the machine holding the leader arm -----------------------
from videosdk_teleop.adapters.lerobot import SO101LeaderAdapter

leader = SO101Leader(SO101LeaderConfig(port="/dev/ttyACM1"))
remote = SO101LeaderAdapter("robot-lab-1", teleop=leader, token=TOKEN,
                            max_misalignment=15.0)

leader.connect()             # local leader arm; the SDK doesn't own this bus
remote.start()
remote.claim_control()
remote.arm_when_aligned()    # hold the leader steady against the follower

for _ in remote.ticks(hz=50):
    remote.observation()     # synced joints + pixels; arms the echo too
```

`SO101FollowerAdapter` / `SO101LeaderAdapter` (`videosdk_teleop.adapters.lerobot`)
are `Follower` / `Leader` subclasses that wrap an ordinary lerobot `Robot` /
`Teleoperator` — see [API.md §4/§5](API.md#4-follower) and
[EXAMPLES.md §3/§4](EXAMPLES.md#3-follower--every-method-and-event) for every
method and event.

### Safety is a config, not a wrapper

```python
follower = SO101FollowerAdapter(
    "robot-lab-1", robot=arm, token=TOKEN,
    safety=SafetyConfig(
        slew=12.0,                  # max change per joint per tick, arm's own units
        watchdog_timeout_s=0.5,     # no actions for this long -> hold position
        on_starvation=FailsafeAction.HOLD))
```

The clamp lives directly above the only call to `write_joints()`, inside
`tick()` — there is no separate wrapper object to construct and no path to the
motors that skips it. It clamps every action, holds if the operator
disappears, and latches on `estop()` until a human calls `clear_estop()`.

| | |
|---|---|
| `on_safe_state(state, reason)` | every failsafe transition |
| `on_limited(event)` | one per clamp, with magnitude |
| `on_applied(action)` | what actually reached the motors |
| `estop()` / `clear_estop()` | latched stop |
| `.state` / `.safety` | current state, limits in force |

### Recording

Same four methods on either side:

```python
session = follower.start_recording("./sessions")
follower.start_episode("pick up the cube")
...                                          # your loop runs
follower.end_episode(success=True)
follower.stop_recording()
```

On the **follower** that records the arm's own observations — pixels as
captured, before an encoder or a network touched them. On the **leader** it
records the synced stream: each frame paired with joints interpolated to that
frame's own capture instant. Export either with
`videosdk-teleops-examples/lerobot-so101/export_lerobot.py`.

### Correlation, for free

`Leader.observation()` remembers which observation it returned, and arms the
next command's echo with it — so every action in a recording points at the
exact observation the operator was looking at, by integer id, with no clock
shared between the machines. `AppliedAction.observation_id` /
`.observation_id_local` on the follower side expose the edge.

### Adding a robot

Nothing here is SO-101 specific. A `Follower` / `Leader` subclass implements
three abstract methods and asks no other questions of the hardware
underneath — see `videosdk-teleops-examples/quickstart/my_arm.py` for the
extension point taught end to end, and
`videosdk-teleops-examples/ros2/ros2_arm.py` for the same lesson over ROS
topics.

---

## The idea

The SDK can answer both directions of one question:

> *For command X, which camera observation did the operator act upon?*
> *For observation Y, which action came out and what actually happened?*

```
  FOLLOWER                                        LEADER
  mint Observation(oid=N) ──── state frame ────►  remember last_oid = N
    wrist frame 500                                       │
    front frame 900                              operator moves the arm
    robot state_seq 4410                                  │
        │                                                 ▼
  ObservationLog.get(N)  ◄──── cmd frame ──── Command(cid=M, oid=N)
    O(1), no clocks involved
        │
  AppliedAction(cid=M, oid=N, target=…, action=…, limits=[…])
        │
  RobotState(sseq=K, cid=M)
```

Each id is minted by the machine that owns the thing it names and only ever
echoed by the other. **No shared counter, no clock sync, no agreement problem.**

Read [SYNC.md](SYNC.md) next — it is the load-bearing document.

---

## Documents

| | |
|---|---|
| [EXAMPLES.md](EXAMPLES.md) | bring your own hardware: complete code for both ends, no SO-101 — §3/§4 are the full follower/leader tour |
| [API.md](API.md) | every public symbol: signatures, defaults, threading, exceptions — §4/§5 cover Follower/Leader |
| [ARCHITECTURE.md](ARCHITECTURE.md) | threads, modules, safety layers, adding a robot or camera |
| [PROTOCOL.md](PROTOCOL.md) | wire format, receive gates, what is frozen |
| [SYNC.md](SYNC.md) | every clock, what is measurable and what is not |
| [RECORDING.md](RECORDING.md) | staging schema, drop accounting, LeRobot export |
| [ROS2.md](ROS2.md) | `teleop_bridge_node`, calibration, what ROS is and is not here |
| [DASHBOARD.md](DASHBOARD.md) | the live stats page: every tile, its source, and what it cannot tell you |
| [DASHBOARD_PARAMETERS.md](DASHBOARD_PARAMETERS.md) | every dashboard number: meaning, stats key, and how it is calculated |

---

## Safety

| Layer | Where |
|---|---|
| lease | transport, receive |
| sequence (duplicate / reorder) | transport, receive |
| staleness | control loop |
| watchdog | control loop, every tick |
| deadman | control loop |
| position + slew clamps | `Clamp` |
| e-stop | **latched**, only a human clears it |

Defaults are timid on purpose. `watchdog_timeout_s` and `max_staleness_ms`
should be re-derived on **your** link under load — the numbers that matter are
the ones your network actually produces.

`HOLD` is the default failsafe, not `TORQUE_OFF`: cutting torque on a
gravity-loaded arm drops it.

Every clamp is recorded with its **magnitude**, not just a count. A joint
clamped by 0.2° is rounding; the same joint clamped by 40° means calibration is
wrong, and a bare counter cannot tell them apart.

---

## Testing

```bash
pip install -e ".[dev]"
pytest -q
```

No test touches a network, a servo, a camera, or `sleep()`. Everything
timing-sensitive takes an explicit monotonic value. `LoopbackTransport`
*subclasses* the real transport and replaces only the three methods that touch
the network, so the receive gates under test are production code.

The same harness is importable for your own `Follower` / `Leader` subclasses:

```python
from videosdk_teleop.testing import LoopbackTransport, connect, grant_lease

arm = MyArm(transport=LoopbackTransport("my-arm"))
operator = FakeLeader(transport=LoopbackTransport("operator"))  # videosdk_teleop.adapters.fake
connect(arm, operator)
```

---

## Known limitations

1. **`oid` is a bound, not a proof.** The leader stamps the newest observation
   it received *on the data channel*; what the operator's eye saw is a video
   frame from a separate RTP path. Closing this needs frame metadata inside the
   encoded video (SEI) and vsaiortc owns the codec. Mitigated by recording both
   the claimed and the locally-current observation, so the doubt is measured.

2. **`observation → command` latency is not reported.** It spans two
   unsynchronised clocks and cannot be measured. `observation → applied` is
   exact and is the number that matters. See [SYNC.md](SYNC.md).

3. **`capture_mono` is arrival, not shutter.** True shutter timestamps are not
   exposed through the vsaiortc MediaPlayer path. Inter-camera *skew* stays
   meaningful; the absolute value carries a fixed unknown offset.

4. **Images are re-encoded.** The capture path decodes MJPG before we see it, so
   the recorder re-encodes JPEG on the writer thread. Off the control loop, but
   not free.

5. **`claim` is first-come and unauthenticated.** Any participant can take the
   lease. E-stop is unauthenticated *by design* — anyone should be able to stop
   the arm — but `claim` should be gated by meeting-level auth before this is
   exposed to untrusted participants.

6. **SO-101 control-table addresses are unverified against your firmware.**
   Feetech ships variants. Run `camera_test.py --servo-probe` once per rig.

7. **`videosdk` does not import on Python 3.14** (pydantic v1 inside
   pymediasoup). The transport degrades to unavailable and everything else
   still runs; use 3.10–3.12 for live meetings.

8. **Single-process, single-robot.** No fleet, no multi-robot session, no
   authenticated operator handover queue.

---

## Next

- Gate `claim` behind meeting-level identity, and add an operator handover queue
- SEI frame metadata if/when the codec is reachable — closes limitation 1
- Direct PyAV demux to keep original MJPG packets — closes limitation 4
- Bench-derived `watchdog_timeout_s` / `max_staleness_ms` from a probe script
- Bus-budget measurement to justify raising `observation_hz` toward camera fps
- Policy playback: replay a LeRobot dataset through the same safety envelope
