Metadata-Version: 2.4
Name: twinrobo-core
Version: 0.1.0
Summary: TwinRobo: physically grounded digital twins of real cameras for robot learning in simulation.
Author: TwinRobo contributors
License-Expression: AGPL-3.0-only AND Apache-2.0
Project-URL: Homepage, https://twinrobo.github.io/
Project-URL: Documentation, https://twinrobo.github.io/
Project-URL: Repository, https://github.com/TwinRobo/TwinRobo-Core
Project-URL: Changelog, https://github.com/TwinRobo/TwinRobo-Core/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/TwinRobo/TwinRobo-Core/issues
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE-EXCEPTION.md
License-File: COMMERCIAL.md
License-File: NOTICE
License-File: twinrobo/catalog/LICENSE
Requires-Dist: deeplens-core>=2.5.6
Requires-Dist: numpy>=1.24
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Provides-Extra: mujoco
Requires-Dist: mujoco==3.3.2; extra == "mujoco"
Requires-Dist: warp-lang>=1.17; extra == "mujoco"
Provides-Extra: isaac
Requires-Dist: isaacsim[all,extscache]==6.1.0.0; extra == "isaac"
Provides-Extra: libero
Requires-Dist: mujoco==3.3.2; extra == "libero"
Requires-Dist: robosuite==1.4.1; extra == "libero"
Requires-Dist: bddl==1.0.1; extra == "libero"
Requires-Dist: gym==0.25.2; extra == "libero"
Requires-Dist: easydict; extra == "libero"
Requires-Dist: future; extra == "libero"
Requires-Dist: h5py; extra == "libero"
Requires-Dist: warp-lang>=1.17; extra == "libero"
Dynamic: license-file

# TwinRobo

**Physically grounded digital twins of real cameras for robot learning in simulation.**

[![CI](https://github.com/TwinRobo/TwinRobo-Core/actions/workflows/ci.yml/badge.svg)](https://github.com/TwinRobo/TwinRobo-Core/actions/workflows/ci.yml)
[![License: AGPL-3.0](https://img.shields.io/badge/license-AGPL--3.0-blue)](https://github.com/TwinRobo/TwinRobo-Core/blob/main/LICENSE)
[![Commercial license](https://img.shields.io/badge/commercial-license-indigo)](https://github.com/TwinRobo/TwinRobo-Core/blob/main/COMMERCIAL.md)
[![Camera catalog: Apache-2.0](https://img.shields.io/badge/camera%20catalog-Apache--2.0-green)](https://github.com/TwinRobo/TwinRobo-Core/blob/main/twinrobo/catalog/LICENSE)
![Python 3.12](https://img.shields.io/badge/python-3.12-blue)
![Status: alpha](https://img.shields.io/badge/status-alpha-orange)
[![Docs](https://img.shields.io/badge/docs-API%20reference-indigo)](https://twinrobo.github.io/)
[![Demo](https://img.shields.io/badge/demo-TwinRobo%20Preview-teal)](https://twinrobo.github.io/TwinRobo-Preview/)

Existing simulators render through a perfect pinhole camera, while in the real world robots see through real cameras: lenses with their own field of view, defocus blur, distortion, aberration and vignetting. Policies trained on pinhole images meet a different image at deployment, causing a significant [domain gap](https://twinrobo.github.io/why/) between simulation and reality. TwinRobo closes that gap: it renders scenes through the actual optics of the camera you will deploy, as a drop-in for the simulators robot learning already uses.

**One environment, any real camera.** TwinRobo is an open environment for aligning
perception with off-the-shelf cameras: pick a camera from the
[catalog](https://twinrobo.github.io/catalog/), and every simulator, rendering method and dataset
replay in TwinRobo sees through it. The catalog grows with the community:
[add a camera](#add-your-camera) and it works everywhere at once; camera makers
can [become partners](#for-camera-makers-become-a-partner) and ship verified
models of their products.

![An Isaac Sim tabletop: the simulator's pinhole (left) and the Stereolabs ZED X 2.2 mm, every pixel ray-traced through its lens (right)](https://raw.githubusercontent.com/TwinRobo/TwinRobo-Core/main/images/isaac-raycast.jpg)

## At a glance

| | MuJoCo (incl. LIBERO, RoboCasa) | NVIDIA Isaac Sim 6.x |
|---|---|---|
| Adapter | `twinrobo.plugins.mujoco.MujocoCameraTwin` | `twinrobo.plugins.isaac.IsaacCameraTwin` |
| `psf`: pinhole + lens blur and distortion | ✅ | ✅ |
| `raycast`: lens rays cast into the scene | ✅ | ✅ |
| RGB on the GPU, ground-truth depth on request | ✅ | ✅ |
| Cameras on robot links, stereo modules | ✅ | any USD camera prim |
| Setup | `pip install` | Docker recipe included |

**Cameras:** 12 in the catalog, including Intel RealSense D435 / D455,
Stereolabs ZED X, ZED X Mini and ZED 2i, Luxonis OAK-D, Logitech C920 and the
Raspberry Pi Camera Module 3, as single cameras or stereo pairs with rectified
output. Every camera works in every simulator and rendering method.
[Add yours](#add-your-camera) or [calibrate one](https://twinrobo.github.io/calibration/).

## Features

- **Real lens optics.** Lenses are traced with
  [DeepLens](https://github.com/vccimaging/DeepLens): depth- and field-dependent
  blur, aberration and vignetting from a lens prescription instead of a
  hand-tuned filter, and distortion from the camera's calibration or the lens'
  own.
- **Two rendering methods, per camera.** A fast 2.5D PSF renderer, and a GPU
  *ray cast* (NVIDIA Warp) that traces every pixel's rays through the lens into
  the scene, exact even for defocused foreground objects.
  [More](https://twinrobo.github.io/rendering/)
- **Drop-in for robot simulators.** MuJoCo, LIBERO (robosuite 1.4), RoboCasa
  kitchens (robosuite 1.5) and NVIDIA Isaac Sim (any USD stage, RTX rendering).
  `CameraTwinLiberoEnv` swaps a camera's observations in place, so policies and
  eval scripts run unchanged; recorded demos replay exactly, so any camera can
  be placed in an existing episode. [More](https://twinrobo.github.io/simulators/)
- **Camera catalog and stereo.** The cameras robots use (RealSense, ZED, OAK-D,
  webcams, Raspberry Pi), stereo modules with rectified output and adjustable
  baselines, and cameras mountable on any robot link. Adding a camera is a YAML
  file and a lens file. [More](https://twinrobo.github.io/cameras/)
- **Calibrate a real unit.** `python -m twinrobo.calibration` turns ChArUco,
  slanted-edge and flat-field captures into a `measured` entry: intrinsics,
  distortion, focus, vignetting and stereo baseline. [More](https://twinrobo.github.io/calibration/)
- **Validated in both simulators.** The PSF renderer matches DeepLens' reference
  renderer at about 46 dB PSNR. With the lens-ray methods, points land within a
  pixel of where the lens' chief rays point (within 0.1 px in Isaac). On a
  defocused foreground object, ray cast reproduces the exact coverage of its
  partially covered edge to within ~0.03 (Isaac) and ~0.05 (MuJoCo), where
  pinhole + PSF errs by 0.13–0.4.

## Installation

TwinRobo requires **Python 3.12** (inherited from DeepLens) and a CUDA GPU
for practical speeds.

The current release, v0.1.0 ([changes](https://github.com/TwinRobo/TwinRobo-Core/blob/main/CHANGELOG.md)),
from PyPI:

```bash
conda create -n twinrobo python=3.12 -y && conda activate twinrobo
pip install "twinrobo-core[mujoco]"     # TwinRobo, DeepLens, MuJoCo and NVIDIA Warp
```

or with Isaac Sim 6.1 (Ubuntu 22.04 or later; one package comes from NVIDIA's index;
use its own environment, as Isaac Sim brings its own MuJoCo):

```bash
pip install "twinrobo-core[isaac]" --extra-index-url https://pypi.nvidia.com
```

From a checkout, for development or the LIBERO support:

```bash
git clone https://github.com/TwinRobo/TwinRobo-Core.git && cd TwinRobo-Core
conda create -n twinrobo python=3.12 -y && conda activate twinrobo
pip install -e ".[dev]"          # core + DeepLens
pip install -e ".[libero]"       # + MuJoCo, NVIDIA Warp and LIBERO support
```

LIBERO itself is used from a checkout (`LIBERO_ROOT`). RoboCasa needs
robosuite 1.5 and gets its own environment; see
[simulators](https://twinrobo.github.io/simulators/#robocasa). **Isaac Sim** also runs in
NVIDIA's container, with TwinRobo installed on Isaac's own Python and torch:

```bash
docker pull nvcr.io/nvidia/isaac-sim:6.1.0        # needs the NVIDIA Container Toolkit
docker/isaac/run.sh examples/01_isaac_camera.py   # builds the image on first use
```

## Quick start: mount a real camera on a robot

Pick a camera from the catalog, mount it on a robot link, and render what it
would record. This runs in robosuite's PickPlace scene (`pip install -e ".[libero]"`
brings robosuite; no LIBERO checkout needed):

```python
import robosuite

from twinrobo import CameraSpec, CameraTwin, CatalogRegistry
from twinrobo.plugins.mujoco import MujocoCameraTwin, RobosuiteRenderer, to_uint8
from twinrobo.plugins.mujoco.mounts import CameraMount, mounted

env = robosuite.make(
    "PickPlace",
    robots="Panda",
    has_renderer=False,
    has_offscreen_renderer=True,
    use_camera_obs=False,
)
env.reset()

# 1. Pick a real camera from the catalog (here read out at 960x600).
path = CatalogRegistry().resolve("stereolabs/zed-x/2.2mm")
twin = CameraTwin.from_spec(CameraSpec.from_yaml(path, width=960, height=600))

# 2. Mount it on a robot link: position (m) and yaw/pitch/roll (deg) in the link's frame.
wrist = CameraMount("wrist", body="robot0_right_hand", pos=(0.08, 0, 0), rpy_deg=(67, -48, 173))

# 3. Render what that camera would record.
with mounted(env.sim.model._model, env.sim.data._data, wrist) as host:
    frame = MujocoCameraTwin(twin, RobosuiteRenderer(env), camera=host).get_frame(force_depth=True)

image = to_uint8(frame.rgb)  # the real camera's image, uint8 [H, W, 3]
pinhole = to_uint8(frame.rgb_ideal)  # the simulator's ideal pinhole, same camera
depth = frame.depth[0, 0]  # ground-truth z-depth (m): a label, not the ZED X's own depth
```

The first call builds the lens' PSF bank with DeepLens (about 20 s on a GPU)
and caches it; after that a frame takes a fraction of a second. The mount moves
with the link, so it follows the robot as it acts or replays a demo.

[`examples/05_mount_camera_on_robot.py`](https://github.com/TwinRobo/TwinRobo-Core/blob/main/examples/05_mount_camera_on_robot.py)
does this for three catalog cameras on the same wrist mount and draws what each
one gives you:

```bash
MUJOCO_GL=egl python examples/05_mount_camera_on_robot.py   # writes outputs/05_mount_camera.jpg
```

![Three catalog cameras on the same wrist mount: pinhole vs TwinRobo image, close-up, difference and depth](https://raw.githubusercontent.com/TwinRobo/TwinRobo-Core/main/images/quickstart_cameras.jpg)

Read it top to bottom:

- **Simulator pinhole / TwinRobo camera:** the field of view is each lens'
  own. The ZED X 2.2 mm's wide lens shows its barrel distortion, which a
  pinhole cannot.
- **Close-up:** the same patch at native resolution; the real lens softens the
  lettering, as the camera would.
- **Difference ×4:** where the optics change the image: distortion shifts
  edges for the 2.2 mm lens; blur changes edges and texture for the others.
- **Ground-truth depth:** the simulator's metric z-depth, aligned pixel for pixel with the
  camera's image; a label for training, not an output of the camera (`force_depth=True`).

### Next steps

- **Ray casting:** pass `render="raycast"` to `MujocoCameraTwin` to trace real
  lens rays into the scene (see
  [Rendering methods](#rendering-methods)).
- **Isaac Sim:** the same twin on any USD camera prim:

  ```python
  from twinrobo.plugins.isaac import IsaacCameraTwin

  twin = CameraTwin.from_catalog("stereolabs/zed-x/2.2mm", build_psf=False)
  cam = IsaacCameraTwin(twin, "/World/Camera", render="raycast")
  frame = cam.get_frame()  # frame.rgb: the ZED X's view, on the GPU
  ```

  `docker/isaac/run.sh examples/01_isaac_camera.py --render raycast` renders the
  image at the top of this page. [More](https://twinrobo.github.io/simulators/#isaac-sim)
- **A LIBERO policy's camera:** wrap an env so its observations come from the
  twin, and eval scripts run unchanged:

  ```python
  from twinrobo.datasets.libero import CameraTwinLiberoEnv, make_env

  env, task, init_states = make_env("libero_spatial", 0, resolution=128)
  env = CameraTwinLiberoEnv(env, {"agentview": twin})
  obs = env.reset()  # obs["agentview_image"] now comes from the twin
  ```

- **An RGB-D frame from any source:** `twin.process(rgb, depth)` takes linear
  RGB `[B,3,H,W]` and metric depth `[B,1,H,W]` on the GPU (depth sets each pixel's
  defocus) and returns the camera's image.
- **Your own camera:** copy a catalog spec, or add one for everyone; see
  [Cameras](https://twinrobo.github.io/cameras/) and [CONTRIBUTING](https://github.com/TwinRobo/TwinRobo-Core/blob/main/CONTRIBUTING.md#adding-a-camera).

## Rendering methods

| Method | How | Per frame, 1920×1200 (RTX 3090) | Best for |
|---|---|---|---|
| `psf` | pinhole render + depth- and field-dependent PSF blur | ~0.1–0.4 s | fast training data |
| `raycast` | lens rays intersected with the scene on the GPU | ~0.9 s (MuJoCo), ~1.5 s (Isaac) | defocus across the aperture, aberration, vignetting; exact defocus around occluders |

Ray casting is simulator independent (`twinrobo.optics.lensrender`): a
simulator supplies the camera pose, views rasterized from points across the
lens pupil (for shading), and the scene's triangles.

[Details and validation](https://twinrobo.github.io/rendering/)

## Supported simulators

| Simulator | Scenes and data | Rendering methods |
|---|---|---|
| MuJoCo | any MJCF scene | `psf`, `raycast` |
| LIBERO | LIBERO tasks and demos; Panda, Sawyer, UR5e, iiwa, Jaco, Gen3, multi-camera variants | `psf`, `raycast` |
| RoboCasa | procedurally generated kitchens, LeRobot demo datasets, PandaOmron | `psf`, `raycast` |
| NVIDIA Isaac Sim 6.x | any USD stage, RTX rendering (Docker recipe included) | `psf`, `raycast` |

## Camera catalog

| Vendor | Cameras | Stereo modules (baseline) |
|---|---|---|
| Intel RealSense | D435 / D435i color and IR imager, D455 color and IR imager | D435 (50 mm), D455 (95 mm) |
| Stereolabs | ZED X 2.2 mm and 4 mm eyes, ZED 2i 2.1 mm eye | ZED X (120 mm), ZED X Mini (50 mm), ZED 2i (120 mm) |
| Luxonis | OAK-D color and mono | OAK-D (75 mm) |
| Logitech | C920 HD Pro Webcam | |
| Raspberry Pi | Camera Module 3, Camera Module 3 Wide | |

All entries are `estimated` today: geometry fitted to the datasheet fields of
view, blur from a surrogate lens of matching field, every number sourced in the
entry's header. The [catalog page](https://twinrobo.github.io/catalog/) lists each with its sensor,
lens and field of view, generated from the catalog files.

> **The camera catalog is Apache-2.0.** Everything in
> [`twinrobo/catalog/`](https://github.com/TwinRobo/TwinRobo-Core/blob/main/twinrobo/catalog) (camera specs, stereo modules and lens
> files) is licensed under the [Apache License 2.0](https://github.com/TwinRobo/TwinRobo-Core/blob/main/twinrobo/catalog/LICENSE),
> separately from the AGPL-3.0 code: use the camera data in any project,
> commercial or not, with or without TwinRobo.

## Add your camera

**The catalog is the heart of TwinRobo, and it is open.** Every camera added
becomes available to everyone, in every simulator, rendering method and dataset
replay, with no simulator-specific work. If your robot uses a camera that is
not here, it is the most valuable contribution you can make:

1. Create `twinrobo/catalog/<vendor>/<model>/<lens>/` with a `camera.yaml`
   (sensor, lens, focus, calibration; copy an existing entry) and a DeepLens
   `lens.json`. No prescription published? Build a surrogate of the same field
   of view and f-number with `tools/catalog/make_surrogate_lens.py`.
2. Run `pytest tests/test_catalog.py` and render it once:
   `CameraTwin.from_catalog("<vendor>/<model>/<lens>")`.
3. Open a pull request, with where each number came from. Catalog
   contributions are published under Apache-2.0, like the rest of the catalog.

Calibrated a real unit? Upgrading an entry from `estimated` to `measured` is
just as welcome. The full guide is in
[CONTRIBUTING](https://github.com/TwinRobo/TwinRobo-Core/blob/main/CONTRIBUTING.md#adding-a-camera); to ask for a camera instead,
[open a camera request](https://github.com/TwinRobo/TwinRobo-Core/blob/main/../../issues/new?template=new_camera.md).

## For camera makers: become a partner

**Put your camera in every robot-learning simulator.** Robot teams can now choose
and validate cameras in simulation before they buy hardware. The TwinRobo
Camera Partner program lets camera and lens makers ship
`manufacturer_verified` models of their products, and lets companies sponsor
the open environment their customers use:

- **You contribute** any of: optical data (a lens prescription or measured
  PSFs), factory calibration, sample units, or sponsorship of compute and
  maintenance. Published entries are Apache-2.0, free for your customers to use
  anywhere; data you cannot publish can stay private.
- **You get** a verified catalog entry that works in MuJoCo, Isaac Sim and
  every rendering method, your company listed as a partner here and on the
  docs site, a showcase example with your camera, and early integration of
  new products.

[Read about the program](https://twinrobo.github.io/partners/) or
[open a partner inquiry](https://github.com/TwinRobo/TwinRobo-Core/blob/main/../../issues/new?template=camera_partner.md).

## Documentation

The documentation site, with the API reference, is at
**[twinrobo.github.io](https://twinrobo.github.io/)**; its source is in
[TwinRobo/twinrobo.github.io](https://github.com/TwinRobo/twinrobo.github.io).

| | |
|---|---|
| [Optics and rendering](https://twinrobo.github.io/rendering/) | PSF pipeline, distortion, the two rendering methods, validation |
| [Cameras and stereo](https://twinrobo.github.io/cameras/) | CameraSpec, catalog, stereo modules, spec overrides |
| [Simulators](https://twinrobo.github.io/simulators/) | MuJoCo, LIBERO, RoboCasa and Isaac Sim: setup, behavior, limits |
| [API reference](https://twinrobo.github.io/api/) | every public class and function, from the docstrings |
| [Development](https://twinrobo.github.io/development/) | layout, tests, status |

## Contributing

Beyond cameras, also wanted: new simulator adapters (a lens scene is three
methods; see `twinrobo.optics.lensrender`), sensor noise and ISP models, and
documentation. See [CONTRIBUTING](https://github.com/TwinRobo/TwinRobo-Core/blob/main/CONTRIBUTING.md), and the [ROADMAP](https://github.com/TwinRobo/TwinRobo-Core/blob/main/ROADMAP.md)
for what is planned (simulated depth cameras first).

## Roadmap

- Simulated depth cameras: depth from the stereo modules, with a real sensor's errors
- Faster ray casting in Isaac Sim (fewer, shared shading views)
- Sensor noise and ISP models (interfaces are in place)
- `measured` catalog entries from calibration captures of real units
- More cameras and lenses in the catalog

## Acknowledgements

TwinRobo builds on [DeepLens](https://github.com/vccimaging/DeepLens) for
lens modeling, [MuJoCo](https://mujoco.org),
[NVIDIA Isaac Sim](https://developer.nvidia.com/isaac/sim),
[robosuite](https://robosuite.ai), [LIBERO](https://libero-project.github.io),
[RoboCasa](https://robocasa.ai) and [NVIDIA Warp](https://github.com/NVIDIA/warp).
Third-party files and their licenses are listed in [`NOTICE`](https://github.com/TwinRobo/TwinRobo-Core/blob/main/NOTICE).

## License

TwinRobo is dual-licensed: the **GNU AGPL-3.0** ([`LICENSE`](https://github.com/TwinRobo/TwinRobo-Core/blob/main/LICENSE), with an
additional permission for NVIDIA Isaac Sim / Omniverse in
[`LICENSE-EXCEPTION.md`](https://github.com/TwinRobo/TwinRobo-Core/blob/main/LICENSE-EXCEPTION.md)), free for research, education and
open-source use, or a **commercial license** for proprietary products and hosted
services ([`COMMERCIAL.md`](https://github.com/TwinRobo/TwinRobo-Core/blob/main/COMMERCIAL.md)). The camera catalog
(`twinrobo/catalog/`) is Apache-2.0, free to use anywhere. Third-party material is
listed in [`NOTICE`](https://github.com/TwinRobo/TwinRobo-Core/blob/main/NOTICE).
