Synthetic Data & Teleoperation

Generate synthetic data with Isaac Sim Replicator as tracked runs, drive the robot through upstream ROS teleop stacks, and record demonstrations that land as first-class datasets — so the chain record → inspect → train → evaluate needs no glue code. Caasi adds no SDG engine and no teleoperation of its own: it resolves what is installed and launches it.

caasi synth

Caasi implements no SDG engine. generate reuses the experiment runner (backend: sim) after checking that the Replicator extension resolves from the sdg catalog domain, and the run writes into CAASI_DATASET_DIR. preview and validate are filesystem-only summaries of the synthetic tree — one rgb/ frame counts as one produced episode and no image is ever decoded.

caasi synth status

caasi synth status [--json]

Reports the two capabilities of the sdg catalog domain — replicator (exts/omni.replicator.core, core) and writer_kit (exts/omni.replicator.writer.kit) — both detected as paths under the Isaac Sim installation. On a machine without Isaac Sim this is the real output:

shellcaasi synth status
Synthetic Data Generation
Capability  State    Detail
replicator  missing  not installed
writer_kit  absent   not installed
0 of 2 capabilities installed.

The core capability shows missing, the optional one absent; the summary line counts installed capabilities. Exit 0 — this is a report, not a gate. JSON carries the full payload: domain, kind, tool, root, total, installed, capabilities (each with key, label, group, core, found, how, value, launch, script, nodes), launcher and running:

shellcaasi synth status --json | jq '{domain, total, installed, running}'
{ "domain": "sdg", "total": 2, "installed": 0, "running": [] }

The same hardware/software surface is gated by doctor's data component, which also covers the bag-storage plugins and the download CLIs:

shellcaasi doctor --component data
Environment Diagnostics

Data & Recording
  ✗ Replicator — not found (exts/omni.replicator.core)
      ↳ Install it, or point Caasi at the renamed upstream with `caasi config set
catalog.data.replicator.paths [...]`.
  • MCAP bag storage — not found (rosbag2_storage_mcap)
  • SQLite3 bag storage — not found (rosbag2_storage_default_plugins)
  • Hugging Face CLI — not found (hf, huggingface-cli)
  • NGC CLI — not found (ngc)
✘ 1 issue(s) found, 0 warning(s). Run with --verbose for hints.
exit code: 1

caasi synth generate

caasi synth generate CONFIG_PATH [--episodes N] [--output PATH] [--dry-run] [-- SCRIPT_ARGS…]

What happens on launch:

1. The Replicator gate runs first: sdg.replicator must resolve, even for --dry-run. When it doesn't, the real failure is Error: Isaac Sim Replicator was not found (exts/omni.replicator.core). Install Isaac Sim, or point Caasi at it with `caasi config set catalog.sdg.replicator.paths [...]`. (exit 1).
2. The dataset directory is created: --output PATH as-is, otherwise <name-slug>-<YYYYmmdd-HHMMSS> in the project's datasets/ or paths.datasets (default ~/.caasi/datasets) — the same rules as dataset generate.
3. The script receives --dataset-dir <dir> and the environment variable CAASI_DATASET_DIR (plus --headless when the experiment declares it) — the generator writes the synthetic tree there.
4. A detached run (kind: dataset, the experiment's backend) is started and metadata.json is written.

The synthetic dataset tree:

sdg-dome-20260907-101500/
├── metadata.json
├── rgb/              # one frame = one produced episode
├── depth/
├── segmentation/
├── bounding_boxes/
└── metadata/

and its metadata.json:

{
  "name": "sdg-dome",
  "created": "2026-09-07T10:15:00+08:00",
  "generator": "replicator",
  "replicator": "/opt/isaac-sim-6.0/exts/omni.replicator.core",
  "experiment": "/home/you/demo/experiments/sdg_dome.yaml",
  "backend": "sim",
  "episodes": 200,
  "status": "generating",
  "run_id": "20260907-101500-sdg-dome"
}

shellcaasi synth generate experiments/sdg_dome.yaml --episodes 200 --dry-run
Dry run — nothing was started:
  command: /opt/isaac-sim-6.0/python.sh /home/you/demo/scripts/sdg_collect.py --episodes 200 --headless --dataset-dir <dir>
  cwd:     /home/you/demo/experiments
caasi synth generate experiments/sdg_dome.yaml --episodes 200
Synthetic data generation started: /home/you/.caasi/datasets/sdg-dome-20260907-101500
  Follow it with: caasi logs 20260907-101500-sdg-dome -f

caasi synth preview

caasi synth preview QUERY [--json]

QUERY is the same argument dataset inspect takes: a path, latest/last/newest, or a unique name/prefix — unknown or ambiguous → Error: No dataset matching '<query>'. (exit 1). Preview counts files, never decodes images:

• episodes produced = number of files in rgb/;
• per-directory file counts (recursive) for rgb, depth, segmentation, bounding_boxes and metadata;
• total file count and byte size of the whole dataset;
• elapsed seconds since the metadata's created timestamp;
• the linked run's id and live status.

shellcaasi synth preview latest
sdg-dome /home/you/.caasi/datasets/sdg-dome-20260907-101500
  generator: replicator
  episodes: 200 produced, 200 declared
  size: 1.9 GB
  elapsed: 3421.5s
  Run: 20260907-101500-sdg-dome (succeeded)
directory       files
rgb               200
depth             200
segmentation      200
bounding_boxes    200
metadata            3
caasi synth preview latest --json | jq '{episodes_produced, episodes_declared}'
{ "episodes_produced": 200, "episodes_declared": 200 }

JSON payload: {"path", "name", "generator", "episodes_declared", "episodes_produced", "counts", "files", "bytes", "elapsed", "run"}. run is null when unlinked or {"id", "status": "unknown"} if the run was deleted; elapsed is null when created is unparseable.

caasi synth validate

caasi synth validate QUERY [--json]

Starts from the structural checks of dataset validate (metadata present, valid JSON object, name set, declared episode count vs episodes/) and adds two SDG rules, with these exact issue strings:

• when metadata declares an integer episodes and rgb/ exists: metadata declares 200 episode(s) but rgb/ holds 197;
• for each of depth/, segmentation/, bounding_boxes/ that exists: segmentation/ holds 0 file(s) but rgb/ holds 197 — each must match the rgb/ count.

shellcaasi synth validate latest
2 issue(s) found:
  • metadata declares 200 episode(s) but rgb/ holds 197
  • segmentation/ holds 0 file(s) but rgb/ holds 197
exit code: 1
caasi synth validate latest --json
{
  "path": "/home/you/.caasi/datasets/sdg-dome-20260907-101500",
  "valid": false,
  "counts": { "rgb": 197, "depth": 197, "segmentation": 0, "bounding_boxes": 197, "metadata": 3 },
  "issues": ["metadata declares 200 episode(s) but rgb/ holds 197",
             "segmentation/ holds 0 file(s) but rgb/ holds 197"]
}
exit code: 1

A consistent dataset prints Dataset at <path> is consistent. and exits 0; any issue exits 1 — in both human and JSON mode.

caasi teleop

Drive a robot, record demonstrations, replay recorded bags. Every teleop command delegates: each backend is an upstream ROS package (or an upstream script) that Caasi only resolves and launches — ros2 launch for the keyboard/joystick stacks, the resolved Isaac Sim XR script, and ros2 bag record/play for demonstrations. Caasi implements no teleoperation itself.

The group exposes exactly four verbs — start, record, stop, replay; there is no teleop status. What is installed lives in the teleop catalog domain:

shellcaasi config catalog teleop
Teleoperation teleop · ros
capability  upstream targets       launch file                        caasi group
keyboard    teleop_twist_keyboard  teleop_twist_keyboard              —
                                   teleop-launch.py
joy         teleop_twist_joy, joy  teleop_twist_joy                   —
                                   teleop-launch.py
xr          isaac-sim.xr.vr.sh     —                                  —
record      rosbag2                —                                  —

Every field is overridable: `caasi config set catalog.<domain>.<capability>.<field> <value>`

caasi teleop start

caasi teleop start [CONFIG.yaml] [ARGS…] [--backend|-b CAPABILITY] [--device PATH]
            [--name NAME] [--dry-run]

Two modes, decided by the first positional argument:

• Stack mode (no YAML): the backend is picked --backend → the catalog.teleop.default setting → the first installed of keyboard, joy, xr (in that order). Launch-file stacks run through ros2 launch <package> <launch-file> and --device becomes the launch argument device:=<path>; a resolved script such as Isaac Sim's isaac-sim.xr.vr.sh runs directly, with --device as its first positional argument.
• Experiment mode: when the first argument ends in .yaml/.yml, that experiment YAML launches the robot's sim through the normal experiment runner.

Everything unrecognized is passed through to the stack or the experiment. The run is tracked with kind: teleop (backend ros for launch-file stacks, otherwise the resolved tool); --name overrides the run name, which defaults to the capability key or the experiment name.

shellcaasi teleop start -b joy --device /dev/input/js0 --dry-run
Dry run — nothing was started:
  command: /opt/ros/jazzy/bin/ros2 launch teleop_twist_joy teleop-launch.py device:=/dev/input/js0
caasi teleop start
Teleop started: 20260907-210145-keyboard
  Follow it with: caasi logs 20260907-210145-keyboard -f

On a machine where nothing is installed, the real failures are:

shellcaasi teleop start -b keyboard --dry-run
Error: 'keyboard' is not available (teleop_twist_keyboard). Install it, or point Caasi at the renamed upstream with `caasi config set catalog.teleop.keyboard.packages [...]`.
exit code: 1
caasi teleop start --dry-run
Error: No teleop capability is installed. Install it, or point Caasi at the renamed upstream with `caasi config set catalog.teleop.<capability>.<field> [...]`.
exit code: 1

caasi teleop record

caasi teleop record [--topics|-t TOPIC]… [--name NAME] [--dry-run] [-- EXTRA…]

ros2 bag record into a dataset directory. -t is repeatable; with no topics everything is recorded (-a). Unrecognized arguments are appended to the ros2 bag record command. The ros2 CLI is required — otherwise Error: ros2 CLI not found; source a ROS 2 distro first. (exit 1); when the current shell has not sourced a distro, the run's environment sources the detected one.

The destination is <name-slug>-<YYYYmmdd-HHMMSS> under the usual datasets base (--name defaults to teleop), and metadata.json is written at launch — the bridge that makes a recording a first-class dataset for dataset inspect:

{
  "name": "pick-place",
  "created": "2026-09-07T21:05:12+08:00",
  "kind": "teleop",
  "topics": ["/cmd_vel", "/joint_states"],
  "status": "recording",
  "run_id": "20260907-210512-pick-place"
}

The run is tracked with kind: bag, backend: ros2 and extra: {topics, dataset}.

shellcaasi teleop record --dry-run
Dry run — nothing was started:
  command: /opt/ros/jazzy/bin/ros2 bag record -a -o /home/you/.caasi/datasets/teleop-20260907-204831
  dest:    /home/you/.caasi/datasets/teleop-20260907-204831
caasi teleop record -t /cmd_vel -t /joint_states --name pick-place
Recording demonstrations: /home/you/.caasi/datasets/pick-place-20260907-210512
  Follow it with: caasi logs 20260907-210512-pick-place -f

caasi teleop stop

caasi teleop stop

Stops every running/paused run of kind teleop or bag, newest first — one line per run; a no-op when nothing is active. This is how a recording is closed out.

shellcaasi teleop stop
Stopped 20260907-210145-keyboard (teleop).
Stopped 20260907-210512-pick-place (bag).
caasi teleop stop
No active teleop or recording runs.

caasi teleop replay

caasi teleop replay BAG [--dry-run] [-- EXTRA…]

Publishes a recorded bag again with ros2 bag play, as a tracked run (kind: replay, backend: ros2). BAG is the path to a recorded bag directory — the dataset directory teleop record wrote works as-is, since ros2 bag record -o pointed at it. Unrecognized arguments are appended to ros2 bag play. Distinct from caasi replay, which opens recorded run artifacts in a viewer.

shellcaasi teleop replay datasets/pick-place-20260907-210512 --dry-run
Dry run — nothing was started:
  command: /opt/ros/jazzy/bin/ros2 bag play datasets/pick-place-20260907-210512
caasi teleop replay /tmp/nonexistent-bag
Error: Bag '/tmp/nonexistent-bag' not found.
exit code: 1

Record → inspect → train → evaluate

The point of datasets being directories with a metadata.json: every producer writes to the same convention, so the consumers need no glue. The chain, in real commands:

1. Produce. Demonstrations: caasi teleop record (above). Synthetic frames: caasi synth generate (above). Simulation-collected data: caasi dataset generate. Existing data: caasi dataset download.
2. Inspect. caasi teleop stop closes the recording, then caasi dataset inspect latest shows metadata, on-disk contents and the linked run; caasi synth preview latest (above) counts produced episodes for SDG sets. Gate with caasi dataset validate or caasi synth validate (above).
3. Train. caasi lab train for Isaac Lab experiments, or caasi train for any experiment — both start tracked background runs.
4. Evaluate. caasi lab evaluate runs the experiment's evaluate_script against a checkpoint; caasi benchmark start measures sim throughput. Review the recording itself with caasi teleop replay, or run artifacts with caasi replay.

shellcaasi teleop record -t /cmd_vel -t /joint_states --name pick-place
Recording demonstrations: /home/you/demo/datasets/pick-place-20260907-210512
  Follow it with: caasi logs 20260907-210512-pick-place -f
# …drive the robot, then close the recording…
caasi teleop stop
Stopped 20260907-210512-pick-place (bag).
caasi dataset inspect latest
Dataset /home/you/demo/datasets/pick-place-20260907-210512
  name: pick-place
  created: 2026-09-07T21:05:12+08:00
  kind: teleop
  topics: ['/cmd_vel', '/joint_states']
  status: recording
  run_id: 20260907-210512-pick-place
  Run: 20260907-210512-pick-place (stopped)
caasi lab train experiments/pick.yaml --steps 100000 --device cuda:0
Run 20260907-212000-pick started in the background.
  Follow it with: caasi logs 20260907-212000-pick -f
caasi lab evaluate experiments/pick.yaml --checkpoint runs/20260907-212000-pick/model_000100000.pt
Run 20260907-224500-pick started in the background.
  Follow it with: caasi logs 20260907-224500-pick -f

caasi dataset download

caasi dataset download REF [--backend hf|ngc|url] [--name NAME] [--dry-run] [-- EXTRA…]

Downloads an existing dataset as a tracked run. The backend is auto-detected from the reference's scheme; --backend forces one. Each backend resolves to an upstream download tool and builds its command:

Failures are honest and early (both exit 1):

shellcaasi dataset download ftp://example.com/x
Error: Cannot detect a download backend for 'ftp://example.com/x' (use hf://, ngc://, http(s):// or --backend hf|ngc|url).
caasi dataset download hf://lerobot/aloha_sim
Error: No download tool found for backend 'hf'.

The dataset name is --name, or the last path segment of the source — hf://lerobot/aloha_sim becomes aloha-sim-<timestamp>. The run is tracked with kind: dataset and backend set to the detected download backend, and metadata.json is written at launch:

{
  "name": "aloha-sim",
  "created": "2026-09-07T21:00:00+08:00",
  "source": "hf://lerobot/aloha_sim",
  "backend": "hf",
  "status": "downloading",
  "run_id": "20260907-210000-aloha-sim"
}

shellcaasi dataset download hf://lerobot/aloha_sim --dry-run
Dry run — nothing was started:
  command: hf download lerobot/aloha_sim --local-dir /home/you/.caasi/datasets/aloha-sim-20260907-205053
  dest:    /home/you/.caasi/datasets/aloha-sim-20260907-205053
caasi dataset download https://example.com/data.tar.gz --dry-run
Dry run — nothing was started:
  command: /usr/bin/curl -L -o /home/you/.caasi/datasets/data-tar-gz-20260907-205053/data.tar.gz https://example.com/data.tar.gz
  dest:    /home/you/.caasi/datasets/data-tar-gz-20260907-205053
caasi dataset download hf://lerobot/aloha_sim
Dataset download started: /home/you/.caasi/datasets/aloha-sim-20260907-210000
  Run: 20260907-210000-aloha-sim — follow it with 'caasi logs 20260907-210000-aloha-sim -f'

Unrecognized flags pass straight through to the tool, between the source and the destination flags — caasi dataset download hf://lerobot/aloha_sim --revision main runs hf download lerobot/aloha_sim --revision main --local-dir <dataset-dir>. Dry runs need no tool installed and leave no directories behind.

Once the files land, the directory is an ordinary dataset: dataset inspect, dataset convert and dataset validate all work on it, and dataset list shows its source and status.