Caasi v0.1.0 Training: train · benchmark start · benchmark report

Training & Benchmarking

Headless-first training and physics throughput benchmarks. Both take the same experiment YAML as sim run, translate purpose-built flags into script arguments, and launch the result as a detached tracked run you can follow, pause, stop or report on from any terminal.

caasi train

caasi train CONFIG_PATH [--steps N] [--envs N] [--resume PATH] [--seed N]
            [--device TEXT] [--dry-run] [-- SCRIPT_ARGS…]
ParameterKindTypeDefaultDescription
CONFIG_PATHargumentpathrequiredExperiment YAML (backend: is typically lab).
--stepsoptionint—Training steps/iterations → passed to the script as --steps N.
--envsoptionint—Parallel environments → --envs N.
--resumeoptionpath—Checkpoint to resume from → --resume PATH.
--seedoptionint—Random seed → --seed N.
--deviceoptionstr—e.g. cuda:0 → --device cuda:0.
--dry-runoptionflagoffShow the launch command without starting.
trailing argspass-throughAppended verbatim after the translated flags.

How flags become script arguments

Caasi does not train anything — your script does. train builds the argv in this order, keeping only what you provided:

--steps N  --envs N  --resume PATH  --seed N  --device D  --headless  <your trailing args>

--headless is appended automatically when the experiment YAML has headless: true (the default). Interpreter/backend resolution is identical to sim run (see experiment YAML).

Example

shellcaasi train experiments/ant.yaml --steps 500000 --envs 4096 --device cuda:0 --dry-run
Dry run — nothing was started:
  command: /opt/IsaacLab/isaaclab.sh -p /opt/IsaacLab/scripts/train_ant.py --steps 500000 --envs 4096 --device cuda:0 --headless
  cwd: /home/you/demo/experiments
caasi train experiments/ant.yaml --steps 500000 --envs 4096
Run 20260905-151207-ant-train started in the background.
  Follow it with: caasi logs 20260905-151207-ant-train -f
caasi logs latest -f
…learning iteration 12/1500  mean reward 87.2 …

The run is recorded with kind: train and extra: {experiment, steps, envs} in its manifest, so you can tell training runs apart in caasi run list --json.

No JSON output

train itself has no JSON mode — use --dry-run to see the command, then caasi run status <id> --json / caasi logs for everything after launch. Exit codes: 0 launched (or dry-run), 1 config/backend error. The training outcome surfaces later as the run's succeeded/failed status.

caasi benchmark

Benchmarks answer “how fast is my physics actually stepping?” — no viewport, no rendering, just throughput. benchmark start launches the run; benchmark report later parses the metrics back out of its log.

caasi benchmark start

caasi benchmark start CONFIG_PATH [--envs N] [--steps N] [--dry-run] [-- SCRIPT_ARGS…]
ParameterKindTypeDefaultDescription
CONFIG_PATHargumentpathrequiredBenchmark experiment YAML.
--envsoptionint—Number of parallel environments → --envs N.
--stepsoptionint—Physics steps to benchmark → --steps N.
--dry-runoptionflagoffShow the launch command without starting.
 
shellcaasi benchmark start experiments/fps.yaml --envs 4096 --steps 20000
Run 20260905-160112-fps started in the background.
  When finished: caasi benchmark report 20260905-160112-fps

Argv construction mirrors train (--envs, --steps, --headless, then pass-through). Run kind: benchmark. Exit codes: 0 launched / dry-run, 1 config error.

caasi benchmark report

caasi benchmark report QUERY [--json]

Reads the run's stdout.log and extracts metrics using two conventions — your benchmark script can use either (or both):

  1. Explicit metric lines (recommended): print caasi_metric <name>=<value> anywhere in a line. Names allow letters, digits, _ . -; values are ints/floats (exponents ok).
  2. Common labelled lines: Simulation FPS: 241.5, Steps/sec: 61696, Real-time factor: 12.3, GPU utilization: 94 %, GPU memory: 17890 — recognized labels are normalized to simulation_fps, steps_per_sec, real_time_factor, gpu_utilization, gpu_memory.

Example benchmark script output

# inside your benchmark script:
print(f"caasi_metric simulation_fps={fps:.1f}")
print(f"caasi_metric steps_per_sec={sps}")
# …or simply:
print("Simulation FPS: 241.5")

Report

shellcaasi benchmark report latest
fps (20260905-160112-fps) — succeeded
      Benchmark Metrics
  Metric              Value
  simulation_fps      241.5
  steps_per_sec       61696
  real_time_factor    12.3
caasi benchmark report latest --json
{
  "id": "20260905-160112-fps",
  "status": "succeeded",
  "metrics": {
    "simulation_fps": 241.5,
    "steps_per_sec": 61696.0,
    "real_time_factor": 12.3
  }
}

Reporting on a run that is still running/paused warns Still running — metrics may be incomplete. but proceeds. If no metrics are found you get a hint about the two conventions.

Exit codes

SituationExit
Metrics parsed (or still-running report), run not failed0
Run status is failed (JSON mode or no-metrics mode)1
Run not found1

Full workflow

experiments/fps.yamlname: fps-bench
backend: lab
script: source/standalone/benchmarks/throughput.py
headless: true
shell — compare two environment countscaasi benchmark start experiments/fps.yaml --envs 1024 --steps 20000
Run 20260905-161204-fps-bench started in the background.
while [ "$(caasi run status fps-bench --json | jq -r .status)" = "running" ]; do sleep 5; done
caasi benchmark report fps-bench --json | jq .metrics
{ "simulation_fps": 241.5, "steps_per_sec": 61696.0 }
caasi benchmark start experiments/fps.yaml --envs 4096 --steps 20000
Run 20260905-162011-fps-bench started in the background.
caasi benchmark report latest
fps-bench (20260905-162011-fps-bench) — succeeded
  Metric              Value
  simulation_fps      612.4
  steps_per_sec       156774

The run name always comes from the experiment's name: field — benchmark start has no --name flag — so that name is a stable query, and latest always means the newest run.