Remote & Container

Same workflows, different machine. caasi remote drives the system ssh client to run experiments on a GPU box; caasi container drives docker or podman to run them inside an official image. Caasi implements no protocol and links no SDK — it builds command lines, and every long-running one becomes a tracked run.

caasi remote

Declaring remotes

Remotes live in your configuration — global (~/.config/caasi/config.yaml) or per-project (./caasi.yaml):

~/.config/caasi/config.yamlremotes:
  gpu-box:
    host: 192.168.1.50
    user: robot
    port: 22
    identity: ~/.ssh/id_ed25519
    path: ~/experiments
  cloud-a100:
    host: a100.example.com
    user: ubuntu
    identity: ~/.ssh/cloud.pem

Add one from the command line without editing YAML:

shellcaasi config set remotes.gpu-box.host 192.168.1.50
Set remotes.gpu-box.host = 192.168.1.50
caasi config set remotes.gpu-box.user robot
Set remotes.gpu-box.user = robot
caasi config set remotes.gpu-box.path '~/experiments'
Set remotes.gpu-box.path = ~/experiments

caasi remote list

caasi remote list [--json]

Prints the configured machines: target, port and working directory.

shellcaasi remote list
Remote machines
  gpu-box          robot@192.168.1.50:22  ~/experiments
  cloud-a100       ubuntu@a100.example.com
caasi remote list --json
[
  { "name": "gpu-box", "host": "192.168.1.50", "user": "robot",
    "port": 22, "identity": "~/.ssh/id_ed25519", "path": "~/experiments" },
  { "name": "cloud-a100", "host": "a100.example.com", "user": "ubuntu",
    "port": null, "identity": "~/.ssh/cloud.pem", "path": null }
]

Nothing configured is not an error — it is a hint:

shellcaasi remote list
No remote machines configured; add one with 'caasi config set remotes.<name>.host <host>'.

caasi remote connect

caasi remote connect NAME [--command|-c TEXT] [--dry-run]

Builds the ssh invocation from the machine entry:

ssh [-p PORT] [-i IDENTITY] [USER@]HOST [COMMAND]

Interactive (no -c): Caasi prints Opening interactive session to robot@192.168.1.50... and then replaces its own process with ssh (os.execvp). You get a real TTY — password prompts, tmux, htop, GUI forwarding all work exactly as with plain ssh. Exiting returns you to your original shell.

shellcaasi remote connect gpu-box --dry-run
Dry run — nothing was launched:
  command: ssh -p 22 -i /home/robot/.ssh/id_ed25519 robot@192.168.1.50
caasi remote connect gpu-box
Opening interactive session to robot@192.168.1.50...
robot@gpu-box:~$ nvidia-smi --query-gpu=name --format=csv,noheader
NVIDIA A100-SXM4-80GB
robot@gpu-box:~$ exit

Batch (-c): one command, output printed, exit code mirrored (negative → 1) — ideal for checks and scripts:

shellcaasi remote connect gpu-box -c 'nvidia-smi --query-gpu=name,memory.total --format=csv,noheader'
NVIDIA A100-SXM4-80GB, 81920 MiB
caasi remote connect gpu-box -c 'df -h ~/experiments | tail -1'
/dev/nvme0n1p1  1.8T  640G  1.1T  37% /home/robot/experiments
caasi remote connect gpu-box -c 'test -f ~/experiments/train.py'
exit code: 1   # file missing — the remote exit code, mirrored

caasi remote run

caasi remote run NAME [--dry-run] [--json] -- COMMAND [ARGS…]

Starts a command on the remote machine as a detached, tracked run. Everything after NAME (and after Caasi's own flags) is the remote command:

ssh [-p PORT] [-i IDENTITY] [USER@]HOST  "cd <path> && <your command>"

The cd is only added when the machine declares a path; arguments are quoted with shlex.join, so spaces and globs survive the trip.

shellcaasi remote run gpu-box --dry-run python train.py --headless --steps 10000
Dry run — nothing was launched:
  command: ssh -p 22 -i /home/robot/.ssh/id_ed25519 robot@192.168.1.50 cd ~/experiments && python train.py --headless --steps 10000
caasi remote run gpu-box python train.py --headless --steps 10000
Remote run started: 20260905-142310-gpu-box-python.
Follow it with: caasi logs 20260905-142310-gpu-box-python -f

A run with no command is refused rather than silently opening a session:

shellcaasi remote run gpu-box
Error: Provide a command to run, e.g.: caasi remote run <name> python train.py
exit code: 1

The record it creates:

shellcaasi remote run gpu-box python train.py --headless --json
{
  "id": "20260905-143012-gpu-box-python",
  "name": "gpu-box-python",
  "backend": "ssh",
  "kind": "remote",
  "command": ["ssh", "-p", "22", "-i", "/home/robot/.ssh/id_ed25519",
              "robot@192.168.1.50", "cd ~/experiments && python train.py --headless"],
  "cwd": "/home/robot/work",
  "created": "2026-09-05T14:30:12+08:00",
  "pid": 48213,
  "paused": false,
  "stopped": false,
  "directory": "/home/robot/.caasi/runs/20260905-143012-gpu-box-python",
  "status": "running"
}

From here the whole run toolbox applies — the logs are the ssh stream, written locally as it arrives:

shellcaasi logs gpu-box-python -f
epoch 1/50  loss 2.41  fps 812
epoch 2/50  loss 1.98  fps 830
^C
caasi run status gpu-box-python
Run gpu-box-python
  Id        20260905-142310-gpu-box-python
  Status    running
  Backend   ssh   Kind remote   Pid 48122
caasi run stop gpu-box-python
Run 20260905-142310-gpu-box-python stopped.

shellcaasi remote connect gpu-box -c 'cd ~/experiments && nohup python train.py --headless > train.log 2>&1 & echo $!'
51244
caasi remote connect gpu-box -c 'tail -3 ~/experiments/train.log'
epoch 47/50  loss 0.31  fps 841

caasi container

Runtime discovery is one line: docker on PATH, else podman. If neither exists, list says so in yellow, check fails, and run exits 1 with Error: No container tool found; install Docker or Podman first.

caasi container list

caasi container list [--json]

Shows the well-known Isaac images (repository references, tags deliberately not pinned) plus whatever Isaac-related image you already have locally.

shellcaasi container list
Containers  (tool: docker)
  Known images:
    nvcr.io/nvidia/isaac-sim             NVIDIA Isaac Sim (NGC)
    nvcr.io/nvidia/isaac-lab             NVIDIA Isaac Lab (NGC)
  Local Isaac images:
    nvcr.io/nvidia/isaac-sim:5.1.0
    nvcr.io/nvidia/isaac-lab:2.3.0
caasi container list --json
{
  "tool": "/usr/bin/docker",
  "known": ["nvcr.io/nvidia/isaac-sim", "nvcr.io/nvidia/isaac-lab"],
  "local": ["nvcr.io/nvidia/isaac-sim:5.1.0", "nvcr.io/nvidia/isaac-lab:2.3.0"]
}

Local images come from <tool> image ls --format '{{.Repository}}:{{.Tag}}', keeping only entries whose name contains isaac, isaacsim or isaaclab (dangling <none> tags are dropped). Nothing pulled yet → (none pulled yet); no runtime → "tool": null and "local": [] in JSON.

caasi container check

caasi container check [--json]

Readiness gate for GPU containers — three checks, run in order, each 20 s timeout:

shellcaasi container check
✓ tool              /usr/bin/docker
✓ daemon            daemon is responding
✓ nvidia-runtime    NVIDIA runtime available
Container stack is ready.
exit code: 0
caasi container check --json
{ "ok": false, "checks": [
    { "check": "tool", "ok": true, "detail": "/usr/bin/docker" },
    { "check": "daemon", "ok": true, "detail": "daemon is responding" },
    { "check": "nvidia-runtime", "ok": false,
      "detail": "NVIDIA runtime missing (install nvidia-container-toolkit)" } ] }
exit code: 1

The nvidia-runtime check is skipped (reported false) when the daemon is down. Exit 0 only when all three pass, in both human and JSON mode — so it composes:

shellcaasi container check && caasi container run nvcr.io/nvidia/isaac-sim:5.1.0 ./runheadless.sh

caasi container run

caasi container run IMAGE [--gpus all] [--dry-run] [--json] [CONTAINER COMMAND…]

Builds and launches, as a tracked run:

<docker|podman> run --rm [--gpus VALUE] IMAGE [CONTAINER COMMAND…]

The container is always started with --rm, so it cleans itself up when the run ends. The run is named after the image: last path component, : → - (nvcr.io/nvidia/isaac-sim:5.1.0 → isaac-sim-5.1.0), with backend: container, kind: container and extra: {image, gpus} in the manifest.

shellcaasi container run nvcr.io/nvidia/isaac-sim:5.1.0 --dry-run ./runheadless.sh --allow-root
Dry run — nothing was launched:
  command: /usr/bin/docker run --rm --gpus all nvcr.io/nvidia/isaac-sim:5.1.0 ./runheadless.sh --allow-root
caasi container run nvcr.io/nvidia/isaac-sim:5.1.0 ./runheadless.sh --allow-root
Container run started: 20260905-150211-isaac-sim-5-1-0.
Follow it with: caasi logs 20260905-150211-isaac-sim-5-1-0 -f
caasi container run nvcr.io/nvidia/isaac-lab:2.3.0 --gpus device=0 --json
{
  "id": "20260905-150433-isaac-lab-2-3-0",
  "name": "isaac-lab-2.3.0",
  "backend": "container",
  "kind": "container",
  "command": ["/usr/bin/docker", "run", "--rm", "--gpus", "device=0",
              "nvcr.io/nvidia/isaac-lab:2.3.0"],
  "cwd": "/home/robot/work",
  "created": "2026-09-05T15:04:33+08:00",
  "pid": 51902,
  "paused": false,
  "stopped": false,
  "directory": "/home/robot/.caasi/runs/20260905-150433-isaac-lab-2-3-0",
  "status": "running"
}

shellcaasi container run nvcr.io/nvidia/isaac-sim:5.1.0 --dry-run
Dry run — nothing was launched:
  command: /usr/bin/docker run --rm --gpus all nvcr.io/nvidia/isaac-sim:5.1.0
docker run --rm --gpus all -v "$PWD":/work -p 8211:8211 nvcr.io/nvidia/isaac-sim:5.1.0 ./runheadless.sh

End-to-end: headless Isaac Sim in a container

shellcaasi container check
Container stack is ready.
caasi container list
  Local Isaac images:
    (none pulled yet)
docker pull nvcr.io/nvidia/isaac-sim:5.1.0
5.1.0: Pulling from nvidia/isaac-sim ...
caasi container run nvcr.io/nvidia/isaac-sim:5.1.0 ./runheadless.sh --allow-root
Container run started: 20260905-151002-isaac-sim-5-1-0.
caasi logs latest -f
[0.000s] Simulation App Starting...
[18.204s] Streaming server started on port 8211
caasi run status isaac-sim-5.1.0 --json | jq '{status, pid}'
{ "status": "running", "pid": 52310 }
caasi run stop 20260905-1510
Run 20260905-151002-isaac-sim-5-1-0 stopped.

Queries resolve as latest, the exact run name (isaac-sim-5.1.0, the image tail with : → -), or a unique id prefix — note that ids start with the timestamp, so a bare isaac-sim matches nothing. See QUERY semantics.

Local, remote, container

That uniformity is the point: whether the work happens on this machine, across SSH or inside a container, you inspect it with the same caasi run and caasi logs commands.