Remote & Container

Same workflows, different machine. caasi remote drives the system ssh client to run experiments on a GPU box; caasi container drives docker or podman to run them inside an official image. Caasi implements no protocol and links no SDK — it builds command lines, and every long-running one becomes a tracked run.

caasi remote

Declaring remotes

Remotes live in your configuration — global (~/.config/caasi/config.yaml) or per-project (./caasi.yaml):

~/.config/caasi/config.yamlremotes:
  gpu-box:
    host: 192.168.1.50
    user: robot
    port: 22
    identity: ~/.ssh/id_ed25519
    path: ~/experiments
  cloud-a100:
    host: a100.example.com
    user: ubuntu
    identity: ~/.ssh/cloud.pem

Field	Required	Meaning
`host`	yes	hostname or IP — entries without it are skipped entirely
`user`	no	SSH user; with it the target becomes `user@host`
`port`	no	adds `-p PORT`
`identity`	no	key file, `~` expanded, adds `-i FILE`
`path`	no	working directory for `remote run` (`cd PATH && …`); shown by `remote list`

Add one from the command line without editing YAML:

shellcaasi config set remotes.gpu-box.host 192.168.1.50
Set remotes.gpu-box.host = 192.168.1.50
caasi config set remotes.gpu-box.user robot
Set remotes.gpu-box.user = robot
caasi config set remotes.gpu-box.path '~/experiments'
Set remotes.gpu-box.path = ~/experiments

Name resolution

Every command that takes NAME accepts the exact name or a unique prefix: with gpu-box and cloud-a100 configured, caasi remote connect gpu resolves to gpu-box. An ambiguous or unknown query fails with Error: No remote machine matching 'g' (configured: gpu-box, cloud-a100). and exit 1.

caasi remote list

caasi remote list [--json]

Prints the configured machines: target, port and working directory.

shellcaasi remote list
Remote machines
  gpu-box          robot@192.168.1.50:22  ~/experiments
  cloud-a100       ubuntu@a100.example.com
caasi remote list --json
[
  { "name": "gpu-box", "host": "192.168.1.50", "user": "robot",
    "port": 22, "identity": "~/.ssh/id_ed25519", "path": "~/experiments" },
  { "name": "cloud-a100", "host": "a100.example.com", "user": "ubuntu",
    "port": null, "identity": "~/.ssh/cloud.pem", "path": null }
]

Nothing configured is not an error — it is a hint:

shellcaasi remote list
No remote machines configured; add one with 'caasi config set remotes.<name>.host <host>'.

caasi remote connect

caasi remote connect NAME [--command|-c TEXT] [--dry-run]

Builds the ssh invocation from the machine entry:

ssh [-p PORT] [-i IDENTITY] [USER@]HOST [COMMAND]

Option	Default	Effect
`--command`, `-c`	none	run one command instead of an interactive session; output captured, exit code mirrored
`--dry-run`	`false`	print the exact `ssh` command line and do nothing

Interactive (no -c): Caasi prints Opening interactive session to robot@192.168.1.50... and then replaces its own process with ssh (os.execvp). You get a real TTY — password prompts, tmux, htop, GUI forwarding all work exactly as with plain ssh. Exiting returns you to your original shell.

shellcaasi remote connect gpu-box --dry-run
Dry run — nothing was launched:
  command: ssh -p 22 -i /home/robot/.ssh/id_ed25519 robot@192.168.1.50
caasi remote connect gpu-box
Opening interactive session to robot@192.168.1.50...
robot@gpu-box:~$ nvidia-smi --query-gpu=name --format=csv,noheader
NVIDIA A100-SXM4-80GB
robot@gpu-box:~$ exit

Batch (-c): one command, output printed, exit code mirrored (negative → 1) — ideal for checks and scripts:

shellcaasi remote connect gpu-box -c 'nvidia-smi --query-gpu=name,memory.total --format=csv,noheader'
NVIDIA A100-SXM4-80GB, 81920 MiB
caasi remote connect gpu-box -c 'df -h ~/experiments | tail -1'
/dev/nvme0n1p1  1.8T  640G  1.1T  37% /home/robot/experiments
caasi remote connect gpu-box -c 'test -f ~/experiments/train.py'
exit code: 1   # file missing — the remote exit code, mirrored

Your SSH setup still applies

Because Caasi only orchestrates the system ssh, everything in ~/.ssh/config (jump hosts, agent forwarding, keep-alives, host aliases) and your ssh-agent work unchanged. You can even set host to an ~/.ssh/config alias.

caasi remote run

caasi remote run NAME [--dry-run] [--json] -- COMMAND [ARGS…]

Starts a command on the remote machine as a detached, tracked run. Everything after NAME (and after Caasi's own flags) is the remote command:

ssh [-p PORT] [-i IDENTITY] [USER@]HOST  "cd <path> && <your command>"

The cd is only added when the machine declares a path; arguments are quoted with shlex.join, so spaces and globs survive the trip.

shellcaasi remote run gpu-box --dry-run python train.py --headless --steps 10000
Dry run — nothing was launched:
  command: ssh -p 22 -i /home/robot/.ssh/id_ed25519 robot@192.168.1.50 cd ~/experiments && python train.py --headless --steps 10000
caasi remote run gpu-box python train.py --headless --steps 10000
Remote run started: 20260905-142310-gpu-box-python.
Follow it with: caasi logs 20260905-142310-gpu-box-python -f

A run with no command is refused rather than silently opening a session:

shellcaasi remote run gpu-box
Error: Provide a command to run, e.g.: caasi remote run <name> python train.py
exit code: 1

The record it creates:

Field	Value
`name`	`<machine>-<first token of your command>` → `gpu-box-python`
`backend` / `kind`	`ssh` / `remote`
`extra` (manifest only)	`{remote: gpu-box, remote_command: "python train.py --headless --steps 10000"}`
`pid`	the local `ssh` client's PID — that is what `run stop` signals

shellcaasi remote run gpu-box python train.py --headless --json
{
  "id": "20260905-143012-gpu-box-python",
  "name": "gpu-box-python",
  "backend": "ssh",
  "kind": "remote",
  "command": ["ssh", "-p", "22", "-i", "/home/robot/.ssh/id_ed25519",
              "robot@192.168.1.50", "cd ~/experiments && python train.py --headless"],
  "cwd": "/home/robot/work",
  "created": "2026-09-05T14:30:12+08:00",
  "pid": 48213,
  "paused": false,
  "stopped": false,
  "directory": "/home/robot/.caasi/runs/20260905-143012-gpu-box-python",
  "status": "running"
}

From here the whole run toolbox applies — the logs are the ssh stream, written locally as it arrives:

shellcaasi logs gpu-box-python -f
epoch 1/50  loss 2.41  fps 812
epoch 2/50  loss 1.98  fps 830
^C
caasi run status gpu-box-python
Run gpu-box-python
  Id        20260905-142310-gpu-box-python
  Status    running
  Backend   ssh   Kind remote   Pid 48122
caasi run stop gpu-box-python
Run 20260905-142310-gpu-box-python stopped.

The run is the ssh connection

Caasi tracks the local ssh process, so the run ends when the connection ends — and a dropped link can leave the remote process running while the local run is marked failed. For long training jobs, start them inside tmux/nohup on the remote side and poll instead:

shellcaasi remote connect gpu-box -c 'cd ~/experiments && nohup python train.py --headless > train.log 2>&1 & echo $!'
51244
caasi remote connect gpu-box -c 'tail -3 ~/experiments/train.log'
epoch 47/50  loss 0.31  fps 841

caasi container

Runtime discovery is one line: docker on PATH, else podman. If neither exists, list says so in yellow, check fails, and run exits 1 with Error: No container tool found; install Docker or Podman first.

Subcommand	What it does
`list`	List known and locally available Isaac container images.
`status`	Container runtime status (tool, daemon, NVIDIA runtime, images).
`check`	Check container readiness (tool, daemon, GPU runtime).
`doctor`	Check container readiness (runtimes, GPU, guidance).
`run`	Run a command inside a container image as a tracked run.

caasi container list

caasi container list [--json]

Shows the well-known Isaac images (repository references, tags deliberately not pinned) plus whatever Isaac-related image you already have locally.

shellcaasi container list
Containers  (tool: docker)
  Known images:
    nvcr.io/nvidia/isaac-sim             NVIDIA Isaac Sim (NGC)
    nvcr.io/nvidia/isaac-lab             NVIDIA Isaac Lab (NGC)
  Local Isaac images:
    nvcr.io/nvidia/isaac-sim:5.1.0
    nvcr.io/nvidia/isaac-lab:2.3.0
caasi container list --json
{
  "tool": "/usr/bin/docker",
  "known": ["nvcr.io/nvidia/isaac-sim", "nvcr.io/nvidia/isaac-lab"],
  "local": ["nvcr.io/nvidia/isaac-sim:5.1.0", "nvcr.io/nvidia/isaac-lab:2.3.0"]
}

Local images come from <tool> image ls --format '{{.Repository}}:{{.Tag}}', keeping only entries whose name contains isaac, isaacsim or isaaclab (dangling <none> tags are dropped). Nothing pulled yet → (none pulled yet); no runtime → "tool": null and "local": [] in JSON.

caasi container status

caasi container status [--json]

list answers “which images?”; status answers “can I run anything at all?”. Five rows, from the same probes check uses (<tool> info, 20 s timeout):

Row	Value
`Runtime`	the tool's basename (`docker` / `podman`), or `(none)`
`Daemon`	`daemon is responding` / `daemon is not responding`
`NVIDIA runtime`	`NVIDIA runtime available` / `NVIDIA runtime missing (install nvidia-container-toolkit)`
`Known images`	how many well-known Isaac images Caasi knows — `2`
`Local images`	the local Isaac images, comma-separated, or `(none pulled yet)`

shellcaasi container status
Containers
  Runtime          docker
  Daemon           daemon is responding
  NVIDIA runtime   NVIDIA runtime available
  Known images     2
  Local images     (none pulled yet)
caasi container status --json | jq '{tool, daemon, nvidia_runtime, ready}'
{
  "tool": "/usr/bin/docker",
  "daemon": true,
  "nvidia_runtime": true,
  "ready": true
}

The full payload is {"tool", "daemon", "nvidia_runtime", "known", "local", "ready"} — tool is the resolved absolute path or null, known the image references, local what is already pulled. ready is true when a tool exists and its daemon answers; the NVIDIA runtime is deliberately not part of it. When no tool resolves, the daemon and image probes are skipped (not run), Runtime prints (none) and the yellow No container tool found; install Docker or Podman first. hint is appended.

Always exits 0 — status is a report, not a gate. For a gate use check; for the reasons behind it, doctor.

caasi container check

caasi container check [--json]

Readiness gate for GPU containers — three checks, run in order, each 20 s timeout:

Check	Passes when	Detail on failure
`tool`	`docker` or `podman` is on `PATH`	`No container tool found; install Docker or Podman first.`
`daemon`	`<tool> info` exits 0	`daemon is not responding` — is the service started? are you in the `docker` group?
`nvidia-runtime`	`<tool> info` mentions `nvidia`	`NVIDIA runtime missing (install nvidia-container-toolkit)`

shellcaasi container check
✓ tool              /usr/bin/docker
✓ daemon            daemon is responding
✓ nvidia-runtime    NVIDIA runtime available
Container stack is ready.
exit code: 0
caasi container check --json
{ "ok": false, "checks": [
    { "check": "tool", "ok": true, "detail": "/usr/bin/docker" },
    { "check": "daemon", "ok": true, "detail": "daemon is responding" },
    { "check": "nvidia-runtime", "ok": false,
      "detail": "NVIDIA runtime missing (install nvidia-container-toolkit)" } ] }
exit code: 1

The nvidia-runtime check is skipped (reported false) when the daemon is down. Exit 0 only when all three pass, in both human and JSON mode — so it composes:

shellcaasi container check && caasi container run nvcr.io/nvidia/isaac-sim:5.1.0 ./runheadless.sh

caasi container doctor

caasi container doctor [--verbose] [--json]

The diagnostic view of the same stack: two check sections — containers and nvidia — plus one container-specific extra, GPU in containers. It uses the shared doctor renderer, so the symbols (✓ ! ✗ •) and the hint rules are the ones you know from caasi doctor: hints print for failures and warnings, and --verbose adds them for passing checks too.

containers — a Docker and a Podman row, each probed with <tool> --version (5 s): ok with the version line, skip when the binary is absent, warn when the probe fails. When both are missing, a third Container runtime row warns no container runtime found with the hint Needed only for 'caasi container' workflows.
nvidia — the driver, one row per GPU and CUDA, all read from nvidia-smi; a single fail row when nvidia-smi is missing or errors.
GPU in containers — the daemon's NVIDIA runtime, reported as guidance rather than a gate: ok (NVIDIA runtime available, hint Expose the GPU with: caasi container run --gpus all <image>), warn (NVIDIA runtime missing (install nvidia-container-toolkit)), or skip when there is no tool or no responding daemon.

shellcaasi container doctor
✓ Docker — Docker version 29.8.0, build 88096ef
• Podman — not installed
✓ NVIDIA driver — driver 580.173.02
✓ GPU — NVIDIA GeForce RTX 3080, 10.0 GiB VRAM
✓ CUDA — CUDA 13.0 (driver-reported)
✓ GPU in containers — NVIDIA runtime available
exit code: 0
caasi container doctor --json | jq '.sections, .checks[-1], .exit_code'
[
  "containers",
  "nvidia"
]
{
  "section": "containers",
  "name": "GPU in containers",
  "status": "ok",
  "detail": "NVIDIA runtime available",
  "hint": "Expose the GPU with: caasi container run --gpus all <image>"
}
0

Exit code: 1 when any row is a fail, 0 otherwise — in both human and JSON mode (exit_code is in the payload). Warnings and skips never fail it, which is exactly how doctor differs from check:

shell — daemon without nvidia-container-toolkitcaasi container doctor
✓ Docker — Docker version 29.8.0, build 88096ef
• Podman — not installed
✓ NVIDIA driver — driver 580.173.02
✓ GPU — NVIDIA GeForce RTX 3080, 10.0 GiB VRAM
✓ CUDA — CUDA 13.0 (driver-reported)
! GPU in containers — NVIDIA runtime missing (install nvidia-container-toolkit)
    ↳ Install nvidia-container-toolkit and restart the container daemon.
exit code: 0   # a warning does not fail the gate — 'container check' exits 1 here

	`container check`	`container doctor`
Scope	three boolean gates: `tool`, `daemon`, `nvidia-runtime`	the `containers` + `nvidia` sections and the GPU in containers extra
Missing NVIDIA runtime	`ok: false` → exit `1`	`warn` + install hint → exit `0`
No container tool at all	exit `1`	`skip`/`warn` rows; exit `1` only if the `nvidia` section itself fails
JSON	`{"ok", "checks": [{"check", "ok", "detail"}]}`	`{"group": "container", "sections", "checks": [{"section", "name", "status", "detail", "hint"}], "exit_code"}`
Use it for	the CI gate in front of `container run`	finding out why the gate fails, with hints

caasi container run

caasi container run IMAGE [--gpus all] [--dry-run] [--json] [CONTAINER COMMAND…]

Builds and launches, as a tracked run:

<docker|podman> run --rm [--gpus VALUE] IMAGE [CONTAINER COMMAND…]

Option	Default	Effect
`--gpus`	`all`	value passed to `--gpus` (`all`, `2`, `device=0,1`). Pass `--gpus ""` to omit the flag and run without GPU access
`--dry-run`	`false`	print the command, start nothing
`--json`	`false`	print the full run record

The container is always started with --rm, so it cleans itself up when the run ends. The run is named after the image: last path component, : → - (nvcr.io/nvidia/isaac-sim:5.1.0 → isaac-sim-5.1.0), with backend: container, kind: container and extra: {image, gpus} in the manifest.

shellcaasi container run nvcr.io/nvidia/isaac-sim:5.1.0 --dry-run ./runheadless.sh --allow-root
Dry run — nothing was launched:
  command: /usr/bin/docker run --rm --gpus all nvcr.io/nvidia/isaac-sim:5.1.0 ./runheadless.sh --allow-root
caasi container run nvcr.io/nvidia/isaac-sim:5.1.0 ./runheadless.sh --allow-root
Container run started: 20260905-150211-isaac-sim-5-1-0.
Follow it with: caasi logs 20260905-150211-isaac-sim-5-1-0 -f
caasi container run nvcr.io/nvidia/isaac-lab:2.3.0 --gpus device=0 --json
{
  "id": "20260905-150433-isaac-lab-2-3-0",
  "name": "isaac-lab-2.3.0",
  "backend": "container",
  "kind": "container",
  "command": ["/usr/bin/docker", "run", "--rm", "--gpus", "device=0",
              "nvcr.io/nvidia/isaac-lab:2.3.0"],
  "cwd": "/home/robot/work",
  "created": "2026-09-05T15:04:33+08:00",
  "pid": 51902,
  "paused": false,
  "stopped": false,
  "directory": "/home/robot/.caasi/runs/20260905-150433-isaac-lab-2-3-0",
  "status": "running"
}

Everything after IMAGE is the container's command

Caasi models the two flags that matter for GPU experiments (--rm, --gpus). Tokens after the image go to the container's entrypoint, not to docker — so volumes, ports and -e are not supported here. When you need them, generate the base line with --dry-run and extend it yourself:

shellcaasi container run nvcr.io/nvidia/isaac-sim:5.1.0 --dry-run
Dry run — nothing was launched:
  command: /usr/bin/docker run --rm --gpus all nvcr.io/nvidia/isaac-sim:5.1.0
docker run --rm --gpus all -v "$PWD":/work -p 8211:8211 nvcr.io/nvidia/isaac-sim:5.1.0 ./runheadless.sh

End-to-end: headless Isaac Sim in a container

shellcaasi container check
Container stack is ready.
caasi container list
  Local Isaac images:
    (none pulled yet)
docker pull nvcr.io/nvidia/isaac-sim:5.1.0
5.1.0: Pulling from nvidia/isaac-sim ...
caasi container run nvcr.io/nvidia/isaac-sim:5.1.0 ./runheadless.sh --allow-root
Container run started: 20260905-151002-isaac-sim-5-1-0.
caasi logs latest -f
[0.000s] Simulation App Starting...
[18.204s] Streaming server started on port 8211
caasi run status isaac-sim-5.1.0 --json | jq '{status, pid}'
{ "status": "running", "pid": 52310 }
caasi run stop 20260905-1510
Run 20260905-151002-isaac-sim-5-1-0 stopped.

Queries resolve as latest, the exact run name (isaac-sim-5.1.0, the image tail with : → -), or a unique id prefix — note that ids start with the timestamp, so a bare isaac-sim matches nothing. See QUERY semantics.

Local, remote, container

	local (`sim run`, `train`)	`remote run`	`container run`
Transport	direct process launch	system `ssh`	`docker` / `podman`
`backend` / `kind`	`sim`\|`lab`\|`python` / `experiment`\|`train`	`ssh` / `remote`	`container` / `container`
Config needed	tool registry (tools)	`remotes:` entry	a runtime + a pulled image
Readiness gate	`caasi doctor` · `sim check`	`remote connect NAME -c 'nvidia-smi'`	`container check`
Logs land in	the same place — `<run dir>/stdout.log`, followed with `caasi logs <query> -f`

That uniformity is the point: whether the work happens on this machine, across SSH or inside a container, you inspect it with the same caasi run and caasi logs commands.

Previous← Native & Shell NextOverview →

Caasi CLI 0.2.0 — these pages live in docs/ and need no build step. Source: github.com/a6y3ap/caasi.