Metadata-Version: 2.4
Name: roboci
Version: 0.0.1
Summary: Local-first CI for physical AI evaluation, replay, and regression testing.
Author: Gagandeep Reehal
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/GagandeepReehal/roboci
Project-URL: Documentation, https://gagandeepreehal.github.io/roboci/
Project-URL: Repository, https://github.com/GagandeepReehal/roboci
Project-URL: Issues, https://github.com/GagandeepReehal/roboci/issues
Project-URL: Changelog, https://github.com/GagandeepReehal/roboci/blob/main/CHANGELOG.md
Keywords: robotics,physical-ai,simulation,ci,regression-testing,closed-loop-evaluation,open-loop-evaluation,replay
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi<1.0,>=0.110
Requires-Dist: jinja2<4.0,>=3.1
Requires-Dist: pydantic<3.0,>=2.6
Requires-Dist: pyyaml<7.0,>=6.0
Requires-Dist: typer<1.0,>=0.12
Requires-Dist: uvicorn<1.0,>=0.27
Provides-Extra: dev
Requires-Dist: build<2.0,>=1.2; extra == "dev"
Requires-Dist: httpx<1.0,>=0.26; extra == "dev"
Requires-Dist: mypy<2.0,>=1.10; extra == "dev"
Requires-Dist: pytest<9.0,>=8.0; extra == "dev"
Requires-Dist: ruff<1.0,>=0.5; extra == "dev"
Requires-Dist: twine<7.0,>=5.1; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs<2.0,>=1.6; extra == "docs"
Requires-Dist: mkdocs-include-markdown-plugin<8.0,>=7.1; extra == "docs"
Requires-Dist: pymdown-extensions<12.0,>=11.0; extra == "docs"
Provides-Extra: carla
Provides-Extra: isaac
Provides-Extra: gazebo
Dynamic: license-file

# RoboCI

[![RoboCI CI](https://github.com/gagandeepreehal/roboci/actions/workflows/roboci-example.yml/badge.svg)](https://github.com/gagandeepreehal/roboci/actions/workflows/roboci-example.yml)
[![Documentation](https://github.com/gagandeepreehal/roboci/actions/workflows/docs.yml/badge.svg)](https://github.com/gagandeepreehal/roboci/actions/workflows/docs.yml)
[![License: Apache-2.0](https://img.shields.io/badge/License-Apache--2.0-blue.svg)](https://github.com/gagandeepreehal/roboci/blob/main/LICENSE)
[![Python 3.11+](https://img.shields.io/badge/Python-3.11%2B-blue.svg)](https://github.com/gagandeepreehal/roboci/blob/main/pyproject.toml)
[![PyPI](https://img.shields.io/pypi/v/roboci.svg)](https://pypi.org/project/roboci/)
[![Status: Public Alpha](https://img.shields.io/badge/Status-Public%20Alpha-orange.svg)](https://github.com/gagandeepreehal/roboci/blob/main/CHANGELOG.md)

**Local-first CI for physical AI: run closed-loop simulations, open-loop dataset checks, replay failures, and compare regressions from one developer workflow.**

RoboCI is for robotics, autonomous vehicle, drone, and embodied AI teams that need repeatable evaluation before code reaches hardware. It writes every run to inspectable local artifacts, returns CI-friendly exit codes, and ships with a dashboard for replay, metrics, and failure triage.

[Documentation](https://gagandeepreehal.github.io/roboci/)

## v0.0.1 Public Alpha

RoboCI v0.0.1 is a local-first public alpha. It is useful for demos, early evaluation, replay, artifact review, and regression checks, but it is not production safety validation and should not be used as certification evidence for robots, vehicles, drones, manipulators, or hardware-in-loop systems.

This release includes the local CLI, turnkey deterministic mock simulator runs, CARLA/Isaac/Gazebo/ROS2 adapter surfaces that require local runtime setup, open-loop sample dataset evaluation, file-backed run artifacts, HTML and markdown reports, regression comparison, a FastAPI artifact API, and a Next.js dashboard for run review, replay, regressions, and failure tagging.

Early contributors can help most by trying the demo, reporting rough edges, adding simulator and agent examples, improving dashboard workflows, and tightening artifact contracts before the first stable release.

```bash
git clone https://github.com/GagandeepReehal/roboci.git
cd roboci
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

roboci demo
roboci report --run runs/latest
```

## Demo

![RoboCI terminal demo](assets/roboci-demo.gif)

The demo shows a passing smoke run, an intentional collision failure, named baseline promotion, regression comparison, and generated HTML/markdown reports using real local RoboCI output.

```bash
roboci demo
roboci demo --openloop
roboci demo --dashboard
```

## Why RoboCI Exists

Physical AI regressions are costly because failures are often slow to reproduce, hard to compare, and scattered across logs, simulator output, metrics, and screenshots. RoboCI keeps that loop tight:

- Run closed-loop scenario suites and open-loop dataset evaluations from the CLI.
- Store replay, metrics, reports, failures, and regression output under `runs/<run_id>/`.
- Search and filter scenarios by metadata tags such as `pedestrian`, `urban`, `crossing`, `rain`, `night`, and `highway`.
- Compare runs against baselines with gates that work in CI.
- Inspect indexed run artifacts locally in a FastAPI + Next.js dashboard, with safe text previews for summary files and local open actions for larger artifacts.

## Screenshots

### Dashboard

![RoboCI dashboard](assets/dashboard.png)

### Replay

![RoboCI replay viewer](assets/replay.png)

### Regression

![RoboCI regression explorer](assets/regression.png)

## Quickstart

Requirements:

- Python 3.11+
- Docker Compose for the default dashboard path
- Node.js 20+ and npm only for local dashboard development without Docker
- macOS, Linux, or any environment that can run the Python CLI

From a cloned repo root:

```bash
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -e ".[dev]"

roboci demo
roboci report --run runs/latest
roboci doctor
```

The run writes artifacts to `runs/<run_id>/`. `runs/latest` resolves to the newest run.
`roboci doctor` checks the local environment after the demo creates the workspace artifacts; dashboard and port checks are advisory for CLI-only use.

Published-package install uses the bundled demo and benchmark assets, so the first demo does not require a cloned repository checkout:

```bash
pip install roboci
roboci demo
roboci report --run runs/latest
```

If this version is not yet available from PyPI, use the editable repository install shown above.

## Installation

Development install:

```bash
git clone <repo-url>
cd roboci
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
```

Default dashboard path, no local Node required:

```bash
docker compose --profile dashboard up --build api dashboard
```

Optional health check:

```bash
roboci doctor
```

`roboci doctor` exits non-zero for required CLI setup failures. Missing dashboard dependencies, an absent `runs/` directory before the first run, and busy or sandbox-blocked ports are reported as advisories.

One-command demo:

```bash
roboci demo
```

Optional sample dataset workflow:

```bash
roboci demo --openloop
```

## Closed-Loop Demo

Run a passing closed-loop suite with the mock simulator:

```bash
roboci run \
  --suite examples/suites/smoke.yaml \
  --agent examples/agents/rule_based_agent.py \
  --sim mock
```

Expected output: a passing run, a run id, and artifacts under `runs/<run_id>/`.

Run an intentional failure:

```bash
roboci run \
  --suite examples/suites/failing_collision.yaml \
  --agent examples/agents/collision_test_agent.py \
  --sim mock
```

Expected output: a non-zero exit after RoboCI writes collision metrics, replay data, failure records, and a report-ready run directory.

The non-zero exit is expected for this demo because the suite intentionally triggers a collision. Use it when you want failure, replay, and tagging artifacts.

External closed-loop simulator adapters for CARLA, Isaac Sim, and Gazebo are documented in the [simulator adapter guide](https://gagandeepreehal.github.io/roboci/closed_loop_simulators/). Their default tests use mocks, so the repository test suite does not require those simulator SDKs.

## Open-Loop Demo

Evaluate a model against the sample open-loop dataset:

```bash
roboci openloop run \
  --dataset examples/datasets/sample_openloop \
  --model examples/agents/rule_based_agent.py \
  --task trajectory_prediction
```

Expected output: aggregate and per-sample open-loop metrics under `runs/<run_id>/`.

## Benchmarks

List bundled public-alpha benchmarks:

```bash
roboci benchmarks list
```

The benchmark catalog is packaged with RoboCI. `roboci benchmarks list` and `roboci benchmarks run <id>` work from any working directory after installation.

Run the dependency-light smoke benchmark:

```bash
roboci benchmarks run smoke
```

Copy `.github/workflows/roboci.yml` into a project to run the smoke benchmark on pull requests, post `report.md` as a PR comment, and upload `roboci-smoke-run.zip` as an artifact.

The current catalog includes `smoke`, `ground-v0`, `drone-v0`, and `manipulator-v0`. They default to the deterministic mock simulator and write normal RoboCI artifacts under `runs/<run_id>/`. External simulators are adapter surfaces: select them explicitly only after local CARLA, Isaac Sim, Gazebo/ROS2, or another runtime is installed and configured.

## Dashboard Demo

Start the API and prebuilt dashboard image with Docker Compose:

```bash
docker compose --profile dashboard up --build api dashboard
```

Default URLs:

- Dashboard: `http://127.0.0.1:3000/dashboard`
- API: `http://127.0.0.1:8000`
- API docs: `http://127.0.0.1:8000/docs`

The dashboard image is built from `frontend/package-lock.json`; it does not run `npm install` when the container starts.

For a local Node development loop, install dashboard dependencies once and run:

```bash
cd frontend
npm install
cd ..
roboci dev
```

If a port is busy, `roboci dev` picks the next available port and prints the selected URLs. Choose ports explicitly with `roboci dev --api-port 8010 --frontend-port 3010`.

Users without local Node can also run:

```bash
roboci dev --docker
```

`roboci doctor` checks whether the default API port is available. If the backend is already running, or if a sandbox blocks local port probing, the port check can report `blocked`; stop the running backend or use the URL printed by `roboci dev`.

## Docker

The default Docker path is CLI/demo focused and does not include simulator SDKs:

```bash
docker compose build roboci
docker compose run --rm roboci roboci doctor
docker compose run --rm roboci roboci demo
docker compose run --rm roboci roboci benchmarks run smoke
```

Optional dashboard profile:

```bash
docker compose --profile dashboard up --build api dashboard
```

The dashboard profile builds a Next.js standalone image and uses `NEXT_PUBLIC_ROBOCI_API=http://127.0.0.1:8000` for browser requests and `ROBOCI_SERVER_API=http://api:8000` for server-rendered dashboard requests inside the Compose network.

CARLA smoke profile:

```bash
docker compose --profile carla up --build carla carla-roboci
```

This starts `carlasim/carla:0.9.15`, builds a RoboCI runner image, waits for the CARLA RPC port, and runs `examples/suites/carla.yaml` end to end. The default runner image does not bundle the CARLA Python API because published CARLA client packages are platform- and Python-version-specific. On prepared Linux/GPU CARLA runners, set `ROBOCI_CARLA_PYTHON_PACKAGE` to a matching CARLA Python API wheel or package before building; otherwise `carla-roboci` exits early with a clear setup error instead of failing during image build.

## Replay And Failure Tagging

Generate a run first:

```bash
roboci run --suite examples/suites/failing_collision.yaml --agent examples/agents/collision_test_agent.py --sim mock
```

Then inspect it in the dashboard:

- Run list: `http://127.0.0.1:3000/runs`
- Run detail: `http://127.0.0.1:3000/runs/<run_id>`
- Scenario replay: `http://127.0.0.1:3000/runs/<run_id>/scenarios/<scenario_id>`
- Failure replay deep link: `http://127.0.0.1:3000/runs/<run_id>/scenarios/<scenario_id>?t=<timestamp>&frame=<frame_index>`

RoboCI stores detected failures as file-backed artifacts, so tags and triage state stay close to the run data they describe.
Failure records include replay location fields when available, and failure links in the dashboard open the replay at the incident frame.
The dashboard failure editor supports a lifecycle workflow with `open`, `acknowledged`, `assigned`, `in_progress`, `fixed`, `verified`, `regressed`, and `wont_fix` statuses. Optional `owner`, `fixed_in_run_id`, and `regressed_in_run_id` fields are stored in `failures.json`, and each status change appends a local `status_history` entry. Closed-loop regression comparisons also backfill lifecycle links: baseline failures that pass in the current run get `fixed_in_run_id`, while new failures on previously passing scenarios get `regressed_in_run_id`.
Scenario replay pages show a synchronized replay scene and event timeline sourced from `replay/<scenario_id>.timeline_events.json`; clicking a timeline event seeks the replay scene to that event timestamp.

## Scenario Metadata

Scenario YAML files can include optional lowercase, slug-style tags:

```yaml
id: pedestrian_crossing_001
name: Pedestrian Crossing
tags:
  - pedestrian
  - crossing
  - urban
```

Tags are written into run scenario results under `runs/<run_id>/metrics.json`, exposed through the API, and shown in the dashboard. Use `/scenarios?tag=<tag>` to filter local examples and marketplace packs, `/scenarios?domain=<domain>` to filter marketplace packs by robotics domain, and `/scenarios?q=<text>` to search pack names, tasks, metrics, requirements, tags, and scenario names. Run detail pages support `/runs/<run_id>?tag=<tag>`, `/runs/<run_id>?domain=<domain>`, and `/runs/<run_id>?pack=<pack-id>` filters.

Marketplace packs are domain-aware. New packs can declare `domain: ground_robotics`, `domain: manipulators`, or `domain: drones`, plus supported tasks, domain metrics, adapter requirements, simulator requirements, tags, version, license, and domain-specific metadata. Workspace-local packs can be added with `examples/scenario_marketplace/<pack-id>/pack.yaml`, and `roboci scenarios install <pack-id> --versioned` installs a version-suffixed copy that can coexist with the default install. Existing packs without a `domain` field default to `ground_robotics` so installed packs and older run artifacts continue to load. Pack name, version, domain, task type, and metric group are propagated into run summaries, provenance, dashboard views, and generated reports.

Scenario YAML files can also include optional behavioral assertions:

```yaml
assertions:
  - name: stop_before_pedestrian
    type: min_distance
    actor_type: pedestrian
    threshold: 1.5
    severity: critical

  - name: max_speed_near_pedestrian
    type: max_speed_when_actor_near
    actor_type: pedestrian
    distance: 10
    max_speed: 2.0
    severity: high
```

Supported assertion types are `no_collision`, `min_distance`, `max_speed`, `max_speed_when_actor_near`, and `goal_reached`. `critical` and `high` failures fail the scenario; lower severities are recorded as advisory results. Assertion results are written to `metrics.json`, summarized in `summary.json`, and shown in both static reports and the dashboard alongside generic failure counts.

## Regression And Reports

Fresh runs write both static reports automatically:

```text
runs/<run_id>/report.html
runs/<run_id>/report.md
```

Regenerate static reports for an existing run:

```bash
roboci report --run runs/latest
roboci report --run runs/latest --format markdown
```

The default report command regenerates `report.html`. The markdown format regenerates `report.md`, which is concise enough for PR comments and CI artifact review.

Static reports use local relative artifact links by default. To embed a dashboard deep link without assuming a port, set `ROBOCI_DASHBOARD_URL` before generating the report, for example to the dashboard URL printed by `roboci dev`.

Compare a run against a baseline:

```bash
roboci baseline create smoke --from runs/latest --description "Known-good smoke run"
roboci baseline promote runs/latest smoke
roboci baseline list
roboci baseline show smoke

roboci compare \
  --current runs/latest \
  --baseline-name smoke \
  --max-regression-percent 5
```

Named baselines are stored under `baselines/<name>/` with `baseline.json` metadata containing the source run id, resolved source directory, creation time, and optional description. `roboci baseline list` only shows directories with `baseline.json`, so unrelated folders under `baselines/` are ignored. `roboci compare --baseline <run-dir>` still works for path-based comparisons.

`roboci compare` returns a non-zero exit when regressions exceed the configured threshold. That behavior is intended for CI gates.

## CLI Reference

| Command | Purpose |
| --- | --- |
| `roboci init` | Create a minimal RoboCI workspace layout. |
| `roboci init --template carla-av` | Generate a CARLA AV starter workspace with agent, simulator config, scenario, and suite. |
| `roboci init --template ros2-mobile-robot` | Generate a ROS2 bridge mobile-robot starter workspace. |
| `roboci init --template drone-navigation` | Generate a mock-drone starter workspace that runs immediately. |
| `roboci doctor` | Check required CLI setup plus advisory run, frontend, and API port status. |
| `roboci demo` | Run the local public-alpha happy path: smoke, expected collision, baseline, compare, reports. |
| `roboci demo --openloop` | Also run the sample open-loop dataset workflow. |
| `roboci demo --dashboard` | Start `roboci dev` after the demo when dashboard dependencies are installed; otherwise print install guidance. |
| `roboci run --suite <path> --agent <path> --sim <name-or-path>` | Run a closed-loop scenario suite. |
| `roboci report --run <run-dir>` | Generate a static HTML report for a run. |
| `roboci report --run <run-dir> --format markdown` | Generate a concise markdown report for PR and CI review. |
| `roboci benchmarks list` | List bundled benchmark metadata. |
| `roboci benchmarks run <id>` | Run `smoke`, `ground-v0`, `drone-v0`, or `manipulator-v0`. |
| `roboci bundle runs/latest --output roboci-run.zip` | Zip one run directory for sharing or CI upload. |
| `roboci baseline create <name> --from <run-dir>` | Create a named golden baseline under `baselines/`. |
| `roboci baseline promote <run-dir> <name>` | Promote a run to a named golden baseline with positional arguments. |
| `roboci baseline list` | List named baselines. |
| `roboci baseline show <name>` | Show baseline metadata. |
| `roboci compare --current <run-dir> --baseline-name <name>` | Compare a current run against a named baseline. |
| `roboci compare --current <run-dir> --baseline <run-dir>` | Compare a current run against a baseline path. |
| `roboci serve` | Start only the FastAPI backend. |
| `roboci dev` | Start the FastAPI backend and local Next.js dashboard. |
| `roboci dev --docker` | Start the FastAPI backend and prebuilt dashboard image through Docker Compose. |
| `roboci openloop run --dataset <path> --model <path> --task <name>` | Run open-loop dataset evaluation. |
| `roboci openloop compare --current <run-dir> --baseline <run-dir>` | Compare open-loop run metrics. |
| `roboci scenarios list` | List bundled and workspace scenario packs. |
| `roboci scenarios install <pack-id>` | Install a scenario pack into the workspace examples. |
| `roboci scenarios install <pack-id> --versioned` | Install a version-suffixed copy of a marketplace pack. |
| `roboci scenarios remove <pack-id>` | Remove an installed marketplace pack and generated suite from a workspace. |

## Architecture Overview

RoboCI is built around a local artifact contract. The CLI runs evaluation work, the runners write structured files under `runs/<run_id>/`, and reports, regression checks, the API, and the dashboard all read the same artifacts. Every run writes `provenance.json` and embeds the same object in `summary.json` so regression comparisons can account for Git state, source file hashes, simulator/runtime versions, and open-loop dataset hashes. The dashboard indexes local artifacts but only previews allowlisted root text files (`summary.json`, `metrics.json`, `regression.json`, `failures.json`, `config.yaml`, and `provenance.json`) through the API; episode logs, replay artifacts, reports, and open-loop files remain local-only for preview and can be opened with the host default app.

```mermaid
flowchart LR
  CLI["Typer CLI"] --> Closed["Closed-loop runner"]
  CLI --> Open["Open-loop runner"]
  CLI --> Compare["Regression comparator"]

  Closed --> Agents["Agent loaders"]
  Closed --> Sims["Simulator adapters"]
  Closed --> Metrics["Metrics"]
  Closed --> Assertions["Assertions"]
  Closed --> Replay["Replay exports"]
  Closed --> Failures["Failure detection"]

  Open --> Dataset["Dataset loader"]
  Open --> Model["Model adapter"]
  Open --> OLMetrics["Open-loop metrics"]

  Metrics --> Runs["runs/run_id"]
  Assertions --> Runs
  Replay --> Runs
  Failures --> Runs
  OLMetrics --> Runs
  Compare --> Runs

  Runs --> Report["HTML and markdown reports"]
  Runs --> API["FastAPI API"]
  API --> Dashboard["Next.js dashboard"]
```

## Project Structure

| Path | What lives there |
| --- | --- |
| `roboci/cli` | Typer command definitions and command handlers. |
| `roboci/core` | Closed-loop runner, runtime types, config, and errors. |
| `roboci/agents` | Python and native agent loading. |
| `roboci/simulators` | Mock, Python, native, CARLA, and ROS2 simulator adapters. |
| `roboci/openloop` | Dataset/model runner and open-loop metrics. |
| `roboci/metrics` | Safety, task, comfort, and open-loop metrics. |
| `roboci/assertions` | Deterministic scenario assertion evaluation. |
| `roboci/replay` | Timeline and GeoJSON replay exporters. |
| `roboci/failures` | Failure detection, schemas, stores, and tags. |
| `roboci/regression` | Baselines, comparators, and regression gates. |
| `roboci/reporting` | JSON, plot, HTML, and markdown report generation. |
| `roboci/benchmarks` | Benchmark metadata loading and validation. |
| `roboci/api` | FastAPI backend used by the dashboard. |
| `benchmarks` | Public-alpha benchmark definitions and reference scenarios. |
| `frontend` | Next.js dashboard. |
| `examples` | Example agents, simulators, suites, datasets, scenario packs, and ROS2 bridge files. |
| `tests` | Unit and integration tests. |
| `docs` | Deeper CLI, API, architecture, frontend, data format, and operations guides. |

## Roadmap

See `ROADMAP.md` for stable v0.1 surfaces, unstable surfaces, current limitations, and the contribution roadmap.

## Known Limitations

- RoboCI is local-first; `ROBOCI_API_TOKEN` is a shared trusted-network token, not a hosted multi-user service.
- RoboCI is not production safety validation. Treat outputs as deterministic regression evidence for development, not certification.
- The mock simulator is turnkey and deterministic. CARLA, Isaac Sim, Gazebo/ROS2, Python simulator extensions, and native simulators require local runtime setup.
- The dashboard reads file-backed artifacts and expects the FastAPI backend to be running.
- Dashboard access defaults to Docker Compose; local dashboard development still requires Node.js/npm.
- Native C++ and Rust examples require local compiler/runtime availability.
- ROS2 support is a reference bridge, not a turnkey production robotics deployment.
- CARLA integration depends on the CARLA Docker image or a local CARLA setup with a matching Python API.
- Demo media is captured from local example runs and should be refreshed when dashboard workflows change.

## Contributing

Contributions are welcome. For most changes:

```bash
python -m pytest
cd frontend
npm run lint
npm run build
```

Guidelines:

- Keep generated `runs/`, named baseline data, and dashboard build output out of commits.
- Add focused tests when changing CLI, API, runner, regression, replay, or dashboard behavior.
- Update docs when command flags, artifact formats, or workflows change.
- Prefer small pull requests with a clear reproduction or demo command.

Start with the [documentation map](https://gagandeepreehal.github.io/roboci/documentation-map/) for deeper project documentation.

Build the documentation site locally with MkDocs:

```bash
pip install -e ".[docs]"
mkdocs serve
mkdocs build --strict
```

## Citation

If RoboCI helps your research or development work, please cite the repo:

```bibtex
@software{roboci_2026,
  title = {RoboCI: Local-first CI for Physical AI Evaluation, Replay, and Regression Testing},
  author = {Reehal, Gagandeep},
  year = {2026},
  version = {0.0.1},
  note = {Public alpha release},
  url = {https://github.com/GagandeepReehal/roboci}
}
```

## License

RoboCI is licensed under Apache-2.0. See `LICENSE`.
