Metadata-Version: 2.4
Name: rpx-benchmark
Version: 0.3.2
Summary: RPX — benchmark harness for evaluating robot-perception models on real-world RGB-D data
Author: RPX Team
Maintainer: RPX Team
License: MIT License
        
        Copyright (c) 2025 Intelligent Robotics and Vision Lab at the University of Texas at Dallas
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/IRVLUTD/RPX
Project-URL: Documentation, https://irvlutd.github.io/RPX/toolkit-docs/
Project-URL: Repository, https://github.com/IRVLUTD/RPX
Project-URL: Issues, https://github.com/IRVLUTD/RPX/issues
Keywords: robot-learning,perception,rgb-d,benchmark,robotics,embodied-ai,huggingface,deployment-readiness
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: Pillow>=9.0
Requires-Dist: requests>=2.31
Requires-Dist: rich>=13
Requires-Dist: scipy>=1.10
Provides-Extra: docs
Requires-Dist: mkdocs-material<10,>=9.6; extra == "docs"
Requires-Dist: pdoc<16,>=15; extra == "docs"
Requires-Dist: pandas>=2.0; extra == "docs"
Provides-Extra: hub
Requires-Dist: huggingface_hub[hf_xet]>=0.26; extra == "hub"
Requires-Dist: pyarrow>=14; extra == "hub"
Provides-Extra: hf-datasets
Requires-Dist: datasets>=2.18; extra == "hf-datasets"
Requires-Dist: huggingface_hub>=0.22; extra == "hf-datasets"
Provides-Extra: schemas
Requires-Dist: pydantic>=2.5; extra == "schemas"
Provides-Extra: esd-fast
Requires-Dist: opencv-python>=4.8; extra == "esd-fast"
Provides-Extra: ar-pose-audit
Requires-Dist: opencv-contrib-python-headless>=4.8; extra == "ar-pose-audit"
Requires-Dist: matplotlib>=3.7; extra == "ar-pose-audit"
Provides-Extra: analysis
Requires-Dist: pandas>=2.0; extra == "analysis"
Requires-Dist: scikit-learn>=1.3; extra == "analysis"
Requires-Dist: scipy>=1.10; extra == "analysis"
Requires-Dist: matplotlib>=3.7; extra == "analysis"
Provides-Extra: viz
Requires-Dist: rerun-sdk>=0.26; extra == "viz"
Requires-Dist: matplotlib>=3.7; extra == "viz"
Requires-Dist: pillow>=10; extra == "viz"
Requires-Dist: tqdm>=4.66; extra == "viz"
Provides-Extra: dev
Requires-Dist: numpy<2.5; extra == "dev"
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-cov>=5; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: black>=24.8; extra == "dev"
Requires-Dist: mypy<2.0,>=1.11; extra == "dev"
Requires-Dist: pre-commit>=3.8; extra == "dev"
Requires-Dist: pydantic>=2.5; extra == "dev"
Requires-Dist: types-requests; extra == "dev"
Dynamic: license-file

# RPX Benchmark Toolkit

Load RPX data, run reference or custom perception models, and compare accuracy,
scene-change robustness and compute cost with Python 3.10–3.12:

```bash
python -m pip install 'rpx-benchmark[hub,schemas]'
git clone https://github.com/IRVLUTD/RPX.git
cd RPX
python benchmark/examples/run_depth.py
```

The example is a CPU-only installation check using synthetic data. Reference
checkpoints need their own runtime, weights and GPU resources. The package
installation alone does not install every model's dependencies.

For toolkit development, replace the PyPI install with an editable source
install from the repository root:

```bash
python -m pip install -e './benchmark[hub,schemas]'
```

## Reference models

Use isolated environments because model dependencies conflict. The linked
READMEs contain build, checkpoint, cache and execution instructions. Run a
model's small inference gate before a full benchmark, and retain its dataset
revision and runtime metadata with the results.

| Evaluation | Models and commands | Environment |
|---|---|---|
| Image depth | Ten canonical models, including DA3, DA-V2 Large, Depth Pro, UniDepth V2, Metric3D V2, MoGe-2, HyDen, Lotus 2, FE2E and ZipDepth; `scripts/run_depth.py` | [Depth](https://github.com/IRVLUTD/RPX/blob/main/docker/depth-smoke/README.md), with dedicated [FE2E](https://github.com/IRVLUTD/RPX/blob/main/docker/depth-fe2e/README.md) and [ZipDepth](https://github.com/IRVLUTD/RPX/blob/main/docker/depth-zipdepth/README.md) overlays |
| Video depth | DA3, DepthCrafter, Video Depth Anything, ChronoDepth, RollingDepth, DVD, GEMDepth, ViGeo, MonST3R and VGGT; `scripts/run_video_depth.py` | [Depth](https://github.com/IRVLUTD/RPX/blob/main/docker/depth-smoke/README.md), [DVD](https://github.com/IRVLUTD/RPX/blob/main/docker/depth-dvd/README.md), [GEMDepth](https://github.com/IRVLUTD/RPX/blob/main/docker/depth-gemdepth/README.md) |
| Relative pose | VGGT, DA3, CUT3R, DUSt3R, MASt3R, MUSt3R, Reloc3r, Pi3X, Fast3R and MonST3R; additional baseline adapters are retained | [Relative pose](https://github.com/IRVLUTD/RPX/blob/main/docker/rcpe-smoke/README.md) |
| Tracking | Mask-, box- and text-initialized adapters, with MOS and Ego protocols; `scripts/run_tracking.py` and `scripts/run_tracking_paper.py` | [Tracking](https://github.com/IRVLUTD/RPX/blob/main/docker/tracking-smoke/README.md) |
| VQA / bbox grounding | Twenty roster entries with native Transformers or vLLM backends, one- and two-image inputs; `scripts/run_vqa_benchmark.py` | [VQA](https://github.com/IRVLUTD/RPX/blob/main/docker/vqa-smoke/README.md) |

For segmentation, detection, open-vocabulary detection, generic grounding,
sparse depth and keypoint matching, the toolkit supplies the callable APIs,
loaders and evaluators below. There is no bundled full pretrained-model sweep
for those tasks. Mask annotation models produce dataset labels; they are not
a substitute for a benchmark adapter.

Example, **inside the selected model environment**, from `benchmark/`:

```bash
python scripts/run_depth.py --model da-v2-large --split easy \
  --repo IRVLUTD/RPX --device cuda --max-samples 10 \
  --output-dir ../rpx_results/da-v2-large/easy-smoke
python scripts/run_video_depth.py --help
python scripts/run_relative_pose.py --help
python scripts/run_tracking.py --help
```

`python benchmark/scripts/run_depth_smoke_matrix.py --list-models` lists the
canonical depth roster without loading weights. Docker matrix files and adapter
registries are the source of model identifiers and pinned checkpoint revisions.
Model availability and successful inference are different checks: the offline
suite tests adapters with controlled inputs, while GPU gates run real weights.

## Bring your own model

Wrap a callable and pass it to a public task runner. The callable may invoke a
local model or your inference service; convert its response to the documented
NumPy output. Authentication and service configuration belong in your wrapper,
not in committed source.

```python
import rpx_benchmark as rpx
from my_model import predict_depth  # your implementation: RGB -> depth in metres

model = rpx.make_numpy_depth_model(predict_depth, name="my-depth")
result, report, paths = rpx.run_monocular_depth(
    rpx.MonocularDepthRunConfig(
        model=model,
        split="easy",
        repo_id="IRVLUTD/RPX",
        revision=rpx.DEFAULT_REVISION,
        device="cpu",  # configure your callable's actual device yourself
        output_dir="rpx_results/my-depth/easy",
    )
)
print(result.aggregated)
print(paths["json"])
```

For a local dataset, supply `manifest_path="/data/manifests/monocular_depth/easy.json"`
to skip downloading. Manifests name a task and dataset root and list samples
with modality paths, scene, phase and difficulty. Local examples and all ten
public workflows are exercised in [tests/test_user_workflows.py](https://github.com/IRVLUTD/RPX/blob/main/benchmark/tests/test_user_workflows.py).
For relative depth, set `depth_output_kind="relative"` on the factory; never
label arbitrary relative values as metric depth.

| Task | Factory (under `rpx`) | Callable input → output | Public runner |
|---|---|---|---|
| Monocular depth | `make_numpy_depth_model` | RGB `(H,W,3)` → depth `(H,W)` | `run_monocular_depth` |
| Video depth | `make_numpy_video_depth_model` | RGB `(T,H,W,3)` → depth `(T,H,W)` | `run_video_depth` |
| Segmentation | `make_numpy_mask_model` | RGB → integer instance mask; 0 is background | `run_segmentation` |
| Detection | `make_numpy_detection_model` | RGB → boxes, scores, labels | `run_object_detection` |
| Open-vocabulary detection | `make_numpy_detection_model(..., task=rpx.TaskType.OPEN_VOCAB_DETECTION)` | RGB → boxes, scores, labels; bind your vocabulary in the callable | `run_open_vocab_detection` |
| Tracking | `make_numpy_tracking_model` | RGB → tracks with persistent IDs and boxes | `run_object_tracking` |
| Grounding | `make_numpy_grounding_model` | RGB, text → boxes and scores | `run_visual_grounding` |
| Relative pose | `make_numpy_pose_model` | RGB A, RGB B → rotation `(3,3)`, translation `(3,)` | `run_relative_pose` |
| Sparse depth | `make_numpy_sparse_depth_model` | RGB, pixel coordinates → depth values | `run_sparse_depth` |
| Keypoint matching | `make_numpy_keypoint_model` | RGB A, RGB B → points0, points1 | `run_keypoint_matching` |

Factory docstrings specify shapes and units. For custom preprocessing, batching
or framework-specific outputs, implement `BenchmarkModel` or compose
`BenchmarkableModel` with input/output adapters. The generic tracking API is a
frame-based contract; use the production tracking runner for the full clip-based
paper protocol and its initialization/scoring rules.

### Custom VQA model or API

VQA uses JSONL question manifests rather than the core task manifests. A custom
callable receives only ordered image paths and the public prompt; ground-truth
answers and boxes are kept inside the evaluator.

```python
from rpx_benchmark.vqa import evaluate_vqa
from my_model import predict_vqa

# predict_vqa(image_paths, prompt, max_new_tokens, output_kind) -> raw text
# image_paths: (target,) or (reference_crop, target), in that order.
report = evaluate_vqa(
    "vqa_manifest.jsonl", predict_vqa,
    model_name="my-vlm", model_revision="my-checkpoint-or-service-version",
    image_cache=".cache/rpx-vqa", output_dir="rpx_results/my-vlm",
)
print(report["metrics"])
```

Return one JSON object with `label` and `bbox` (XYXY coordinates normalized to
0–1000); binary questions require `yes` or `no`. Parsing failures and inference
errors remain in the denominator. Runs save raw outputs, errors, timing,
manifest hash, model revision and scored results. Use a fresh output directory.
The [VQA runtime README](https://github.com/IRVLUTD/RPX/blob/main/docker/vqa-smoke/README.md) explains manifest
preparation and the reference-model benchmark. The preserved Molmo adapter in
`scripts/vqa_models/molmo_backend.py` is optional and outside the frozen roster.

## Data and results

The Hub loader fetches only required modalities and reuses its cache. Pin a
dataset commit for comparable runs; do not mix results from moving revisions.
The [data README](https://github.com/IRVLUTD/RPX/blob/main/benchmark/data/README.md) identifies the local canonical splits and
metadata. Dataset publishing tools are documented separately in
[rpx_benchmark/dataset_hub/README.md](https://github.com/IRVLUTD/RPX/blob/main/benchmark/rpx_benchmark/dataset_hub/README.md).

Generic frame runs write `result.json`, `summary.md`, `cells.parquet` and
`per_sample_metrics.parquet`. Video runs write clip/cell results. Hardware,
precision and batch size belong in each cell; unavailable measurements stay
unavailable. Custom API wall time includes transport and is not a GPU-only
latency measurement. Production runners add prediction caches, protocol
metrics and provenance records.

## Testing

From `benchmark/`, in a clean virtual environment:

```bash
python -m pip install -e '.[dev,hub,schemas,hf-datasets,analysis,ar-pose-audit]' \
  imageio imageio-ffmpeg av transformers build
python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
python -m pip install --no-deps 'git+https://github.com/JonathonLuiten/TrackEval.git@12c8791b303e0a0b50f753af204249e622d0281a'
pytest tests -q --cov=rpx_benchmark --cov-report=term-missing
pytest ../data/mask_annotation/visual_grounding_gt/vqa_gt/tests -q
ruff check rpx_benchmark tests scripts
mypy rpx_benchmark
python -m build
```

Tests cover task contracts, local datasets, public custom-model workflows,
metrics, reports, download behavior, model-call signatures and VQA parsing.
They do not download model checkpoints. Run the linked CUDA inference gates
on the target machine to validate real models, available memory and dataset
access. Sensor capture and annotation UI checks require their own hardware.

## Extend the toolkit

See the [metric registration guide](https://irvlutd.github.io/RPX/toolkit-docs/metrics/) and [task extension guide](https://irvlutd.github.io/RPX/toolkit-docs/tasks/). The guides include runnable examples, validation rules, and the source changes needed for a new task identity. They are also bundled in the installed package under `rpx_benchmark/guides/`.

Browse the [toolkit documentation](https://irvlutd.github.io/RPX/toolkit-docs/) for installation, model integration, and API reference pages.

To locate the bundled guides after installation:

```python
from importlib.resources import files
print(files("rpx_benchmark").joinpath("guides/tasks/README.md").read_text())
```

To build the documentation from a repository checkout:

```bash
pip install -e "./benchmark[hub,schemas,docs]"
python tools/docs/build_pages.py --output _site
python -m http.server --directory _site 8000
```

Open `http://localhost:8000/toolkit-docs/`. The build generates guides and API pages from the same source.
