Metadata-Version: 2.5
Name: XCalib
Version: 0.3.0
Summary: Camera-LiDAR cross-modal matching for edge devices: pretrained matchers, targetless calibration, ONNX/TensorRT export, training, and one-shot adaptation.
Project-URL: Repository, https://github.com/radar-lab/XCalib
Project-URL: Documentation, https://radar-lab.github.io/XCalib/
Author-email: Lihao Guo <leolihao@arizona.edu>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: camera-lidar,cross-modal matching,extrinsic calibration,jetson,onnx,tensorrt
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: huggingface-hub>=0.30
Requires-Dist: loguru<0.8,>=0.7
Requires-Dist: numpy<3,>=2.0
Requires-Dist: opencv-python<5,>=4.10
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: torch>=2.2
Requires-Dist: tqdm<5,>=4.67
Provides-Extra: dev
Requires-Dist: h5py<4,>=3.11; extra == 'dev'
Requires-Dist: mkdocs-material>=9.5; extra == 'dev'
Requires-Dist: mkdocstrings[python]>=0.26; extra == 'dev'
Requires-Dist: onnx<2,>=1.16; extra == 'dev'
Requires-Dist: onnxruntime<2,>=1.18; extra == 'dev'
Requires-Dist: pyside6<7,>=6.6; extra == 'dev'
Requires-Dist: pytest<9,>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: gui
Requires-Dist: pyside6<7,>=6.6; extra == 'gui'
Provides-Extra: onnx
Requires-Dist: onnx<2,>=1.16; extra == 'onnx'
Requires-Dist: onnxruntime<2,>=1.18; extra == 'onnx'
Provides-Extra: train
Requires-Dist: h5py<4,>=3.11; extra == 'train'
Description-Content-Type: text/markdown

# XCalib

[![CI](https://github.com/radar-lab/XCalib/actions/workflows/ci.yml/badge.svg)](https://github.com/radar-lab/XCalib/actions/workflows/ci.yml)
[![Docs](https://github.com/radar-lab/XCalib/actions/workflows/docs.yml/badge.svg)](https://radar-lab.github.io/XCalib/)
[![PyPI](https://img.shields.io/pypi/v/xcalib)](https://pypi.org/project/XCalib/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)
[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/radar-lab/XCalib/blob/main/demo/xcalib_quickstart.ipynb)

Camera-LiDAR cross-modal matching for edge devices, behind one small API:

- **Match** every camera box against every LiDAR box with a pretrained matcher, then pick
  the pairs: optional one-to-one assignment, gated by projection geometry once you pass a
  calibration ([pair selection](docs/pair-selection.md)).
- **Calibrate** without targets (PnP/RANSAC), or nudge a site's existing matrix and keep
  the change only if the reprojection error improves ([deployed site](docs/deployed-site.md)).
- **Adapt** on site without labels (one-shot), or keep learning from human-verified pairs
  with a gated continual refine ([continual refine](docs/continual-refine.md)).
- **Capture and review** the frames the matcher is lost on, and fix the wrong pairs in a
  Qt window ([review GUI](docs/review-gui.md)).
- **Deploy** with ONNX/TensorRT export and **train** on HDF5 caches.

`xcalib` is the deployment package for the matching front-end studied in
*Position Encoding in Detection-Based LiDAR–Camera Matching: A Diagnostic
Study at Infrastructure Sites* (accepted at IEEE Sensors Letters). The same weights run on a
lab workstation and an NVIDIA Jetson AGX Thor. It builds on the lab's prior
targetless calibration framework
[CalibRefine](https://github.com/radar-lab/Lidar_Camera_Automatic_Calibration)
(Cheng et al., IEEE TIM 2026, [arXiv:2502.17648](https://arxiv.org/abs/2502.17648)).

> **Status** — publicly released and actively developed. The public API
> (`xcalib.__all__`) is stable across minor versions; internals may change
> without notice. Broader visibility is expected once the accompanying IEEE
> Sensors Letters paper is published. New versions are cut from `main` as tagged
> releases.

**Docs: [radar-lab.github.io/XCalib](https://radar-lab.github.io/XCalib/)**

## Install

```bash
pip install XCalib            # inference
pip install "XCalib[onnx]"    # + ONNX export / onnxruntime parity checks
pip install "XCalib[train]"   # + h5py (HDF5 training / fine-tuning)
pip install "XCalib[gui]"     # + PySide6 (the xcalib-review window)
```

For normal use, install the wheel from PyPI once it is published; downloading
or building from source is not required. Source checkouts are for development,
paper validation, or local experimentation.

Requires Python 3.10+ (CI tests 3.10 through 3.14). The wheel pins
`torch>=2.2` only. On Jetson AGX Thor the default PyPI torch (>= 2.12,
aarch64 + bundled CUDA 13) works out of the box; on x86 install the CUDA
build you need first and pip will leave it alone. After install the
`xcalib` CLI is on `$PATH` and configs resolve from package data — no checkout
required.

For development, [pixi](https://pixi.sh) drives everything (the manifest
lives in `pyproject.toml` under `[tool.pixi.*]`):

```bash
pixi install            # per-platform torch (CUDA 12.8 on x86, PyPI CUDA-13 on Thor)
pixi run test           # full suite
pixi run smoke          # quick synthetic-frame smoke tests
```

| Platform                | torch source                                      |
|-------------------------|---------------------------------------------------|
| `linux-aarch64` (Thor)  | PyPI (`torch>=2.12`, cp31x manylinux_2_28 + cu13) |
| `linux-64` / `win-64`   | `https://download.pytorch.org/whl/cu128`          |
| `osx-arm64`             | PyPI (CPU build, used only for code review)       |

## Quickstart

> Prefer a runnable, end-to-end walkthrough (install → Hub weights/data → match →
> calibrate, with inline overlays)? Open
> [`demo/xcalib_quickstart.ipynb`](demo/xcalib_quickstart.ipynb) locally or in
> [Colab](https://colab.research.google.com/github/radar-lab/XCalib/blob/main/demo/xcalib_quickstart.ipynb).
> The continual-refine flow has its own notebook,
> [`demo/refine_unification.ipynb`](demo/refine_unification.ipynb).

```python
import cv2, numpy as np
from xcalib import Matcher

matcher = Matcher.from_pretrained("crlite", site="a9_dataset_r02_s01")
# ... or local: Matcher.from_pretrained("crlite", weights="checkpoints/crlite_custom_best.pth")

image       = cv2.cvtColor(cv2.imread("frame.jpg"), cv2.COLOR_BGR2RGB)  # H,W,3 uint8
point_cloud = np.load("frame_lidar.npy").astype(np.float32)             # N,3 (X,Y,Z)
bboxes_2d   = np.array([[100, 200, 180, 320]], dtype=np.float32)        # K,4 (x1,y1,x2,y2)
bboxes_3d   = np.array([[5.0, -2.0, -1.5, 8.0, 0.5, 0.5]], dtype=np.float32)  # M,6

result = matcher.match(image, point_cloud, bboxes_2d, bboxes_3d,
                       top_k=5, validate="warn")
print(result.similarity)    # K x M float32 in [0, 1]
print(result.matches)       # [(img_idx, lid_idx, score), ...]
```

Everything a deployment needs hangs off the `Matcher`:

```python
matcher.pair(...)                                  # alias of match()
matcher.build("onnx", output_dir="onnx/crlite")    # export + torch parity check
matcher.build("trt", precision="fp16")             # Jetson: trtexec + shape profiles

K = xcalib.CameraIntrinsics(fx=1418., fy=1422., cx=976., cy=606.)
calib = matcher.calibrate(image, pc, b2, b3, intrinsics=K)   # targetless PnP/RANSAC extrinsics

result = matcher.match(image, pc, b2, b3,          # result.assignment: optimal one-to-one pairs,
                       one_to_one=True,            #   gated by the calibration's projection
                       calibration=calib)

session = matcher.oneshot(K)                       # label-free adaptation:
session.observe(image, pc, b2, b3)                 #   confident matches -> projection
session.adapt(steps=50)                            #   -> pseudo-labels -> adapter updates
session.save("runs/site_adapted")

matcher.train("train.h5", "val.h5", epochs=10)     # fine-tune loaded weights
matcher.save_pretrained("runs/my_model")           # weights + config
```

A deployed site can nudge a matrix it already trusts, and save the frames the matcher gets
lost on so a person can fix the pairs in [`xcalib-review`](docs/review-gui.md):

```python
rec = xcalib.SceneRecorder("capture/site1", camera="cam_front")
session = matcher.oneshot(K, recorder=rec)   # saves frames with detections but no confident pair
session.set_calibration(calib)               # start from the matrix the site already trusts
session.observe(image, pc, b2, b3)           # ... once per frame
session.calibrate(refine=True)               # nudge it; kept only if the median error drops
```

The whole flow is in [deployed site](docs/deployed-site.md).

Training and dataset helpers:

```python
from xcalib import train, load_dataset

best = train("crlite", "a9_dataset_r02_s01", "a9_dataset_r02_s01",
             site="a9_dataset_r02_s01", epochs=100, warmup_epochs=5,
             optimizer="adam", scheduler="cosine")          # paper recipe
loader = load_dataset("a9_dataset_r02_s01", split="test")   # Hub or local cache
```

To keep improving a site's matcher from human-verified pairs instead of HDF5 caches, see
[Continual refine](docs/continual-refine.md).

External configs work the same way as packaged configs, so users can keep a
custom YAML next to custom weights:

```python
matcher = Matcher.from_pretrained(
    "crlite",
    weights="runs/site42/best.pth",
    config="runs/site42/crlite_site42.yaml",   # outside the wheel/package
)
```

Architecture keys are read from that YAML before the model is built. Common
knobs include `embed_dim`, `token_len`, `top_k`, `crop_size`,
`point_cloud_size`, `vit.depth`, `vit.num_heads`, `gnn.num_layers`,
`gnn.hidden_dim`, `similarity_head.hidden_dims`, and the CalibRefine
`dense.*` widths. Changing shapes requires weights trained with the same cfg;
released checkpoints should stay with their packaged configs.

The exact input contract (RGB `uint8`, finite `float32` meters, shape and
sync requirements) is specified in
[docs/protocol.md](docs/protocol.md) and enforced by
`validate="strict" | "warn" | "off"`.

## See it in action

Rendered from the public A9 (TUMTraf intersection) test cache with real
checkpoints (`scripts/make_demo_media.py`):

![Matching demo on an A9 test frame](docs/assets/a9_matching.gif)

`matcher.match()` scores every camera x LiDAR detection pair and locks the
confident argmax per camera box (green links span the image and the bird's-eye
LiDAR view); below-threshold rows stay unmatched on purpose.

![Calibrated camera-LiDAR projection on A9](docs/assets/a9_calibration.gif)

Those matches are what `matcher.calibrate()` feeds to PnP/RANSAC to recover
the camera-LiDAR extrinsics without targets — once calibrated, the cloud and
3D detections land on the scene. One-shot adaptation (`matcher.oneshot()`)
then uses that projection as a label-free supervisor to keep adapting the
matcher on-site. Details and caveats: [docs](https://radar-lab.github.io/XCalib/#see-it-in-action).

## Model zoo

| Model             | Backbone            | Position encoding                        | Tier            |
|-------------------|---------------------|------------------------------------------|-----------------|
| `crlite`          | ResNet + PointNet   | Enhanced 3D (2D-sin + depth MLP + 3D MLP)| paper           |
| `crlite_2dpe`     | ResNet + PointNet   | 2D-only (`pe_mode=2d_only`)              | paper           |
| `crlite_vit_exp1` | crop-ViT + PointNet | none                                     | paper           |
| `crlite_vit_exp3` | crop-ViT + PointNet | Enhanced 3D                              | paper           |
| `calibrefine`     | ResNet + PointNet2  | 2D sinusoidal (pairwise)                 | paper baseline  |
| `crlite_vit_exp4` | crop-ViT + GAT      | edge-aware GAT over boxes & centres      | delivery only   |

`calibrefine` is our standalone port of the Common Feature Discriminator from
[CalibRefine](https://github.com/radar-lab/Lidar_Camera_Automatic_Calibration)
(Cheng et al., IEEE TIM 2026, [arXiv:2502.17648](https://arxiv.org/abs/2502.17648)),
kept as the pairwise baseline the paper compares against.

Each released model ships with a matching checkpoint and config. See
[docs/hub.md](docs/hub.md) for public loading and download conventions;
[docs/adding-models.md](docs/adding-models.md) for how new models land here.

## Datasets

HDF5 matching caches (schema: [docs/hdf5-format.md](docs/hdf5-format.md)):

- `a9_dataset_r02_s01`: public Hub artifact for the accepted-paper release.

```bash
xcalib pull-dataset --site a9_dataset_r02_s01 --split test
```

`load_dataset(site, split)` prefers a checkout-local cache and falls back to
released Hub artifacts when available. The A9 caches derive from the TUM
Traffic (A9) dataset (CC BY-NC-ND 4.0); users remain responsible for following
upstream dataset terms.

The A9 caches also carry each LiDAR box as an oriented cuboid. By default the loader hands
out the boxes exactly as the cache stores them, which is what every released weight and
reported number uses. `box_source="envelope"` (accepted by `load_dataset`, `train` and
`Matcher.train`) hands out the extents that enclose each cuboid instead, so the axis-aligned
point-cloud crop keeps vehicles that are turned against the axes whole. Weights are tied to
the source they were trained with; see
[3D boxes: stored and envelope](docs/hdf5-format.md#3d-boxes-stored-and-envelope).

## CLI

```bash
xcalib demo         --model crlite_vit_exp1 --weights ... --device cpu
xcalib export-onnx  --model crlite --site a9_dataset_r02_s01
xcalib build-trt    --model crlite --site a9_dataset_r02_s01 --precision fp16
xcalib pull-weights --model crlite --site a9_dataset_r02_s01
xcalib pull-dataset --site a9_dataset_r02_s01 --split test
xcalib version
xcalib-review --scene capture/site1 --camera cam_front   # review window; needs XCalib[gui]
```

## Reproducing the paper

Validation commands, public A9 Table I numbers, frozen UTC reference artefacts,
and latency benchmarks moved to **[docs/paper.md](docs/paper.md)**.

Public reproducibility is anchored on A9. UTC3/UTC4 entries are frozen
reference artefacts for manuscript reconciliation and can only be rerun by
partners who hold the institutional UTC HDF5 caches and weights; those data are
not shipped in this public repository.

```bash
pixi run validate-a9          # public A9 Table I column
pixi run benchmark            # synthetic latency figure, no UTC data required
```

Frozen evaluation summaries cited by the supplemental:
[docs/evidence/index.md](docs/evidence/index.md).

## Repository layout

```text
./
├── pyproject.toml            packaging + pixi env + ruff (single config file)
├── mkdocs.yml                docs site (GitHub Pages)
├── AGENTS.md                 entry point for contributors and coding agents
├── src/xcalib/               the package (ships in the wheel)
│   ├── engine/               matcher, wrappers, exporter, trt, trainer
│   ├── models/               architectures + registry
│   ├── oneshot/              PnP/RANSAC calibration + refinement, pseudo-labels, adapters
│   ├── continual/            corrections corpus, gated continual refine, revisions
│   ├── gui/                  `xcalib-review` Qt window + scene IO
│   ├── hub/                  Hub download helpers for weights + datasets
│   ├── data/                 cropping helpers + HDF5 loader (lazy h5py)
│   ├── utils/                config, io, metrics, torch_check
│   ├── cfg/                  packaged per-(model, site) YAML configs
│   ├── assign.py             one-to-one assignment + geometric gate
│   ├── capture.py            SceneRecorder: save frames for review
│   ├── calibration_refine.py physical-pose refinement candidate + validation gate
│   ├── visualization.py      matching + calibration overlays
│   ├── protocol.py           input contract (validate_frame_inputs, intrinsics)
│   └── cli.py                `xcalib` console entry point
├── scripts/                  public demos, smoke helpers, and docs guards
│   └── paper/                paper validation, export, TRT, and benchmark scripts
├── demo/                     quickstart + continual-refine notebooks, numbered scripts
├── tests/                    grouped pytest suite (smoke/unit/integration)
├── docs/                     docs site sources (see mkdocs.yml nav)
│   ├── agents/               contributor workflow, code index, conventions
│   └── evidence/             frozen JSON evidence linked from the paper docs
├── checkpoints/              optional local .pth files (gitignored)
└── datasets/                 optional local HDF5 caches (gitignored)
```

## API stability

Only the symbols exported by `xcalib.__all__` (`Matcher`, `train`,
`load_dataset`, result dataclasses, pair-selection and deployed-site helpers,
protocol types) are stable across minor versions. Anything imported from
submodules is internal and may change without notice.

## License and citation

Code is licensed under [Apache-2.0](LICENSE). The A9/TUM Traffic dataset has
its own CC BY-NC-ND 4.0 terms.

If you use this package in academic work, please cite the xcalib paper. The
entry below is accepted-paper metadata; replace the paper URL / DOI once the
publisher page is live:

```bibtex
@article{guo2026xcalib,
  author  = {Guo, Lihao and Tang, Jiahao and Bang, Tam and Zhang, Tianya and
             Harris, Austin and Sartipi, Mina and Cao, Siyang},
  title   = {Position Encoding in Detection-Based LiDAR--Camera Matching:
             A Diagnostic Study at Infrastructure Sites},
  journal = {IEEE Sensors Letters},
  year    = {2026},
  note    = {Accepted. Paper URL pending. Code:
             https://github.com/radar-lab/XCalib},
}
```

For the `calibrefine` baseline and the auto-calibration framework this work
builds on, cite:

```bibtex
@article{cheng2026calibrefine,
  author  = {Cheng, Lei and Guo, Lihao and Zhang, Tianya and Bang, Tam and
             Harris, Austin and Hajij, Mustafa and Sartipi, Mina and
             Cao, Siyang},
  title   = {CalibRefine: Deep Learning-Based Online Automatic Targetless
             LiDAR-Camera Calibration With Iterative and Attention-Driven
             Post-Refinement},
  journal = {IEEE Transactions on Instrumentation and Measurement},
  volume  = {75},
  year    = {2026},
  note    = {arXiv:2502.17648. Code:
             https://github.com/radar-lab/Lidar_Camera_Automatic_Calibration},
}
```

Contact: Lihao Guo — `leolihao@arizona.edu`.
