Metadata-Version: 2.4
Name: flash-rt
Version: 0.2.0
Summary: FlashRT — realtime CUDA inference engine for small-batch, latency-sensitive AI workloads. Ships VLA frontends for Pi0, Pi0.5, Pi0-FAST, GROOT N1.6/N1.7.
License: Apache-2.0
Project-URL: Homepage, https://github.com/flashrt-project/FlashRT
Project-URL: Repository, https://github.com/flashrt-project/FlashRT
Project-URL: Issues, https://github.com/flashrt-project/FlashRT/issues
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: GPU :: NVIDIA CUDA
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pyyaml
Provides-Extra: torch
Requires-Dist: torch; extra == "torch"
Requires-Dist: safetensors; extra == "torch"
Requires-Dist: sentencepiece; extra == "torch"
Provides-Extra: chameleon
Requires-Dist: torch; extra == "chameleon"
Requires-Dist: safetensors; extra == "chameleon"
Requires-Dist: transformers<4.56,>=4.43; extra == "chameleon"
Requires-Dist: Pillow; extra == "chameleon"
Provides-Extra: jax
Requires-Dist: jax; extra == "jax"
Requires-Dist: jaxlib; extra == "jax"
Requires-Dist: ml_dtypes; extra == "jax"
Requires-Dist: orbax-checkpoint; extra == "jax"
Requires-Dist: flax; extra == "jax"
Provides-Extra: server
Requires-Dist: fastapi; extra == "server"
Requires-Dist: uvicorn; extra == "server"
Provides-Extra: eval
Requires-Dist: opencv-python; extra == "eval"
Requires-Dist: tqdm; extra == "eval"
Provides-Extra: motus
Requires-Dist: nvidia-cublas<14,>=13.1; extra == "motus"
Provides-Extra: minimax-remover
Requires-Dist: diffusers; extra == "minimax-remover"
Requires-Dist: einops; extra == "minimax-remover"
Requires-Dist: scipy; extra == "minimax-remover"
Requires-Dist: sageattention; extra == "minimax-remover"
Provides-Extra: flash-attn
Requires-Dist: flash-attn; extra == "flash-attn"
Provides-Extra: thor-fa4
Requires-Dist: nvidia-cutlass-dsl==4.5.1; extra == "thor-fa4"
Requires-Dist: quack-kernels==0.4.1; extra == "thor-fa4"
Provides-Extra: all
Requires-Dist: flash-rt[chameleon,eval,jax,minimax-remover,motus,server,torch]; extra == "all"
Dynamic: license-file

<p align="center">
  <img src="FlashRT.png" alt="FlashRT" width="640">
</p>

# FlashRT

**FlashRT is a high-performance realtime inference engine for small-batch, latency-sensitive AI workloads.**

<p align="center">
  | <a href="https://arxiv.org/abs/2606.20537"><b>Paper</b></a>
  | <a href="https://github.com/flashrt-project/FlashRT-HF-kernels"><b>HF Kernels</b></a>
  | <a href="https://github.com/LiangSu8899/FlashRT-Nexus"><b>Nexus</b></a> |
</p>

A general kernel library composed into static graphs — no ONNX export, no engine compilation, no per-driver rebuild. Hand-written kernels (norm / activation / fusion / RoPE / FP8 / NVFP4 GEMM / attention) cover standard transformer, DiT, and SigLIP primitives. The composition pattern itself is hardware-agnostic; today the codebase ships with NVIDIA implementations spanning edge to server (Jetson AGX Thor through A100 / RTX 4090 / 5090).

The flagship integration today is **VLA control** — production frontends for Pi0, Pi0.5, GROOT N1.6, GROOT N1.7, and Pi0-FAST, validated on LIBERO where applicable. The same kernel set also powers BAGEL world-model research paths, Higgs Audio v3 TTS, Wan2.2 / Motus video-policy paths, and **single-stream LLM inference** with Qwen3.6-27B NVFP4 long-context serving. The pattern is workload-shaped (small-batch realtime), not model-class-shaped.

Existing inference tooling is shaped for different workloads — TensorRT for tactic-search compile to frozen engines, vLLM / SGLang for high-batch LLM serving. FlashRT targets the small-batch realtime cell with hand-tuned kernels and no compile step.

## Demos

Each film is one checkpoint executed several ways, side by side on **one wall
clock**, each pane advancing at the rate it was actually measured at. The GIFs
below are sped up to fit the page; the [real-time recordings](https://github.com/flashrt-project/FlashRT-assets/tree/main/demo/mp4) and the
[full walkthrough](https://huggingface.co/spaces/liangsu9988/fast-kernels-are-not-fast-pipelines) play one second per second.

### VLA

**π0.5 on a Jetson AGX Thor.** LIBERO-spatial task 1, one flow-matching sample
held fixed. Two hosts as their authors ship them, then the same model
accelerated two ways. Per-decision latency **266.0 / 303.6 → 36.1 → 29.2 ms**,
3.8 Hz to **34.2 Hz** of policy decisions; the task finishes in **6.0 s**
instead of 23.5 s, and all four arms complete it.

<img src="https://github.com/flashrt-project/FlashRT-assets/raw/main/demo/gif/thor_pi05.gif" alt="thor_pi05" width="100%">

<sub>2× playback. <a href="https://github.com/flashrt-project/FlashRT-assets/blob/main/demo/mp4/thor_pi05.mp4">Real-time recording</a>.</sub>

<details>
<summary><strong>More VLA films</strong> — one checkpoint four ways, and the same checkpoint on two hosts</summary>

**π0.5 · RTX 5090 — four ways of executing one checkpoint.** The host as
shipped, the same host compiled, the same host with structures attached, and
the hand-written FlashRT pipeline. **107.5 → 58.9 → 25.6 → 21.6 ms**, i.e.
9.3 → 46.5 Hz. Same task, same initial state in every pane.

<img src="https://github.com/flashrt-project/FlashRT-assets/raw/main/demo/gif/pi05_race.gif" alt="pi05_race" width="100%">

<sub>1.5× playback. <a href="https://github.com/flashrt-project/FlashRT-assets/blob/main/demo/mp4/pi05_race.mp4">Real-time recording</a>.</sub>

**GR00T N1.7 · RTX 5090.** LIBERO-10 task 4, two camera views.
**44.4 → 25.4 → 17.8 → 15.8 ms**, 22.5 → 63.1 Hz.

<img src="https://github.com/flashrt-project/FlashRT-assets/raw/main/demo/gif/groot_race.gif" alt="groot_race" width="100%">

<sub>3× playback. <a href="https://github.com/flashrt-project/FlashRT-assets/blob/main/demo/mp4/groot_race.mp4">Real-time recording</a>.</sub>

**GR00T N1.7 · Jetson AGX Thor.** The same checkpoint and one flow-matching
sample shared by all four panes. **109.5 / 154.2 → 28.4 → 28.1 ms** — the
explicit structure book and the hand-written pipeline land 0.25 ms apart.

<img src="https://github.com/flashrt-project/FlashRT-assets/raw/main/demo/gif/thor_groot.gif" alt="thor_groot" width="100%">

<sub>5× playback. <a href="https://github.com/flashrt-project/FlashRT-assets/blob/main/demo/mp4/thor_groot.mp4">Real-time recording</a>.</sub>

**π0.5 under two independent hosts.** LeRobot **107.5 → 25.6 ms**, OpenPI
**41.7 → 28.8 ms**. Two hosts that start 2.58× apart end within 12% of each
other; each is measured against the form its own authors ship.

<img src="https://github.com/flashrt-project/FlashRT-assets/raw/main/demo/gif/pi05_cross.gif" alt="pi05_cross" width="100%">

<sub>1.5× playback. <a href="https://github.com/flashrt-project/FlashRT-assets/blob/main/demo/mp4/pi05_cross.mp4">Real-time recording</a>.</sub>

**GR00T N1.7 under Isaac-GR00T and the LeRobot port.** One NVIDIA checkpoint,
two hosts: **44.4 → 17.8 ms** and **41.8 → 18.6 ms**. The two hosts agree to a
cosine of 0.999995 on the one decision both made from a byte-identical
observation.

<img src="https://github.com/flashrt-project/FlashRT-assets/raw/main/demo/gif/groot_cross.gif" alt="groot_cross" width="100%">

<sub>3× playback. <a href="https://github.com/flashrt-project/FlashRT-assets/blob/main/demo/mp4/groot_cross.mp4">Real-time recording</a>.</sub>

</details>

### VLM

**Qwen3-VL-8B on the unedited `transformers` host.** One image, one prompt,
greedy. Three arms: the host as shipped, the host compiled with a static cache,
and the host with structures attached. Decode **65.6 → 82.9 → 145.7 tok/s**,
**1.76×** over the host's own compiled form; TTFT ~30 ms in all three. 180
seats, no host edit.

<img src="https://github.com/flashrt-project/FlashRT-assets/raw/main/demo/gif/qwen3vl.gif" alt="qwen3vl" width="100%">

<sub>1× playback. <a href="https://github.com/flashrt-project/FlashRT-assets/blob/main/demo/mp4/qwen3vl.mp4">Real-time recording</a>.</sub>

### LLM

**Qwen3.6-35B-A3B on one 32 GB card.** 67 GB of BF16 weights do not fit;
`quantize_on_adopt` regrids the expert banks before the model reaches the
device (22.3 GiB resident) and the model runs. Decode **51.6 → 203.5 → 284.9
tok/s**, the third pane being the checkpoint's own draft head at 3.43 tokens
accepted per round — shown on the prompt where it wins, dropped on the one
where it does not.

<img src="https://github.com/flashrt-project/FlashRT-assets/raw/main/demo/gif/qwen36.gif" alt="qwen36" width="100%">

<sub>1× playback. <a href="https://github.com/flashrt-project/FlashRT-assets/blob/main/demo/mp4/qwen36.mp4">Real-time recording</a>.</sub>

**Attached inside vLLM, under batch.** The same 35B mixture-of-experts served
by vLLM on a Jetson AGX Thor at 1, 4, 8 and 16 concurrent requests — four
chapters in one file, each pane a live stream. Aggregate throughput
**38.4 → 77.4**, **77.5 → 182.8**, **98.9 → 245.5**, **211.4 → 302.1 tok/s**;
every level gains. A routed mixture of experts does not dilute the way a dense
model does — each token reads its own experts, so expert weight traffic grows
with the batch instead of being shared — which is why the gain climbs to batch
8 rather than falling away.

<img src="https://github.com/flashrt-project/FlashRT-assets/raw/main/demo/gif/thor_concurrency.gif" alt="thor_concurrency" width="100%">

<sub>3.5× playback. <a href="https://github.com/flashrt-project/FlashRT-assets/blob/main/demo/mp4/thor_concurrency.mp4">Real-time recording</a>.</sub>

<details>
<summary><strong>More LLM films</strong> — inside vLLM and SGLang, single stream</summary>

**Inside two serving engines.** Qwen3-8B, single stream, 144 seats bound by a
hook that fires after the engine loads and before its first trace. vLLM
**99.1 → 145.3 tok/s**, SGLang **101.2 → 203.0 tok/s**, TTFT falls in both.
Neither engine is forked: each keeps its scheduler, its memory planner and its
own graph, and the seats go inside that.

<img src="https://github.com/flashrt-project/FlashRT-assets/raw/main/demo/gif/engines.gif" alt="engines" width="100%">

<sub>1× playback. <a href="https://github.com/flashrt-project/FlashRT-assets/blob/main/demo/mp4/engines.mp4">Real-time recording</a>.</sub>

</details>

### Video

**Wan2.2 TI2V-5B — one clip, four ways of making it.** 480×480, 33 frames, 20
denoise steps, one prompt and one seed, all four arms in one process against
one baseline. **6.48 → 5.23 → 2.11 → 1.68 s**; per transformer call
**162.2 → 41.8 ms**. The third arm runs 4-bit on every call whose own per-step
score holds the band and hands the rest to FP8 — 171 calls at four bits, 50 at
eight, decided per call by measurement. Every pane plays its own clip at the
end.

<img src="https://github.com/flashrt-project/FlashRT-assets/raw/main/demo/gif/wan22.gif" alt="wan22" width="100%">

<sub>1× playback. <a href="https://github.com/flashrt-project/FlashRT-assets/blob/main/demo/mp4/wan22.mp4">Real-time recording</a>.</sub>

---

## FlashRT is fast with:

- **hand-written CUDA kernels**: norm, activation, residual+norm+quant fusion, RoPE / qkv-split, FP8 / NVFP4 GEMM, cuBLASLt FP8, CUTLASS SM100 FP8, vendored Flash-Attention 2, Thor CUTLASS FMHA
- **Static CUDA Graph capture** of the entire forward — zero Python overhead at replay
- **Production FP8 (E4M3) and NVFP4** with automatic per-tensor calibration, JSON-cached to disk
- **No compile, no export**: direct safetensors / Orbax loading, first call ~3 s, every call after is graph replay
- Survives CUDA driver upgrades, GPU swaps, and prompt changes without rebuild
- **Serving hosts** for OpenAI-compatible LLM/audio endpoints and robot execution-state scenarios

## FlashRT is easy to use with:

- **3-line API**: `flash_rt.load_model(...).predict(images, prompt)`
- **Auto-dispatched hardware**: same code path on Jetson Thor / RTX 5090 / RTX 4090
- **PyTorch and JAX frontends** share one kernel binary, equivalent results (cosine ≥ 0.999)
- **Plugin model registration** — add a new VLA via one frontend file + a declarative `WEIGHT_SPEC`, no fork required
- **LIBERO benchmark integration** out of the box; ~6 minutes from `git clone` to first inference
- **FlashRT Structures** — attach the kernel catalog onto an **unmodified** PyTorch host, no fork and no edit to its source; works on `lerobot`, Isaac-GR00T, `openpi`, `transformers`, `diffusers`, and inside vLLM / SGLang. Its own package and repository: [FlashRT-Structures](https://github.com/flashrt-project/FlashRT-Structures)

```python
import flashrt_structures as structures    # pip install flashrt-structures

plan = structures.attach(model, forward)   # discover → calibrate → gate → activate
print(structures.explain(plan))            # bound / routed / kept-at-host / refused, with reasons

loop = structures.decode_loop(model, max_len=4096)      # serving door
out  = loop.generate(input_ids, max_new_tokens=256)
```

Structures: [repository](https://github.com/flashrt-project/FlashRT-Structures) · [explicit pipeline examples](https://github.com/flashrt-project/FlashRT-Structures/tree/main/examples) · [walkthrough with films](https://huggingface.co/spaces/liangsu9988/fast-kernels-are-not-fast-pipelines)

See [Supported Models](#supported-models), [Hardware Support](#hardware-support), and [Benchmark](#benchmark) for the current map.

## News

- **Aug 2026** — **AMD Instinct MI350X (CDNA4)** joins the VLA line: **Pi0.5 at 16.4 ms** (2.47× over `torch.compile` max-autotune, 7.1× over eager) and **GROOT N1.7 at 16.0 ms** full-frame (4.24× over eager). A standalone ROCm/HIP backend — hand-written HIP kernels, hipBLASLt FP8 GEMM, MFMA attention, HIP graph capture — arch-gated to gfx950 at both build and run time, carrying no CUDA assumptions. See [AMD deployment](docs/deployment_amd.md) and [Pi0.5 on MI350X](docs/deployment_amd_pi05.md).
- **Aug 2026** — **FlashRT Structures** ships: the kernel catalog attaches to an *unmodified* PyTorch host — `lerobot`, Isaac-GR00T, `openpi`, `transformers`, `diffusers`, and inside vLLM / SGLang — with no fork and no edit to the host. Measured on VLA, VLM, LLM and video models across RTX 5090 and Jetson AGX Thor. Now its own package: [FlashRT-Structures](https://github.com/flashrt-project/FlashRT-Structures). See also the [walkthrough with films](https://huggingface.co/spaces/liangsu9988/fast-kernels-are-not-fast-pipelines).
- **Jul 2026** — **Cosmos3-Edge on Jetson AGX Thor** reaches **6.60x** no-cache AV denoise speedup; its NVFP4 Reasoner decodes text/image/video at **104.3 / 112.6 / 108.7 tok/s**. The earlier **Cosmos3-Nano RTX 5090** path reaches **4.8 s E2E** with FP8 for a 480p, 49-frame, 10-step workload. See [Cosmos3-Edge](docs/cosmos3_edge_thor.md) and [Cosmos3-Nano](docs/cosmos3_video_usage.md).
- **Jul 2026** — **Native C++ PI0.5** now owns checkpoint loading, preprocessing, graph capture, FP8 calibration, and action postprocessing behind `frt_model_runtime_v1`. [FlashRT Nexus](https://github.com/LiangSu8899/FlashRT-Nexus) adopts that ABI for embedded robot loops, HTTP serving, and execution-state capsules.
- **Jun 2026** — **Qwen3-VL-8B on RTX 5090** adds an NVFP4 language stack, FP8 ViT, whole-prefill CUDA Graph, image/multi-image/video inputs, **~100 ms** full-resolution TTFT, and **~150 tok/s** decode. At 0.5 MP, TTFT is **~32 ms**. See [Qwen3-VL RTX 5090](docs/qwen3_vl_nvfp4.md).

<details>
<summary><strong>More</strong></summary>

- **Jun 2026** — **FlashRT HF Kernels** are published through the Hugging Face Kernel Hub under the `flashrt` namespace. See [source packages](https://github.com/flashrt-project/FlashRT-HF-kernels) and [huggingface.co/flashrt](https://huggingface.co/flashrt).
- **Jul 2026** — Thanks to [@chenping9999](https://github.com/chenping9999) for the [MiniMax-Remover FP8/NVFP4 pipeline](docs/minimax_remover_usage.md), including fused transformer and VAE kernels.
- **Jul 2026** — Thanks to [@diantoudedianshan](https://github.com/diantoudedianshan) for [RTC temporal fusion](docs/rtc_temporal_fusion.md) and the [VLASh projected-state asynchronous runner](docs/vlash.md).
- **Jul 2026** — Thanks to [@heiheiha798](https://github.com/heiheiha798) for Qwen3-VL SM89 FP8 decode optimization for 8B/2B, bounded graph caches, and GROOT N1.7 SM89 graph fixes. See [Qwen3-VL SM89](docs/qwen3_vl_fp8_sm89.md).
- **Jul 2026** — Thanks to [@strayberry](https://github.com/strayberry) for the Qwen3-VL Jetson Orin BF16 frontend.
- **Jun 2026** — **Higgs Audio v3 TTS-4B** adds a kernelized FP8/BF16 decode path, streaming-friendly generation, and FastAPI serving. See [usage](docs/higgs_audio_v3.md#3-quickstart), [performance](docs/higgs_audio_v3.md#performance), and [serving](serving/higgs_audio_agent/README.md).
- **Jun 2026** — The native `serving/` and runtime contracts were introduced for OpenAI-compatible LLM/audio hosts and robot execution-state hosts. See [serving](serving/README.md), [runtime architecture](docs/architecture.md), and [model-runtime API](docs/model_runtime_api.md).
- **Jun 2026** — Thanks to [@shideqin](https://github.com/shideqin) for the OmniVoice BF16/FP4 acceleration backend, Python pipeline, quickstart, and serving agent. See [OmniVoice serving](serving/omnivoice_agent/README.md).
- **Jun 2026** — Thanks to [@xym808](https://github.com/xym808) for the [MelBandRoformer FP8 audio source-separation pipeline](docs/melband_roformer_usage.md), measured at **3.1x** over PyTorch on RTX 5060 Ti.
- **Jun 2026** — Thanks to [@heiheiha798](https://github.com/heiheiha798) for validated GROOT N1.7 SM89 routing, Pi0.5 JAX FP4 validation on Thor, and deployment build isolation.
- **May 2026** — **Qwen3.6-27B NVFP4** supports 256 K context on one RTX 5090, OpenAI-compatible serving, FP8-KV long-context verify, and **145 tok/s** warm decode at 256 K. See [Qwen3.6 NVFP4](docs/qwen36_nvfp4.md).
- **May 2026** — **Qwen3-8B NVFP4** text serving reaches **9.1 ms TTFT at P=64** and **150 tok/s** warm decode on RTX 5090. See [Qwen3-8B NVFP4](docs/qwen3_8b_nvfp4.md).
- **May 2026** — **Wan2.2 TI2V-5B**, **LingBot-VLA**, and the **Motus RTX beta** were added. See [Wan2.2](docs/wan22_usage.md), [LingBot](docs/lingbot_usage.md), and [Motus](docs/motus_usage_beta.md).
- **May 2026** — Thanks to [@cuihengrui35](https://github.com/cuihengrui35) for RTX 5060 Ti Pi0.5 results (**41.4 ms / ~24 Hz**, LIBERO Spatial **344/350**) and [@wangerforcs](https://github.com/wangerforcs) for NVIDIA L40 results (**26.6 ms / 38 Hz**).
- **May 2026** — Thanks to [@gugudeshubao](https://github.com/gugudeshubao) for the Pi0.5 Jetson AGX Orin INT8 port, kernels, frame-cache inference, deployment guide, and benchmarks; thanks to [@strayberry](https://github.com/strayberry) for Orin BF16 validation. See [Orin deployment](docs/deployment_orin.md).

</details>

<a name="performance"></a>

## Benchmark

Baseline comparisons and source methodology live in [Benchmark Comparison](docs/benchmark_comparison.md).

#### Pi0.5

| Hardware | Mode | Latency | Throughput | Source |
|---|---|---:|---:|---|
| Jetson AGX Thor | FP8, 2-view | **44.0 ms** | **23 Hz** | [Thor VLA](examples/thor/README.md#thor-vla-performance) |
| Jetson AGX Thor | NVFP4 + FA4, 1-view † | **23.01 ms** | **43 Hz** | [Pi0.5 Thor NVFP4](docs/pi05_thor_decoder_fp4_e2e.md) |
| Jetson AGX Thor | NVFP4 + FA4, 2-view | **27.17 ms** | **37 Hz** | [Pi0.5 Thor NVFP4](docs/pi05_thor_decoder_fp4_e2e.md) |
| Jetson AGX Thor | NVFP4 + FA4, 3-view | **31.74 ms** | **32 Hz** | [Pi0.5 Thor NVFP4](docs/pi05_thor_decoder_fp4_e2e.md) |
| RTX 5090 | FP8, 2-view | **17.58 ms** | **57 Hz** | [Blackwell VLA](examples/blackwell/README.md#vla-latency-rtx-5090) |
| AMD Instinct MI350X | FP8, CDNA4 | **16.4 ms** | **61 Hz** | [AMD Pi0.5](docs/deployment_amd_pi05.md) |

For the Pi0.5 Thor rows, use about 300 warmup calls; 20 can be insufficient.
See the [warmup and latency reference notes](docs/pi05_thor_decoder_fp4_e2e.md#warmup-and-latency-references).

† At 1 view the per-sample action cosine against the FP8 reference does
not clear the 0.995 gate (worst sample 0.971): with single-view input the
flow-matching field itself is near a decision boundary on some samples,
so trajectories land in a different action basin. See
[one-view fidelity](docs/pi05_thor_decoder_fp4_e2e.md#one-view-fidelity-diagnosis-and-passing-configuration-2026-07-27)
for the diagnosis and a configuration that does clear the gates at 1 view.

#### Pi0

| Hardware | Mode | Latency | Throughput | Source |
|---|---|---:|---:|---|
| Jetson AGX Thor | FP8, 2-view | **45.8 ms** | **22 Hz** | [Thor VLA](examples/thor/README.md#thor-vla-performance) |
| RTX 5090 | FP8, 1-view | **18.43 ms** | **54 Hz** | [API snippets](#api-snippets) |
| RTX 5090 | FP8, 2-view | **21.16 ms** | **47 Hz** | [API snippets](#api-snippets) |
| RTX 5090 | FP8, 3-view | **24.48 ms** | **41 Hz** | [API snippets](#api-snippets) |

#### GROOT N1.6

| Hardware | Mode | Latency | Throughput | Source |
|---|---|---:|---:|---|
| Jetson AGX Thor | T=16 | **41 ms** | **24 Hz** | [Thor VLA](examples/thor/README.md#thor-vla-performance) |
| Jetson AGX Thor | T=50 | **45 ms** | **22 Hz** | [Thor VLA](examples/thor/README.md#thor-vla-performance) |
| RTX 5090 | T=16, 2-view | **12.53 ms** | **80 Hz** | [Blackwell VLA](examples/blackwell/README.md#vla-latency-rtx-5090) |
| RTX 5090 | T=50, 2-view | **13.08 ms** | **76 Hz** | [Blackwell VLA](examples/blackwell/README.md#vla-latency-rtx-5090) |

#### GROOT N1.7

| Hardware | Mode | Latency | Throughput | Source |
|---|---|---:|---:|---|
| Jetson AGX Thor | NVFP4 + FA4, LIBERO 1-view | **23.7 ms** | **42 Hz** | [N1.7 Thor NVFP4](USAGE.md#groot-n17-thor) |
| Jetson AGX Thor | FP8, LIBERO 1-view | **36.8 ms** | **27 Hz** | [Benchmark comparison](docs/benchmark_comparison.md#groot-n17-on-jetson-agx-thor) |
| Jetson AGX Thor | NVFP4 + FA4, 2-view | **29.9 ms** | **33 Hz** | [N1.7 Thor NVFP4](USAGE.md#groot-n17-thor) |
| Jetson AGX Thor | FP8, 2-view | **50.2 ms** | **20 Hz** | [GROOT N1.7 API](#groot-n17-rtx) |
| RTX 5090 | FP8, 2-view base, full graph | **16.6 ms** | **60 Hz** | [GROOT N1.7 API](#groot-n17-rtx) |
| AMD Instinct MI350X | FP8 backbone + BF16 DiT, full frame | **16.0 ms** | **62 Hz** | [AMD deployment](docs/deployment_amd.md) |

#### Pi0-FAST

| Hardware | Mode | Latency | Throughput | Source |
|---|---|---:|---:|---|
| Jetson AGX Thor | max-perf | **8.1 ms/token** | **123 tok/s** | [Thor VLA](examples/thor/README.md#thor-vla-performance) |
| RTX 5090 | max-perf | **2.39 ms/token** | **418 tok/s** | [Blackwell VLA](examples/blackwell/README.md#vla-latency-rtx-5090) |

#### LingBot-VLA

| Hardware | Mode | Latency | Throughput | Source |
|---|---|---:|---:|---|
| Jetson AGX Thor | FA4, 10 steps | **64.1 ms** | **16 Hz** | [LingBot usage](docs/lingbot_usage.md#5-accuracy--latency-thor-sm_110-cuda-graph-replay) |
| Jetson AGX Thor | FA4, 25 steps | **97.5 ms** | **10 Hz** | [LingBot usage](docs/lingbot_usage.md#5-accuracy--latency-thor-sm_110-cuda-graph-replay) |
| Jetson AGX Thor | FA4, 50 steps | **155.8 ms** | **6 Hz** | [LingBot usage](docs/lingbot_usage.md#5-accuracy--latency-thor-sm_110-cuda-graph-replay) |

#### Qwen3.6-27B

RTX 5090:

| Mode | Prefill | Decode | Source |
|---|---:|---:|---|
| NVFP4, 128 | **31.4 ms** | **144.9 tok/s** | [Qwen3.6 NVFP4](docs/qwen36_nvfp4.md) |
| NVFP4, 4 K | **350.9 ms** | **129.6 tok/s** | [Qwen3.6 NVFP4](docs/qwen36_nvfp4.md) |
| NVFP4, 16 K | **1.570 s** | **175.4 tok/s** | [Qwen3.6 NVFP4](docs/qwen36_nvfp4.md) |
| NVFP4, 256 K | **87.976 s** | **144.6 tok/s** | [Qwen3.6 NVFP4](docs/qwen36_nvfp4.md) |

Jetson AGX Thor:

| Mode | Prefill | Decode | Source |
|---|---:|---:|---|
| NVFP4, 128 | **268 ms** | **42.8 tok/s** | [Qwen3.6 Thor](docs/qwen36_nvfp4.md#jetson-agx-thor-numbers) |
| NVFP4, 16 K | **19.23 s** | **52.9 tok/s** | [Qwen3.6 Thor](docs/qwen36_nvfp4.md#jetson-agx-thor-numbers) |

DGX Spark / GB10:

| Mode | Prefill | Decode | Source |
|---|---:|---:|---|
| NVFP4, 128 | **170.1 ms** | **40.42 tok/s** | [Qwen3.6 Spark](docs/qwen36_spark.md#performance) |
| NVFP4, 16 K | **8.545 s** | **54.94 tok/s** | [Qwen3.6 Spark](docs/qwen36_spark.md#performance) |

#### Qwen3.6-35B-A3B

RTX 5090:

| Mode | Prefill | Decode | Source |
|---|---:|---:|---|
| NVFP4, 64 | **40.42 ms** | **257.95 tok/s** | [Qwen3.6-MoE usage](docs/qwen36_moe_usage.md#validation) |

Jetson AGX Thor, against vLLM 0.26.0 on the same part and protocol:

| Mode | Prefill | Decode | Source |
|---|---:|---:|---|
| NVFP4, 20 | **89.5 ms** (vLLM 102.3) | | [Qwen3.6-MoE Thor](docs/qwen36_moe_usage.md#jetson-agx-thor-numbers) |
| NVFP4, 1 K | **216.0 ms** (vLLM 319.4) | **87.1 tok/s** (vLLM 31.6) | [Qwen3.6-MoE Thor](docs/qwen36_moe_usage.md#jetson-agx-thor-numbers) |
| NVFP4, 2 K | **379.6 ms** (vLLM 495.0) | **86.3 tok/s** (vLLM 31.5) | [Qwen3.6-MoE Thor](docs/qwen36_moe_usage.md#jetson-agx-thor-numbers) |
| NVFP4, 32 K | **7207.5 ms** (vLLM 7231.8) | | [Qwen3.6-MoE Thor](docs/qwen36_moe_usage.md#jetson-agx-thor-numbers) |

TTFT leads at every length from 20 to 32768 tokens; 128 K context reaches the
board at 2470 tok/s of prefill. The decode column was taken before a later
round that moved the steady step from 89.0 to 102.6 tok/s, and vLLM's side is
unaffected by it, so the ratios shown are lower bounds.

Speculative decode with the MTP head reaches **106.74 tok/s** against 100.35
plain in the same process, emitting the same tokens as greedy decoding. See
[speculative decode](docs/qwen36_moe_usage.md#speculative-decode).

#### Qwen3-8B

| Hardware | Mode | Prefill | Decode | Source |
|---|---|---:|---:|---|
| RTX 5090 | P=64 | **9.1 ms** | **150 tok/s** | [Qwen3-8B NVFP4](docs/qwen3_8b_nvfp4.md) |
| RTX 5090 | P=1024 | **24.8 ms** | **150 tok/s** | [Qwen3-8B NVFP4](docs/qwen3_8b_nvfp4.md) |

#### Qwen3-VL-8B

RTX 5090, NVFP4 language stack + FP8 ViT, image + text:

| Input profile | Prompt tokens | TTFT | Decode | Source |
|---|---:|---:|---:|---|
| Full resolution | 1581 | **~100 ms** | **~150 tok/s** | [Qwen3-VL RTX 5090](docs/qwen3_vl_nvfp4.md#1-headline-performance) |
| 0.5 MP cap | 473 | **~32 ms** | **~150 tok/s** | [Qwen3-VL resolution sweep](docs/qwen3_vl_nvfp4.md#ttft-vs-resolution-the-dominant-knob) |

#### Qwen3-VL-2B on Jetson

Official BF16 checkpoint, full-resolution image + text (1581 prompt tokens,
6256 vision patches), with opt-in weight-only decode quantization:

| Hardware | Decode tier | Prefill | Decode | Source |
|---|---|---:|---:|---|
| Jetson AGX Thor | BF16 | **230 ms** (eager) | **66.4 tok/s** | [Qwen3-VL Jetson Thor](docs/qwen3_vl_thor.md#jetson-thor-validation) |
| Jetson AGX Thor | `w8` (FP8 e4m3) | 230 ms (eager) | **106.6 tok/s** | [Measured tiers](docs/qwen3_vl_thor.md#measured-tiers) |
| Jetson AGX Thor | `w4` (NVFP4 e2m1) | 230 ms (eager) | **158.1 tok/s** | [Measured tiers](docs/qwen3_vl_thor.md#measured-tiers) |
| Jetson AGX Orin | BF16 | 927 ms (graph) | 36.8 tok/s | [Qwen3-VL Jetson Orin](docs/qwen3_vl_rtx_bf16.md#jetson-orin-validation) |

Thor prefill is eager by measurement, not omission: a prefill CUDA Graph
prototype bought 0.3% there (GPU-bound), so it was not shipped; see
[Where prefill time goes](docs/qwen3_vl_thor.md#where-prefill-time-goes).

#### Higgs Audio v3

| Hardware | Mode | Latency | TTFA | Throughput | Source |
|---|---|---:|---:|---:|---|
| RTX 5090 | FP8 AR decode | **3.6 ms/frame** | **~79 ms** | RTF **0.09** | [Higgs performance](docs/higgs_audio_v3.md#performance) |
| RTX 5090 | BF16 AR decode | **6.0 ms/frame** | **~127 ms** | RTF **0.151** | [Higgs performance](docs/higgs_audio_v3.md#performance) |

#### Motus Stage3

| Hardware | Mode | Latency | Throughput | Source |
|---|---|---:|---:|---|
| RTX 5090 | fast profile | **167 ms** | **6.0 Hz** | [Motus usage](docs/motus_usage_beta.md) |
| RTX 5090 | TeaCache | **100 ms** | **10 Hz** | [Motus usage](docs/motus_usage_beta.md) |

#### Wan2.2 TI2V-5B

| Hardware | Mode | Generation time | Source |
|---|---|---:|---|
| RTX 5090 | 720p, 121f, 20 steps | **178.6 s** | [Wan2.2 benchmarks](docs/wan22_usage.md#benchmarks) |
| RTX 5090 | TeaCache 0.3 | **114.2 s** | [Wan2.2 benchmarks](docs/wan22_usage.md#benchmarks) |

#### Cosmos3-Edge AV inverse dynamics and Reasoner

Jetson AGX Thor, 30-step denoise boundary, 15 hot back-to-back iterations:

| Mode | Denoise P50 | Speedup | Source |
|---|---:|---:|---|
| Official eager | 33.14 s | 1.00x | [Cosmos3-Edge usage](docs/cosmos3_edge_thor.md#performance) |
| FlashRT FP8, no step cache | 5.792 s | **5.72x** | [Cosmos3-Edge usage](docs/cosmos3_edge_thor.md#performance) |
| FlashRT FP8 + NVFP4 FFN, no step cache | 5.020 s | **6.60x** | [Cosmos3-Edge usage](docs/cosmos3_edge_thor.md#performance) |

Reasoner decode throughput uses batch 1, greedy decode, 128 output tokens, and
the public 1705/911/1263-token text/image/video profile:

| Mode | Text | Image | Video | Source |
|---|---:|---:|---:|---|
| FlashRT NVFP4 + whole-step CUDA Graph | **104.3 tok/s** | **112.6 tok/s** | **108.7 tok/s** | [Cosmos3-Edge Reasoner usage](docs/cosmos3_edge_thor.md) |

See the [complete Cosmos3-Edge Thor usage guide](docs/cosmos3_edge_thor.md) for
the SM110 build flags, official AV baseline and fixture workflow, FP8/NVFP4
commands, Reasoner text/image/video inputs, accuracy checks, and limitations.

#### Cosmos3-Nano text-to-video

RTX 5090, 480p, 49 frames, 10-step UniPC:

| Mode | Denoise | VAE decode | E2E | Source |
|---|---:|---:|---:|---|
| Official eager | ~53.5 s | ~2.3 s | **~55.8 s** | [Cosmos3-Nano usage](docs/cosmos3_video_usage.md#1-precision--speed-rtx-5090--sm120) |
| FlashRT BF16, no step cache | 4.4 s | 2.3 s | **6.7 s** | [Cosmos3-Nano usage](docs/cosmos3_video_usage.md#1-precision--speed-rtx-5090--sm120) |
| FlashRT FP8, no step cache | 2.5 s | 2.3 s | **4.8 s** | [Cosmos3-Nano usage](docs/cosmos3_video_usage.md#1-precision--speed-rtx-5090--sm120) |

## Getting Started

- [Install FlashRT](#build--install)
- [Quick Start](#quick-start)
- [API snippets — Pi0 / Pi0.5 / GROOT / Pi0-FAST / Qwen3.6](#api-snippets)
- [Supported Models](#supported-models) · [Hardware Support](#hardware-support) · [Benchmark](#benchmark)
- [Native C++ PI0.5](docs/pi05_native_cpp.md) · [native FP8 calibration](docs/pi05_native_calibration.md) · [model-runtime C ABI](docs/model_runtime_api.md)
- [FlashRT Nexus](https://github.com/LiangSu8899/FlashRT-Nexus) — embedded robot loops, serving, and execution-state capsules
- [Asynchronous VLA execution](docs/vlash.md) · [RTC temporal fusion](docs/rtc_temporal_fusion.md) · [legacy chunk runner](docs/rtc_lite_design.md)
- [Serving](serving/README.md) · [Architecture](docs/architecture.md)
- [Qwen3.6-27B NVFP4 LLM path — quickstart, K selection, measured throughput](docs/qwen36_nvfp4.md) · [Spark usage](docs/qwen36_spark.md) · [parameter reference](docs/qwen36_usage.md) · [OpenAI-compatible server example](serving/qwen36_agent/README.md)
- [Adding a new model](docs/adding_new_model.md)
- [Contributing](CONTRIBUTING.md)
- [Architecture](docs/architecture.md)

## Quick Start

> Already built? Run the snippet below. **Not yet built? See [Build & install](#build--install) first** — `cmake .. && make -j` produces the kernel `.so` files this snippet imports. About 6 minutes from `git clone` to first inference.

```python
import flash_rt   # Python module name; project is FlashRT (see About)

model = flash_rt.load_model(
    checkpoint="/path/to/pi05_checkpoint",
    config="pi05",          # or "pi0", "groot", "groot_n17", "pi0fast"
    framework="torch",      # or "jax"
)

actions = model.predict(
    images=[base_img, wrist_img],
    prompt="pick up the red block",
)
# Pi0.5: actions shape (10, 7) — 10 future steps, 7 DOF
```

First call: ~3 s (calibration + CUDA Graph capture). Every subsequent call: 44 ms graph replay on Thor. No `.engine` file, no rebuild after restart. Full snippets for Pi0 / GROOT / Pi0-FAST in [API snippets](#api-snippets).



## Start here

| If you want to … | Read |
|---|---|
| **Run your first inference** | [Build & install](#build--install) — Docker and native Linux paths |
| **See API examples for all 4 VLA models + the Qwen3.6 LLM** | [API snippets](#api-snippets) |
| **Run Qwen3.6-27B NVFP4 (LLM, 256 K on RTX 5090; Spark/GB10 supported)** | [`docs/qwen36_nvfp4.md`](docs/qwen36_nvfp4.md) — quickstart, K selection, measured throughput · [`docs/qwen36_spark.md`](docs/qwen36_spark.md) — DGX Spark usage and performance · [`docs/qwen36_usage.md`](docs/qwen36_usage.md) — full parameter reference · [`serving/qwen36_agent/`](serving/qwen36_agent/README.md) — OpenAI-compatible HTTP server |
| **Run Qwen3-8B NVFP4 text serving** | [`docs/qwen3_8b_nvfp4.md`](docs/qwen3_8b_nvfp4.md) · [`examples/qwen3_openai_server.py`](examples/qwen3_openai_server.py) |
| **Run Higgs Audio v3 TTS** | [`docs/higgs_audio_v3.md`](docs/higgs_audio_v3.md) — usage + performance · [`serving/higgs_audio_agent/`](serving/higgs_audio_agent/README.md) — HTTP serving |
| **Run Motus RTX beta, TeaCache, or legacy async chunk runner** | [`docs/motus_usage_beta.md`](docs/motus_usage_beta.md) · [`docs/rtc_lite_design.md`](docs/rtc_lite_design.md) |
| **Run Wan2.2 TI2V-5B official-pipeline baseline** | [`docs/wan22_usage.md`](docs/wan22_usage.md) |
| **Run Cosmos3-Edge AV inverse dynamics or Reasoner on Jetson AGX Thor** | [`docs/cosmos3_edge_thor.md`](docs/cosmos3_edge_thor.md) — SM110 build, official AV baseline, optimized denoise, Reasoner text/image/video usage, correctness checks, and benchmarks |
| **Run Cosmos3-Nano text-to-video on RTX 5090** | [`docs/cosmos3_video_usage.md`](docs/cosmos3_video_usage.md) — SM120 build, BF16/FP8 denoise, quality metrics, and benchmark workflow |
| **Use FlashRT kernels through Hugging Face Kernel Hub** | [`flashrt-project/FlashRT-HF-kernels`](https://github.com/flashrt-project/FlashRT-HF-kernels) · [`huggingface.co/flashrt`](https://huggingface.co/flashrt) |
| **Deploy the Python-free PI0.5 C++ pipeline** | [`docs/pi05_native_cpp.md`](docs/pi05_native_cpp.md) — build, configure, calibrate, and execute · [`docs/pi05_native_calibration.md`](docs/pi05_native_calibration.md) — FP8 artifacts · [`docs/model_runtime_api.md`](docs/model_runtime_api.md) — stable C ABI |
| **Integrate FlashRT with Nexus** | [`FlashRT-Nexus`](https://github.com/LiangSu8899/FlashRT-Nexus) — embedded and HTTP deployment over `frt_model_runtime_v1` · [`docs/runtime_contract.md`](docs/runtime_contract.md) — FlashRT-side contract |
| **Run VLA inference asynchronously** | [`docs/vlash.md`](docs/vlash.md) — projected-state VLASh · [`docs/rtc_temporal_fusion.md`](docs/rtc_temporal_fusion.md) — overlapping chunk fusion · [`docs/rtc_lite_design.md`](docs/rtc_lite_design.md) — legacy chunk runner |
| **Run serving hosts** | [`serving/README.md`](serving/README.md) — scenario hosts · [`docs/serving_design.md`](docs/serving_design.md) — capsules and roadmap · [`docs/serving_production.md`](docs/serving_production.md) — production notes |
| **Look up the stable Python API surface** | [`docs/stable_api.md`](docs/stable_api.md) |
| **Integrate a new model into FlashRT** | [`docs/adding_new_model.md`](docs/adding_new_model.md) — end-to-end walkthrough; external plugin pattern in [`docs/plugin_model_template.md`](docs/plugin_model_template.md) |
| **Contribute a bug fix, benchmark, or model path** | [`CONTRIBUTING.md`](CONTRIBUTING.md) — development rules, validation expectations, and PR checklist |
| **Understand the architecture** | [`docs/architecture.md`](docs/architecture.md) — the 8 infrastructure components and how they compose |
| **Use a load-bearing API** (weight loading, attention, calibration) | [`docs/extension/weight_spec.md`](docs/extension/weight_spec.md) · [`docs/extension/attention_backend.md`](docs/extension/attention_backend.md) · [`docs/extension/calibration.md`](docs/extension/calibration.md) |
| **See supported model list** | [Supported Models](#supported-models) |
| **See measured performance** | [Benchmark](#benchmark) · [Benchmark comparison](docs/benchmark_comparison.md) |
| **Know which GPUs have been tested (and how to contribute a run)** | [Hardware Support](#hardware-support) · [Community benchmarks](#community-benchmarks) |
| **Know what kernels ship and whether they fit your model** | [`docs/kernel_catalog.md`](docs/kernel_catalog.md) — the "parts list" with a re-use decision tree |
| **See which fusion patterns exist and why some were rejected** | [`docs/kernel_fusion.md`](docs/kernel_fusion.md) |
| **Understand FP8 calibration mechanics** | [`docs/calibration.md`](docs/calibration.md) |
| **Train a Pi0.5 LoRA fine-tune (FP8 + LoRA, plain or RECAP/ACP-conditioned, PyTorch *or* JAX)** | [`training/README.md`](training/README.md). JAX companion at [`training/jax/README.md`](training/jax/README.md) |
| **Run advantage-conditioned (RECAP / π\*0.6) policies with classifier-free guidance** | [`docs/rl_inference.md`](docs/rl_inference.md) — PyTorch + JAX frontends both supported |
| **See how FlashRT differs from TensorRT / vLLM / SGLang** | [`docs/inference_engine_differences.md`](docs/inference_engine_differences.md) |

---

## Key techniques

The short version: kernel fusion + static FP8 + captured CUDA Graph
+ vendored in-SO Flash-Attention 2. Hand-written CUDA kernels cover
only the memory-bound ops (norm, activation, fusion, quant);
compute-bound GEMM / attention are delegated to cuBLASLt, CUTLASS,
and the vendored FA2.

Full details by topic:

- [`docs/kernel_catalog.md`](docs/kernel_catalog.md) — every kernel
  shipped, grouped by function, with a re-use decision tree for
  non-VLA models.
- [`docs/kernel_fusion.md`](docs/kernel_fusion.md) — production
  fusion patterns, the four historical dead-end optimizations, and
  why the current fusion set converged where it did.
- [`docs/calibration.md`](docs/calibration.md) — FP8 static
  calibration mechanics.
- [`docs/pi05_native_calibration.md`](docs/pi05_native_calibration.md) —
  native PI0.5 single-view, multi-view and dataset artifact workflow.
- [`docs/optimization-details.md`](docs/optimization-details.md) —
  line-by-line Pi0.5 latency breakdown (44 ms vs 70 ms baseline).

---

## API snippets

Already built? Jump to API examples below. Not yet built? See
[Build & install](#build--install) for the full Docker / native
Linux flow, then come back.

### 3 Lines of Code

```python
import flash_rt

model = flash_rt.load_model(
    checkpoint="/path/to/checkpoint",
    framework="torch",    # or "jax"
    autotune=3,           # 0=off, 3=default, 5=thorough
)

actions = model.predict(
    images=[base_img, wrist_img],
    prompt="pick up the red block",
)
# Pi0.5: actions shape (10, 7) — 10 future steps, 7 DOF

# State is part of the VLA observation. Pi0/GROOT N1.6 consume it during
# inference; token-based variants encode it in the prompt prefix.

# Pi0 (continuous state input):
model = flash_rt.load_model(
    checkpoint="/path/to/pi0_checkpoint",
    config="pi0",
)
actions = model.predict(
    images=[base_img, wrist_img],
    prompt="pick up the red block",
    state=state,
)
# Pi0: actions shape (10, 7)

# GROOT N1.6:
model = flash_rt.load_model(
    checkpoint="/path/to/groot_checkpoint",
    config="groot",
)
actions = model.predict(
    images=[base_img, wrist_img],
    prompt="pick up the red block",
    state=state,
)
# GROOT: actions shape (50, 128) — 50 steps, 128-dim padded

# Pi0-FAST (autoregressive — discrete token generation, not diffusion):
model = flash_rt.load_model(
    checkpoint="/path/to/pi0_fast_base",  # Orbax (jax) or safetensors-converted (torch)
    config="pi0fast",
    framework="torch",  # or "jax"
)
actions = model.predict(images=[base_img, wrist_img], prompt="pick up the red block")
# Pi0-FAST: action sequence is generated as discrete FAST tokens then decoded
# to continuous actions via the FAST tokenizer (DCT inverse).

# Pi0-FAST max-performance mode (for fixed-prompt 24h deployment):
model = flash_rt.load_model(
    checkpoint="/path/to/pi0_fast_base",
    config="pi0fast",
    decode_cuda_graph=True,       # capture decode loop as CUDA Graph
    decode_graph_steps=46,        # action tokens per inference (50 total with text prefix)
)
```

#### Qwen3.6-27B NVFP4 (LLM, RTX 5090)

The LLM path uses a dedicated frontend — same kernel binary, separate
generation API since chat completion has a different surface from VLA
control. See [`docs/qwen36_usage.md`](docs/qwen36_usage.md) for the
full parameter reference and [`docs/qwen36_nvfp4.md`](docs/qwen36_nvfp4.md)
for the K-curve / measured throughput / model-dependency notes.

```python
import os
import torch
from flash_rt.frontends.torch.qwen36_rtx import Qwen36TorchFrontendRtx

# The NVFP4 ckpt has no MTP head; point this env var at a paired
# FP8 ckpt directory that contains mtp.safetensors. Without it,
# speculative decode is disabled (pure-decode still works at ~36 tok/s).
os.environ["FLASHRT_QWEN36_MTP_CKPT_DIR"] = "/path/to/qwen36_fp8_ckpt"

fe = Qwen36TorchFrontendRtx(
    "/path/to/qwen36_nvfp4",   # prithivMLmods/Qwen3.6-27B-NVFP4
    quant="nvfp4",
)

prompt = "Explain quantum entanglement in one short paragraph."
input_ids = fe._tokenizer(prompt, return_tensors="pt").input_ids.cuda()

out = fe.generate_own_speculative_KN_nvfp4(
    input_ids, max_new_tokens=256, K=6,   # K=6 peaks at NTOK<=128
)
text = fe._tokenizer.decode(out[0, input_ids.shape[1]:].tolist())
print(text)
```

For an OpenAI-API-compatible HTTP server (chat completions, drop-in
replacement for `OpenAI(base_url=...)`), use the production agent server
[`serving/qwen36_agent/`](serving/qwen36_agent/) (see its
[`README.md`](serving/qwen36_agent/README.md)):

```bash
pip install fastapi uvicorn
export FLASHRT_QWEN36_MTP_CKPT_DIR=/path/to/qwen36_mtp_ckpt
python -m serving.qwen36_agent.server \
    --checkpoint /path/to/qwen36_nvfp4 \
    --port 8000
# Then: curl http://localhost:8000/v1/chat/completions ...
```

### Framework Choice

| Checkpoint Format | `framework=` | Source |
|-------------------|:---:|--------|
| **safetensors** (HuggingFace/PyTorch) | `"torch"` | `model.safetensors` |
| **Orbax** (JAX/Physical Intelligence) | `"jax"` | `checkpoint/` dir |

Both frontends produce equivalent results (cosine > 0.999) and share the same `flash_rt_kernels.so`.

### Hardware Auto-Dispatch

User code does **not** need to know which GPU it's running on.
`load_model()` inspects `torch.cuda.get_device_capability()` at call
time and routes to the best-matching backend automatically:

| Compute capability | GPU | Backend |
|---|---|---|
| SM110 (11.0) | Jetson AGX Thor | `flash_rt.hardware.thor.*` |
| SM120 (12.0) | RTX 5090 Blackwell | `flash_rt.hardware.rtx.*`, falling back to Thor for models without a 5090-native class (Pi0-FAST uses Thor's in-file SM120 runtime fork) |
| SM89  (8.9)  | RTX 4090 Ada | `flash_rt.hardware.rtx.*` |

Override with `hardware="thor"` / `"rtx_sm120"` / `"rtx_sm89"` for
cross-hardware debugging — `"auto"` (default) is what you almost
always want. Unsupported SM levels raise a clear `RuntimeError` at
`load_model` time rather than falling back silently, because a wrong
backend at runtime is more expensive to debug than a clean crash.

```python
# Same code path on every supported GPU. On an RTX 5090 this resolves
# to RtxTorchGroot; on Jetson Thor it resolves to ThorPipelineTorchGroot.
model = flash_rt.load_model(
    "/path/to/groot_checkpoint",
    config="groot",
    embodiment_tag="gr1",     # see GROOT embodiment slots below
)
```

### GROOT N1.6 embodiment slots

GROOT's per-embodiment MLPs (state encoder / action encoder / action
decoder) live in 32 parallel slots inside a single checkpoint. In the
`GR00T-N1.6-3B` base checkpoint only a subset of those slots are
actually trained — the rest are at initialization std ~0.02 and emit
noise-like actions regardless of input. **Pick a trained slot for any
demo or deployment**:

| `embodiment_tag=` | Slot | Description |
|---|---|---|
| `gr1` | 20 | GR1 humanoid, 1 camera view. Good default for single-cam demos. |
| `robocasa_panda_omron` | 13 | Tabletop arm + mobile base, 3 camera views |
| `behavior_r1_pro` | 24 | BEHAVIOR humanoid, 3 camera views |
| `new_embodiment` | 10 | Placeholder for fine-tuning (UNTRAINED in base) |

Any other tag in the map (`libero_panda`, `oxe_google`, `oxe_widowx`,
`unitree_g1`, `oxe_droid`) is **untrained** in the base 3B checkpoint
and logs a warning at load time. Fine-tune one of those slots
yourself or pick a trained tag for immediate use.

### GROOT N1.7 RTX

```python
import flash_rt

model = flash_rt.load_model(
    "/path/to/GR00T-N1.7-3B",
    framework="torch",
    config="groot_n17",
    hardware="rtx_sm120",
    num_views=2,
    embodiment_tag="oxe_droid_relative_eef_relative_joint",
)

model.set_prompt(aux=aux, prompt="put the blue block in the green bowl")
actions_normalized = model.infer(
    state_normalized,
    aux=aux,  # fresh observation/model inputs
    initial_noise=initial_noise,
    use_dit_graph=True,
)
```

GROOT N1.7 is registered as `config="groot_n17"` for the RTX torch
path (`rtx_sm120` and `rtx_sm89`). `rtx_sm120` keeps the shared RTX
registration and `load_model()` refines that default to the FP8
production frontend when `use_fp16=False`; `rtx_sm89` is registered
directly to its dedicated FP8 frontend. `use_fp16=True, use_fp8=False`
selects the explicit RTX reference frontend for the selected hardware. It
uses the N1.7 `set_prompt(aux=...)` / normalized-state `infer(...)`
contract. On the RTX FP8 path, the first `infer(aux=...)` lazily captures the
backbone graph; later compatible calls replay it before the existing action
graph, covering the complete backbone-to-action execution without rebuilding
per-call scratch buffers. Omitting `aux` keeps the original eager backbone from
`set_prompt()` and does not capture the optional graph. See
[USAGE.md](USAGE.md#groot-n17-rtx).

### Autotune

CUDA Graph instantiation is non-deterministic on Thor — the same kernels can produce different schedules with ~2ms variance. `autotune` recaptures until a fast schedule is found:

| `autotune=` | Behavior | Extra Startup |
|-------------|----------|---------------|
| `0` or `False` | Off — single capture, may be 2ms slower | 0 |
| `3` (default) | Retry up to 3× — usually finds fast graph on trial 0 | ~1s |
| `5` | Retry up to 5× — better chance for JAX | ~2.5s |
| `True` | Same as `3` | ~1s |

### Pi0-FAST Performance Modes

Pi0-FAST supports two decode modes, controlled by `decode_cuda_graph`:

| Parameter | `set_prompt` (cold) | `set_prompt` (cached) | 50-token E2E | Best for |
|-----------|--------------------:|----------------------:|-------------:|----------|
| `decode_cuda_graph=False` (default) | ~2.5 s | **~0.1 s** | **~464 ms** | Frequent prompt changes |
| `decode_cuda_graph=True` | ~4.0 s | **~1.5 s** | **~431 ms** | Fixed prompt, 24h deployment |

**How it works:**

- **Default mode** (`decode_cuda_graph=False`): Each decode token runs through a
  Python loop with per-step kernel launches. Lowest startup cost. FP8 calibration
  scales are cached to `~/.flash_rt/calibration/` after the first run — subsequent
  `set_prompt` calls with the same checkpoint skip the 2.4s calibration entirely.

- **Max-performance mode** (`decode_cuda_graph=True`): The action-phase decode loop
  is captured as a single CUDA Graph (same technique as Pi0's diffusion loop).
  Eliminates all Python dispatch overhead during decode. Adds ~1.5s to `set_prompt`
  for graph capture, but saves ~33 ms per 50-token inference.
  Break-even at ~45 inferences.

```python
# Default: good for interactive / multi-prompt scenarios
model = flash_rt.load_model(checkpoint, config="pi0fast")
model.set_prompt("pick up the red block", state=state)
# set_prompt: 0.1s (cached) / 2.5s (first run)
# infer: ~464 ms per 50-token sequence

# Max-performance: best for fixed-prompt continuous control
model = flash_rt.load_model(
    checkpoint, config="pi0fast",
    decode_cuda_graph=True,
    decode_graph_steps=46,    # covers sequences up to 46 action tokens (50 total)
)
model.set_prompt("pick up the red block", state=state)
# set_prompt: 1.5s (cached) / 4.0s (first run)
# infer: ~431 ms per 50-token sequence
```

**Calibration caching**: FP8 activation scales are automatically cached per
checkpoint and sequence length. Delete `~/.flash_rt/calibration/` to force
recalibration. The first `infer()` call always recalibrates with real image
data regardless of cache.

### NVFP4 encoder FFN (Pi0.5 only)

Optional NVFP4 (Blackwell block-scaled FP4) quantization on the Pi0.5 encoder
FFN stack. Implemented for **Pi0.5 torch and JAX on Thor** — passing
`use_fp4=True` with any other config (pi0 / groot / pi0fast) emits a warning
and uses the FP8 route. (`config="groot_n17"` on Thor has its own NVFP4
tier behind the same flag — see [GROOT N1.7 Thor](USAGE.md#groot-n17-thor).)

```python
model = flash_rt.load_model(
    checkpoint,
    config="pi05",
    use_fp4=True,    # single flag → enables the production-validated preset
)
```

`use_fp4=True` resolves to the best-known production preset automatically:
- `fp4_layers` = all 17 live encoder FFN layers (0-16)
- `use_awq` = `True` — activation-aware weight quantization (AWQ)
- `use_p1_split_gu` = `True` — P1 split-GU 2-GEMM path

Adding `use_fp4_decoder=True` selects the complete Thor NVFP4 tier the
latency table above is measured with, resolving the remaining sub-flags to
`awq_alpha=0.8`, `use_fp4_encoder_attn=True`, `use_fp4_siglip_ffn=True` and
`encoder_p1_combiner="epilogue_hw"`:

```python
model = flash_rt.load_model(
    checkpoint, config="pi05", framework="torch", hardware="thor",
    num_views=3,
    use_fp4=True, use_fp4_decoder=True,
    use_fa4=True,    # attention backend; independent of the NVFP4 flags
)
```

The decoder path is a strict opt-in: unavailable kernels and unsupported
modes raise errors rather than selecting FP8 projections. `use_fa4=True` is
also supported on the FP8 frontend and is what the FP8 baseline in that
table runs, so the reported speedup isolates NVFP4 rather than the
attention backend.

Advanced users can override any sub-flag explicitly at `load_model()` call
time (e.g. `fp4_layers=(7, 8, 9), use_awq=False` reverts to the conservative
L7-9 subset).

**What it does**:
- Gate+Up and Down GEMMs across all 17 live encoder FFN layers run in NVFP4
  (block-size 16, UE4M3 block scales) instead of FP8.
- **AWQ** applies activation-aware per-input-channel pre-scaling to the
  quantized weights, with the inverse scale fused into pre-GEMM kernels
  (`residual_add_rms_norm_mul_fp4_sfa`, `geglu_two_mul_fp4_to_fp4`). This
  preserves precision under full-scope FP4 (without AWQ, full-scope FP4 cos
  drops from ~0.998 to ~0.33 due to cumulative multi-layer drift).
- **P1 split-GU** splits the merged Gate+Up GEMM into separate gate_proj /
  up_proj NVFP4 GEMMs that emit packed FP4 + SFA directly (via
  `LinCombBlockScaleFactor` epilogue), combined by a dedicated
  `geglu_two_mul_fp4_to_fp4` kernel. Eliminates ~31 MB/layer of DRAM
  round-trips vs the merged-GU path.
- Residual stream stays fp16 through the FP4 region (NVIDIA
  `enable_llm_nvfp4` style — `output_quantizer` disabled).

**Requirements**:
- SM100+ GPU (validated on Thor SM110). Non-SM100 hardware logs a warning
  and uses the FP8 route.
- `flash_rt_fp4.so` extension (built alongside `flash_rt_kernels.so`).

**Measured on Thor SM110, Pi0.5 / LIBERO Spatial 10 × 50 = 500 episodes**:

| Config | Task success | E2E P50 (normal) |
|---|---|---|
| FP8 baseline | 491 / 500 (98.2%) | ~43.5 ms |
| **NVFP4 full-17 + AWQ + P1 (`--use_fp4`)** | **491 / 500 (98.2%)** | **~43.5 ms** |

Task-level parity with the FP8 baseline (491/500 for both — P1 + AWQ
preserves FP4 precision across all 17 live FFN layers).

**Replay-latency benchmark (1-view / 2-view / 3-view, N=8 LIBERO
stratified calibration, 50 graph replays, Thor SM110)**:

| Config | 1-view | 2-view | 3-view | cos vs PyTorch FP32 ref (3v) |
|---|---|---|---|---|
| FP8 baseline (torch) | 34.06 ms | 41.79 ms | 55.46 ms | 0.999236 |
| **NVFP4 encoder (torch)** | **31.91 ms** | **39.78 ms** | **51.51 ms** | **0.998932** |
| **NVFP4 encoder (jax, Orbax)** | **34.39 ms** | **43.65 ms** | **56.90 ms** | **0.999030** |

Encoder FP4 preserves cosine **≥ 0.9989** vs the same-origin PyTorch
reference in these Thor replay-latency checks. The JAX FP4 path derives
NVFP4 weights directly from the Orbax checkpoint (no torch dependency at
runtime) and uses the same two-phase multi-sample calibration flow as the
torch FP4 path. Treat the table as Thor correctness / availability evidence,
not a broad performance claim across every view count or host.
Reproduce with
[`tests/bench_pi05_thor_views.py`](tests/bench_pi05_thor_views.py)
(defaults now include `jax_fp4`).

**What's next**:
- Decoder FP4 is implemented as a draft all-projection W4A4 path. The current
  multi-view latency and precision blockers are tracked in
  [`docs/pi05_thor_decoder_fp4_e2e.md`](docs/pi05_thor_decoder_fp4_e2e.md).
- `geglu_two_mul` SFA-prefetch optimization (O1, ~-0.5-1.1 ms)
- SigLIP FFN FP4 / AWQ auto-tune / Pi0.6 port

---

## Install from PyPI (pure Python, no local build)

```bash
pip install flash-rt
```

The published distribution is **pure Python**: it carries the frontends,
the structure catalog (`flash_rt.catalog`) and the kernel sources, and no
`.so` at all. That is enough to read the catalog and inspect a binding.
It is *not* enough for the native `flash_rt.load_model` path, which needs
compiled extensions; asking for them without a build produces a refusal
that names the cmake line rather than a missing-module traceback. Build
them with the section below.

Attaching structures to an unmodified PyTorch host, or inside vLLM /
SGLang, is the job of the separate
[FlashRT-Structures](https://github.com/flashrt-project/FlashRT-Structures) package
(`pip install flashrt-structures`). It depends on this distribution for
the catalog and the calibration contract, and obtains its kernels from
the kernel hub at bind time. Its README covers which door a host takes,
how to read a refusal, and the hub build-matrix boundaries.

## Build & install

This is the hands-on "go from a fresh machine to a green benchmark"
section. For a single-page install reference (prerequisites,
troubleshooting table, JAX/transformers pin rationale) see
[`docs/INSTALL.md`](docs/INSTALL.md).

Docker and native Linux paths both produce the same two
extension modules:

| Artifact | Size | What it contains |
|---|---|---|
| `flash_rt/flash_rt_kernels.so` | ~3 MB | Hand-written memory-bound kernels (norm, activation, fusion, FP8 quant, cuBLASLt wrappers, Thor FMHA). **Always built.** |
| `flash_rt/flash_rt_fa2.so` | ~135 MB | Vendored Flash-Attention 2 v2.7.4.post1 fwd (fp16 + bf16, SM80/86/89/120). **Built automatically on RTX targets.** Thor skips it by default and uses `fvk.attention_qkv_fp16` (cuBLAS-decomposed) instead; `-DFLASHRT_ENABLE_THOR_FA2=ON` builds it there for the one model whose long prefill needs it (Qwen3.6, bf16 head_dim 256 — a single instantiation). |

**Crucially — no `pip install flash-attn` required.** The FA2 kernel
is vendored at source level and built into `flash_rt_fa2.so` during
`cmake`/`make`; at runtime `import flash_rt` loads both .so files
directly, so you never hit the `flash-attn` wheel's
`torch × CUDA × driver × glibc` compatibility matrix. Setting
`FVK_RTX_FA2=0` is still supported as a fall-back to `pip flash-attn`
for debugging, but the default path has zero pip-wheel dependency.

### Option A — Prebuilt Docker image (fastest, recommended)

The published image already has CUDA 13.0, PyTorch 2.9, the
FlashRT kernels prebuilt, and CUTLASS vendored — pull and run, no
local compile, no `flash-attn` wheel hunting:

```bash
docker pull ghcr.io/liangsu8899/flashrt:latest
docker run --rm --gpus all -it ghcr.io/liangsu8899/flashrt:latest
# Drops you in a Python REPL with `flash_rt` already imported.
```

For Modal / RunPod / Vast and other cloud runners, point the image
config at the same registry — Modal cold-start drops from a 10-minute
kernel compile to a ~30-second pull:

```python
image = modal.Image.from_registry("ghcr.io/liangsu8899/flashrt:0.2.0")
```

Tags + advanced usage (build args, slim variants, mounting checkpoints):
see [`docker/README.md`](docker/README.md).

> **Thor (SM110)** is not covered by this image — Jetson is ARM64 and
> uses a different NVIDIA base. Thor users follow Option C below.

### Option B — Build the Docker image yourself

If you need a different GPU arch, want to pin a specific commit, or
prefer to vet the image source:

```bash
git clone https://github.com/flashrt-project/FlashRT.git
cd FlashRT
docker build -t flashrt:dev -f docker/Dockerfile .
docker run --rm --gpus all -it flashrt:dev
```

Build args (`GPU_ARCH`, `FA2_HDIMS`, `BASE_IMAGE`, `CUTLASS_REF`)
documented in [`docker/README.md`](docker/README.md). Cold build on a
fresh host is ~25 min (NGC pull + FA2 codegen); warm rebuild ~12 min.

### Option C — Native Linux (no Docker)

System requirements:

| Component | Minimum | Notes |
|---|---|---|
| GPU | SM80+ (A100, 30xx+, Thor, 4090, 5090, DGX Spark) | |
| NVIDIA driver | 545+ for CUDA 13, 525+ for CUDA 12.4 | 5090 needs 550+ |
| CUDA Toolkit | 12.4+ (Thor/Hopper) or 12.8+ (Blackwell) | CUDA 13 recommended on 5090 |
| Python | 3.10 / 3.11 / 3.12 | 3.12 on the default NGC image |
| GCC/G++ | 11+ with C++17 | |
| CMake | 3.24+ | |

**Create an isolated Python environment first.** The build step calls
`python3 -m pybind11 --cmakedir` to locate pybind11 headers, so the
Python that runs `cmake ..` MUST be the same interpreter the `.so`
files will be imported from. System-Python + conda-Python mix-ups are
the #1 native-install failure mode.

```bash
python3.12 -m venv .venv         # 3.10 / 3.11 / 3.12 all supported
source .venv/bin/activate
```

Minimum pip list (for the `torch` frontend; everything **must** be
installed *before* `cmake ..`):

```bash
# 1. PyTorch matching your CUDA:
pip install torch --index-url https://download.pytorch.org/whl/cu128   # 5090 / CUDA 12.8+
# or
pip install torch --index-url https://download.pytorch.org/whl/cu124   # 4090 / A100 / Thor

# 2. Build helpers
pip install pybind11 cmake "numpy>=1.24" safetensors

# 3. Runtime / benchmarking
#    transformers is pinned <4.56 because the Pi0.5 PaliGemma tokenizer
#    path broke in 4.56+; drop the upper bound once we verify the new
#    tokenizer API.
pip install "transformers<4.56" pandas pillow pyarrow

# 4. JAX-side (optional — only if you will load Orbax checkpoints).
#    Versions are pinned because the Orbax/jaxlib/PJRT plugin ABI is
#    not stable across minor releases; upgrading any of the four
#    without matching the others is a reliable way to get cryptic
#    "PJRT device not registered" errors at import time. Pin bump is
#    tracked upstream — see docs/INSTALL.md §JAX for rationale.
pip install jax==0.5.3 jax-cuda12-pjrt==0.5.3 jax-cuda12-plugin==0.5.3 ml_dtypes==0.5.3
```

Then build:

```bash
git clone https://github.com/flashrt-project/FlashRT.git
cd FlashRT
git clone --depth 1 --branch v4.4.2 \
    https://github.com/NVIDIA/cutlass.git third_party/cutlass

pip install -e ".[torch]"          # or "[jax]" / "[all]"
# NOTE: editable mode (-e) is required. The cmake build below drops
# compiled .so files into flash_rt/ in the source tree; editable
# install makes that directory importable directly. A non-editable
# `pip install .` would install a copy BEFORE the .so files exist and
# `import flash_rt` would fail at runtime with a missing-module error.

cmake -B build -S .                 # auto-detects GPU arch
cmake --build build -j$(nproc)
# CMake writes .so files directly into flash_rt/ — no `cp` /
# `make install` / `ninja install` step needed.
```

### GPU arch override

CMake reads `nvidia-smi --query-gpu=compute_cap` to pick the target
arch. Override for cross-compilation or when auto-detect fails:

```bash
cmake -B build -S . -DGPU_ARCH=110   # Jetson AGX Thor   (FA2 opt-in, CUTLASS SM100 path ON)
cmake -B build -S . -DGPU_ARCH=121   # DGX Spark / GB10   (FA2 sm_121 AOT, NVFP4 ON)
cmake -B build -S . -DGPU_ARCH=120   # RTX 5090           (FA2 sm_120 AOT, NVFP4 ON)
cmake -B build -S . -DGPU_ARCH=89    # RTX 4090           (FA2 sm_80 AOT natively runs on Ada)
cmake -B build -S . -DGPU_ARCH=86    # RTX 3090 / A10     (FA2 sm_80 AOT)
cmake -B build -S . -DGPU_ARCH=80    # A100               (FA2 sm_80 AOT)
```

FA2 is enabled by CMake when `GPU_ARCH ∈ {80, 86, 89, 120, 121}`, and on
Thor SM110 when `-DFLASHRT_ENABLE_THOR_FA2=ON` is passed. Other arches
(notably SM90 Hopper, and Thor without that flag) route attention
through the cuBLAS-decomposed `fvk.attention_qkv_fp16` path instead of
FA2 — `flash_rt_fa2.so` simply isn't built, and no runtime error
results.

### Build timing (one-time)

On a 5090 with CUDA 13 in a warm container, `make -j$(nproc)`:

| Target | Time |
|---|---|
| `flash_rt_kernels` (main kernels) | ~2 min |
| `flash_rt_fa2` (FA2 vendor, default — 12 kernel .cu files × sm_80 + sm_120/sm_121 + Blackwell PTX fallback) | **~4.5 min** (267 s) |
| Full `make -j$(nproc)` | ~6.5 min |

Subsequent rebuilds of only the hand-written kernels take ~2 min —
FA2 is a separate CMake target and is only re-linked, not recompiled,
unless the vendored source itself changes.

### Slim-build flags (developer iteration speed)

FA2's CUTLASS 3.x templates dominate cold-build cost. The default
matrix covers every RTX family card × fp16+bf16 × all 3 hdim
buckets, which is right for distribution but overkill when you're
iterating on a single 5090/4090 and a single model family. Three
opt-in CMake flags trade binary coverage for iteration speed:

| Flag | Default | What it does | `fa2` cold build on 5090 |
|---|---|---|---|
| — | (none) | 12 .cu × sm_80 + sm_120/sm_121 + Blackwell PTX fallback | **267 s (4.5 min)** |
| `-DFA2_ARCH_NATIVE_ONLY=ON` | OFF | Only emit SASS for the detected GPU; skip sm_80 + PTX passes | **110 s** (−59%) |
| `-DFA2_HDIMS="96;256"` | `"96;128;256"` | Drop `head_dim=128` (shipped models don't use it; reserved for future DiT variants) | **210 s** (−21%) |
| `-DFA2_DTYPES="fp16"` | `"fp16;bf16"` | Drop bf16 (Pi0 is fp16-only; Pi0.5 / GROOT need bf16) | **179 s** (−33%) |
| `-DFA2_ARCH_NATIVE_ONLY=ON -DFA2_HDIMS="96;256" -DFA2_DTYPES="fp16"` | — | All three combined (single-card + pi0-only) | **87 s** (−67%) |

Shipped `flash_rt_fa2.so` size also shrinks — the all-three-slim
build produces **17.8 MB** (vs 135 MB default), a **87% reduction**
in binary size on the FA2 module.

Dropped entries still resolve at the Python layer — calling a
stubbed entry (e.g. `fa2.fwd_bf16` on a build with
`FA2_DTYPES="fp16"`) aborts the process with a clear
"rebuild with -DFA2_DTYPES=…" message instead of linker errors or
silent wrong output.

### ccache (iterative C++ rebuild speedup)

If `ccache` is on PATH at CMake-config time, it is enabled
automatically for both C++ and CUDA compiles. First build is
unchanged. Hit rate on the `.cpp` side (pybind bindings) is high,
so repeat edits to `csrc/bindings.cpp` / `csrc/fa2_bindings.cpp` get
fast rebuilds. CUDA .cu files — nvcc's invocation style makes
`ccache` hit rate unreliable, so treat CUDA speedup as a bonus
rather than a guarantee. Tip: set `CCACHE_DIR` to a host-mounted
path so the cache survives container rebuilds.

Install via `apt-get install ccache` (Ubuntu) or equivalent.

### Verify

```bash
python examples/quickstart.py \
    --checkpoint /path/to/pi05_checkpoint \
    --benchmark 20
```

Expected (default `--num_views 2`): `P50: ~44 ms (23 Hz)` on Thor.
On RTX 5090 pure replay is ~17.4 ms (57 Hz); `quickstart.py` reports
end-to-end wall clock (~19.5 ms / 51 Hz) because it wraps
`model.predict(...)` with `time.perf_counter` and therefore also
counts image normalization, upload, download, and un-normalization.
For the pure-replay number, time `model._pipe._enc_ae_graph.replay()`
between `cuda.Event` markers — see [Measurement protocol](#measurement-protocol).

### Verify

```bash
python examples/quickstart.py \
    --checkpoint /path/to/pi05_checkpoint \
    --benchmark 20
```

Expected (default `--num_views 2`): `P50: ~44 ms (23 Hz)` on Thor.
On RTX 5090 pure replay is ~17.6 ms (57 Hz); `quickstart.py` reports
the end-to-end wall clock (~19.5 ms / 51 Hz) because it wraps
`model.predict(...)` with `time.perf_counter` and therefore also
counts the graph-external image normalization, upload, download, and
un-normalization. For the pure-replay number, time
`model._pipe._enc_ae_graph.replay()` between `cuda.Event` markers —
see [Measurement protocol](#measurement-protocol).

**GROOT N1.6:**
```bash
python examples/quickstart.py \
    --checkpoint /path/to/groot_checkpoint \
    --config groot \
    --benchmark 20
```

Expected: `P50: ~44 ms (23 Hz)` on Thor.

---

## Supported Models

- **Pi0.5** (`config="pi05"`) — [quickstart](#quick-start), [API reference](USAGE.md#api-reference), [NVFP4 notes](USAGE.md#nvfp4-pi05-only), [Thor example](examples/thor/README.md), [RTX 5090 example](examples/blackwell/README.md)
- **Pi0** (`config="pi0"`) — [API snippets](#api-snippets), [usage guide](USAGE.md#api-reference)
- **GROOT N1.6** (`config="groot"`) — [API snippets](#api-snippets), [GROOT embodiment slots](#groot-n16-embodiment-slots)
- **GROOT N1.7** (`config="groot_n17"`) — 23.7 ms on Jetson AGX Thor (NVFP4 + FA4 tier, LIBERO 1-view; 36.8 ms FP8), 16.6 ms on RTX 5090 (2-view base, full graph); [usage guide](USAGE.md#groot-n17-rtx), [API snippet](#groot-n17-rtx)
- **Pi0-FAST** (`config="pi0fast"`) — [usage guide](USAGE.md#pi0-fast), [performance modes](#pi0-fast-performance-modes)
- **LingBot-VLA** — [LingBot usage](docs/lingbot_usage.md), [Thor latency](docs/lingbot_usage.md#5-accuracy--latency-thor-sm_110-cuda-graph-replay)
- **Motus Stage3 RTX beta** (`config="motus"`) — [Motus usage](docs/motus_usage_beta.md), [legacy async chunk runner](docs/rtc_lite_design.md)
- **Wan2.2 TI2V-5B** (`config="wan22_ti2v_5b"`) — [Wan2.2 usage](docs/wan22_usage.md)
- **Cosmos3-Nano text-to-video** (`config="cosmos3_video"`) — RTX 5090 BF16/FP8 denoise and complete benchmark workflow; [usage and performance](docs/cosmos3_video_usage.md)
- **Cosmos3-Edge AV inverse dynamics and Reasoner** (`config="cosmos3_edge"`) — Jetson AGX Thor official baseline, 6.60x no-cache AV denoise, and NVFP4 multimodal chat decode; [complete usage and performance](docs/cosmos3_edge_thor.md)
- **Qwen3-VL-8B** — RTX 5090 NVFP4/FP8 multimodal path, RTX 4090 official-FP8 path, and Jetson BF16 paths for Thor and Orin; [RTX 5090 usage](docs/qwen3_vl_nvfp4.md), [RTX 4090 usage](docs/qwen3_vl_fp8_sm89.md), [Jetson Thor usage](docs/qwen3_vl_thor.md), [Jetson Orin usage](docs/qwen3_vl_rtx_bf16.md)
- **Chameleon-7B** — Jetson Thor dynamic-FP8 prefill (~120 ms E2E, ~30 tok/s decode) and Jetson Orin INT8/QuaRot-INT4 path; [usage](docs/chameleon_usage.md), [Thor notes](docs/chameleon_thor_sm110.md), [Orin SM87 notes](docs/chameleon7b_rtx_sm87.md)
- **MiniMax-Remover** — FP8 transformer + NVFP4 VAE video inpainting; [usage and performance](docs/minimax_remover_usage.md)
- **MelBandRoformer** — kernelized FP8 audio source separation; [usage and performance](docs/melband_roformer_usage.md)
- **OmniVoice TTS** — BF16/FP4 acceleration and HTTP serving; [serving quickstart](serving/omnivoice_agent/README.md)
- **Higgs Audio v3 TTS-4B** — [Higgs usage](docs/higgs_audio_v3.md#3-quickstart), [Higgs serving](serving/higgs_audio_agent/README.md)
- **Qwen3.6-27B NVFP4** — [Qwen3.6 usage](docs/qwen36_nvfp4.md), [parameter reference](docs/qwen36_usage.md), [serving](serving/qwen36_agent/README.md)
- **Qwen3-8B NVFP4** — [Qwen3-8B usage](docs/qwen3_8b_nvfp4.md), [OpenAI server example](examples/qwen3_openai_server.py)
- **BAGEL world-model path** — research preview; see [kernel catalog](docs/kernel_catalog.md) and [adding a model](docs/adding_new_model.md)
- **Reusable HF kernel packages** — [flashrt-project/FlashRT-HF-kernels](https://github.com/flashrt-project/FlashRT-HF-kernels), [huggingface.co/flashrt](https://huggingface.co/flashrt)

---

## Hardware Support

FlashRT ships CUDA and ROCm implementations today. The kernel composition
pattern is not vendor-specific; what is vendor-specific is which artifacts
have been built and tested, and the dispatch map that selects them.

| Hardware | SM | Status | Validated paths / notes |
|---|---:|---|---|
| Jetson AGX Thor | SM110 | Production target | Pi0, Pi0.5, GROOT N1.6, Pi0-FAST, Qwen3.6 Thor path, Lingbot, Cosmos3-Edge AV/Reasoner, and Qwen3-VL BF16 with opt-in W8/W4 decode ([docs](docs/qwen3_vl_thor.md)); CUTLASS FMHA / Thor attention paths; Pi0.5 FP8 and NVFP4 validation live in [examples/thor](examples/thor/README.md#thor-vla-performance). |
| RTX 5090 | SM120 | Production target | Pi0/Pi0.5/GROOT/Pi0-FAST RTX paths, Qwen3.6, Qwen3-8B, Qwen3-VL, Higgs Audio v3 FP8, Motus, Wan2.2, Cosmos3-Nano, and HF Kernel Hub package validation; see [RTX 5090 latency](examples/blackwell/README.md#vla-latency-rtx-5090). |
| AMD Instinct MI350X | gfx950 (CDNA4) | Production target | Pi0.5 at **16.4 ms** FP8 and GROOT N1.7 at **16.0 ms** full-frame, on a standalone ROCm/HIP backend: hand-written HIP kernels, hipBLASLt FP8 GEMM, MFMA attention, HIP graph capture. Arch-gated to gfx950 at both build and run time. See [AMD deployment](docs/deployment_amd.md) and [Pi0.5 on MI350X](docs/deployment_amd_pi05.md). |
| AMD Instinct MI300 | gfx942 (CDNA3) | Refused, not ported | The backend rejects any device whose architecture does not start with `gfx950`, because the kernels use CDNA4-specific MFMA shapes. `FLASHRT_AMD_ALLOW_ARCH=1` is a build-script escape hatch for porting work, not a supported path. |
| RTX 4090 | SM89 | Validated / supported target | RTX VLA build path and deployment recipe; Higgs BF16 path compiles/configures. See [deployment_rtx4090.md](docs/deployment_rtx4090.md). |
| RTX 5060 Ti | SM120 | Community validated | Pi0.5 FP8 and LIBERO Spatial submission; see [Community benchmarks](#community-benchmarks). |
| RTX 4060 Ti | SM89 | Validated build/run target | Included in current tested hardware list; run local benchmarks before making model-specific latency claims. |
| NVIDIA L40 | SM89 | Community validated | Pi0.5 FP8 submission; see [Community benchmarks](#community-benchmarks). |
| Jetson AGX Orin | SM87 | Community port | Pi0.5 INT8/BF16 paths, Orin tile dispatch, frame-cache inference, and Qwen3-VL BF16 with opt-in INT8/INT4 decode ([docs](docs/qwen3_vl_rtx_bf16.md)); see [deployment_orin.md](docs/deployment_orin.md). |
| A100 / A10 / RTX 3090 / RTX 3080 / A5000 / A6000 and other SM80/86/89 GPUs | SM80/86/89 | Build target | CMake and FA2 gates cover Ampere/Ada shapes. Treat unlisted cards as expected to build until a benchmark or regression row is submitted. |

Feature notes:

- `flash_rt_kernels.so` is the always-built core extension.
- RTX targets build `flash_rt_fa2.so`; Thor routes attention through Thor-specific kernels.
- SM100+ targets can build the NVFP4/FP4 extension where the model frontend uses it.
- Unsupported SM levels fail at dispatch/build time instead of silently selecting an incorrect backend.

---


<a name="community-benchmarks"></a>

## Community Hardware Benchmarks

These runs are external hardware submissions using public quickstart or
deployment scripts. Exact latency depends on driver, CUDA, clock state,
warmup count, and checkpoint.

| Contributor | Hardware | Model | Cameras | Warmup | Cache | P50 | P95 / range | Throughput | Check |
|---|---|---|---:|---:|---:|---:|---:|---:|---:|
| [@cuihengrui35](https://github.com/cuihengrui35) | RTX 5060 Ti, SM120, 16 GB | Pi0.5 FP8 | 2 | 200 | - | **41.4 ms** | 40.9-43.2 ms | **24 Hz** | - |
| [@wangerforcs](https://github.com/wangerforcs) | NVIDIA L40, SM89 | Pi0.5 FP8 | 2 | 500 | - | **26.6 ms** | 26.2-27.3 ms | **38 Hz** | - |
| [@gugudeshubao](https://github.com/gugudeshubao) | Jetson AGX Orin 64 GB, SM87 | Pi0.5 DROID INT8 | 2 | - | 1 | **124 ms** | - | **8.04 Hz** | 1.000 |
| [@gugudeshubao](https://github.com/gugudeshubao) | Jetson AGX Orin 64 GB, SM87 | Pi0.5 DROID INT8 | 2 | - | 2 | **127 / 39 ms** | - | **12.2 Hz** | 0.991 |
| [@strayberry](https://github.com/strayberry) | Jetson AGX Orin 32 GB, SM87 | Pi0.5 BF16 | 2 | - | 1 | **215.9 ms** | 217.1 ms | **4.6 Hz** | - |
| [@strayberry](https://github.com/strayberry) | Jetson AGX Orin 32 GB, SM87 | Pi0.5 BF16 | 2 | - | 2 | **137 ms** | 218 ms | **7.3 Hz** | - |

Task-level submission:

| Contributor | Hardware | Task | Trials | Success | Rate |
|---|---|---|---:|---:|---:|
| [@cuihengrui35](https://github.com/cuihengrui35) | RTX 5060 Ti, SM120, 16 GB | Pi0.5 LIBERO Spatial | 350 | **344** | **98.3%** |

If you contribute a hardware benchmark, include the exact command, warmup count,
driver/CUDA/PyTorch versions, and `nvidia-smi` output. For new cards, start with
`python examples/quickstart.py --checkpoint <...> --benchmark 20`.

---

## Citation

If you use FlashRT for your research, please cite our paper:

```bibtex
@misc{su2026executionstatecapsules,
  title={Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving},
  author={Liang Su},
  year={2026},
  eprint={2606.20537},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  doi={10.48550/arXiv.2606.20537},
  url={https://arxiv.org/abs/2606.20537},
}
```

---



## Acknowledgments

- [CUTLASS](https://github.com/NVIDIA/cutlass) — GEMM templates and FMHA kernels
- [FlashAttention](https://github.com/Dao-AILab/flash-attention) — Attention backend for SM89/SM120
- [Physical Intelligence](https://www.physicalintelligence.company/) — Pi0/Pi0.5 model architecture
- [OpenPI](https://github.com/Physical-Intelligence/openpi) — Reference PyTorch implementation
- [NVIDIA Isaac GR00T](https://github.com/NVIDIA/Isaac-GR00T) — GROOT N1.6 model
