Metadata-Version: 2.4
Name: vlalab
Version: 0.1.3
Summary: A toolbox for tracking and visualizing the real-world deployment process of VLA models
Author: VLA-Lab Contributors
License: MIT
Project-URL: Homepage, https://github.com/VLA-Lab/VLA-Lab
Project-URL: Documentation, https://github.com/VLA-Lab/VLA-Lab#readme
Project-URL: Repository, https://github.com/VLA-Lab/VLA-Lab
Project-URL: Issues, https://github.com/VLA-Lab/VLA-Lab/issues
Keywords: robotics,vla,deployment,visualization,logging
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.20.0
Requires-Dist: matplotlib>=3.5.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: fastapi<1.0.0,>=0.110.0
Requires-Dist: uvicorn<1.0.0,>=0.30.0
Requires-Dist: opencv-python-headless>=4.5.0
Requires-Dist: pyarrow>=15.0.0
Requires-Dist: click>=8.0.0
Requires-Dist: rich>=12.0.0
Requires-Dist: plotly>=5.10.0
Provides-Extra: zarr
Requires-Dist: zarr>=2.10.0; extra == "zarr"
Provides-Extra: full
Requires-Dist: zarr>=2.10.0; extra == "full"
Requires-Dist: scipy>=1.7.0; extra == "full"
Requires-Dist: pillow>=9.0.0; extra == "full"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=3.0.0; extra == "dev"
Requires-Dist: black>=22.0.0; extra == "dev"
Requires-Dist: ruff>=0.0.250; extra == "dev"
Dynamic: license-file

<div align="center">
  
# 🦾 VLA-Lab

### The Missing Toolkit for Vision-Language-Action Model Deployment

[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
[![PyPI version](https://img.shields.io/badge/pypi-v0.1.3-orange.svg)](https://pypi.org/project/vlalab/)

**Evaluate · Replay · Inspect Data** — All-in-one toolkit for real-world VLA deployment debugging

[🚀 Quick Start](#-quick-start) · [📸 Screenshots](#-screenshots) · [🎯 Features](#-features) · [🔧 Installation](#-installation)

</div>

---

## 🎯 Why VLA-Lab?

Deploying VLA models to real robots is **hard**. You face:

- 🕵️ **Black-box inference** — Can't see what the model "sees" or why it fails
- ⏱️ **Hidden latencies** — Transport delays, inference bottlenecks, control loop timing issues
- 📊 **Fragmented logging** — Every framework logs differently, making cross-model comparison painful
- 🔄 **Tedious debugging** — Replaying failures requires manual log parsing and visualization

**VLA-Lab solves this.** A unified logging format + interactive visualization dashboard covering the debugging loop from open-loop evaluation to real-robot replay and dataset inspection.

---

## ✨ Features

<table>
<tr>
<td width="50%">

### 🎯 Open-Loop Eval
Start offline: compare predicted actions against ground truth before touching the robot. Inspect MSE / MAE summaries, temporal alignment, error heatmaps, and 3D trajectory overlays.

### 🔬 Real-Robot Runs
Replay deployment logs step-by-step: multi-camera observations, state/action curves, action chunks, latency breakdowns, attention overlays, and Rerun-compatible recordings.

</td>
<td width="50%">

### 📊 Dataset Viewer
Inspect LeRobot / GR00T / Zarr datasets frame-by-frame with video playback, state/action/gripper curves, sampled frames, and workspace distributions.

### 🧭 OOD Diagnosis
When real-world success stays low, compare deployment observations against the dataset to catch out-of-distribution lighting, camera framing, object placement, or task-state mismatches.

</td>
</tr>
</table>

### 🔧 Supported Frameworks

| Framework | Status |
|:---|:---:|
| **Diffusion Policy** | ✅ Supported |
| **Isaac-GR00T** | ✅ Supported |
| **Pi 0.5** | ✅ Supported |
| **DreamZero** | ✅ Supported |
| **VITA** | ✅ Supported |

> VLA-Lab uses a unified logging protocol — adapting a new framework takes only a few lines of glue code.

---

## 🧭 Recommended Debugging Workflow

1. **Run open-loop evaluation first.** If predicted actions do not match ground truth offline, fix the checkpoint, action normalization, horizon, or model inputs before running the robot.
2. **Then inspect real-robot Runs.** Check whether camera observations, model paths, prompts, states, action chunks, timing, and attention maps look sane during deployment.
3. **Finally inspect the dataset for OOD cases.** If open-loop eval and real-robot wiring both look healthy but success remains low, compare the run observations against training episodes. The failure may come from lighting, camera placement, object configuration, or states that were absent from the dataset.

---

## 📸 Screenshots

<table>
<tr>
<td align="center" width="50%">

#### 1. 🎯 Open-Loop Evaluation
<img src="assets/readme_openloop_eval_workbench.png" width="100%"/>
<sub>Default eval folders · GT vs Pred curves · trajectory metrics</sub>

</td>
<td align="center" width="50%">

#### 2. 🔬 Real-Robot Run Replay
<img src="assets/readme_run_replay_workbench.png" width="100%"/>
<sub>Multi-camera observations · state/action curves · action chunks · Rerun export</sub>

</td>
</tr>
<tr>
<td colspan="2" align="center">

#### 3. 📊 Dataset / OOD Inspection
<img src="assets/readme_dataset_ood_workbench.png" width="100%"/>
<sub>Default dataset folders · video playback · state/action/gripper curves · workspace checks</sub>

</td>
</tr>
</table>

---

## 🔧 Installation

```bash
pip install vlalab
```

Install with full dependencies (including Zarr dataset support):

```bash
pip install vlalab[full]
```

Or install from source:

```bash
git clone https://github.com/ky-ji/VLA-Lab.git
cd VLA-Lab
pip install -e .
```

---

## 🚀 Quick Start

### Minimal Example (3 Lines!)

```python
import vlalab

# Initialize a run
run = vlalab.init(project="pick_and_place", config={"model": "diffusion_policy"})

# Log during inference
vlalab.log({"state": obs["state"], "action": action, "images": {"front": obs["image"]}})
```

### Full Example

```python
import vlalab

# Initialize with detailed config
run = vlalab.init(
    project="pick_and_place",
    config={
        "model": "diffusion_policy",
        "action_horizon": 8,
        "inference_freq": 10,
    },
)

# Access config anywhere
print(f"Action horizon: {run.config.action_horizon}")

# Inference loop
for step in range(100):
    obs = get_observation()
    
    t_start = time.time()
    action = model.predict(obs)
    latency = (time.time() - t_start) * 1000
    
    # Log everything in one call
    vlalab.log({
        "state": obs["state"],
        "action": action,
        "images": {"front": obs["front_cam"], "wrist": obs["wrist_cam"]},
        "inference_latency_ms": latency,
    })

    robot.execute(action)

# Auto-finishes on exit, or call manually
vlalab.finish()
```

### Launch Visualization

```bash
# Default web dashboard (FastAPI + Next.js)
vlalab view
```

### Launch Web Dashboard (FastAPI + Next.js)

```bash
# 1. Install Python package
pip install -e .

# 2. Install web dependencies once
cd web
npm install
cd ..

# 3. Start FastAPI + Next.js together
vlalab view
```

Useful variants:

```bash
# Start only the API backend
vlalab serve --no-frontend

# Change the frontend port
vlalab view --port 3100

# Override the run directory used by the API
vlalab view --run-dir /path/to/vlalab_runs
```

The web UI covers the main workflows:

- deploy dashboard
- overview / run list / run detail
- dataset viewer
- open-loop eval viewer

---

## 📖 Documentation

### Core Concepts

**Run** — A single deployment session (one experiment, one episode, one evaluation)

**Step** — A single inference timestep with observations, actions, and timing

**Artifacts** — Images, point clouds, and other media saved alongside logs

### API Reference

<details>
<summary><b>vlalab.init() — Initialize a run</b></summary>

```python
run = vlalab.init(
    project: str = "default",     # Project name (creates subdirectory)
    name: str = None,             # Run name (auto-generated if None)
    config: dict = None,          # Config accessible via run.config.key
    dir: str = "./vlalab_runs",   # Base directory (or $VLALAB_DIR)
    tags: list = None,            # Optional tags
    notes: str = None,            # Optional notes
)
```

</details>

<details>
<summary><b>vlalab.log() — Log a step</b></summary>

```python
vlalab.log({
    # Robot state
    "state": [...],                    # Full state vector
    "pose": [x, y, z, qx, qy, qz, qw], # Position + quaternion
    "gripper": 0.5,                    # Gripper opening (0-1)
    
    # Actions
    "action": [...],                   # Single action or action chunk
    
    # Images (multi-camera support)
    "images": {
        "front": np.ndarray,           # HWC numpy array
        "wrist": np.ndarray,
    },
    
    # Timing (any *_ms field auto-captured)
    "inference_latency_ms": 32.1,
    "transport_latency_ms": 5.2,
    "custom_metric_ms": 10.0,
})
```

</details>

<details>
<summary><b>RunLogger — Advanced API</b></summary>

For fine-grained control over logging:

```python
from vlalab import RunLogger

logger = RunLogger(
    run_dir="runs/experiment_001",
    model_name="diffusion_policy",
    model_path="/path/to/checkpoint.pt",
    task_name="pick_and_place",
    robot_name="franka",
    cameras=[
        {"name": "front", "resolution": [640, 480]},
        {"name": "wrist", "resolution": [320, 240]},
    ],
    inference_freq=10.0,
)

logger.log_step(
    step_idx=0,
    state=[0.5, 0.2, 0.3, 0, 0, 0, 1, 1.0],
    action=[[0.51, 0.21, 0.31, 0, 0, 0, 1, 1.0]],
    images={"front": image_rgb},
    timing={
        "client_send": t1,
        "server_recv": t2,
        "infer_start": t3,
        "infer_end": t4,
    },
)

logger.close()
```

</details>

### CLI Commands

```bash
# Launch the default FastAPI + Next.js dashboard
vlalab view [--port 3000] [--api-port 8000]

# Launch FastAPI backend and optional Next.js frontend
vlalab serve [--api-port 8000] [--web-port 3000] [--frontend/--no-frontend]

# Convert legacy logs (auto-detects format)
vlalab convert /path/to/old_log.json -o /path/to/output

# Inspect a run
vlalab info /path/to/run_dir
```

### Attention Backend Integration

`VLA-Lab` now treats attention extraction as a model-provided backend with a stable interface.
The built-in default still auto-discovers the existing `Isaac-GR00T` backend, but new models should register their own backend explicitly:

```bash
export VLALAB_ATTENTION_BACKEND=/abs/path/to/backend.py
export VLALAB_ATTENTION_PYTHON=/abs/path/to/python  # optional
```

Interface contract and migration checklist:

- [Attention Backend Guide](docs/attention_backend.md)
- [Deploy Dashboard Guide](docs/deploy_dashboard.md)

---

## 📁 Run Directory Structure

```
vlalab_runs/
└── pick_and_place/                 # Project
    └── run_20240115_103000/        # Run
        ├── meta.json               # Metadata (model, task, robot, cameras)
        ├── steps.jsonl             # Step records (one JSON per line)
        └── artifacts/
            └── images/             # Saved images
                ├── step_000000_front.jpg
                ├── step_000000_wrist.jpg
                └── ...
```

---

## 🗺️ Roadmap

- [x] Core logging API & unified run format
- [x] Streamlit visualization suite (5 pages)
- [x] Diffusion Policy adapter
- [x] Isaac-GR00T adapter
- [x] Pi 0.5 adapter
- [x] DreamZero adapter
- [x] VITA adapter
- [x] Open-loop evaluation pipeline
- [ ] Cloud sync & team collaboration
- [ ] Real-time streaming dashboard
- [ ] Automatic failure detection
- [ ] Integration with robot simulators

---

## 🤝 Contributing

We welcome contributions! 

```bash
git clone https://github.com/ky-ji/VLA-Lab.git
cd VLA-Lab
pip install -e ".[dev]"
```

---

## 📄 License

MIT License — see [LICENSE](LICENSE) for details.

---

<div align="center">
  
**⭐ Star us on GitHub if VLA-Lab helps your research!**

*Built with ❤️ for the robotics community*

</div>
