Metadata-Version: 2.4
Name: runway-robotics-sdk
Version: 0.1.0
Summary: A Python client for simulating robot actions and generating augmented data using Runway's Robotics World Model API
Project-URL: Homepage, https://github.com/runwayml/robotics-sdk-python
Project-URL: Repository, https://github.com/runwayml/robotics-sdk-python
Project-URL: Issues, https://github.com/runwayml/robotics-sdk-python/issues
Author-email: RunwayML <dev-feedback@runwayml.com>
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: OS Independent
Classifier: Operating System :: POSIX
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx<1,>=0.28.1
Requires-Dist: imageio-ffmpeg<1,>=0.5.1
Requires-Dist: imageio<3,>=2.36.1
Requires-Dist: numpy<3,>=1.24.0
Requires-Dist: pillow<14,>=11.0.0
Requires-Dist: runwayml<5,>=3.22.0
Requires-Dist: tqdm<5,>=4.67.1
Provides-Extra: demo
Requires-Dist: gradio>=5.8.0; extra == 'demo'
Description-Content-Type: text/markdown

# Runway Robotics Python SDK

[![PyPI version](https://img.shields.io/pypi/v/runway-robotics-sdk)](https://pypi.org/project/runway-robotics-sdk/)
[![Python](https://img.shields.io/pypi/pyversions/runway-robotics-sdk)](https://pypi.org/project/runway-robotics-sdk/)
[![License](https://img.shields.io/pypi/l/runway-robotics-sdk)](LICENSE)

A Python SDK for interacting with Runway's General World Model (GWM-1) and video generation APIs, featuring action-conditioned world simulation and video-to-video transformation for robotics applications.

## Features

- **World Model** — Generate predicted video sequences from robot actions and camera frames
- **Image-to-Video** — Generate video from a single reference image using text prompts
- **Video-to-Video** — Transform footage using text prompts and style references

## Requirements

- Python 3.10+
- A [Runway API](https://dev.runwayml.com/) key

## Installation

```bash
pip install runway-robotics-sdk
```

Then set your API key as an environment variable:

```bash
export RUNWAYML_API_SECRET=your_api_key_here
```

## Quick Start

All SDK functionality is accessed through the `RunwayRobotics` client:

```python
from runway_robotics_sdk import RunwayRobotics

client = RunwayRobotics()
```

The client reads your API key from `RUNWAYML_API_SECRET` by default. You can also pass it explicitly:

```python
client = RunwayRobotics(api_key="your_api_key_here")
```

| Parameter | Type | Description |
|-----------|------|-------------|
| `api_key` | `str \| None` | Runway API secret key. Defaults to the `RUNWAYML_API_SECRET` environment variable. |
| `base_url` | `str \| None` | Override the API base URL. Defaults to `RUNWAYML_BASE_URL` environment variable or the production endpoint. Use this if connecting to a custom environment.|

## Robotics World Model

Generate predicted video sequences from initial camera frames and robot action trajectories using `client.world_model.create()`.

### Single-View Mode

```python
world = client.world_model.create(
    base_view=Image.open("data/base_view.jpg"),
)

actions = load_your_action_sequence()

for i in range(0, len(actions), world.chunk_size):
    obs = world.step(actions[i : i + world.chunk_size])

world.save("outputs/trajectory.mp4")
```

### Multi-View Mode

```python
world = client.world_model.create(
    base_view=Image.open("data/base_view.jpg"),
    wrist_view=Image.open("data/wrist_view.jpg"),
)

actions = load_your_action_sequence()

for i in range(0, len(actions), world.chunk_size):
    obs = world.step(actions[i : i + world.chunk_size])
    # obs.base_image  — latest base camera frame
    # obs.wrist_image — latest wrist camera frame

world.save("outputs/trajectory.mp4")
```

### Action Format

Each action is a 7-element list:

| Index | Value |
|-------|-------|
| 0–5 | End-effector pose (x, y, z, roll, pitch, yaw) |
| 6 | Gripper state (continuous) |

Actions are processed in chunks of `world.chunk_size` (17) frames. Shorter sequences are automatically padded to the chunk size.

### Supported Robot Arms

The current model works well with the following robot arms:

| Robot Arm | Training Data |
|-----------|--------------|
| **Franka Emika Panda** | [DROID dataset](https://droid-dataset.github.io/) |
| **WidowX 250 S** | [BridgeData V2](https://rail-berkeley.github.io/bridgedata/) |

### API Reference

**`client.world_model.create()`**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `base_view` | `PIL.Image \| np.ndarray` | *required* | Starting base camera view |
| `wrist_view` | `PIL.Image \| np.ndarray \| None` | `None` | Starting wrist camera view. If `None`, single-view mode is used. |
| `timeout` | `float` | `60` | Maximum seconds to wait for each API call |
| `show_progress` | `bool` | `False` | Display a progress bar |

**Returns:** A `WorldModelEnv` instance.

**`WorldModelEnv` Methods:**

| Method | Description |
|--------|-------------|
| `step(action)` | Execute actions, return a `StepOutput` with `.base_image` and `.wrist_image` |
| `save(path)` | Save all generated frames as a video file (15 FPS) |

**`WorldModelEnv` Properties:**

| Property | Description |
|----------|-------------|
| `frames` | List of all generated frames |
| `chunk_size` | Number of actions per API call (17) |
| `last_base_view` | Most recent base camera observation |
| `last_wrist_view` | Most recent wrist camera observation |

## Image-to-Video

Generate video from a single reference image via `client.image_to_video.generate()`. Provide a starting frame and an optional text prompt describing the desired motion or scene.

### Basic Usage

```python
result = client.image_to_video.generate(
    image="robot_workspace.png",
    prompt="The robot arm reaches forward and grasps the red cup",
    show_progress=True,
)

# Save the raw video to disk
result.save("outputs/generated.mp4")

# Or access decoded frames directly
frames = result.frames  # np.ndarray of shape (T, H, W, C)
```

### Input Formats

The `image` parameter accepts multiple formats:

| Format | Example |
|--------|---------|
| URI | `"https://example.com/image.png"` |
| Local file | `"image.png"` or `Path("image.png")` |
| PIL Image | `Image.open("image.png")` |
| Numpy array | `(H, W, C)` array |

### API Reference

**`client.image_to_video.generate()`**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `image` | `str \| Path \| PIL.Image \| np.ndarray` | *required* | Input image |
| `prompt` | `str \| None` | `None` | Text description of desired output (1–1000 chars) |
| `ratio` | `str \| None` | `None` | Output aspect ratio. Auto-selects from input dimensions if `None`. Options: `"1280:720"`, `"720:1280"`, `"1104:832"`, `"832:1104"`, `"960:960"`, `"1584:672"` |
| `duration` | `int` | `10` | Output video duration in seconds |
| `seed` | `int \| None` | `None` | Random seed for reproducibility |
| `timeout` | `float` | `300` | Maximum seconds to wait for API response |
| `show_progress` | `bool` | `False` | Display a progress bar |

**Returns:** A `GenerationResult` with `.bytes` (raw video), `.frames` (lazy-decoded `np.ndarray` of shape `(T, H, W, C)`), and `.save(path)`.

## Video-to-Video

Transform videos using text prompts via `client.video_to_video.generate()`. Common use cases include:

- **Scene modification** — Add or change objects on a workspace surface
- **Lighting conditions** — Simulate different lighting environments
- **Material appearance** — Change object textures, colors, or materials

### Basic Usage

```python
result = client.video_to_video.generate(
    video="rollout.mp4",
    prompt="Make the lighting in the scene dark as if the power was out in the warehouse.",
    show_progress=True,
)

result.save("outputs/transformed.mp4")
frames = result.frames  # np.ndarray of shape (T, H, W, C)
```

### Input Formats

The `video` parameter accepts multiple formats:

| Format | Example |
|--------|---------|
| URI | `"https://example.com/video.mp4"` |
| Local file | `"video.mp4"` or `Path("video.mp4")` |
| Numpy array | Single `(T, H, W, C)` array |
| Frame list | List of `(H, W, C)` arrays |

### Style References

Guide the transformation with a reference image:

```python
result = client.video_to_video.generate(
    video="rollout.mp4",
    prompt="Transform the scene to match the reference lighting",
    references=[Image.open("reference.jpg")],
)
result.save("outputs/transformed.mp4")
```

### API Reference

**`client.video_to_video.generate()`**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `video` | `str \| Path \| np.ndarray \| list[np.ndarray]` | *required* | Input video |
| `prompt` | `str` | *required* | Text description of desired transformation (1–1000 chars) |
| `ratio` | `str \| None` | `None` | Output aspect ratio. Auto-selects from input dimensions if `None`. Options: `"1280:720"`, `"720:1280"`, `"1104:832"`, `"960:960"`, `"832:1104"`, `"1584:672"`, `"848:480"`, `"640:480"` |
| `seed` | `int \| None` | `None` | Random seed for reproducibility |
| `references` | `list \| None` | `None` | Up to 1 reference image for style guidance |
| `timeout` | `float` | `300` | Maximum seconds to wait for API response |
| `show_progress` | `bool` | `False` | Display a progress bar |

**Returns:** A `GenerationResult` with `.bytes` (raw video), `.frames` (lazy-decoded `np.ndarray` of shape `(T, H, W, C)`), and `.save(path)`.

## Error Handling

The SDK raises the following exceptions:

| Exception | When |
|-----------|------|
| `ValueError` | Invalid inputs (bad aspect ratio, too many references, empty actions) |
| `GenerationError` | A generation task failed, timed out, or the output could not be downloaded |
| `AuthenticationError` | Invalid API key (HTTP 401) |
| `RateLimitError` | Too many requests (HTTP 429) |
| `APIStatusError` | Any other API HTTP error (4xx/5xx) — has `.status_code` and `.body` |
| `APIConnectionError` | Network connectivity failure |

All exceptions can be imported directly from `runway_robotics_sdk`:

```python
from runway_robotics_sdk import GenerationError, AuthenticationError

try:
    obs = world.step(actions)
except GenerationError as e:
    print(f"Simulation failed: {e}")
except AuthenticationError:
    print("Check your API key")
```

## Examples

```bash
uv run python examples/world_model.py       # World model action rollout (multi-view)
uv run python examples/image_to_video.py     # Generate rollout video from an image
uv run python examples/video_to_video.py     # Transform a rollout video with a text prompt
```

### Interactive Web Demo

Launch the Gradio-based demo for interactive world model simulation:

```bash
pip install runway-robotics-sdk[demo]
uv run python demo/app.py
```

Open [http://localhost:7880](http://localhost:7880) to upload camera frames, provide action sequences, and generate predicted trajectories.

## Contributing

Clone the repository and install development dependencies:

```bash
git clone https://github.com/runwayml/robotics-sdk-python.git
cd robotics-sdk-python
uv sync --all-extras
uv run lefthook install
```

Git hooks (via [Lefthook](https://github.com/evilmartians/lefthook)) run automatically:

- **Pre-commit** — `ruff format` and `ruff check --fix` on staged files (auto-fixes and re-stages)
- **Pre-push** — `mypy` type checking and `pytest` with coverage

You can also run checks manually:

```bash
uv run ruff format .          # Format code
uv run ruff check --fix .     # Lint (with auto-fix)
uv run mypy runway_robotics_sdk/ demo/ examples/  # Type check
uv run pytest                  # Run tests
```

All checks are also enforced via GitHub Actions CI.

## Data Attribution

Example data in this repository uses the [DROID](https://droid-dataset.github.io/) (Distributed Robot Interaction Dataset), licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). See [CITATION.md](CITATION.md) for full citation details.

## License

This project is licensed under the Apache License 2.0. See [LICENSE](LICENSE) for details.
