Metadata-Version: 2.4
Name: daft-physical-ai
Version: 0.1.0
Summary: Physical-AI data processing on Daft, starting with hand tracking.
Project-URL: Repository, https://github.com/Eventual-Inc/daft-physical-ai
Author: Eventual
Maintainer: Eventual
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: <3.14,>=3.10
Requires-Dist: daft>=0.7.17
Requires-Dist: numpy
Provides-Extra: all
Requires-Dist: einops; extra == 'all'
Requires-Dist: hydra-colorlog; extra == 'all'
Requires-Dist: hydra-core; extra == 'all'
Requires-Dist: mediapipe>=0.10; (python_version < '3.13') and extra == 'all'
Requires-Dist: numpy<2; extra == 'all'
Requires-Dist: opencv-python-headless; extra == 'all'
Requires-Dist: pandas; extra == 'all'
Requires-Dist: pyrootutils; extra == 'all'
Requires-Dist: pytorch-lightning==2.1.3; extra == 'all'
Requires-Dist: rich; extra == 'all'
Requires-Dist: scikit-image; extra == 'all'
Requires-Dist: smplx==0.1.28; extra == 'all'
Requires-Dist: timm; extra == 'all'
Requires-Dist: torch; extra == 'all'
Requires-Dist: torchvision; extra == 'all'
Requires-Dist: ultralytics==8.1.34; extra == 'all'
Requires-Dist: webdataset; extra == 'all'
Requires-Dist: xtcocotools; extra == 'all'
Requires-Dist: yacs; extra == 'all'
Provides-Extra: mediapipe
Requires-Dist: mediapipe>=0.10; (python_version < '3.13') and extra == 'mediapipe'
Provides-Extra: wilor
Requires-Dist: einops; extra == 'wilor'
Requires-Dist: hydra-colorlog; extra == 'wilor'
Requires-Dist: hydra-core; extra == 'wilor'
Requires-Dist: numpy<2; extra == 'wilor'
Requires-Dist: opencv-python-headless; extra == 'wilor'
Requires-Dist: pandas; extra == 'wilor'
Requires-Dist: pyrootutils; extra == 'wilor'
Requires-Dist: pytorch-lightning==2.1.3; extra == 'wilor'
Requires-Dist: rich; extra == 'wilor'
Requires-Dist: scikit-image; extra == 'wilor'
Requires-Dist: smplx==0.1.28; extra == 'wilor'
Requires-Dist: timm; extra == 'wilor'
Requires-Dist: torch; extra == 'wilor'
Requires-Dist: torchvision; extra == 'wilor'
Requires-Dist: ultralytics==8.1.34; extra == 'wilor'
Requires-Dist: webdataset; extra == 'wilor'
Requires-Dist: xtcocotools; extra == 'wilor'
Requires-Dist: yacs; extra == 'wilor'
Description-Content-Type: text/markdown

# daft-physical-ai

Physical-AI data processing on [Daft](https://github.com/Eventual-Inc/Daft), starting with hand
tracking. The methods run as Daft UDFs, so they slot into any Daft
pipeline and execute lazily, batched, and distributed.

## API

The package operates on a Daft image column and returns a hand-pose column. A
LeRobot dataset is a natural source: Daft's native reader
`daft.datasets.lerobot` (added in [Daft #7090](https://github.com/Eventual-Inc/Daft/pull/7090))
decodes each camera into an image column with `load_video_frames`.

```python
import daft
from daft.datasets import lerobot
from daft_physical_ai.hands import track_hands

# one row per frame; the camera key is decoded into an image column.
# egodex-test is a tiny EgoDex sample (3 episodes / 632 frames) in LeRobot v3 format.
df = lerobot.read("pepijn223/egodex-test", load_video_frames="observation.image")

# pick a method (each returns the same schema):
# mediapipe -> CPU, 2D only, permissive license, no weights to supply
# wilor     -> GPU, 3D MANO keypoints (MANO weights user-supplied)
df = df.with_column("hands", track_hands(df["observation.image"], method="mediapipe"))

df.write_parquet("annotated/")
```

Install the method you need as an extra: `pip install daft-physical-ai[mediapipe]`
(CPU, 2D), `pip install daft-physical-ai[wilor]` (GPU, 3D), or
`pip install daft-physical-ai[all]` for both. WiLoR additionally needs a CUDA
`torch` build and `chumpy` from git
(`pip install 'chumpy @ git+https://github.com/mattloper/chumpy'`, omitted from the
extra because PyPI metadata can't carry direct references), plus a user-supplied
`MANO_RIGHT.pkl` ([research-gated](docs/mano.md)).

> **Note:** the LeRobot reader with batched video decode
> ([Daft #7184](https://github.com/Eventual-Inc/Daft/pull/7184)) is not yet in a
> stable Daft release (latest is v0.7.17). Until it is, this repo resolves `daft`
> from the nightly index - `uv sync` handles it via the `daft-nightly` index in
> `pyproject.toml`.

## Output schema

One unified output schema regardless of method: each frame yields a list of
0-2 detected hands. A single hand value (MediaPipe):

```python
{
    "handedness": "right",        # "left", "right", or "unknown"
    "confidence": 0.979,
    "kp2d": [[1412.1, 1111.1],    # 21 image-space [x, y] keypoints
             [1357.9, 1075.9],
             ...],
    "kp3d": None,                 # 21 [x, y, z] keypoints, or null for 2D-only methods
}
```

The Daft type is `list[struct{ handedness: string, confidence: float32, kp2d:
list[list[float32]], kp3d: list[list[float32]] }]`, defined as `HANDS_DTYPE` in
`daft_physical_ai/hands/schema.py`.

## Example

A complete walkthrough - read a dataset, run `track_hands` (MediaPipe), draw the
keypoints, and score against EgoDex ground truth:

![track_hands keypoints](examples/demo_keypoints.png)

Available in three equivalent forms:

- **[examples/demo.md](examples/demo.md)** - read it start to finish; code and outputs inline.
- **[examples/demo.ipynb](examples/demo.ipynb)** - runnable notebook (outputs included).
- **[examples/demo.py](examples/demo.py)** - plain script.

Generate your own (other methods, a Modal GPU runtime, with/without eval) with the
`daft-physical-ai hands` command - run it with no flags for an interactive
walkthrough, or pass flags:

```bash
# No flags - interactive walkthrough that asks a few questions
daft-physical-ai hands

# --no-input skips all prompts; flags supply the answers, the rest use defaults
daft-physical-ai hands --method mediapipe --output-dir my-demo --no-input
daft-physical-ai hands --method wilor --runtime modal --mano-path ./MANO_RIGHT.pkl --no-input
```

Hand tracking is the first capability; each new one will be its own subcommand
(`daft-physical-ai <command>` lists what's available).

The bare `daft-physical-ai` command works once the package is installed. Until
it's published to PyPI, run it from a clone instead:

```bash
uv sync                          # installs the daft-physical-ai console script
uv run daft-physical-ai          # generate a demo (prefix the commands above with `uv run`)
```

To *run* a generated demo you also need its inference stack (`uv sync` already
brings the nightly Daft with the LeRobot reader). Install the extras into the
venv, then run from the activated venv - not `uv run`, which re-syncs the env
and would drop them:

```bash
source .venv/bin/activate
uv pip install -U av mediapipe scipy opencv-python matplotlib jupyterlab
jupyter lab hand-tracking-demo/demo.ipynb
```

Once the LeRobot reader lands in a stable Daft release (> v0.7.17), the nightly
pin in `pyproject.toml` goes away and this collapses to
`pip install daft-physical-ai`.

## Development

```bash
uv sync                      # set up env + install deps
uv run pre-commit install    # install lint/format hooks
uv run pytest tests/ -v      # run the test suite
```

## Versioning

Versions are derived from git tags via `hatch-vcs`. Tag releases as `v0.1.0`,
`v0.2.0`, etc.

## Publishing

Publishing a GitHub release triggers `.github/workflows/publish-package.yml`,
which builds a wheel and sdist with `uv build` and uploads both to PyPI via
[trusted publishing](https://docs.pypi.org/trusted-publishers/). Configure the
trusted publisher on PyPI for this repository before the first release.
