Metadata-Version: 2.4
Name: datachain
Version: 0.62.6
Summary: Wrangle unstructured AI data at scale
Author-email: Dmitry Petrov <support@dvc.org>
License-Expression: Apache-2.0
Project-URL: Documentation, https://www.datachain.ai/docs
Project-URL: Issues, https://github.com/datachain-ai/datachain/issues
Project-URL: Source, https://github.com/datachain-ai/datachain
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Development Status :: 2 - Pre-Alpha
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml
Requires-Dist: tomlkit
Requires-Dist: tqdm
Requires-Dist: numpy<3,>=1
Requires-Dist: pandas>=2.0.0
Requires-Dist: ujson>=5.10.0
Requires-Dist: packaging
Requires-Dist: pyarrow>=22
Requires-Dist: typing-extensions
Requires-Dist: python-dateutil>=2
Requires-Dist: dateparser>=1.3.0
Requires-Dist: attrs>=21.3.0
Requires-Dist: fsspec>=2024.2.0
Requires-Dist: s3fs>=2026.6.0
Requires-Dist: gcsfs!=2026.8.0,!=2026.8.1,>=2026.3.0
Requires-Dist: adlfs>=2024.2.0
Requires-Dist: azure-storage-blob>=12.28.0
Requires-Dist: dvc-data<4,>=3.10
Requires-Dist: mcap>=1.6.0
Requires-Dist: dvc-objects<6,>=4
Requires-Dist: shtab<2,>=1.3.4
Requires-Dist: sqlalchemy>=2
Requires-Dist: multiprocess>=0.70.16
Requires-Dist: cloudpickle
Requires-Dist: pydantic>=2.11
Requires-Dist: jmespath>=1.0
Requires-Dist: datamodel-code-generator[genson]>=0.96.0
Requires-Dist: Pillow<14,>=13.0.0
Requires-Dist: msgpack<2,>=1.0.4
Requires-Dist: psutil
Requires-Dist: huggingface_hub
Requires-Dist: iterative-telemetry>=0.0.10
Requires-Dist: platformdirs
Requires-Dist: filelock
Requires-Dist: dvc-studio-client<1,>=0.23.0
Requires-Dist: tabulate
Requires-Dist: websockets
Requires-Dist: litellm<2,>=1.103.1
Provides-Extra: docs
Requires-Dist: mkdocs>=1.5.2; extra == "docs"
Requires-Dist: mkdocs-gen-files>=0.5.0; extra == "docs"
Requires-Dist: mkdocs-material==9.8.0; extra == "docs"
Requires-Dist: mkdocstrings-python>=1.6.3; extra == "docs"
Requires-Dist: mkdocs-literate-nav>=0.6.1; extra == "docs"
Requires-Dist: mkdocs-section-index>=0.3.10; extra == "docs"
Requires-Dist: mkdocs-redirects>=1.2.1; extra == "docs"
Requires-Dist: eval-type-backport; extra == "docs"
Provides-Extra: torch
Requires-Dist: torch>=2.1.0; extra == "torch"
Requires-Dist: torchvision; extra == "torch"
Requires-Dist: transformers>=4.36.0; extra == "torch"
Provides-Extra: audio
Requires-Dist: soundfile; extra == "audio"
Provides-Extra: remote
Requires-Dist: lz4; extra == "remote"
Requires-Dist: requests>=2.22.0; extra == "remote"
Provides-Extra: vector
Requires-Dist: usearch; extra == "vector"
Requires-Dist: numkong!=7.7.1; sys_platform == "darwin" and extra == "vector"
Provides-Extra: hf
Requires-Dist: numba>=0.60.0; extra == "hf"
Requires-Dist: datasets[vision]>=4.0.0; extra == "hf"
Requires-Dist: datasets[audio]>=4.0.0; (sys_platform == "linux" or sys_platform == "darwin") and extra == "hf"
Requires-Dist: fsspec>=2024.12.0; extra == "hf"
Provides-Extra: video
Requires-Dist: av>=16.1.0; extra == "video"
Requires-Dist: ffmpeg-python; extra == "video"
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9.0; extra == "postgres"
Provides-Extra: zarr
Requires-Dist: zarr>=3.0.0; extra == "zarr"
Provides-Extra: tests
Requires-Dist: datachain[audio,hf,postgres,remote,torch,vector,video,zarr]; extra == "tests"
Requires-Dist: pytest<9.3,>=8; extra == "tests"
Requires-Dist: pytest-asyncio; extra == "tests"
Requires-Dist: pytest-sugar>=0.9.6; extra == "tests"
Requires-Dist: pytest-cov>=4.1.0; extra == "tests"
Requires-Dist: pytest-mock>=3.12.0; extra == "tests"
Requires-Dist: pytest-servers[all]>=0.5.14; extra == "tests"
Requires-Dist: pytest-benchmark[histogram]>=5.3.1; extra == "tests"
Requires-Dist: pytest-codspeed>=5.0.4; extra == "tests"
Requires-Dist: pytest-xdist>=3.3.1; extra == "tests"
Requires-Dist: pytest-split>=0.10.0; extra == "tests"
Requires-Dist: pytest-dotenv; extra == "tests"
Requires-Dist: virtualenv; extra == "tests"
Requires-Dist: dulwich; extra == "tests"
Requires-Dist: hypothesis; extra == "tests"
Requires-Dist: aiotools>=1.7.0; extra == "tests"
Requires-Dist: requests-mock; extra == "tests"
Requires-Dist: scipy; extra == "tests"
Requires-Dist: ultralytics; extra == "tests"
Provides-Extra: dev
Requires-Dist: datachain[docs,tests]; extra == "dev"
Requires-Dist: mypy==2.4.1; extra == "dev"
Requires-Dist: types-python-dateutil; extra == "dev"
Requires-Dist: types-dateparser; extra == "dev"
Requires-Dist: types-pytz; extra == "dev"
Requires-Dist: types-PyYAML; extra == "dev"
Requires-Dist: types-requests; extra == "dev"
Requires-Dist: types-tabulate; extra == "dev"
Provides-Extra: examples
Requires-Dist: datachain[tests]; extra == "examples"
Requires-Dist: defusedxml; extra == "examples"
Requires-Dist: accelerate; extra == "examples"
Requires-Dist: anthropic; extra == "examples"
Requires-Dist: huggingface_hub[hf_transfer]; extra == "examples"
Requires-Dist: transformers>=5.0.0; extra == "examples"
Requires-Dist: ultralytics; extra == "examples"
Requires-Dist: open_clip_torch; extra == "examples"
Requires-Dist: openai; extra == "examples"
Requires-Dist: torchaudio; extra == "examples"
Dynamic: license-file

# ![DataChain](https://raw.githubusercontent.com/datachain-ai/datachain/main/docs/assets/datachain.svg) Data preparation and exploration for Physical AI

[![PyPI](https://img.shields.io/pypi/v/datachain.svg)](https://pypi.org/project/datachain/)
[![Python Version](https://img.shields.io/pypi/pyversions/datachain)](https://pypi.org/project/datachain/)
[![Tests](https://github.com/datachain-ai/datachain/actions/workflows/tests.yml/badge.svg)](https://github.com/datachain-ai/datachain/actions/workflows/tests.yml)

DataChain is a Python library for processing large file collections in S3, GCS, Azure and local storage.
Run your readers and models over images, video, audio and sensor recordings, then save the results as typed, versioned datasets.

Query saved metadata to find relevant files, episodes and clips. Build new training selections from existing
labels, embeddings and features without repeating the original processing. Your coding agent can inspect
the saved results and producing code, then continue the work in a new session.

- **Training and evaluation:** combine metadata and model outputs, select useful examples, and export datasets to training tools.
- **Physical AI and robotics:** select recordings by sensor topics, extract video frames, and curate data for robot learning.
- **Data exploration:** investigate model failures, find rare conditions and gaps in data coverage, and answer new questions from saved results.
- **Large inventories:** discover millions of objects using storage metadata, before reading their contents.

DataChain handles parallel workers, download overlap and checkpoints; your Python defines the processing.
`File` gives readers streaming access or a reusable local cache; delta updates process changed files and reuse saved results.
LeRobot episodes, MCAP time windows and WebDataset samples use the same Python API.

[Quickstart](#quickstart) · [Python API](#the-python-pipeline) · [Examples](#python-and-more-examples)

## Performance

### Object storage

DataChain was **~10× faster than rclone and s5cmd** when listing and saving **10M S3 object versions**.
Below: **257K objects per store**.

| Tool | S3 | GCS | Azure |
|---|---:|---:|---:|
| **DataChain** | **15 s** | **7 s** | **26 s** |
| rclone | 23 s (1.5×) | 20 s (2.9×) | 95 s (3.7×) |
| obstore | 20 s (1.3×) | 15 s (2.1×) | 66 s (2.5×) |
| AWS CLI / gcloud / azcopy | 66 s (4.4×) | 175 s (25×) | 94 s (3.6×) |

Times include saving. [Full comparison, including faster single-cloud tools](https://github.com/datachain-ai/datachain/blob/main/docs/benchmarks.md#comparison-with-dedicated-listing-tools).

### Training-data workflows

DataChain created and read these training selections **2.1–8.1× faster**:

| Workload | Compared with | Speedup |
|---|---|---:|
| LeRobot: 300 DROID episodes (S3) | LeRobot streaming (HF Hub) | **8.1×** |
| MCAP: 207 grasp clips | `mcap filter` | **2.1×** |

Development runs; DROID compares S3 with HF Hub. [Details](https://github.com/datachain-ai/datachain/blob/main/docs/benchmarks.md#training-subsets-across-file-formats).

### Coding agents

DataChain + skill delivered verified results **up to 2.7× faster** than no skill.

| Agent / LLM | Task | Speedup |
|---|---|---:|
| Claude Code / Opus 5.5 | Held-out media tasks | **2.69×** |
| Codex / GPT-6.1 Sol | Bucket inventory | **1.67×** |
| Codex / GPT-6.1 Sol | Video updates | **2.05×** |

Codex: two runs per development task; no overall gain in held-out tests. [Details](https://github.com/datachain-ai/datachain/blob/main/docs/benchmarks.md#codex-on-development-tasks).

## Quickstart

In an empty project directory, install DataChain:

```bash
pip install 'datachain[video]'
```

For a coding agent, also install the skill and start a new agent session in that directory:

```bash
datachain skill install --target claude --local
# Or: --target codex, cursor, copilot, pi
```

Ask your agent:

```text
Prepare a training selection from the robot demonstrations at
s3://robolake/droid/droid_100/
Score wrist-camera motion over the first second: mean absolute pixel change.
Keep scores of at least 10. Save the scores, camera references and training manifest.
Save and run build_episodes.py.
```

### The Python pipeline

Here is the same task in Python. Save it as `build_episodes.py` and run `python build_episodes.py`:

```python
import datachain as dc


def motion(camera: dc.VideoFragment) -> float:
    start = round(camera.start * 15)
    end = round(camera.end * 15) - 1
    first = camera.video.get_frame(start).get_np().astype("float32")
    later = camera.video.get_frame(min(start + 15, end)).get_np()
    return float(abs(later - first).mean())


episodes = dc.read_lerobot("s3://robolake/droid/droid_100/", anon=True)
scored = (
    episodes.settings(parallel=4, prefetch=0)
    .map(motion=motion, params=["videos.observation_images_wrist_image_left"])
    .save("robot_episodes")
)
training = scored.filter(dc.C("motion") >= 10).save("training_episodes")
training.to_parquet("training_manifest.parquet")
```

This [DROID](https://droid-dataset.github.io/) sample records at 15 fps. Each episode keeps the
camera files and time windows as references; no video repacking, model download or GPU is needed.
A reference run produced:

```text
robot_episodes@1.0.0      100 rows
training_episodes@1.0.0   63 rows
training_manifest.parquet
```

For MCAP sensor logs, create time windows with the same API:

```python
windows = dc.read_mcap("recordings/*.mcap", topics="/joint_states", duration=10)
windows.save("joint_state_windows")
```

The [standalone MCAP example](https://www.datachain.ai/docs/guide/file-pipelines/#catalog-mcap-recordings) includes local recordings to try.

### Continue in a new session

Keep the project directory: it contains `.datachain/`, the Python script and the manifest.
Start a new agent session in the same project and ask:

```text
Which episodes did we exclude? How many would pass at threshold 5? Use the saved scores.
```

Inspect the saved scores, camera windows and producing code:

```bash
datachain dataset describe training_episodes --json
datachain show training_episodes --limit 3
```

See the [walkthrough](https://www.datachain.ai/docs/guide/file-pipelines/) for versioned reads, robot demonstrations and an image-to-PyTorch example.

**Agents reading this:** start with the [skill](https://github.com/datachain-ai/datachain/blob/main/src/datachain/skill/datachain/SKILL.md).
[llms.txt](https://github.com/datachain-ai/datachain/blob/main/llms.txt) links to the API and documentation.

## Python and more examples

- [Robot learning: curate LeRobot episodes](https://www.datachain.ai/docs/guide/file-pipelines/#robot-demonstrations): process camera clips and reuse saved selections.
- [WebDataset: revise a saved selection](https://www.datachain.ai/docs/guide/file-pipelines/#webdataset-samples): filter samples inside TAR shards without repacking them.
- [Video datasets: extract frames and timestamps](https://www.datachain.ai/docs/references/data-types/videofile/): read video with `datachain[video]`.
- [Training: load a saved selection into PyTorch](https://www.datachain.ai/docs/guide/file-pipelines/#feed-pytorch): use a pinned version in a new process.
- [Public image pipeline](https://www.datachain.ai/docs/guide/file-pipelines/#public-images): typed `.map()`, selection and manifest.
- [Multi-cloud inventory](https://www.datachain.ai/docs/guide/file-pipelines/#inventory-across-clouds): aggregate S3, GCS and Azure listings.
- [Updates and resume](https://www.datachain.ai/docs/guide/file-pipelines/#update-and-resume): refresh saved results and continue interrupted work.

## Where it runs

Locally, dataset rows and provenance live in `.datachain/db` (SQLite); downloaded files use `.datachain/cache`.
Keep that project state for later sessions. Local use needs no DataChain account or server.

Optional [DataChain Studio](https://datachain.ai) adds shared datasets, versions and jobs, with AI-generated descriptions.
People and agents can reuse the same catalog through the UI and [MCP](https://www.datachain.ai/docs/studio/user-guide/agents/mcp/).
Jobs run with or without a cluster; clusters process large workloads across dozens or hundreds of machines.

Cloud reads incur normal provider request and transfer charges. The library is
[Apache-2.0](https://github.com/datachain-ai/datachain/blob/main/LICENSE). [Storage configuration](https://www.datachain.ai/docs/guide/remotes) · [Dataset DB](https://www.datachain.ai/docs/concepts/dataset-db)

[Documentation](https://www.datachain.ai/docs) · [Examples](https://github.com/datachain-ai/datachain/blob/main/examples) ·
[Contributing](https://www.datachain.ai/docs/contributing) · [Issues](https://github.com/datachain-ai/datachain/issues) ·
[Discord](https://discord.com/invite/datachain)
