# DataChain

> DataChain is a Python library for file pipelines across S3, GCS, Azure and local storage. Use it for inventories, Python readers and model inference, then save typed, versioned results for later queries. Local pipelines use the project's catalog without a DataChain account.

## Agent entry point

- [DataChain skill](https://github.com/datachain-ai/datachain/blob/main/src/datachain/skill/datachain/SKILL.md): task selection, inventories, media metadata, batched inference, saved results and updates.
- [Python and agent quickstarts](https://github.com/datachain-ai/datachain/blob/main/README.md).
- [API reference](https://docs.datachain.ai/references/).

Start saved-result questions with the catalog and recorded code. Refresh sources when current freshness is requested; file delta does not detect model or preprocessing changes.

## Import Convention

```python
import datachain as dc
```

Access everything through `dc.*`: `dc.read_storage()`, `dc.Column()`, `dc.func.*`, `dc.File`, `dc.ImageFile`, etc.

For annotation types: `from datachain import model`
For custom types: `from pydantic import BaseModel`

Never use: `from datachain import File, C, func`

## Key Patterns

### Read data
- `dc.list_storage("s3://bucket/path/")` -- list storage files without a chain or saved dataset ([guide](https://docs.datachain.ai/guide/listing-storage/))
- `dc.read_storage("s3://bucket/path/", type="image")` -- files from cloud storage
- `dc.read_csv()`, `dc.read_json()`, `dc.read_parquet()` -- structured formats
- `dc.read_lerobot("hf://datasets/lerobot/pusht")` -- LeRobot v3 robotics dataset, one row per episode with camera `VideoFragment`s
- `dc.read_database(query, connection_string)` -- SQL databases
- `dc.read_dataset("name", version="1.0.0")` -- an explicit saved version
- `datachain dataset describe NAME --json` -- recorded schema, versions, lineage and code; metadata only
- `dc.read_pandas(df)`, `dc.read_hf("dataset")` -- in-memory sources

### Transform data
- `.filter(dc.C("col") > value)` -- filter saved rows in SQL
- `.mutate(new_col=dc.func.path.file_ext("file.path"))` -- SQL-compiled column derivation
- `.batch_map(result=my_batch_function, batch=32)` -- lists of inputs with one output per input; subclass `dc.BatchMapper` for model setup ([guide](https://docs.datachain.ai/guide/batch-inference/))
- `.map(result=my_function)` -- Python operation per record (needs `settings(parallel=N)` for perf)
- `.gen(chunks=split_function)` -- expand one record to many
- `.agg(summary=aggregate_fn, partition_by="group_col")` -- group and reduce
- `.merge(other, on="key")` -- join two chains
- `.group_by(count=dc.func.count(), partition_by="col")` -- aggregate analytics

### Save and export
- `.save("dataset_name")` -- versioned, named dataset (PREFERRED terminal operation)
- `.persist()` -- anonymous cache for branching
- `.to_pandas()`, `.to_parquet()`, `.to_csv()`, `.to_json()` -- export formats
- `.to_pytorch(transform=..., tokenizer=...)` -- iterable PyTorch Dataset for a DataLoader
- `.to_storage("s3://output/")` -- write files to storage
- `.to_database("table", "connection_string")` -- write to SQL database

## Critical Rules

1. Annotate Python operation return values or pass output=; unannotated functions default to str
2. Use a directory URI such as `"s3://bucket/path/"` or an explicit file glob such as `"s3://bucket/path/*.jpg"`
3. Pass `anon=True` for public buckets
4. Use `save()` before `filter()` on expensive Python operation output
5. Use `settings(parallel=True)` for ML/LLM operations
6. Use `batch_map(batch=N)` when the model takes a list of inputs per call
7. Use data-engine operations (filter, mutate, group_by) over Python when possible
8. `map` and `batch_map` support named functions for multiple signals; use a Pydantic model for related fields from one function
9. Preserve source File identities when expanding records with gen, especially for file-delta updates
10. Filter and aggregate saved rows with dc.func; use pandas, DuckDB or a warehouse for table-heavy analytics

## Documentation Sections

- Getting Started: /getting-started/
- Core Concepts: /concepts/ (dataset db, datasets, chain, execution model, files and types)
- Guides: /guide/ (reading data, operations, Python operations, exporting, datasets, functions, scaling, best practices)
- API Reference: /references/ (auto-generated from docstrings)
- CLI Reference: /commands/
- Best Practices: /guide/best-practices/
