Metadata-Version: 2.4
Name: datajig
Version: 0.7.0
Summary: Agent-native data control plane for verified, versioned, training-ready datasets
Project-URL: Repository, https://github.com/liukejun7/DataJig
Project-URL: Issues, https://github.com/liukejun7/DataJig/issues
Project-URL: Documentation, https://github.com/liukejun7/DataJig#readme
Author-email: Kejun Liu <liukj7@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data-quality,datasets,version-control
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Version Control
Requires-Python: >=3.11
Requires-Dist: blake3<2,>=1.0
Requires-Dist: imagehash<5,>=4.3
Requires-Dist: jinja2<4,>=3.1
Requires-Dist: numpy<3,>=2.0
Requires-Dist: pillow<13,>=11
Requires-Dist: scipy<2,>=1.14
Provides-Extra: dev
Requires-Dist: build<2,>=1.2; extra == 'dev'
Requires-Dist: hatchling<2,>=1.27; extra == 'dev'
Requires-Dist: mypy<2,>=1.14; extra == 'dev'
Requires-Dist: numpy<2.5,>=2.0; extra == 'dev'
Requires-Dist: ruff<1,>=0.9; extra == 'dev'
Provides-Extra: duckdb
Requires-Dist: duckdb==1.5.6; extra == 'duckdb'
Description-Content-Type: text/markdown

# DataJig

**DataJig is an agent-native data control plane that turns raw datasets into
verified, versioned, training-ready inputs.**

DataJig gives AI agents a safe, auditable path from raw data to model training.
It wraps data work in explicit plans, deterministic identities, semantic checks,
immutable revisions, and evidence of which records crossed a verified training
adapter boundary.

```text
raw source -> pin -> prepare/transform -> review -> seal -> export -> train
                `------ content-addressed evidence -------'  `-> receipt
```

DataJig is local-first and ships as a Python wheel containing its Rust core. It
supports single files and deterministic recursive datasets in CSV, flat
Parquet, or keyed JSONL, plus ImageFolder workflows.

## Install

```bash
python -m pip install datajig
datajig --version
datajig capabilities
datajig tutorial ./datajig-tutorial
```

Add the exactly pinned DuckDB provider for bounded joins, projections, filters,
and aggregations:

```bash
python -m pip install 'datajig[duckdb]'
```

## Where it fits

DataJig does not replace Git, DVC, Oxen, lakeFS, DuckDB, Polars, Spark,
PyTorch, or Hugging Face. It sits between those systems as an agent-facing
control plane:

- pin exact upstream bytes and preserve provenance;
- authorize bounded preparation and SQL transform plans;
- scope agent edits and reject stale review evidence;
- seal content-addressed, immutable dataset revisions;
- export deterministic subsets, splits, and training shards;
- verify records at the training adapter boundary and write an at-least-once
  receipt for that boundary.

Every task, candidate, review, revision, subset, transform, and training bundle
receives a deterministic identity. If data changes after review, a plan drifts,
or an agent touches bytes outside its declared task, DataJig fails closed.

## Agent-native interface

Commands return bounded JSON with actionable `next_actions`. Discovery exposes
a deterministic `agent_contract_id`, machine-readable command descriptors, and
artifact schemas. `repository-install` can install a version-matched Agent Skill,
pre-commit guard, SHA-pinned GitHub Actions workflow, and content-addressed lock
into another repository.

```bash
datajig repository-install --root .
datajig repository-check --root .
```

## Typical workflow

```bash
datajig inspect data/papers.parquet --id-field paper_id
datajig prepare-plan data/papers.parquet \
  --recipe recipes/papers.prepare.json \
  --output data/papers.jsonl \
  --plan artifacts/papers.prepare.plan.json
datajig prepare-apply artifacts/papers.prepare.plan.json \
  --accept-plan prep_...
datajig init data/papers.jsonl --id-field paper_id
```

The optional DuckDB provider follows the same plan-and-accept pattern. SQL is
authorized by the Rust core before execution, input aliases are staged privately,
resources are bounded, and canonical JSONL output is independently verified.

## Project links

- Repository: https://github.com/liukejun7/DataJig
- Releases: https://github.com/liukejun7/DataJig/releases
- Changelog: https://github.com/liukejun7/DataJig/blob/main/CHANGELOG.md
- Security policy: https://github.com/liukejun7/DataJig/blob/main/SECURITY.md
- License: Apache-2.0
