Metadata-Version: 2.4
Name: datajig
Version: 0.8.3
Summary: Agent-native data control plane for verified, versioned, training-ready datasets
Project-URL: Repository, https://github.com/liukejun7/DataJig
Project-URL: Issues, https://github.com/liukejun7/DataJig/issues
Project-URL: Documentation, https://github.com/liukejun7/DataJig#readme
Author-email: Kejun Liu <liukj7@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data-quality,datasets,version-control
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Version Control
Requires-Python: >=3.11
Requires-Dist: blake3<2,>=1.0
Requires-Dist: imagehash<5,>=4.3
Requires-Dist: jinja2<4,>=3.1
Requires-Dist: numpy<3,>=2.0
Requires-Dist: pillow<13,>=11
Requires-Dist: pyyaml<7,>=6.0.2
Requires-Dist: scipy<2,>=1.14
Provides-Extra: dev
Requires-Dist: build<2,>=1.2; extra == 'dev'
Requires-Dist: hatchling<2,>=1.27; extra == 'dev'
Requires-Dist: mypy<2,>=1.14; extra == 'dev'
Requires-Dist: numpy<2.5,>=2.0; extra == 'dev'
Requires-Dist: ruff<1,>=0.9; extra == 'dev'
Requires-Dist: types-pyyaml<7,>=6.0.12; extra == 'dev'
Provides-Extra: duckdb
Requires-Dist: duckdb==1.5.6; extra == 'duckdb'
Description-Content-Type: text/markdown

# DataJig

**DataJig is an agent-native data control plane that turns raw datasets into
verified, versioned, training-ready inputs.**

DataJig gives AI agents a safe, auditable path from raw data to model training.
It wraps data work in explicit plans, deterministic identities, semantic checks,
immutable revisions, and evidence of which records crossed a verified training
adapter boundary.

Version 0.8.3 preserves bounded DuckDB conversion diagnostics, publishes the
complete pipeline configuration schema and example, and continuously exercises
a three-day raw-log-to-training journey in parallel CI.

```text
raw source -> pin -> prepare/transform -> review -> seal -> export -> train
                `------ content-addressed evidence -------'  `-> receipt
```

DataJig is local-first and ships as a Python wheel containing its Rust core. It
supports single files and deterministic recursive datasets in CSV, flat
Parquet, or keyed JSONL, plus ImageFolder workflows.

## Install

```bash
python -m pip install datajig
datajig --version
datajig capabilities
datajig tutorial ./datajig-tutorial
```

Add the exactly pinned DuckDB provider for bounded joins, projections, filters,
and aggregations:

```bash
python -m pip install 'datajig[duckdb]'
```

## Where it fits

DataJig does not replace Git, DVC, Oxen, lakeFS, DuckDB, Polars, Spark,
PyTorch, or Hugging Face. It sits between those systems as an agent-facing
control plane:

- pin exact upstream bytes and preserve provenance;
- authorize bounded preparation and SQL transform plans;
- scope agent edits and reject stale review evidence;
- seal content-addressed, immutable dataset revisions;
- export deterministic subsets, splits, and training shards;
- verify records at the training adapter boundary and write an at-least-once
  receipt for that boundary.

Every task, candidate, review, revision, subset, transform, and training bundle
receives a deterministic identity. If data changes after review, a plan drifts,
or an agent touches bytes outside its declared task, DataJig fails closed.

## Agent-native interface

Commands return bounded JSON with actionable `next_actions`. Discovery exposes
a deterministic `agent_contract_id`, machine-readable command descriptors, and
artifact schemas. `repository-install` can install a version-matched Agent Skill,
pre-commit guard, SHA-pinned GitHub Actions workflow, and content-addressed lock
into another repository.

```bash
datajig repository-install --root .
datajig repository-check --root .
```

## Typical workflow

For a complete local training-data delivery, install the DuckDB extra and use a
single reviewed pipeline:

```bash
datajig artifact-schema pipeline-config
datajig pipeline plan --config pipeline.yaml --plan pipeline-plan.json
datajig pipeline apply pipeline-plan.json --accept-plan pipe_...
datajig pipeline info deliveries/my-training-data/pipeline-receipt.json --verify
datajig lineage pipe_... --state workspace/.datajig --format text
```

The plan binds source bytes, SQL, target paths, export splits, and external run
IDs. Apply uses OS locks, a durable recovery journal, detached revision sealing,
HEAD compare-and-swap, and a delivery commit marker. Interrupted work requires
explicit `--resume`; an identical update still delivers a bundle and consumer
plans without inventing a new revision.

`lineage` resolves committed pipeline receipts and related pipeline artifacts.
For an atomic manual workflow, inspect its transform receipt and workspace log
separately; a bare manual revision is not a pipeline lineage root.

An abbreviated configuration looks like this:

```yaml
schema_version: 1
pipeline: {name: papers-train, provider: duckdb}
target: {dataset: data/papers.jsonl, state: workspace/.datajig, mode: create}
delivery: {output: deliveries/papers-train}
inputs:
  - {alias: papers, path: raw/papers.parquet, format: parquet}
transform:
  sql: SELECT paper_id AS id, title FROM papers ORDER BY id
  id_field: id
export: {split: [train=9, test=1], max_shard_records: 10000}
consumption_plan:
  - {consumer: huggingface, split: train, run_id: experiment-42}
```

Atomic commands remain available when a pipeline is not the right abstraction:

```bash
datajig inspect data/papers.parquet --id-field paper_id
datajig prepare-plan data/papers.parquet \
  --recipe recipes/papers.prepare.json \
  --output data/papers.jsonl \
  --plan artifacts/papers.prepare.plan.json
datajig prepare-apply artifacts/papers.prepare.plan.json \
  --accept-plan prep_...
datajig init data/papers.jsonl --id-field paper_id
```

The optional DuckDB provider follows the same plan-and-accept pattern. SQL is
authorized by the Rust core before execution, input aliases are staged privately,
resources are bounded, and canonical JSONL output is independently verified.

## Project links

- Repository: https://github.com/liukejun7/DataJig
- Releases: https://github.com/liukejun7/DataJig/releases
- Changelog: https://github.com/liukejun7/DataJig/blob/main/CHANGELOG.md
- Security policy: https://github.com/liukejun7/DataJig/blob/main/SECURITY.md
- License: Apache-2.0
