Metadata-Version: 2.4
Name: dset-cli
Version: 0.1.0
Summary: Version control for datasets — commit, diff, and roll back millions of files with semantic diffs.
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# dset — version control for datasets

Commit, diff, and roll back millions of files the way you already do with
code — and see what actually changed: samples, classes, label distributions.
Not bytes.

Pure Python, zero dependencies, single file.

## Install

```
pip install .
```

(from this folder — or just run `python dset/cli.py ...` directly)

## Quick start

```
cd my-dataset
dset init
dset add images/ labels.csv
dset commit "June survey, initial labels"

# ... collect more data, fix labels ...

dset add .
dset commit "July imagery + relabeling pass"

dset diff v1 v2
```

Output:

```
comparing v1 → v2  (301 → 406 files)

  files       +120   −15   ~1 modified
  images      +120   −15
  labels      ~1 modified
  size        1.5 MB → 2.1 MB  (+568.2 KB)

  class       acacia  19.7% → 38.8%  (+19.1%)
  class       shrub   43.7% → 33.3%  (−10.3%)
  class       bare    36.7% → 27.9%  (−8.8%)
```

Go back to any version, exactly:

```
dset checkout v1
```

## Commands

| command | what it does |
|---|---|
| `dset init` | start tracking the current directory |
| `dset add <paths>` | stage files or folders (incremental — unchanged files are skipped) |
| `dset status` | staged / modified / deleted / untracked files |
| `dset commit "msg"` | snapshot as a new version (v1, v2, ...) with dataset stats |
| `dset log` | list versions with file counts and sizes |
| `dset diff <a> <b>` | semantic diff: file counts by type, size, class distribution shift |
| `dset checkout <ref>` | restore the working tree to a version (`--force` to discard changes) |

Refs can be a tag (`v3`), `HEAD`, or a commit-id prefix.

## How it works

- **Content-addressed storage.** Every file is hashed (SHA-256) and stored
  once under `.dset/objects/`, like git. A new version costs only what
  changed; re-adding identical data costs nothing.
- **Commits are manifests** — JSON maps of path → hash. Checkout rebuilds
  the working tree from objects, restoring and deleting as needed.
- **Semantic stats are computed at commit time** (file counts by type, total
  size, class distribution parsed from CSV/TSV label files with a
  `label`/`class`/`category`/`target` column), so `dset diff` is instant even
  on huge datasets.
- **Safe by construction.** Objects are copied (never hard-linked), written
  atomically, and stored read-only, so editing a working file can never
  corrupt history. On btrfs/XFS the copy is a free copy-on-write clone.
- Uncommitted changes block `dset checkout` unless you pass `--force`.

## Current limits (MVP)

- Class stats read CSV/TSV label files only (COCO/YOLO parsers are next).
- No remote yet — `dset push` / `dset pull` to a self-hosted server is the
  next milestone.
- No `.dsetignore` yet; hidden files and dot-directories are always skipped.
