Metadata-Version: 2.4
Name: framepin
Version: 0.3.1
Summary: A lockfile for your video/sequence-ML datasets and experiments — reproduce any run without copying a single frame.
Author: boogy-ro
License: MIT
Project-URL: Homepage, https://boogy-ro.github.io/framepin/
Project-URL: Repository, https://github.com/boogy-ro/framepin
Project-URL: Documentation, https://github.com/boogy-ro/framepin#readme
Project-URL: Changelog, https://github.com/boogy-ro/framepin/releases
Project-URL: Issues, https://github.com/boogy-ro/framepin/issues
Keywords: mlops,dataset-versioning,experiment-tracking,reproducibility,video,machine-learning,data-lineage,data-version-control,dvc-alternative,ci-gate,dataset-drift,content-hashing,merkle,pytorch
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Version Control
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# framepin

[![tests](https://github.com/boogy-ro/framepin/actions/workflows/ci.yml/badge.svg)](https://github.com/boogy-ro/framepin/actions/workflows/ci.yml) [![PyPI](https://img.shields.io/pypi/v/framepin)](https://pypi.org/project/framepin/)

**A lockfile for your video/sequence-ML datasets and experiments — reproduce any run without copying a single frame.**

`framepin` pins the *exact* data a training run saw, links it to the run's
params/metrics/commit, and tells you — when a metric moves — whether it was the
**code** or the **data**. Zero dependencies, no server, no data copies. Just
JSON that commits cleanly to git.

```bash
pip install framepin   # zero dependencies — pure standard library
```

> Status: **v0.1 alpha.** Core (snapshot / diff / track / regress) works and is
> tested. APIs may shift before 1.0.

---

## The problem

If you train models on video or sequence data, this has happened to you:

- A run from three weeks ago had a better `val_loss`. Which dataset version was
  it? You re-sampled frames and re-labeled twice since. **Gone.**
- `val_loss` got worse after a change. Was it your code… or did someone swap out
  30% of the clips? **Nobody can say quickly.**
- `wandb`/`mlflow` track the *run* but treat the dataset as an opaque string.
  `dvc` versions the *data* but is heavy and lives apart from your metrics.
  Neither answers **"is this run reproducible from these exact bytes?"**

Video datasets are big, churn constantly, and get reorganized — so a directory
reshuffle looks like "half my data changed" in a naive diff.

## What framepin does

```python
import framepin

with framepin.track(name="baseline", params={"lr": 3e-4}) as run:
    run.use_dataset("data/clips")      # content-hashes + pins this exact version
    model = train(...)
    run.log_metric("val_loss", 0.21)   # pinned to that version forever
```

Then, three weeks later:

```bash
$ framepin regress <old_run> <new_run> -m val_loss
regress 02189c0fbd14 -> ab09cc84a35a
  val_loss: 0.21 -> 0.28  (Δ +0.07)

  ⚠ DATA CHANGED between these runs — the dataset version differs.
    A metric move here cannot be attributed to code alone.
```

That last line is the whole point. **framepin separates "the code regressed"
from "the data changed under you."**

## Before / after

**Before** — the dataset is a string you hope is still true:

```python
run.log({"dataset": "data/clips (v3?)", "val_loss": 0.21})
# 3 weeks later: what was v3? which frames? was it re-labeled? 🤷
```

**After** — the dataset is a content hash you can reproduce and diff:

```python
with framepin.track(name="baseline") as run:
    run.use_dataset("data/clips")   # -> version c29aa729c669 (Merkle root of every byte)
    run.log_metric("val_loss", 0.21)
# later: `framepin show <run>` -> exact version; `framepin diff v1 v2` -> what changed
```

## 60-second quickstart

```bash
cd your-project
framepin init                     # creates .framepin/  (commit it to git)

framepin snapshot data/clips      # -> snapshot c29aa729c669 (N files, no copies)
# ... change / relabel / resample your data ...
framepin snapshot data/clips      # -> a new version id

framepin diff c29aa729c669 <new>  # added / removed / modified / MOVED
framepin log                       # list every pinned version, newest first
framepin gc --apply                # prune versions no run references (dry-run without --apply)
```

Large datasets: add `--jobs 16` to snapshot/verify for parallel hashing.
Scripts, CI and AI agents: add `--json` to snapshot/verify/log for
machine-readable output.

### Datasets defined by path-list files (train.txt of absolute paths)

Many teams don't train on "a directory" — they train on **txt manifests**: one
absolute path per line (500k clips, several lists concatenated per experiment).
framepin pins that whole construct — the list files *and* the bytes they
point at:

```bash
framepin snapshot --from-list train_urban.txt train_highway.txt
# -> one version id covering: both txt files + every referenced clip
#    re-encode one clip, or edit one line  -> different version id
#    a listed path that no longer exists   -> "⚠ recorded as missing"
```

Repeated paths across lists are deduped. For 100k+ files, content hashes are
cached (`.framepin/hashcache.json`) so later snapshots only re-read files
whose size/mtime changed; `--fast` skips content reads entirely (size+mtime
fingerprint) for routine checks — take a periodic full snapshot as your anchor
of record.

```python
man = framepin.snapshot_from_lists(["train_urban.txt", "train_highway.txt"])
with framepin.track(name="exp-8") as run:
    run.use_dataset(man)               # pins the exact list+content version
```

Track experiments from Python:

```python
import framepin

with framepin.track(name="exp-7", params={"lr": 3e-4, "aug": "mixup"}) as run:
    run.use_dataset("data/clips")
    for epoch in range(epochs):
        ...
    run.log_metric("val_loss", best_val)
    run.log_metric("map50", 0.63)
```

**PyTorch**: nothing about your `Dataset`/`DataLoader` changes — see
[examples/pytorch_integration.py](examples/pytorch_integration.py) for the
3-line pattern in a real train loop.

Inspect lineage and compare:

```bash
framepin runs                     # every run, its metrics, and its data version
framepin show <run>               # full lineage: run -> dataset version -> files
framepin regress <a> <b> -m map50 # metric delta + "was it code or data?"
```

### CI gate: fail the build when the data drifted

Pin a baseline version once, then let CI refuse changes that silently alter
the dataset:

```bash
framepin verify data/clips --against c29aa729c669     # exit 0 = intact
# ✗ DATASET DRIFT vs pinned c29aa729c669:  +1 -0 ~2 →0   -> exit 3, build fails
framepin verify --from-list train_a.txt train_b.txt --against <version>
```

Better: don't hardcode the version in CI config. `framepin pin` writes it to a
one-line **pinfile** you commit next to your code, and `--against-file` reads
it back — accepting an intentional data change becomes a normal reviewed
commit, not a CI-config edit:

```bash
framepin pin data/clips                                # snapshot + write framepin.pin
git add framepin.pin .framepin && git commit -m "pin dataset baseline"
# in CI:
framepin verify data/clips --against-file framepin.pin
# data changed on purpose? re-pin and commit — the diff shows up in review:
framepin pin data/clips
```

`pin` also takes `--from-list`, `--split`, `--file <path>` (default
`framepin.pin`), and `--version <id>` to pin an already-stored version without
re-hashing. Ready-made GitHub Actions workflows — including one that opens a
baseline-update PR automatically when the data drifts — are in
[`examples/ci/`](examples/ci/).

Expected churn (say, labels get re-exported every week) doesn't have to break
the build — allow-list it:

```bash
framepin verify data/clips --against c29aa729c669 --allow 'labels/*'
# ✓ allowed drift vs pinned c29aa729c669:  +0 -0 ~3 →0  (all changed paths match --allow)
```

`--allow` is repeatable; globs match manifest-relative paths (`*` matches
across `/`). The gate still fails if *any* changed path falls outside the
allow-list — allowed changes are just summarized instead of listed. With
`--json` you get `"gate": "pass"|"fail"` plus an `"allowed"` count.

### Stratified datasets: per-split version ids

One dataset, several splits, one manifest — give each split its own version id
so drift reports tell you *which* split moved:

```bash
framepin snapshot data/clips --split train='train/*' --split val='val/*'
# snapshot a208968bd4cf  (500 files, 81236 bytes)
#   split train  2519d0cc5791  (400 files, 64988 bytes)
#   split val    985005ef8b59  (100 files, 16248 bytes)
```

Splits are pure metadata — the top-level version id is unchanged whether or
not you record them. On `verify` drift, the report adds
`splits changed: train` / `splits unchanged: val` (and `"splits_changed"`
under `--json`), so a re-labeled val set never sends you digging through the
train half of the diff.

### Already on W&B or MLflow? Export the lineage

framepin doesn't compete with your tracker — it feeds it. One call tags the
active W&B / MLflow run with the exact pinned dataset version id(s), so the
data version shows up right next to your metrics in their UI:

```python
with framepin.track(name="baseline") as run:
    run.use_dataset("data/clips")
    framepin.integrations.to_mlflow(run)   # tags: framepin.run_id, framepin.dataset.*
    # or: framepin.integrations.to_wandb(run)
```

Neither library is a dependency — the imports are lazy, framepin stays at
zero deps.

## Why not W&B / MLflow / DVC?

| | framepin | W&B / MLflow | DVC |
|---|---|---|---|
| Pins run ↔ **exact dataset version** | ✅ first-class | ⚠️ manual string/artifact | ⚠️ separate from metrics |
| "Code vs data" regression answer | ✅ `regress` | ❌ | ❌ |
| Detects **moved/renamed** files (not add+remove) | ✅ | — | partial |
| Copies your data | ❌ never (hashes only) | ⚠️ artifacts | ✅ cache/remote |
| Server / account required | ❌ | ⚠️ (or self-host) | ❌ |
| Runtime dependencies | **0** | many | several |
| Git-friendly plain-JSON store | ✅ | ❌ | ✅ (pointers) |

framepin is deliberately small. It is **not** a full experiment platform — it
does the one thing those tools under-serve: making the *dataset version* a
first-class, reproducible, diffable part of every run. Use it alongside W&B if
you like; framepin just owns the data-lineage question.

## How it works

- **Snapshot** walks a directory, streams a SHA-256 over each file, and records a
  deterministic manifest keyed by a **Merkle root** of `(path, content-hash)`
  pairs. Same bytes → same version id, regardless of walk order or timestamps.
  Your data is never moved or copied — the manifest is a few KB of JSON.
- **Diff** classifies changes into added / removed / modified / **moved**, so a
  reorg of identical clips reads as moves, not churn.
- **Track** records a run's params, metrics, git commit, and the dataset
  version(s) it consumed, into `.framepin/runs/<id>.json`.
- **Regress** compares two runs' metrics *and* their pinned dataset versions.

Everything lives under `.framepin/` as plain, sorted JSON. Commit it.

## Install from source

```bash
git clone https://github.com/boogy-ro/framepin
cd framepin
pip install -e .          # or just add the repo to PYTHONPATH — no deps
python -m unittest discover -s tests
```

Requires Python ≥ 3.9. No third-party packages.

See the whole workflow in ~2 seconds (builds a throwaway dataset, simulates a
re-label + re-encode, then shows the code-vs-data verdict):

```bash
python3 examples/quickstart_demo.py
```

## Roadmap

- remote manifest registry for teams
- richer CI gate (`verify --allow`, pinfiles and baseline auto-update PRs ship today)

Feedback and issues very welcome — the niche (video/sequence ML data lineage) is
exactly where this should earn its keep or die. Tell me where it falls short.

**Using this on a team?** Comment on [the team-features issue](https://github.com/boogy-ro/framepin/issues/1)
(CI gate / shared registry / audit reports) — it decides what gets built next.

## FAQ

**How do I version an ML dataset without copying it?**
`framepin snapshot data/clips` content-hashes every file into an immutable
version id (a Merkle root). Your data never moves; the snapshot is a few KB of
JSON you commit to git.

**How do I know which dataset version an old training run used?**
If the run was wrapped in `framepin.track(...)` with `run.use_dataset(...)`,
`framepin show <run>` prints its exact pinned dataset version and file list.

**My metric regressed — was it my code or my data?**
`framepin regress <old_run> <new_run> -m val_loss` compares the two runs'
metrics *and* their pinned dataset versions, and prints either
`⚠ DATA CHANGED` or `✓ same dataset version` so you know where to look.

**My dataset is a txt file of absolute paths (500k clips), not a directory.**
`framepin snapshot --from-list train_a.txt train_b.txt` pins the list files
and every file they reference, deduped across lists. Dead paths are recorded
as missing. Use `--fast` for size+mtime fingerprints on huge datasets.

**Does framepin replace W&B, MLflow, or DVC?**
No. It runs alongside them and owns one thing they under-serve: the
dataset-version ↔ run link and the code-vs-data question. No server, no
account, zero dependencies. It can also export its lineage into MLflow/W&B
(`framepin.integrations`).

**Is there a lightweight DVC alternative that doesn't copy or move my data?**
That's framepin's design point: it stores content hashes, never bytes. No
cache directory, no remote storage, no daemon — a snapshot is a few KB of
JSON in `.framepin/` that you commit to git. If you need data *transfer* and
remote storage, use DVC; if you need data *identity* and lineage, framepin.

**How do I make CI fail when my training data changes?**
`framepin pin data/` once (commits a one-line `framepin.pin`), then run
`framepin verify data/ --against-file framepin.pin` in CI — exit 3 on drift,
with an `--allow` glob list for expected churn. Copy-paste workflows are in
[`examples/ci/`](examples/ci/).

**How do I detect that files were renamed/reorganized rather than changed?**
`framepin diff v1 v2` pairs identical-content files across paths and reports
them as MOVED instead of added+removed.

## License

MIT — see [LICENSE](LICENSE).
