Metadata-Version: 2.4
Name: vd3
Version: 0.6.0
Summary: DVC-backed media database for computer vision and ML developers
Project-URL: Homepage, https://github.com/jmuncaster/vd3
Project-URL: Repository, https://github.com/jmuncaster/vd3
Project-URL: Issues, https://github.com/jmuncaster/vd3/issues
Author-email: Justin Muncaster <justin@muncasterconsulting.com>
License-Expression: MIT
Keywords: computer-vision,dataset,dvc,video
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: duckdb>=1.1.0
Requires-Dist: dvc-azure>=3.0
Requires-Dist: dvc-gdrive>=3.0
Requires-Dist: dvc-gs>=3.0
Requires-Dist: dvc-s3>=3.0
Requires-Dist: dvc>=3.50.0
Requires-Dist: ffmpeg-python>=0.2.0
Requires-Dist: orjson>=3.10.0
Requires-Dist: pillow>=10.0.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.12.0
Description-Content-Type: text/markdown

# VD3Storage

[![test](https://github.com/muncasterconsulting/vd3/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/muncasterconsulting/vd3/actions/workflows/test.yml)
[![PyPI version](https://img.shields.io/pypi/v/vd3.svg)](https://pypi.org/project/vd3/)

A [DVC](https://dvc.org)-backed media database for computer vision and ML developers. Tracks video and imageset assets, annotations, and worksets as MP4/JSON media with CSV-based metadata, so datasets stay versioned and reproducible across local disks and remote storage backends.

## Installation

```bash
uv sync
```

To use as a dependency:

```toml
# pyproject.toml
[project]
dependencies = ["vd3"]
```

## Quick Start

```bash
# Initialize a content database in the current directory
vd3 db init

# ...or in a specific directory
vd3 db init /path/to/mydb

# Add a video under a datasource
vd3 datasource add-video my-datasource clip.mp4 -p /path/to/mydb

# Add multiple videos with a glob (quote to prevent shell expansion)
vd3 datasource add-video my-datasource '*.mp4' -p /path/to/mydb

# List assets in a datasource
vd3 datasource assets my-datasource -p /path/to/mydb

# Show media availability
vd3 media status -p /path/to/mydb
```

Every command reads `vd3 <noun> <verb> [args]`. The nouns are `db`,
`datasource`, `workset`, `asset`, `layer`, and `media`. Containers
(`datasource`, `workset`) own ingest verbs and contained-asset listings; the
`asset` noun is reserved for operations on a known asset.

All commands that accept `--path` also honor the `VD3_DB` environment variable, so
you can point at a database once and drop the `-p` flag from individual commands.
On startup `vd3` automatically loads a `.env` file from the current directory or
the nearest ancestor, so a project-level `.env` containing:

```env
VD3_DB=/path/to/mydb
```

is enough — no `export` or shell sourcing required:

```bash
vd3 asset list
vd3 media status
```

Precedence: explicit `--path` > shell-exported `VD3_DB` > `.env` `VD3_DB` > current
directory. Shell-exported values win over `.env` so you can do one-off overrides
without editing the file.

`--path` on a specific command still overrides the env var.

## Core Concepts

- **Datasource** — groups assets by origin (e.g. `dashcam-2024`, `test-data`). Required when importing.
- **Asset** — a single video (MP4 + JSON metadata) or imageset (directory of images).
- **Workset** — a named subset of assets, optionally organized into packages (folders). Independent of storage layout.
- **Annotation layer** — detections or tracks attached to an asset, with a key (e.g. `gt`, `det/yolo-v8`) and a `human` or `machine` source. Layers are **git-backed** by default (small, always present after clone); large layers can be **DVC-backed** (fetched on demand like media) via `add_annotation_layer(..., dvc=True)` / `vd3 layer add-vd3 --dvc`, or converted later with `vd3 layer migrate-to-dvc`.
- **Per-image metadata** — arbitrary non-bbox metadata attached per image (labels, attributes, …), stored as a DVC-backable `image_metadata` layer via `add_image_metadata` / `read_image_metadata`. Keeps bulky per-image metadata out of the git-tracked `imageset.json` manifest; `vd3 layer migrate-imageset-extra` moves existing inline `extra` into one.

## Adding Assets

### Videos

```bash
# Single file
vd3 datasource add-video dashcam clip.mp4

# Glob (recursive)
vd3 datasource add-video dashcam 'rawdata/**/*.mp4'

# Force re-import of a duplicate (matched by SHA-256)
vd3 datasource add-video dashcam clip.mp4 --force

# Add and assign to a workset/package
vd3 datasource add-video dashcam clip.mp4 -w my-workset -k batch1
```

### Imagesets

```bash
# Directory of images
vd3 datasource add-imageset my-datasource /path/to/images

# Tar archive
vd3 datasource add-imageset my-datasource images.tar
```

### Annotation layers

VD3 JSON detections/tracks into an existing asset. The second positional is a
layer-name prefix prepended to every layer key in the file (e.g. importing a
file with `det/yolo` under `run-1` produces `run-1/det/yolo`):

```bash
vd3 layer add-vd3 clip run-1 results.json -p /path/to/mydb
```

### COCO

Import COCO annotations into an existing imageset:

```bash
vd3 layer add-coco my-imageset gt annotations.json \
    --source human --reviewed-all
```

Import a full COCO dataset (creates the imageset and imports annotations in
one step):

```bash
vd3 datasource add-imageset-from-coco my-datasource gt annotations.json \
    --image-root /path/to/images
```

## Worksets

```bash
# Create
vd3 workset create "My Experiment"

# Add assets by name or ID
vd3 workset add my-experiment clip-001 clip-002

# ...or by media-path glob (run from the database root; files must be on disk)
cd /path/to/mydb
vd3 workset add my-experiment 'db/media/videos/fc/*.mp4'

# Inspect
vd3 workset list
vd3 workset show my-experiment      # metadata + layers + packages
vd3 workset assets my-experiment    # assets in the workset

# Remove an asset / delete the workset
vd3 workset remove my-experiment clip-001
vd3 workset delete my-experiment
```

## Remote Storage

Media files are tracked by DVC. A content database has a single configured remote.

```bash
# Set the remote (replaces any existing one)
vd3 media remote set gs://my-bucket/vd3-data
vd3 media remote show

# Sync (push and pull both accept --workset/-w, --asset/-a, --datasource/-d, --all)
vd3 media push --all
vd3 media pull --workset my-experiment
vd3 media status
```

Supported backends:

| Backend | URL form | Notes |
|---|---|---|
| Google Cloud Storage | `gs://bucket/path` | `gcloud auth application-default login` |
| Amazon S3 | `s3://bucket/path` | Standard AWS credential chain |
| Azure Blob Storage | `azure://container/path` | |
| Google Drive | `gdrive://folder-id` | via `dvc-gdrive` |
| Local / NAS | `/mnt/nas/vd3-backup` | |

## Listing & Inspection

```bash
vd3 asset list                       # all assets (cross-container)
vd3 datasource list                  # all datasources
vd3 datasource assets dashcam        # assets in a datasource
vd3 datasource assets dashcam --paths      # one media path per line
vd3 datasource assets dashcam --filenames  # one filename per line
vd3 datasource layers dashcam        # annotation layers across the datasource
vd3 workset assets my-experiment     # assets in a workset
vd3 workset layers my-experiment     # annotation layers across the workset
vd3 asset layers clip                # annotation layers on an asset
vd3 asset show clip                  # asset details
vd3 db info                          # database overview
vd3 db query "SELECT ..."            # raw DuckDB SQL against the CSV tables
```

## Exporting

```bash
# Extract frames from a video or images from an imageset
vd3 asset export-frames clip -o ./out
```

## Library API

The CLI is a thin wrapper around `VD3Storage`, which is also usable directly.

```python
from vd3storage import VD3Storage, Asset, Workset  # Tag, WorksetAsset also exported

# Open an existing database (or use VD3Storage.init(path) to create one)
storage = VD3Storage("/path/to/mydb")

# Browse assets
for a in storage.list_assets(datasource="dashcam"):
    print(f"{a.name} ({a.asset_type}): {a.frame_count} frames @ {a.nominal_fps} fps")

# Look up by (datasource, name) or by ID
clip = storage.get_asset("dashcam", "clip-001")
clip = storage.get_asset_by_id("3f1a...")

# Import a video
asset = storage.import_video("clip.mp4", datasource="dashcam")

# Resolve where the media file lives on disk
storage.resolve_media_path(clip)

# Annotation layers
storage.list_annotation_layers(clip.asset_id)
storage.read_annotation_layer(clip.asset_id, "gt")

# Worksets
ws = storage.create_workset("My Experiment")
storage.add_asset_to_workset(ws.workset_id, clip.asset_id, package="batch1")
storage.list_workset_assets(ws.workset_id)

# Raw DuckDB SQL against the underlying CSV tables
rows = storage.execute_sql("SELECT name, frame_count FROM assets WHERE asset_type = 'video'")
```

Other useful methods: `import_imageset`, `import_coco`, `import_coco_dataset`, `import_result`, `export_coco`, `open_video`, `open_imageset`, `get_frame_image`, `add_tag`, `is_media_available`, `pull`, `push`. Inspect `help(VD3Storage)` for the full surface.

## Versioning & stability

The package follows [Semantic Versioning](https://semver.org). All notable
changes are recorded in [CHANGELOG.md](CHANGELOG.md), and every release is
tagged `vX.Y.Z` in git.

**While the package is in 0.x**, minor bumps (0.2 → 0.3) may contain breaking
changes; patch bumps (0.2.1 → 0.2.2) are backwards compatible. Every breaking
change is called out under a `### Breaking` heading in the CHANGELOG entry for
that release. Once the package reaches 1.0 it will follow strict SemVer
(MAJOR = breaking, MINOR = additive, PATCH = fix).

### Public API

A change is "breaking" only if it alters one of the following:

1. **Names re-exported from the top-level `vd3storage` package** (i.e. listed
   in `vd3storage.__all__`):
   - `VD3Storage` and its documented methods
   - `AlreadyInitializedError`
   - The model classes `Asset`, `Evaluation`, `Tag`, `Workset`, `WorksetAsset`
     (including their field names and types)
   - `__version__`
2. **The `vd3` CLI** — command names, option names, exit codes, and the
   documented input file formats (VD3 JSON, COCO).
3. **The on-disk layout of a content database** — directory structure under
   `db/`, CSV table schemas (tracked by `SCHEMA_VERSION` in `db/tables/`), the
   shape of `video.json` / `imageset.json` / annotation JSON files, and the
   structure of generated `pyproject.toml` / `.gitignore`.

Everything else is **internal** and may change without a major-version bump
even if it is reachable via an import path. That includes the `vd3storage.orm`,
`vd3storage.dvc`, `vd3storage.media`, `vd3storage.importers`,
`vd3storage.exporters`, and `vd3storage.cli` submodules; helper functions in
`vd3storage.storage` that start with `_`; and the on-disk format of files
written into `.dvc/` (those belong to DVC).

### Deprecations

When a public API needs to change incompatibly, the old form keeps working and
emits `DeprecationWarning` for at least one minor release before being removed.
Current deprecations are listed in the CHANGELOG under each release's
`### Deprecated` heading.

To surface them in your own code:

```bash
python -W "default::DeprecationWarning:vd3storage" your_script.py
```

## CLI Reference

```
vd3 --help                       Top-level help
vd3 <noun> --help                Help for a noun
vd3 <noun> <verb> --help         Help for a specific command
```

Every command reads `vd3 <noun> <verb> [args]`. Positionals carry identity
(target → composite parts → payload); flags carry modifiers
(`--paths`, `--filenames`, `--source human`, ...).

| Command | Description |
|---|---|
| `db init` | Initialize a content database (defaults to cwd) |
| `db info` | Show database overview |
| `db query` | Run raw DuckDB SQL against the CSV tables |
| `datasource list` | List datasources |
| `datasource show` | Show datasource stats |
| `datasource assets` | List assets in a datasource |
| `datasource layers` | List annotation layers across a datasource (per-layer coverage) |
| `datasource add-video` | Import video files into a datasource |
| `datasource add-imageset` | Import an imageset (directory or tar) into a datasource |
| `datasource add-imageset-from-coco` | Import a COCO dataset as imageset + layer |
| `workset create` | Create a workset |
| `workset list` | List worksets |
| `workset show` | Show workset metadata + packages |
| `workset assets` | List assets in a workset |
| `workset layers` | List annotation layers across a workset (per-layer coverage) |
| `workset add` | Add assets to a workset |
| `workset remove` | Remove an asset from a workset |
| `workset delete` | Delete a workset (assets are kept) |
| `asset list` | List every asset (cross-container) |
| `asset layers` | List annotation layers on an asset |
| `asset show` | Show asset details |
| `asset remove` | Delete an asset |
| `asset export-frames` | Extract frames from a video or imageset |
| `layer add-coco` | Import COCO annotations into an existing imageset |
| `layer add-vd3` | Import VD3 JSON detections/tracks under a layer-name prefix (`--dvc` stores them DVC-backed) |
| `layer migrate-to-dvc` | Convert existing git-backed annotation layers to DVC-backed (`--all`, `--min-size`, `--dry-run`) |
| `media status` | Show media availability |
| `media push` | Push media to remote storage |
| `media pull` | Pull media from remote storage |
| `media remote set` | Set the remote storage URL |
| `media remote show` | Show the configured remote |

## Development

### Tests

```bash
uv sync --all-groups        # install dev dependencies including pytest-cov
uv run pytest tests -q      # run the suite
```

### Coverage

The repo ships with line + branch coverage measurement via `pytest-cov`. The
quickest entry points:

```bash
./scripts/coverage.sh              # term-missing + HTML (htmlcov/) + XML (coverage.xml)
./scripts/coverage.sh --quick      # term-missing only, no HTML/XML write
./scripts/coverage.sh --check      # CI mode — fails if below the fail_under floor
```

The configured floor lives in `[tool.coverage.report]` of `pyproject.toml`
and is a **ratchet**: it tracks just below the current overall percentage so
that breaking the gate is always an actual regression. Bumping the floor up
is the responsibility of whoever raises coverage.

Branch coverage is enabled from day one. Files explicitly excluded from
measurement (constants, release metadata) live in `[tool.coverage.run].omit`.

### CI

The [`test`](https://github.com/muncasterconsulting/vd3/actions/workflows/test.yml)
GitHub Actions workflow runs the same `./scripts/coverage.sh --check` on every
push to `main` and every pull request, on `ubuntu-latest` / Python 3.12. The
generated `coverage.xml` is uploaded as a 14-day workflow artifact so
`diff-cover` (or a human) can inspect per-file deltas after the fact.

### Linting / Type-checking

The monorepo uses [`prek`](https://github.com/prek-tools/prek) (pre-commit
runner) configured at the repo root in `.pre-commit-config.yaml`. Run
`prek --all-files` from the repo root to apply ruff (lint + format) and
`ty` (type-check) against `vd3storage/`. The same hooks run as a pre-commit
step on staged files.

### Publishing

See `.claude/skills/publish/SKILL.md` (invoked as `/publish` from Claude
Code) for the release flow. It walks through version selection, CHANGELOG
maintenance, the quality gates (prek, pytest, coverage, build), and the
final upload + tag with confirmation prompts at every destructive step.

For a fully manual release, `scripts/publish.sh` is the underlying script;
`--yes` skips the confirmation prompt, `--test` targets TestPyPI.
