Metadata-Version: 2.4
Name: vd3
Version: 0.3.0
Summary: DVC-backed media database for computer vision and ML developers
Project-URL: Homepage, https://github.com/jmuncaster/vd3
Project-URL: Repository, https://github.com/jmuncaster/vd3
Project-URL: Issues, https://github.com/jmuncaster/vd3/issues
Author-email: Justin Muncaster <justin@muncasterconsulting.com>
License-Expression: MIT
Keywords: computer-vision,dataset,dvc,video
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Multimedia :: Video
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Requires-Dist: duckdb>=1.1.0
Requires-Dist: dvc-azure>=3.0
Requires-Dist: dvc-gdrive>=3.0
Requires-Dist: dvc-gs>=3.0
Requires-Dist: dvc-s3>=3.0
Requires-Dist: dvc>=3.50.0
Requires-Dist: ffmpeg-python>=0.2.0
Requires-Dist: orjson>=3.10.0
Requires-Dist: pillow>=10.0.0
Requires-Dist: pydantic>=2.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: rich>=13.0.0
Requires-Dist: typer>=0.12.0
Description-Content-Type: text/markdown

# VD3Storage

A [DVC](https://dvc.org)-backed media database for computer vision and ML developers. Tracks video and imageset assets, annotations, and worksets as MP4/JSON media with CSV-based metadata, so datasets stay versioned and reproducible across local disks and remote storage backends.

## Installation

```bash
uv sync
```

To use as a dependency:

```toml
# pyproject.toml
[project]
dependencies = ["vd3"]
```

## Quick Start

```bash
# Initialize a content database in the current directory
vd3 db init

# ...or in a specific directory
vd3 db init /path/to/mydb

# Add a video under a datasource
vd3 datasource add-video my-datasource clip.mp4 -p /path/to/mydb

# Add multiple videos with a glob (quote to prevent shell expansion)
vd3 datasource add-video my-datasource '*.mp4' -p /path/to/mydb

# List assets in a datasource
vd3 datasource assets my-datasource -p /path/to/mydb

# Show media availability
vd3 media status -p /path/to/mydb
```

Every command reads `vd3 <noun> <verb> [args]`. The nouns are `db`,
`datasource`, `workset`, `asset`, `layer`, and `media`. Containers
(`datasource`, `workset`) own ingest verbs and contained-asset listings; the
`asset` noun is reserved for operations on a known asset.

All commands that accept `--path` also honor the `VD3_DB` environment variable, so
you can point at a database once and drop the `-p` flag from individual commands.
On startup `vd3` automatically loads a `.env` file from the current directory or
the nearest ancestor, so a project-level `.env` containing:

```env
VD3_DB=/path/to/mydb
```

is enough — no `export` or shell sourcing required:

```bash
vd3 asset list
vd3 media status
```

Precedence: explicit `--path` > shell-exported `VD3_DB` > `.env` `VD3_DB` > current
directory. Shell-exported values win over `.env` so you can do one-off overrides
without editing the file.

`--path` on a specific command still overrides the env var.

## Core Concepts

- **Datasource** — groups assets by origin (e.g. `dashcam-2024`, `test-data`). Required when importing.
- **Asset** — a single video (MP4 + JSON metadata) or imageset (directory of images).
- **Workset** — a named subset of assets, optionally organized into packages (folders). Independent of storage layout.
- **Annotation layer** — detections or tracks attached to an asset, with a key (e.g. `gt`, `det/yolo-v8`) and a `human` or `machine` source.

## Adding Assets

### Videos

```bash
# Single file
vd3 datasource add-video dashcam clip.mp4

# Glob (recursive)
vd3 datasource add-video dashcam 'rawdata/**/*.mp4'

# Force re-import of a duplicate (matched by SHA-256)
vd3 datasource add-video dashcam clip.mp4 --force

# Add and assign to a workset/package
vd3 datasource add-video dashcam clip.mp4 -w my-workset -k batch1
```

### Imagesets

```bash
# Directory of images
vd3 datasource add-imageset my-datasource /path/to/images

# Tar archive
vd3 datasource add-imageset my-datasource images.tar
```

### Annotation layers

VD3 JSON detections/tracks into an existing asset. The second positional is a
layer-name prefix prepended to every layer key in the file (e.g. importing a
file with `det/yolo` under `run-1` produces `run-1/det/yolo`):

```bash
vd3 layer add-vd3 clip run-1 results.json -p /path/to/mydb
```

### COCO

Import COCO annotations into an existing imageset:

```bash
vd3 layer add-coco my-imageset gt annotations.json \
    --source human --reviewed-all
```

Import a full COCO dataset (creates the imageset and imports annotations in
one step):

```bash
vd3 datasource add-imageset-from-coco my-datasource gt annotations.json \
    --image-root /path/to/images
```

## Worksets

```bash
# Create
vd3 workset create "My Experiment"

# Add assets by name or ID
vd3 workset add my-experiment clip-001 clip-002

# ...or by media-path glob (run from the database root; files must be on disk)
cd /path/to/mydb
vd3 workset add my-experiment 'db/media/videos/fc/*.mp4'

# Inspect
vd3 workset list
vd3 workset show my-experiment      # metadata + layers + packages
vd3 workset assets my-experiment    # assets in the workset

# Remove an asset / delete the workset
vd3 workset remove my-experiment clip-001
vd3 workset delete my-experiment
```

## Remote Storage

Media files are tracked by DVC. A content database has a single configured remote.

```bash
# Set the remote (replaces any existing one)
vd3 media remote set gs://my-bucket/vd3-data
vd3 media remote show

# Sync (push and pull both accept --workset/-w, --asset/-a, --datasource/-d, --all)
vd3 media push --all
vd3 media pull --workset my-experiment
vd3 media status
```

Supported backends:

| Backend | URL form | Notes |
|---|---|---|
| Google Cloud Storage | `gs://bucket/path` | `gcloud auth application-default login` |
| Amazon S3 | `s3://bucket/path` | Standard AWS credential chain |
| Azure Blob Storage | `azure://container/path` | |
| Google Drive | `gdrive://folder-id` | via `dvc-gdrive` |
| Local / NAS | `/mnt/nas/vd3-backup` | |

## Listing & Inspection

```bash
vd3 asset list                       # all assets (cross-container)
vd3 datasource list                  # all datasources
vd3 datasource assets dashcam        # assets in a datasource
vd3 datasource assets dashcam --paths      # one media path per line
vd3 datasource assets dashcam --filenames  # one filename per line
vd3 datasource layers dashcam        # annotation layers across the datasource
vd3 workset assets my-experiment     # assets in a workset
vd3 workset layers my-experiment     # annotation layers across the workset
vd3 asset layers clip                # annotation layers on an asset
vd3 asset show clip                  # asset details
vd3 db info                          # database overview
vd3 db query "SELECT ..."            # raw DuckDB SQL against the CSV tables
```

## Exporting

```bash
# Extract frames from a video or images from an imageset
vd3 asset export-frames clip -o ./out
```

## Library API

The CLI is a thin wrapper around `VD3Storage`, which is also usable directly.

```python
from vd3storage import VD3Storage, Asset, Workset  # Tag, WorksetAsset also exported

# Open an existing database (or use VD3Storage.init(path) to create one)
storage = VD3Storage("/path/to/mydb")

# Browse assets
for a in storage.list_assets(datasource="dashcam"):
    print(f"{a.name} ({a.asset_type}): {a.frame_count} frames @ {a.nominal_fps} fps")

# Look up by (datasource, name) or by ID
clip = storage.get_asset("dashcam", "clip-001")
clip = storage.get_asset_by_id("3f1a...")

# Import a video
asset = storage.import_video("clip.mp4", datasource="dashcam")

# Resolve where the media file lives on disk
storage.resolve_media_path(clip)

# Annotation layers
storage.list_annotation_layers(clip.asset_id)
storage.read_annotation_layer(clip.asset_id, "gt")

# Worksets
ws = storage.create_workset("My Experiment")
storage.add_asset_to_workset(ws.workset_id, clip.asset_id, package="batch1")
storage.list_workset_assets(ws.workset_id)

# Raw DuckDB SQL against the underlying CSV tables
rows = storage.execute_sql("SELECT name, frame_count FROM assets WHERE asset_type = 'video'")
```

Other useful methods: `import_imageset`, `import_coco`, `import_coco_dataset`, `import_result`, `export_coco`, `open_video`, `open_imageset`, `get_frame_image`, `add_tag`, `is_media_available`, `pull`, `push`. Inspect `help(VD3Storage)` for the full surface.

## Versioning & stability

The package follows [Semantic Versioning](https://semver.org). All notable
changes are recorded in [CHANGELOG.md](CHANGELOG.md), and every release is
tagged `vX.Y.Z` in git.

**While the package is in 0.x**, minor bumps (0.2 → 0.3) may contain breaking
changes; patch bumps (0.2.1 → 0.2.2) are backwards compatible. Every breaking
change is called out under a `### Breaking` heading in the CHANGELOG entry for
that release. Once the package reaches 1.0 it will follow strict SemVer
(MAJOR = breaking, MINOR = additive, PATCH = fix).

### Public API

A change is "breaking" only if it alters one of the following:

1. **Names re-exported from the top-level `vd3storage` package** (i.e. listed
   in `vd3storage.__all__`):
   - `VD3Storage` and its documented methods
   - `AlreadyInitializedError`
   - The model classes `Asset`, `Tag`, `Workset`, `WorksetAsset` (including
     their field names and types)
   - `__version__`
2. **The `vd3` CLI** — command names, option names, exit codes, and the
   documented input file formats (VD3 JSON, COCO).
3. **The on-disk layout of a content database** — directory structure under
   `db/`, CSV table schemas (tracked by `SCHEMA_VERSION` in `db/tables/`), the
   shape of `video.json` / `imageset.json` / annotation JSON files, and the
   structure of generated `pyproject.toml` / `.gitignore`.

Everything else is **internal** and may change without a major-version bump
even if it is reachable via an import path. That includes the `vd3storage.orm`,
`vd3storage.dvc`, `vd3storage.media`, `vd3storage.importers`,
`vd3storage.exporters`, and `vd3storage.cli` submodules; helper functions in
`vd3storage.storage` that start with `_`; and the on-disk format of files
written into `.dvc/` (those belong to DVC).

### Deprecations

When a public API needs to change incompatibly, the old form keeps working and
emits `DeprecationWarning` for at least one minor release before being removed.
Current deprecations are listed in the CHANGELOG under each release's
`### Deprecated` heading.

To surface them in your own code:

```bash
python -W "default::DeprecationWarning:vd3storage" your_script.py
```

## CLI Reference

```
vd3 --help                       Top-level help
vd3 <noun> --help                Help for a noun
vd3 <noun> <verb> --help         Help for a specific command
```

Every command reads `vd3 <noun> <verb> [args]`. Positionals carry identity
(target → composite parts → payload); flags carry modifiers
(`--paths`, `--filenames`, `--source human`, ...).

| Command | Description |
|---|---|
| `db init` | Initialize a content database (defaults to cwd) |
| `db info` | Show database overview |
| `db query` | Run raw DuckDB SQL against the CSV tables |
| `datasource list` | List datasources |
| `datasource show` | Show datasource stats |
| `datasource assets` | List assets in a datasource |
| `datasource layers` | List annotation layers across a datasource (per-layer coverage) |
| `datasource add-video` | Import video files into a datasource |
| `datasource add-imageset` | Import an imageset (directory or tar) into a datasource |
| `datasource add-imageset-from-coco` | Import a COCO dataset as imageset + layer |
| `workset create` | Create a workset |
| `workset list` | List worksets |
| `workset show` | Show workset metadata + packages |
| `workset assets` | List assets in a workset |
| `workset layers` | List annotation layers across a workset (per-layer coverage) |
| `workset add` | Add assets to a workset |
| `workset remove` | Remove an asset from a workset |
| `workset delete` | Delete a workset (assets are kept) |
| `asset list` | List every asset (cross-container) |
| `asset layers` | List annotation layers on an asset |
| `asset show` | Show asset details |
| `asset remove` | Delete an asset |
| `asset export-frames` | Extract frames from a video or imageset |
| `layer add-coco` | Import COCO annotations into an existing imageset |
| `layer add-vd3` | Import VD3 JSON detections/tracks under a layer-name prefix |
| `media status` | Show media availability |
| `media push` | Push media to remote storage |
| `media pull` | Pull media from remote storage |
| `media remote set` | Set the remote storage URL |
| `media remote show` | Show the configured remote |
