Metadata-Version: 2.4
Name: datasetlint
Version: 0.0.1
Summary: Lightweight, local-first linting for robotics and physical AI datasets.
Author: Gagandeep Reehal
Maintainer: Gagandeep Reehal
License-Expression: MIT
Project-URL: Homepage, https://github.com/gagandeepreehal/datasetlint
Project-URL: Source, https://github.com/gagandeepreehal/datasetlint
Project-URL: Issues, https://github.com/gagandeepreehal/datasetlint/issues
Project-URL: Documentation, https://gagandeepreehal.github.io/datasetlint/
Project-URL: Changelog, https://github.com/gagandeepreehal/datasetlint/blob/main/CHANGELOG.md
Keywords: robotics,datasets,validation,physical-ai,lint
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Requires-Dist: pydantic>=2
Requires-Dist: pyyaml>=6
Requires-Dist: typer>=0.9
Requires-Dist: rich>=13
Provides-Extra: adapters
Requires-Dist: pyyaml>=6; extra == "adapters"
Provides-Extra: hf
Requires-Dist: datasets; extra == "hf"
Provides-Extra: mcap
Requires-Dist: mcap; extra == "mcap"
Provides-Extra: ros
Requires-Dist: rosbags; extra == "ros"
Provides-Extra: nuscenes
Requires-Dist: nuscenes-devkit; extra == "nuscenes"
Provides-Extra: waymo
Requires-Dist: waymo-open-dataset-tf-2-12-0; extra == "waymo"
Provides-Extra: all-adapters
Requires-Dist: pyyaml>=6; extra == "all-adapters"
Requires-Dist: datasets; extra == "all-adapters"
Requires-Dist: mcap; extra == "all-adapters"
Requires-Dist: rosbags; extra == "all-adapters"
Requires-Dist: nuscenes-devkit; extra == "all-adapters"
Requires-Dist: waymo-open-dataset-tf-2-12-0; extra == "all-adapters"
Provides-Extra: dev
Requires-Dist: numpy<2.5; extra == "dev"
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: mypy>=1.8; extra == "dev"
Requires-Dist: build>=1; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"
Provides-Extra: docs
Requires-Dist: mkdocs>=1.6; extra == "docs"
Dynamic: license-file

# DatasetLint

[![CI](https://github.com/gagandeepreehal/datasetlint/actions/workflows/ci.yml/badge.svg)](https://github.com/gagandeepreehal/datasetlint/actions/workflows/ci.yml)
[![Docs](https://github.com/gagandeepreehal/datasetlint/actions/workflows/docs.yml/badge.svg)](https://github.com/gagandeepreehal/datasetlint/actions/workflows/docs.yml)
![Python](https://img.shields.io/badge/python-3.10%2B-blue)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

DatasetLint is a lightweight, local-first dataset validation, dataset QA, and linting toolkit for robotics and physical AI datasets. It checks dataset structure, metadata, timestamps, labels, calibration, trajectories, simple distribution stats, dataset diffs, and adapter compatibility before bad data reaches training or evaluation.

It runs locally on folder-based datasets with Python, CSV, and JSON. It does not require robots, simulators, GPUs, ROS, cloud services, or model runtimes.

Documentation: [DatasetLint docs](https://gagandeepreehal.github.io/datasetlint/)

Docs map:

| Need | Start here |
| --- | --- |
| First install and example run | [Getting Started](https://gagandeepreehal.github.io/datasetlint/getting-started/) |
| Exact CLI commands and exit codes | [CLI Reference](https://gagandeepreehal.github.io/datasetlint/cli/) |
| JSON, Markdown, and HTML reports | [Reports](https://gagandeepreehal.github.io/datasetlint/reports/) |
| External formats and adapter limits | [Adapters](https://gagandeepreehal.github.io/datasetlint/adapters/) |
| Contributor workflow | [Development](https://gagandeepreehal.github.io/datasetlint/development/) |

## Current Limitations

- Deep validation works best on the native DatasetLint folder format.
- Adapters provide manifest inspection, manifest export, and adapter validation for external formats.
- nuScenes, MCAP, ROS bag, Waymo, and Hugging Face have bounded `--deep` adapter modes: nuScenes inspects referenced camera/lidar/radar payload headers, MCAP and ROS bag parse channel/topic metadata, Waymo parses frame/label/calibration metadata, and Hugging Face samples rows. If a requested deep parser cannot parse the input, validation fails instead of reporting a valid deep pass.
- Hugging Face `--deep` uses bounded row sampling rather than scanning entire remote datasets by default.
- DatasetLint is not a dataset management platform, model evaluation framework, simulator, replay tool, or data host.
- Large-dataset performance has not been benchmarked yet.
- Adapter support is strongest at metadata/manifest validation; some payload formats still need format-specific decoders for full semantic checks.

## Try In 60 Seconds

After installing from source, run one passing dataset, one failing dataset, and
one machine-readable report:

```bash
datasetlint --version
datasetlint examples/minimal_dataset
datasetlint examples/bad_dataset
datasetlint examples/bad_dataset --format json
datasetlint examples/bad_dataset --format html > report.html
datasetlint diff examples/minimal_dataset examples/bad_dataset --fail-on-regression
```

`examples/minimal_dataset` should pass. `examples/bad_dataset` and the diff
command should exit non-zero because they intentionally contain robotics data
quality problems.

Common next steps:

```bash
datasetlint DATASET_PATH --checks calibration,labels
datasetlint DATASET_PATH --config DATASET_PATH/datasetlint.yaml --fail-on warning
datasetlint validate DATASET_PATH --adapter auto --format json
datasetlint validate logs/run.mcap --adapter mcap --deep --format json
```

Use the docs when you need the exact workflow:

- [Getting Started](https://gagandeepreehal.github.io/datasetlint/getting-started/) for the first local run
- [Configuration](https://gagandeepreehal.github.io/datasetlint/configuration/) for per-rule enable/disable and severity overrides
- [Adapters](https://gagandeepreehal.github.io/datasetlint/adapters/) for MCAP, ROS bag, Argoverse 2, LeRobot, COCO, KITTI, nuScenes, Waymo, Hugging Face, and plugins
- [Troubleshooting](https://gagandeepreehal.github.io/datasetlint/troubleshooting/) for reading failures and fixing noisy checks

## Why This Exists

Robotics datasets often fail in quiet ways:

- dataset schema drift between collection, labeling, and training
- broken or duplicated timestamps
- missing metadata, calibration, sensor files, or referenced frames
- inconsistent labels and track IDs
- silent frame drops or distribution shifts
- adapter-specific ingestion problems discovered too late

DatasetLint catches those issues at the dataset folder boundary so teams can fail fast in local development and CI.

## What DatasetLint Catches

Concrete native-folder checks include:

- missing metadata
- broken file references
- duplicate timestamps
- timestamp gaps
- sensor sync gaps
- missing calibration
- invalid camera intrinsics
- non-normalized quaternion
- invalid label geometry
- duplicate track ID at same timestamp
- unrealistic trajectory speed
- dataset diff regression

## Who It Is For

- robotics ML engineers validating training and evaluation data
- physical AI teams maintaining local dataset collections
- dataset maintainers reviewing schema and metadata quality
- researchers sharing small reproducible datasets
- CI users blocking bad dataset changes before merge

## Current Capabilities

DatasetLint v0.1 supports deep rule validation for the native folder dataset format and normalized manifest workflows for common external dataset formats. Current capabilities include:

- required `metadata.json` and `calibration.json`
- empty CSV files, broken referenced paths, and duplicate filenames
- metadata schema, declared sensors, duration consistency, and version presence
- monotonic, duplicate, and large-gap timestamps
- camera, IMU, GPS, and generic sensor columns
- camera dimensions, expected sensor rates, and likely missing frames
- sensor time overlap, start offsets, pairwise sync gaps, burst gaps, and jitter
- calibration intrinsics, extrinsics, and quaternion normalization
- label columns, confidence, geometry, timestamp range, class switches, duplicate tracks, short tracks, missing labels, box jumps, and size changes
- trajectory columns, finite values, speed, acceleration, yaw range, and stationary motion
- dataset statistics and folder-to-folder diffs
- normalized adapter manifests for generic folders, COCO, KITTI, Argoverse 2, LeRobot, nuScenes, Waymo, ROS bag, MCAP, and Hugging Face datasets
- adapter validation, inspection, discovery, and manifest export commands
- shared manifest-rule summaries for decoded adapter records, including frame references, timestamp consistency, sensor links, calibration shape, annotation links, and split references where the adapter exposes those records
- bounded deep adapter diagnostics for nuScenes payload headers, MCAP/ROS bag timestamp metadata, Waymo frame metadata, and sampled Hugging Face rows
- single-file adapter roots such as `.bag`, `.mcap`, and `.tfrecord` resolve relative frame/file records from the containing directory

## Installation

DatasetLint requires Python 3.10 or newer. Install from source:

```bash
git clone https://github.com/gagandeepreehal/datasetlint.git
cd datasetlint
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e ".[dev,docs]"
```

Use `python3.10`, `python3.11`, or `python3.12` if that is the executable name on your machine. The macOS system `python3` may be Python 3.9, which is too old for this project.

This repository has publishing metadata and a publish workflow, but this checkout has no release tags. Until the first PyPI release is published, use the source install above instead of `pip install datasetlint`.

Adapter extras are opt-in so the base install stays lightweight:

```bash
python -m pip install -e ".[adapters]"
python -m pip install -e ".[hf]"
python -m pip install -e ".[mcap]"
python -m pip install -e ".[ros]"
python -m pip install -e ".[nuscenes]"
python -m pip install -e ".[waymo]"
python -m pip install -e ".[all-adapters]"
```

## Quickstart

Validate the passing example dataset:

```bash
datasetlint lint examples/minimal_dataset
datasetlint examples/minimal_dataset
```

Expected result:

```text
DatasetLint report for .../examples/minimal_dataset: passed with 0 issue(s) (error=0, warning=0, info=0).
```

Run a focused check group:

```bash
datasetlint examples/minimal_dataset --checks labels
datasetlint examples/minimal_dataset --checks sync
```

Render machine-readable output:

```bash
datasetlint report examples/minimal_dataset --out report.json
datasetlint examples/minimal_dataset --format json
datasetlint examples/minimal_dataset --format markdown
datasetlint examples/minimal_dataset --format html > report.html
```

Inspect the intentionally failing dataset:

```bash
datasetlint examples/bad_dataset
```

`examples/bad_dataset` exits non-zero because it contains missing files, invalid calibration, timestamp issues, label problems, and trajectory anomalies.

## CLI Usage

DatasetLint exposes one console script:

```bash
datasetlint --help
datasetlint --version
```

Validate a dataset:

```bash
datasetlint lint DATASET_PATH
datasetlint DATASET_PATH
datasetlint DATASET_PATH --checks labels,sync
datasetlint DATASET_PATH --config DATASET_PATH/datasetlint.yaml
datasetlint DATASET_PATH --adapter folder
datasetlint DATASET_PATH --adapter auto
datasetlint DATASET_PATH --format console
datasetlint DATASET_PATH --format json
datasetlint DATASET_PATH --format markdown
datasetlint DATASET_PATH --format html
datasetlint DATASET_PATH --fail-on warning
```

Write validation reports:

```bash
datasetlint report DATASET_PATH --out report.json
datasetlint report DATASET_PATH --out report.md
datasetlint report DATASET_PATH --out report.html
```

Compute statistics:

```bash
datasetlint stats DATASET_PATH
datasetlint stats DATASET_PATH --format json
datasetlint stats DATASET_PATH --format markdown
```

Compare two datasets:

```bash
datasetlint diff OLD_DATASET NEW_DATASET
datasetlint diff OLD_DATASET NEW_DATASET --format json
datasetlint diff OLD_DATASET NEW_DATASET --fail-on-regression
```

Inspect adapter detection:

```bash
datasetlint adapters DATASET_PATH
datasetlint adapters list
datasetlint adapters detect DATASET_PATH
datasetlint adapters DATASET_PATH --format json
```

Inspect, validate, or export normalized manifests for common formats:

```bash
datasetlint inspect DATASET_PATH --adapter coco
datasetlint inspect DATASET_PATH --auto-detect
datasetlint validate DATASET_PATH --adapter kitti
datasetlint validate DATASET_PATH --adapter nuscenes --deep
datasetlint validate DATASET_PATH --adapter mcap --deep
datasetlint inspect DATASET_PATH --adapter waymo --deep --max-rows 1000
datasetlint validate hf://namespace/dataset --adapter huggingface --split train --deep --max-rows 1000
datasetlint export-manifest DATASET_PATH --adapter nuscenes --output manifest.json
```

Exit codes:

- `0`: command completed and did not meet the configured failure threshold
- `1`: validation failed the `--fail-on` threshold, adapter validation failed, or diff regressions were found with `--fail-on-regression`
- `2`: invalid usage, unknown check group, bad adapter, or invalid config

## Python API Usage

```python
from datasetlint import compare_datasets, compute_dataset_stats, lint_dataset
from datasetlint.adapters import detect_adapters, get_adapter, load_dataset, validate_dataset

report = lint_dataset("examples/minimal_dataset")
print(report.summary())
print(report.count_by_severity())
print(report.to_markdown())

label_report = lint_dataset("examples/minimal_dataset", checks="labels")
sync_report = lint_dataset("examples/minimal_dataset", checks=["sync"])

stats = compute_dataset_stats("examples/minimal_dataset")
print(stats.frame_counts)

diff = compare_datasets("examples/minimal_dataset", "examples/bad_dataset")
print(diff.summary)

detections = detect_adapters("examples/minimal_dataset")
adapter = get_adapter("examples/minimal_dataset", "auto")

manifest = load_dataset("tests/fixtures/coco_dataset", adapter="coco")
validation = validate_dataset("tests/fixtures/kitti_object", adapter="kitti")
```

Public imports from `datasetlint` are `lint_dataset`, `compare_datasets`, `compute_dataset_stats`, `Issue`, `LintConfig`, `LintReport`, `DatasetStats`, and `DatasetDiffReport`.

## Examples

| Path | Expected result | Demonstrates |
| --- | --- | --- |
| `examples/minimal_dataset` | pass | valid folder dataset with camera, IMU, GPS, labels, calibration, and trajectory |
| `examples/bad_dataset` | fail | broad failure surface used for validation and diff examples |
| `examples/invalid_missing_metadata` | fail | missing `metadata.json` |
| `examples/invalid_timestamp_drift` | fail with `--fail-on warning` | timestamp gaps and frequency drift |
| `examples/invalid_label_consistency` | fail | track class switch and duplicate track timestamp |

See [examples/README.md](examples/README.md) for commands and expected outcomes.

Generated sample reports are committed under `examples/reports/`:

- `examples/reports/minimal_report.json`
- `examples/reports/bad_report.json`
- `examples/reports/bad_report.md`

## Configuration

DatasetLint uses defaults when no config is provided. A dataset can include `datasetlint.yaml`, or the CLI can receive `--config path/to/datasetlint.yaml`.

Supported keys:

```yaml
timestamp_gap_threshold_sec: 0.5
frequency_tolerance_fraction: 0.30
missing_frame_gap_multiplier: 1.5
max_speed_mps: 70
max_accel_mps2: 12
stationary_distance_threshold_m: 0.05
duration_tolerance_sec: 1.0
label_max_position_jump_px: 200
label_max_size_change_ratio: 3.0
label_min_track_length: 3
label_class_switch_threshold: 0
max_pairwise_sync_gap_sec: 0.05
min_overlap_ratio: 0.8
frequency_jitter_ratio: 0.2
frame_count_drop_ratio_warning: 0.1
duration_drop_ratio_warning: 0.1
issue_regression_severity: warning
rules:
  enabled: null
  disabled: []
  severity: {}
expected_sensor_rates:
  camera_front: 10
  camera_rear: 10
  imu: 100
  gps: 10
```

The config reader supports normal YAML lists and nested maps. Unknown keys fail validation so stale configs do not silently pass. Rule policy can live in config:

```yaml
rules:
  enabled:
    - calibration
    - labels
  disabled:
    - check_sensor_frequency
  severity:
    check_large_timestamp_gaps: info
```

## Reports

Validation reports include:

- summary pass/fail state
- issue severity: `error`, `warning`, or `info`
- check name
- file path and 1-based CSV row when available
- message and structured metadata
- stats such as issue counts, sensor counts, and sync diagnostics

Output formats are console, JSON, Markdown, and static HTML:

```bash
datasetlint examples/minimal_dataset --format console
datasetlint examples/minimal_dataset --format json
datasetlint examples/minimal_dataset --format markdown
datasetlint examples/minimal_dataset --format html > report.html
datasetlint report examples/bad_dataset --out report.html
```

Issue row numbers use spreadsheet-style rows: the CSV header is row 1 and the first data row is row 2.

## CI Usage

Minimal GitHub Actions example:

```yaml
name: Dataset Lint
on:
  pull_request:
  push:
    branches: [main]

jobs:
  datasetlint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: python -m pip install -e ".[dev]"
      - run: datasetlint examples/minimal_dataset
```

Use `--fail-on warning` for stricter validation, or `datasetlint diff OLD_DATASET NEW_DATASET --fail-on-regression` when comparing dataset revisions.

More CI templates, including report artifacts and dataset diffs, are in [docs/ci.md](docs/ci.md).

## Supported Formats And Adapters

| Dataset / Format | Adapter | Status | Optional Dependency | Notes |
| --- | --- | --- | --- | --- |
| Native DatasetLint folders | `folder` | supported | none | CSV/JSON format used by existing lint checks |
| Generic folders | `generic` | supported | none | Recursive inferred schema for images, point clouds, videos, labels, and timestamps |
| COCO | `coco` | supported | none | Direct JSON parser for images, categories, bbox, and segmentation references |
| KITTI | `kitti` | supported | none | Object and odometry layouts with camera, lidar, labels, calibration, and timestamps |
| Argoverse 2 | `argoverse2` | supported | none | Sensor/scenario file indexing for AV2 logs, annotations, calibration, and timestamped filenames |
| LeRobot | `lerobot` | supported | none | Local LeRobot metadata, episode parquet/jsonl, task metadata, and videos |
| nuScenes | `nuscenes` | manifest + deep payload summaries | optional `nuscenes-devkit` | Direct metadata-table parser available without the devkit; `--deep` checks referenced camera/lidar/radar payload headers |
| Waymo | `waymo` | index + optional deep metadata | optional Waymo/TensorFlow package | TFRecord indexing by default; `--deep` parses frame, label, sensor, and calibration metadata, not image/lidar payload bytes |
| ROS bag | `rosbag` | index + optional deep metadata | optional `rosbags` | ROS1/ROS2 file indexing by default; `--deep` parses topics, message types, counts, and timestamps |
| MCAP | `mcap` | index + optional deep metadata | optional `mcap` | File indexing by default; `--deep` parses channels, schemas, and message timestamps |
| Hugging Face | `huggingface` | index + optional deep sampling | optional `datasets` | Cache/local metadata by default; `--deep` samples rows, projects label/bbox-like columns, and fails malformed sampled bbox payloads |
| Custom adapters | `datasetlint.adapters` entry point | supported | adapter-specific | Install a package exposing a DatasetLint adapter entry point; see `examples/third_party_adapter` for a minimal template |

## Roadmap

Near term:

- stronger dataset diff coverage
- more targeted example datasets
- CI templates for common repository layouts
- clearer rule plugin examples

Medium term:

- modality-specific decoding beyond current metadata manifests: MCAP and ROS bag message payloads, Waymo image/lidar payload bytes, richer Argoverse 2/LeRobot payload semantics, full nuScenes devkit/map/point-cloud semantics, and dataset-specific Hugging Face row schemas
- conversion helpers from normalized manifests to the native lintable folder format
- richer sensor synchronization checks
- richer static HTML report styling while keeping reports dependency-free

Long term:

- rule plugin system
- benchmark sample datasets
- performance profiling on larger logs
- adapter compatibility test suite

Non-goals:

- replacing dataset version control systems
- storing or hosting datasets
- running model evaluation
- simulating robotics environments

## Contributing

Read [CONTRIBUTING.md](CONTRIBUTING.md) for local setup, tests, linting, type checks, docs, and guidance for adding rules or adapters.

## License

DatasetLint is released under the [MIT License](LICENSE).
