Metadata-Version: 2.4
Name: trajlens
Version: 0.4.0
Summary: The quality and synthesis layer for the open robot-learning data ecosystem.
Project-URL: Homepage, https://github.com/yourusername/trajlens
Project-URL: Repository, https://github.com/yourusername/trajlens
Project-URL: Issues, https://github.com/yourusername/trajlens/issues
Author: trajlens contributors
License: Apache-2.0
License-File: LICENSE
Keywords: dataset,huggingface,lerobot,linter,robotics,validation
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Requires-Dist: av<19,>=12.0
Requires-Dist: numpy<3,>=1.26
Requires-Dist: pyarrow<25,>=19.0
Requires-Dist: pydantic<3,>=2.5
Requires-Dist: rich<16,>=13.0
Requires-Dist: structlog<27,>=24.0
Requires-Dist: typer<1,>=0.12
Requires-Dist: watchfiles<2,>=0.22
Provides-Extra: dev
Requires-Dist: fastapi<1,>=0.110; extra == 'dev'
Requires-Dist: httpx<1,>=0.27; extra == 'dev'
Requires-Dist: huggingface-hub<2,>=0.23; extra == 'dev'
Requires-Dist: hypothesis<7,>=6.100; extra == 'dev'
Requires-Dist: jsonschema<5,>=4.20; extra == 'dev'
Requires-Dist: mypy<3,>=1.10; extra == 'dev'
Requires-Dist: pip-audit<3,>=2.7; extra == 'dev'
Requires-Dist: pre-commit<5,>=3.7; extra == 'dev'
Requires-Dist: pytest-cov<8,>=5.0; extra == 'dev'
Requires-Dist: pytest<10,>=8.0; extra == 'dev'
Requires-Dist: ruff<1,>=0.4; extra == 'dev'
Requires-Dist: types-pyaudio; extra == 'dev'
Requires-Dist: uvicorn<1,>=0.29; extra == 'dev'
Provides-Extra: hub
Requires-Dist: huggingface-hub<2,>=0.23; extra == 'hub'
Provides-Extra: web
Requires-Dist: fastapi<1,>=0.110; extra == 'web'
Requires-Dist: uvicorn<1,>=0.29; extra == 'web'
Description-Content-Type: text/markdown

# trajlens

The quality and synthesis layer for the open robot-learning data ecosystem.

ruff for robot data — lint, fix, and generate clean LeRobotDataset datasets.

![trajlens demo](docs/assets/demo.gif)

## Status

v0.4.0, under active development.

`lint` is implemented and audited against the public Hub (see [Real-world audit](#real-world-audit-of-the-hub) below). `fix` (repair engine: timestamp drift, stats recomputation, episode reindexing, task-index repair, video/info.json fps sync, orphan-shard reporting) and `web` (read-only local dashboard, with a per-episode findings view) both ship, with `lint --share` for redacted issue-report sharing, `lint --baseline`/`--update-baseline` for CI adoption on existing datasets, `lint --parallel <n>` for multi-process scanning, and `trajlens watch` for live linting during recording.

## Install

```bash
pip install trajlens
```

For dev work:

```bash
git clone https://github.com/<your-username>/trajlens
cd trajlens
uv venv
source .venv/bin/activate
uv pip install -e ".[dev,hub,web]"
```

The `[hub]` extra pulls in `huggingface_hub`; it's only required to lint datasets by Hub repo id rather than local path. The `[web]` extra pulls in FastAPI + uvicorn; it's only required for `trajlens web`.

## Usage

```bash
trajlens lint <path-or-org/dataset>          # human-readable terminal report
trajlens lint <path-or-org/dataset> --json   # machine-readable JSON report
trajlens lint <path-or-org/dataset> --report out.html
trajlens lint <path-or-org/dataset> --sarif out.sarif   # SARIF 2.1.0, for CI annotations
trajlens lint <path-or-org/dataset> --deep   # also decode video and verify per-frame stats
trajlens lint <path-or-org/dataset> --share  # redacted summary, for pasting into a GitHub issue
trajlens lint <path-or-org/dataset> --share --share-out out.json  # write it to a file instead
trajlens lint <path-or-org/dataset> --parallel 4   # run thread-safe checks across worker processes
trajlens lint <local-path> --baseline .trajlens-baseline.json          # suppress already-known findings
trajlens lint <local-path> --update-baseline .trajlens-baseline.json  # (re)generate the baseline file
trajlens watch <local-path>   # live lint as episode shards are written during recording
```

Exit codes follow lint-tool convention: `0` = clean, `1` = WARN present, `2` = FAIL or load ERROR — so `trajlens lint` composes directly into CI gates.

By default, checks that require materializing a lot of data over the network (full video decode, per-frame stats reconciliation) are skipped for Hub datasets and reported as INFO/skipped rather than run. Pass `--deep` to force them; expect this to be significantly slower and to fetch the full dataset.

The `--json`/`--report` output also includes a per-episode view: the three checks with independent per-episode signal (`STRUCTURAL.METADATA_DATA_AGREEMENT`, `TEMPORAL.TIMESTAMP_MONOTONIC`, `STATISTICAL.PER_EPISODE_STATS_MATCH`) attach a `per_episode` breakdown to their result, and the report surfaces the worst-5 episodes ranked by trust contribution. `--share` never includes this free text (see below) — only counts.

`--share` is a redacted single-file JSON summary safe to paste publicly: trajlens version, trust score, grade, format version, per-check finding counts, the worst-5 episode list (indices and counts only), and a `dataset_ref` (the Hub repo id for Hub datasets, or the local directory's basename only — never any parent path component). It deliberately omits every check's free-text `message`/`details`/`per_episode` fields, since those can embed local filesystem paths.

### Repairing issues: `trajlens fix`

```bash
trajlens fix <local-path>                       # dry-run: preview the diff, write nothing
trajlens fix <local-path> --apply --out <path>  # write a repaired copy
trajlens fix <local-path> --json                # machine-readable dry-run/apply report
trajlens fix <local-path> --only REPAIR.TASK_INDEX_REPAIR       # run only these fixer(s)
trajlens fix <local-path> --except REPAIR.ORPHAN_SHARD_REPORT   # run all applicable fixers except these
trajlens fix <local-path> --apply --out <path> --quarantine     # orphan_shard_report: move orphans instead of just reporting
```

`fix` lints the dataset, selects whichever fixers apply to the findings, and runs them in a fixed order. Six fixers exist: `REPAIR.TIMESTAMP_DEDRIFT`, `REPAIR.STATS_RECOMPUTE`, `REPAIR.EPISODE_REINDEX`, `REPAIR.TASK_INDEX_REPAIR`, `REPAIR.VIDEO_METADATA_SYNC`, and `REPAIR.ORPHAN_SHARD_REPORT`. It's copy-on-write — the source is never mutated — and dry-run by default. Only local datasets can be repaired: a Hub ref's data/video shards are streamed on demand and never fully present on disk to copy, so `fix` refuses Hub refs with a clean error rather than silently producing an incomplete repair.

`--only`/`--except` take comma-separated fixer ids and are validated against the known fixer set before any work starts; an unknown id, or the same id in both flags, exits 2 with a message naming the offending id(s). `--only` also bypasses the normal WARN+ selection threshold, which is the only way to reach `REPAIR.VIDEO_METADATA_SYNC` today — its target check, `VIDEO.RESOLUTION_FPS_MATCH`, is catalog-only and not yet implemented as a standalone check, so it never fires on its own. `--quarantine` only affects `REPAIR.ORPHAN_SHARD_REPORT`: without it, orphan shards are reported but never moved; with it, `apply()` relocates them to `<output>/.trajlens-quarantine/` and writes a manifest.

Exit codes: `0` = nothing to fix, `1` = fixes proposed or applied, `2` = could not fix (load failure, invalid usage, or a fixer's refusal because the underlying data has no consistent repair).

### Viewing a report: `trajlens web`

```bash
trajlens web <path-or-org/dataset>
```

Lints the dataset once and serves a read-only local dashboard over the result (binds to `127.0.0.1` only, no flag to widen the bind). It's a thin FastAPI shell over the same report the terminal/JSON renderers use — no separate lint logic, no writes, no route that accepts a path/ref/dataset id from the browser.

## CI integration

Lint datasets automatically on push or PR using the bundled GitHub Action:

```yaml
- uses: Kunal-Somani/trajlens/.github/actions/lint@v0.4.0
  with:
    dataset-ref: path/to/your/dataset
```

See [docs/github-action.md](docs/github-action.md) for the full input/output reference and a complete workflow example.

## Found something?

Run `trajlens lint <path-or-org/dataset> --share` and paste the output into one of the issue forms below. False positives are the most valuable report you can file — if a check flagged something that isn't actually wrong, [tell us](.github/ISSUE_TEMPLATE/false_positive.yml). There's also a form for [bugs](.github/ISSUE_TEMPLATE/bug_report.yml) and one for [new corruption classes](.github/ISSUE_TEMPLATE/new_corruption_class.yml) trajlens doesn't check for yet.

## Architecture

```mermaid
graph TD
  subgraph Interfaces
    CLI[CLI - typer]
    WEB[Web dashboard - FastAPI + static HTML/JS, optional]
    SDK[Python SDK / import]
  end

  subgraph Core
    LOADER[Dataset Source Layer<br/>local + Hub, version-aware]
    MODEL[Canonical Dataset Model<br/>typed in-memory view]
    REGISTRY[Check Registry<br/>pluggable rules]
    ENGINE[Check Engine<br/>runs checks, bounded]
    REPORT[Report Builder<br/>terminal / json / html / sarif]
    REPAIR[Repair Engine<br/>dry-run, diff, opt-in]
  end

  subgraph Synthesis [Pillar 3, later]
    SIMBK[Sim Backend Protocol<br/>MuJoCo default]
    AUG[Trajectory Augmenter<br/>MimicGen-style]
    DR[Domain Randomizer]
    WRITER[LeRobotDataset Writer]
  end

  CLI --> LOADER
  WEB --> LOADER
  SDK --> LOADER
  LOADER --> MODEL
  MODEL --> ENGINE
  REGISTRY --> ENGINE
  ENGINE --> REPORT
  MODEL --> REPAIR
  REPAIR --> WRITER
  SIMBK --> AUG --> DR --> WRITER
  WRITER --> MODEL
  REPORT --> WEB
  HUB[(Hugging Face Hub)] <--> LOADER
  HUB <--> WRITER
```

## What it checks

trajlens validates a [LeRobotDataset](https://github.com/huggingface/lerobot) (v2.0, v2.1, or v3.0) against its own declared metadata, independent of any particular consumer's assumptions. Checks are grouped by category and run as a check engine over each dataset:

| Category | Check | Severity | What it catches |
|---|---|---|---|
| STRUCTURAL | `VERSION_DETECTED` | INFO | Reports the detected `codebase_version`. |
| STRUCTURAL | `SCHEMA_CONSISTENCY` | FAIL | Parquet column dtypes/widths disagree with `info.json`'s declared feature shapes. |
| STRUCTURAL | `INDEX_CONTINUITY` | FAIL | Gaps or duplicates in `frame_index`/`episode_index`/global `index` columns. |
| STRUCTURAL | `METADATA_DATA_AGREEMENT` | FAIL | Declared episode lengths/`from`-`to` boundaries disagree with actual Parquet row counts (catches [#2401](https://github.com/huggingface/lerobot/issues/2401)-class corruption). |
| STRUCTURAL | `PATH_TEMPLATE_RESOLVES` | FAIL | A declared shard path (data or video) doesn't resolve to a readable file. |
| STRUCTURAL | `ORPHAN_SHARD` | WARN | A data/video shard exists on disk (v3.0 only) that no episode record references — the reverse of `PATH_TEMPLATE_RESOLVES`. |
| SEMANTIC | `FEATURE_DIMENSIONALITY` | FAIL | A feature's actual column width doesn't match its declared `shape`. |
| SEMANTIC | `TASK_INTEGRITY` | FAIL | A `task_index` reference has no corresponding, non-empty task description. |
| SEMANTIC | `LANGUAGE_PRESENT` | WARN | An episode has no non-empty language/task description. |
| SEMANTIC | `CAMERA_INTRINSICS_PLAUSIBLE` | INFO | Advisory; skipped where the LeRobot format carries no intrinsics field. |
| TEMPORAL | `TIMESTAMP_MONOTONIC` | FAIL | Timestamps are not strictly increasing within an episode. |
| TEMPORAL | `TIMESTAMP_SPACING` | WARN | Timestamp spacing is inconsistent with declared `fps` beyond decoder tolerance. |
| STATISTICAL | `STATS_MATCH_DATA` | FAIL | Recomputed global Welford stats diverge from `meta/stats.json`. Skipped over Hub HTTP by default — too slow without `--deep`. |
| STATISTICAL | `PER_EPISODE_STATS_MATCH` | WARN | Same, per-episode. Skipped over Hub HTTP by default. |
| STATISTICAL | `VALUE_SANITY` | WARN | Out-of-range or NaN/Inf values in numeric features. Skipped over Hub HTTP by default. |
| VIDEO | `DECODABLE_SPOTCHECK` | FAIL | A sampled video segment fails to decode. |
| KNOWNBUG | `TIMESTAMP_DRIFT` | FAIL | Cumulative timestamp drift matching the known lerobot [#3177](https://github.com/huggingface/lerobot/issues/3177) bug pattern. |

Every check's full result — message, severity, and structured `details` — is included in the JSON/HTML/SARIF report; the table above is the summary.

## Real-world audit of the Hub

`scripts/audit_hub.py` runs `trajlens lint --json` against a random sample of public Hub datasets tagged `lerobot`, each in an isolated subprocess with a 60s timeout, and aggregates the results. It's how this project validates itself against the actual long tail of community datasets rather than only its own fixtures.

A 100-dataset run (2026-06-24) produced:

| Status | Count | Meaning |
|---|---|---|
| PASS | 19 | No issues found. |
| WARN | 0 | — |
| FAIL | 13 | A real check fired — schema mismatch, metadata/data disagreement, missing language, etc. |
| ERROR | 47 | Dataset failed to *load* (unsupported v2.x Hub streaming, malformed/missing `meta/`, mistagged or deleted repos) — never reached the check engine. |
| TIMEOUT | 21 | Exceeded the 60s per-dataset budget. |

These figures are from a single 100-dataset random sample (raw results: see the `v0.1.0` release assets); `audit_hub.py` samples a fresh random subset of `lerobot`-tagged Hub datasets on each run, so rerunning it will produce a similarly-shaped but not identical distribution.

Of the 47 load-time ERRORs, none are trajlens bugs: about half (24) are the documented v0.1 limitation that v2.x Hub datasets can't be lazily streamed (shard paths are implicit and require a local filesystem to glob), and the rest are dead/mistagged Hub references, repos that aren't actually LeRobotDatasets (no `meta/` directory at the repo root), or genuinely malformed `meta/info.json` (wrong dtype, missing required fields) on the dataset's side.

TIMEOUTs were investigated as a possible performance bug rather than accepted as an inherent network ceiling: profiling two small, previously-timing-out datasets (`abdul004/so101_multi_task_v1`, 125 episodes; `Elvinky/pick_green_block_into_box`, 102 episodes) found that loading a dataset's metadata over Hub HTTP was issuing dozens of small, separately-latency-bound reads per Parquet shard, and downloading the `meta/` file tree one file at a time. Fixing both (single whole-shard fetch instead of scattered reads; parallelized `meta/` download) brought those two datasets from 60s+ timeouts down to 33s and 11s respectively, and cut the audit's overall TIMEOUT count and mean per-dataset duration by roughly a third in before/after sampling. The remaining TIMEOUTs are concentrated in genuinely large multi-thousand-episode shards, where 60s is a real infra ceiling rather than a fixable inefficiency.

### Launch audit findings

Of the 81 datasets that reached a grade (excluding ERROR/TIMEOUT, where no check ever ran), two known upstream `lerobot` bugs accounted for a meaningful share of the failures:

| Known bug | Prevalence (of successfully-linted datasets) |
|---|---|
| `KNOWNBUG.TIMESTAMP_DRIFT` ([#3177](https://github.com/huggingface/lerobot/issues/3177)) | 3.1% |
| `STRUCTURAL.METADATA_DATA_AGREEMENT` ([#2401](https://github.com/huggingface/lerobot/issues/2401)) | 18.8% |

`audit_hub.py` resamples a fresh random subset of `lerobot`-tagged Hub datasets on every run, so these are not a fixed, reproducible distribution — rerunning the audit will not return the same percentages, only a similarly-shaped one. Raw per-dataset results behind these specific numbers are attached to the `v0.1.0` GitHub release as `audit_results_100.json` and `audit_summary_100.txt`.

## Performance note: Hub vs. local

Linting a 100-episode dataset locally takes under 30 seconds.

Linting a Hub dataset directly (`trajlens lint org/dataset`) streams metadata and data shards over HTTP. It will inherently be slower than a local copy — typically under a minute for small-to-medium datasets, more for very large ones — because of unavoidable network round trips. For repeated linting, downloading the dataset locally first is still faster.

## License

Apache-2.0
