Metadata-Version: 2.4
Name: deepen-grade
Version: 0.2.2
Summary: Free, local CLI that grades robot-learning datasets (MCAP/rosbag/LeRobot) on hygiene, episode quality, and calibration sanity -- every metric traced to a named paper.
Project-URL: Homepage, https://github.com/mmusa/deepen-grade
Project-URL: Repository, https://github.com/mmusa/deepen-grade
Project-URL: Issues, https://github.com/mmusa/deepen-grade/issues
Project-URL: Deepen AI, https://deepen.ai
Project-URL: Full Audit, https://deepen-robograde.pages.dev
Author-email: Deepen AI <oss@deepen.ai>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data-quality,imitation-learning,lerobot,mcap,robot-learning,robotics,rosbag
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: numpy>=1.24
Requires-Dist: rich>=13.7
Provides-Extra: all
Requires-Dist: huggingface-hub>=0.24; extra == 'all'
Requires-Dist: mcap-ros2-support>=0.5.3; extra == 'all'
Requires-Dist: mcap>=1.1.1; extra == 'all'
Requires-Dist: pandas>=2.0; extra == 'all'
Requires-Dist: pyarrow>=14.0; extra == 'all'
Requires-Dist: rosbags>=0.10.4; extra == 'all'
Provides-Extra: dev
Requires-Dist: huggingface-hub>=0.24; extra == 'dev'
Requires-Dist: mcap-ros2-support>=0.5.3; extra == 'dev'
Requires-Dist: mcap>=1.1.1; extra == 'dev'
Requires-Dist: pandas>=2.0; extra == 'dev'
Requires-Dist: pyarrow>=14.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: rosbags>=0.10.4; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: lerobot
Requires-Dist: huggingface-hub>=0.24; extra == 'lerobot'
Requires-Dist: pandas>=2.0; extra == 'lerobot'
Requires-Dist: pyarrow>=14.0; extra == 'lerobot'
Provides-Extra: mcap
Requires-Dist: mcap-ros2-support>=0.5.3; extra == 'mcap'
Requires-Dist: mcap>=1.1.1; extra == 'mcap'
Provides-Extra: ros
Requires-Dist: rosbags>=0.10.4; extra == 'ros'
Description-Content-Type: text/markdown

# deepen-grade

A free, local, `pip install`-able CLI that answers **"is this robot dataset
training-grade?"** in one graded report.

`deepen grade` reads `.mcap` (ROS 2), `.bag` (ROS 1), `.db3` (ROS 2 sqlite), or
a [LeRobot](https://huggingface.co/docs/lerobot) dataset (local path or a
HuggingFace `repo-id`), and runs three layers of checks -- **hygiene**,
**episode quality**, and **calibration sanity** -- entirely on your machine.
Nothing uploads by default. Every number in the report is traced to a named,
public source: an open standard, an ROS REP, or a peer-reviewed paper.

```
$ deepen grade my_episode.mcap

╭─ deepen-grade report ──────────────────────────────╮
│ my_episode.mcap                                    │
│ format: mcap   episodes: 1                          │
╰──────────────────────────────────────────────────────╯
Overall grade: B (82/100)   self-assessment -- hygiene & episode quality only
Calibration:   NOT ASSESSED
               no calibration metadata found anywhere in this dataset -- calibration
               quality is unknown, not good; verification requires the deep audit

         Per-episode grades
┏━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━┓
┃ Episode     ┃ Grade ┃ Score ┃ Task ┃
┡━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━┩
│ my_episode  │ B     │ 82    │ -    │
└─────────────┴───────┴───────┴──────┘

           Checks -- my_episode
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳...
│ Topic frequency & drops │ PASS │ 4 topics, no drops detected
│ Cross-modal sync skew   │ WARN │ max skew 28.4ms exceeds 15ms budget
│ tf-tree integrity       │ PASS │ 6 frames, single-parent tree
│ Idle / stall detection  │ PASS │ 3.1% of episode idle
│ LDJ smoothness          │ PASS │ LDJ = -14.2
│ Joint-limit saturation  │ PASS │ worst dim 'joint_3' saturated 4.2%
│ Gripper chatter         │ PASS │ 2 direction reversals (0.31 Hz)
│ Action-state consistency│ PASS │ consistency score 0.94
│ Calibration sanity      │ INFO │ NOT ASSESSED -- no camera calibration metadata found
└──────────────────────────────────────────────────────┘

╭─ Funnel ─────────────────────────────────────────────╮
│ What this can't tell you locally:                    │
│   - Whether camera calibration is geometrically       │
│     correct (not just present) -- targetless          │
│     verification is Deepen's sealed, patented audit.  │
│   - A signed acceptance certificate for a data buy.   │
│   - Robot Dataset Quality Index leaderboard entry.    │
│   - Influence-function trainability scoring.          │
│                                                        │
│ Run the full sealed audit: https://deepen-robograde.pages.dev │
╰────────────────────────────────────────────────────────╯

Deepen AI -- https://deepen.ai   |   Full audit: https://deepen-robograde.pages.dev
```

## Install

The base install is deliberately light (numpy, click, rich -- no torch, no
ROS). Add the extra(s) for the formats you actually use:

```bash
pip install deepen-grade                    # base: no format readers yet
pip install "deepen-grade[mcap]"            # .mcap (ROS 2 default)
pip install "deepen-grade[ros]"             # .bag (ROS 1) / .db3 (ROS 2 sqlite)
pip install "deepen-grade[lerobot]"         # LeRobot local path or HF repo-id
pip install "deepen-grade[all]"             # everything
```

For the latest development version: `pip install "deepen-grade[all] @ git+https://github.com/mmusa/deepen-grade"`

`[ros]` uses the pure-Python [`rosbags`](https://pypi.org/project/rosbags/)
library -- no ROS installation required. `[lerobot]` reads the LeRobot
parquet/JSON dataset format directly (`pandas` + `pyarrow` +
`huggingface_hub`) -- it deliberately does **not** depend on the `lerobot`
training package, which pulls in torch and is much heavier than a CLI needs.
Video files are never downloaded or decoded: every check operates on
recorded state/action arrays and timestamps, not pixels.

If you invoke `deepen grade` on a format whose extra isn't installed, you get
a clear one-line fix (`pip install "deepen-grade[...]"`), not a stack trace.

## Usage

```bash
deepen grade my_episode.mcap
deepen grade my_ros1.bag
deepen grade path/to/ros2_bag/            # directory containing metadata.yaml + *.db3
deepen grade path/to/lerobot_dataset/     # local LeRobotDataset v2/v3 directory
deepen grade some-org/some-dataset        # HuggingFace repo-id (downloaded, no video)

deepen grade my_episode.mcap --json                 # machine-readable, for CI
deepen grade my_episode.mcap --json -o report.json
deepen grade my_episode.mcap --min-grade B          # exit 1 if grade < B (CI gate)

deepen grade some-org/big-dataset --sample 200 --seed 0    # grade a random 200-episode sample
deepen grade some-org/big-dataset --max-episodes 200       # grade the first 200 episodes
deepen grade some-org/container-repo --subdataset team_a/session1   # grade one nested dataset only
deepen grade some-org/gated-dataset --hf-token hf_xxx       # or just set HF_TOKEN
deepen grade my_episode.mcap --quiet                        # no HF progress bars, no CLI chatter
```

### Container repos: datasets nested inside a repo

Some Hub repos aren't a single LeRobot dataset at the root -- they nest one or
more complete LeRobot datasets one or two directories down (a
per-contributor directory, an `ImitationLearning/` wrapper, etc). `deepen
grade` detects this automatically (no `meta/info.json` at the root, but one
exists one or two levels down) and grades every nested dataset together,
prefixing each episode ID with its sub-dataset's relative path
(`team_a/session1::episode_000042`). A warning in the report lists every
sub-dataset found. Use `--subdataset <relpath>` to grade just one of them --
that behaves identically to grading it as a standalone dataset.

### Sampling large corpora: `--sample` / `--max-episodes`

Some Hub repos have 100k+ episodes; downloading and reading the whole thing
can OOM a laptop or blow a CI time budget long before grading starts.
`--sample N` grades a uniform random sample of N episodes (seeded via
`--seed`, default `0`, so the same command always picks the same episodes);
`--max-episodes N` grades the first N episodes in natural order instead.
Either way, only the parquet files that actually contain a selected episode
are ever read -- episodes are always materialized one file at a time, never
concatenated into one corpus-sized DataFrame, so peak memory is bounded by
the largest single file, not the corpus. A `"sampling"` block appears in the
JSON report (`{"mode": "sample"|"head", "n", "seed", "episodes_total"}`) and
the terminal report prints a `GRADED ON A SAMPLE` banner; grade letters
themselves are unaffected.

### Partial reports: `--json -o` writes incrementally

With `--json -o report.json`, the report is written to disk incrementally --
at a progress-proportional interval (roughly every 10% of graded episodes,
never more often than every 200), not just at the end -- via a write-to-temp-
file-then-rename so the file on disk is always a complete, parseable JSON
document, never a half-written one. An in-progress write has `"partial":
true`; the final write sets it to `false`. This means a crash or CI timeout
mid-run still leaves a valid, gradeable-so-far report behind instead of
nothing.

### HuggingFace downloads: retries and rate limits

Repo-id downloads retry up to 3 times with exponential backoff on a dropped
connection or other transient Hub error. An HTTP 429 (rate limited) fails
immediately with an actionable message instead of retrying into the same
wall: anonymous downloads have a lower rate limit than authenticated ones, so
pass `--hf-token` (or set `HF_TOKEN`) to raise it.

## What it checks (and what each check cites)

### A. Hygiene -- deterministic, applies to any recording
| Check | What it measures | Cited source |
|---|---|---|
| Topic frequency & drops | per-topic observed rate, gaps > 3x median interval | ISO/WD 26264-1 companion (arXiv:2606.19769) |
| Cross-modal timestamp skew | ms skew between vision/proprioception/lidar streams | arXiv:2606.19769 |
| tf-tree integrity | single-parent tree, no static/dynamic conflicts, no cycles | REP 105 |
| Message-schema & state/action-dim consistency | consistent types/dims across a dataset's episodes | arXiv:2606.19769, ASAM OpenLABEL |

### B. Episode quality -- published metric implementations
| Check | What it measures | Cited source |
|---|---|---|
| Idle / stall | near-zero normalized action/state velocity runs | DQAF (arXiv:2605.26349), SCIZOR (arXiv:2505.22626) |
| Smoothness | log-dimensionless-jerk of the speed profile | Balasubramanian et al. 2015 (IEEE TBME), DQAF |
| Joint-limit saturation | time spent near the episode's own observed value range | DQAF |
| Gripper chatter | direction-reversal rate of gripper position | DQAF |
| Action-state consistency | normalized tracking error between commanded action and state | DQAF, SCIZOR |

**On thresholds:** the papers above define *what to measure*; the specific
pass/warn/fail cutoffs (e.g. "stall_frac > 30% is a FAIL") are deepen-grade's
own conservative, documented defaults -- see
[`src/deepen_grade/checks/episode_quality.py`](src/deepen_grade/checks/episode_quality.py)
for every constant in one place. No universal numeric cutoff across every
robot embodiment and control rate exists in the literature; these are sane
starting points, not claims about what the cited papers themselves recommend.

### C. Calibration -- its own verdict, never part of the grade
deepen-grade checks that intrinsics/extrinsics are **present** and not
**obviously broken** (missing, NaN, an untouched identity/zero default).
That's it. It does not verify that a calibration is *geometrically correct*
-- doing that from a recording alone (reprojection, epipolar consistency,
drift over a session) is Deepen's patented targetless calibration
verification, which stays sealed and server-side.

Because a well-formed but geometrically *wrong* calibration passes every
local check, calibration is deliberately **excluded from the letter grade**
and reported as its own top-level verdict instead:

- `NOT ASSESSED` -- no calibration metadata found; quality is unknown, not good.
- `PRESENT -- ACCURACY NOT VERIFIED` -- metadata present and structurally sound.
- `STRUCTURALLY BROKEN` -- metadata present but obviously broken (NaN/unset default).

See
[`src/deepen_grade/checks/calibration_sanity.py`](src/deepen_grade/checks/calibration_sanity.py)
for the firewall this module enforces on itself.

Calibration metadata itself is read from whatever the source format actually
carries: for `.mcap`/`.bag`/`.db3`, from a `sensor_msgs/CameraInfo` message on
each camera topic (intrinsics) paired with a matching `/tf_static` transform
by frame_id (extrinsics, left `None` -- never invented -- if no matching
transform was published); for LeRobot, from the `meta/calibration.json`
sidecar convention or (narrowly) a DROID-style `meta/cam2base_extrinsics.json`
/ `meta/info.json["calibration"]` block.

### D. Plausibility -- does this even look like robot data?
A structurally valid LeRobot/mcap/bag file can still contain something that
isn't robot-learning data at all -- a non-robotics dataset reshaped to fit the
schema, for instance. `Robot-data plausibility` is a cheap, dataset-level
heuristic check that flags this: no state/action trajectory anywhere, a state
that never actually changes across any episode, or no usable timestamps.
It's always `INFO` (or `N/A` when nothing looks wrong), never affects the
score, and is deliberately **uncited** -- "does this look like robot data" is
not a metric any of the papers above define. See
[`src/deepen_grade/checks/plausibility.py`](src/deepen_grade/checks/plausibility.py).

## What this can't tell you locally

Every report ends with a funnel section, because it's true, not because it's
a marketing footer:

- Whether calibration is geometrically **correct**, not just present.
- A signed, reproducible acceptance certificate for a data purchase.
- Entry in the public Robot Dataset Quality Index leaderboard.
- Influence-function trainability scoring (which episodes help your policy).

Full sealed audit: **https://deepen-robograde.pages.dev**

## A self-assessment, not a certification

deepen-grade runs on your machine, on your data, with nothing checked by
Deepen -- so its report is a **self-assessment**, and it says so on its face
(`report_type: "self-assessment"` in `--json`). There is deliberately no
claimable "verified" badge here: a trust mark Deepen never saw would be an
honor-system claim, which is exactly what this tool exists to replace.
Signed, third-party attestation -- a report a counterparty can rely on --
is the sealed deep audit at [deepen-robograde.pages.dev](https://deepen-robograde.pages.dev).

## CI: fail the data PR on a bad episode

```bash
deepen grade path/to/dataset --json --min-grade B
```
exits non-zero if the overall grade is worse than `B`. A ready-to-use
composite GitHub Action is in [`action/`](action/action.yml); see
[`examples/ci-gate.yml`](examples/ci-gate.yml) (copy it into `.github/workflows/` in your repo)
for a full workflow that runs it on every PR touching a dataset directory.

```yaml
- uses: mmusa/deepen-grade@v0
  with:
    path: path/to/dataset
    min-grade: B
    extras: lerobot   # mcap | ros | lerobot | all
```

## Privacy

Nothing uploads by default -- full stop. `--share-report` is an explicit
opt-in flag for a future sealed/signed report from deepen-robograde.pages.dev; **it
is not implemented in this release** (see
[`src/deepen_grade/report/share.py`](src/deepen_grade/report/share.py)) and
performs no network call. Use `--share-report --share-report-dry-run
payload.json` to inspect exactly what the eventual payload would contain.

## Development

```bash
git clone https://github.com/mmusa/deepen-grade
cd deepen-grade
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest
```

## License

Apache-2.0 -- see [LICENSE](LICENSE).

---

Built by [Deepen AI](https://deepen.ai), the data infrastructure layer for
physical AI. `deepen-grade` is the free, always-local doorway; the full
sealed audit (targetless calibration verification, acceptance certification)
lives at [deepen-robograde.pages.dev](https://deepen-robograde.pages.dev).
