Metadata-Version: 2.4
Name: humancoin-lerobot
Version: 0.1.1
Summary: Multi-frequency sensor datasets with independent storage and time-aligned queries, built on LeRobot
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/RoboCoin-BAAI/HumanCOIN-LeRobot
Project-URL: Repository, https://github.com/RoboCoin-BAAI/HumanCOIN-LeRobot
Project-URL: Issues, https://github.com/RoboCoin-BAAI/HumanCOIN-LeRobot/issues
Requires-Python: <3.14,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: lerobot==0.3.3
Requires-Dist: numpy<3,>=1.24
Requires-Dist: torch<2.8,>=2.2.1
Requires-Dist: datasets<=3.6.0,>=2.19
Requires-Dist: pyarrow>=14
Requires-Dist: av>=14.2
Requires-Dist: Pillow>=10
Requires-Dist: huggingface-hub<1,>=0.34.2
Requires-Dist: modelscope-hub<1,>=0.7.1
Provides-Extra: visualization
Requires-Dist: matplotlib>=3.8; extra == "visualization"
Requires-Dist: scipy>=1.11; extra == "visualization"
Provides-Extra: conversion
Requires-Dist: PyYAML>=6; extra == "conversion"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=6; extra == "dev"
Dynamic: license-file

# HumanCOIN-LeRobot

> 📖 中文文档：[README_zh-CN.md](README_zh-CN.md)

**A multi-sensor, multi-frequency dataset extension for real-world data, built on LeRobot** — sensors are stored at their independent sampling rates, one parquet file per sensor, dynamically aligned by time window at query time.

---

## Table of Contents

- [HumanCOIN-LeRobot](#humancoin-lerobot)
  - [Table of Contents](#table-of-contents)
  - [1. Motivation: Why Multi-Frequency Support](#1-motivation-why-multi-frequency-support)
  - [2. Installation](#2-installation)
  - [3. Quick Start](#3-quick-start)
  - [4. Demo & Visualization](#4-demo--visualization)
    - [4.1 Running the Demo](#41-running-the-demo)
    - [4.2 Demo 1: Writing (01write_simple.py)](#42-demo-1-writing-01write_simplepy)
    - [4.3 Demo 2: Reading (01read_simple.py)](#43-demo-2-reading-01read_simplepy)
    - [4.4 Demo 3: Alignment Tolerance (02tolerance_example.py)](#44-demo-3-alignment-tolerance-02tolerance_examplepy)
    - [4.5 Demo 4: Window Modes (03window_example.py)](#45-demo-4-window-modes-03window_examplepy)
    - [4.6 Demo 5: Delta Timestamps (04delta_example.py)](#46-demo-5-delta-timestamps-04delta_examplepy)
    - [4.7 Demo 6: Asynchronous Sensor Startup (05async_startup_example.py)](#47-demo-6-asynchronous-sensor-startup-05async_startup_examplepy)
    - [4.8 Demo 7: Audio (06audio_example.py)](#48-demo-7-audio-06audio_examplepy)
    - [4.9 Visualizer Controls](#49-visualizer-controls)
  - [5. Core Concepts](#5-core-concepts)
    - [5.1 Per-Feature Independent Storage](#51-per-feature-independent-storage)
    - [5.2 Task as the Master Clock](#52-task-as-the-master-clock)
    - [5.3 Window Parameter](#53-window-parameter)
  - [6. API Reference](#6-api-reference)
    - [Creating a Dataset](#creating-a-dataset)
    - [Feature Definition Format](#feature-definition-format)
    - [Writing](#writing)
    - [Reading](#reading)
    - [Return Structure](#return-structure)
  - [7. Architecture](#7-architecture)
  - [8. Comparison with Native LeRobotDataset](#8-comparison-with-native-lerobotdataset)
  - [9. Acknowledgements](#9-acknowledgements)


---

## 1. Motivation: Why Multi-Frequency Support

Real-world data collection often requires multiple sensors to capture the same process: cameras record visual information, IMUs measure motion, and EEG and EMG capture physiological signals. Human behavior research, wearable devices, experimental recording, and environmental sensing all involve combining such data. Sensors differ in sampling rate, startup time, and data format, so their individual time series need to be preserved and aligned when used.

Common sensors operate at very different sampling rates:

| Sensor | Typical rate | Shape | Example |
|--------|---------|------|------|
| Camera RGB | 30 / 60 Hz | 480×640×3 | scene camera + wearable camera |
| IMU | 50-1000 Hz | 6 (accel×3 + gyro×3) | accelerometer + gyroscope |
| EEG | 256 / 512 / 1000 Hz | 8-256 channels | brain signals |
| EMG | 20-500 Hz | 4-16 channels | muscle signals |

Forcing streams with different sampling rates into a table at a single frame rate typically requires upsampling low-rate data and downsampling or aggregating high-rate data, which can discard original samples. This project builds on LeRobot’s dataset infrastructure to support independent sensor storage and time-based queries.

**The humancoin_lerobot solution**: every sensor is stored independently as a time-indexed parquet file. At read time, a `window` parameter defines "how to aggregate data around the query timestamp" — no pre-resampling needed.

**Writing** — `add_frame` routes each key to its feature buffer; `save_episode()` flushes them to disk:

```mermaid
flowchart LR
    AF["add_frame(key, value)"]
    IX["MasterIndex record_frame(ts, task)"]
    PF["ParquetFeature buffer.append(ts, data)"]
    VF["VideoFeature save PNG + timestamp list"]
    AuF["AudioFeature buffer chunks + timestamp list"]

    MI[("master_index.parquet timestamp | task | frame_index")]
    P1[("imu.parquet 1000Hz × 6d")]
    P2[("eeg.parquet 256Hz × 8d")]
    P3[("state.parquet 30Hz × 7d")]
    C1[("head_rgb.parquet timestamps only")]
    V1[("head_rgb.mp4 30fps video")]
    A1[("mic.parquet timestamp | num_samples")]
    W1[("mic.wav 16kHz audio")]

    AF -->|"key = task"| IX
    AF -->|"key = sensor<br/>imu / eeg / state"| PF
    AF -->|"key = video"| VF
    AF -->|"key = audio"| AuF
    IX --> MI
    PF --> P1
    PF --> P2
    PF --> P3
    VF --> C1
    VF --> V1
    AuF --> A1
    AuF --> W1

    style AF fill:#fafafa,stroke:#9e9e9e,color:#000
    style IX fill:#e1f5fe,stroke:#4fc3f7,color:#000
    style PF fill:#fafafa,stroke:#9e9e9e,color:#000
    style VF fill:#f3e5f5,stroke:#9c27b0,color:#000
    style AuF fill:#e8f5e9,stroke:#66bb6a,color:#000
    style MI fill:#e1f5fe,stroke:#4fc3f7,color:#000
    style P1 fill:#fff3e0,stroke:#ff9800,color:#000
    style P2 fill:#fff3e0,stroke:#ff9800,color:#000
    style P3 fill:#fce4ec,stroke:#e91e63,color:#000
    style C1 fill:#f3e5f5,stroke:#9c27b0,color:#000
    style V1 fill:#f3e5f5,stroke:#9c27b0,color:#000
    style A1 fill:#e8f5e9,stroke:#4caf50,color:#000
    style W1 fill:#e8f5e9,stroke:#4caf50,color:#000
```

**Reading** — look up the master-clock timestamp in `master_index`, then query each feature by that timestamp:

```mermaid
flowchart TD
    IDX["ds[10]"] -->|"① look up frame 10"| MI[("master_index.parquet timestamp | task | frame_index")]
    MI -->|"② get timestamp"| TS["t = 0.333s"]
    TS -->|"③ nearest<br/>closest row in time"| P3[("state.parquet 30Hz × 7d")]
    TS -->|"③ window=(-0.033, 0] all rows in range"| P1[("imu.parquet 1000Hz × 6d")]
    TS -->|"③ window=(-0.033, 0] all rows in range"| P2[("eeg.parquet 256Hz × 8d")]
    TS -->|"③ nearest, closest row in time"| C1[("head_rgb.parquet timestamp")]
    C1 -->|"④ get frame_index"| V1[("rgb → 480×640×3")]
    P3 -->|"1 row"| RS["state → [7]"]
    P1 -->|"33 rows"| RI["imu → [33, 6]"]
    P2 -->|"8 rows"| RE["eeg → [8, 8]"]
    V1 -->|"1 frame"| RV["head_rgb → [3, 480, 640]"]

    style IDX fill:#fafafa,stroke:#9e9e9e,color:#000
    style MI fill:#e1f5fe,stroke:#4fc3f7,color:#000
    style TS fill:#fafafa,stroke:#9e9e9e,color:#000
    style P3 fill:#fce4ec,stroke:#e91e63,color:#000
    style P1 fill:#fff3e0,stroke:#ff9800,color:#000
    style P2 fill:#e8f5e9,stroke:#4caf50,color:#000
    style C1 fill:#f3e5f5,stroke:#9c27b0,color:#000
    style V1 fill:#f3e5f5,stroke:#9c27b0,color:#000
    style RS fill:#fce4ec,stroke:#e91e63,color:#000
    style RI fill:#fff3e0,stroke:#ff9800,color:#000
    style RE fill:#e8f5e9,stroke:#4caf50,color:#000
    style RV fill:#f3e5f5,stroke:#9c27b0,color:#000
```

The result of `ds[10]` (t=0.333s):
- `state` → nearest match → `[7]`
- `imu` → all 33 readings in the `(-0.033, 0]` window → `[33, 6]`
- `eeg` → all 8 readings in the `(-0.033, 0]` window → `[8, 8]`
- `head_rgb` → nearest row in timestamp parquet → `frame_index` → decode that frame → `[3, 480, 640]`

## 2. Installation

Python **3.10–3.13** (verified with Python 3.12). LeRobot is currently pinned to **0.3.3**.

Install from [PyPI](https://pypi.org/project/humancoin-lerobot/):

```bash
pip install humancoin-lerobot
```

Or install from source:

```bash
git clone https://github.com/RoboCoin-BAAI/HumanCOIN-LeRobot
cd HumanCOIN-LeRobot
pip install -e .
# Optional: desktop and web visualization
pip install -e ".[visualization]"
```

Dependencies:

| Dependency | Purpose |
|------|------|
| `lerobot` | base dataset framework (inherits LeRobotDataset) |
| `pyarrow` | parquet file I/O |
| `numpy` / `torch` | data processing |
| `av` (PyAV) | video encoding/decoding |
| `matplotlib` | visualization (optional) |
| `scipy` | web visualizer signal filtering (optional) |

### Download HumanCOIN datasets

Browse dataset names on [Hugging Face](https://huggingface.co/HumanCOIN) or [ModelScope](https://www.modelscope.cn/organization/HumanCOIN). These are organization pages; pass a dataset name from the selected hub to the command below.

```bash
pip install humancoin-lerobot

# Replace DATASET_NAME with an actual dataset repository name
humancoin-download --hub huggingface --datasets DATASET_NAME --target-dir ./data
humancoin-download --hub modelscope --datasets DATASET_NAME --target-dir ./data

# Download only selected files (partial download; may not be loadable as a full dataset)
humancoin-download --datasets DATASET_NAME --include "meta/*" README.md --target-dir ./data

# Multiple datasets; preview without downloading
humancoin-download --datasets DATASET_A DATASET_B --target-dir ./data --dry-run
# Also available as: python -m humancoin_lerobot.download ...
```

The default namespace is `HumanCOIN`; explicit `owner/name` IDs are also accepted. `--ds_lists` is an alias for `--datasets`. Files are stored under `<target-dir>/HumanCOIN/<dataset-name>`; the default target directory is `~/.cache/huggingface/lerobot`. Repeating a download reuses files through the hub SDK. Use `--revision` to select a version and `--max-workers` to control parallel downloads. Private/gated datasets require access and `HF_TOKEN` or `MODELSCOPE_TOKEN` (or `--token`); ModelScope uses the lightweight `modelscope-hub` SDK; credentials are passed to the client or read from its login cache. `MS_TOKEN` is also accepted. A failed dataset does not stop the remaining downloads; the command exits nonzero if any fail.

```python
from humancoin_lerobot import HumancoinLeRobotDataset

ds = HumancoinLeRobotDataset(
    repo_id="HumanCOIN/DATASET_NAME",
    root="data/HumanCOIN/DATASET_NAME",
)
item = ds[0]
```

Downloads preserve the repository files; loading requires the multi-frequency dataset layout supported by this project. Downloading does not convert other formats.

## 3. Quick Start

```python
import numpy as np
from humancoin_lerobot import HumancoinLeRobotDataset

ds = HumancoinLeRobotDataset.create(
    repo_id="my_dataset", root="data/", fps=30,
    features={
        # camera: dtype="video" auto-encodes to MP4
        "observation.images.cam": {
            "dtype": "video", "shape": (480, 640, 3),
            "names": ["h", "w", "c"], "fps": 30,
        },
        # high-rate IMU: window defines "return all readings in (t-33ms, t]"
        "observation.imu": {
            "dtype": "float32", "shape": (6,),
            "names": ["ax", "ay", "az", "gx", "gy", "gz"],
            "fps": 1000,
            "window": (-0.033, 0.0),
            "tolerance_s": 0.002,
        },
        # mid-rate EEG
        "observation.eeg": {
            "dtype": "float32", "shape": (8,),
            "names": ["Fz", "Cz", "Pz", "Oz", "F3", "F4", "C3", "C4"],
            "fps": 256,
            "window": (-0.033, 0.0),
        },
        # end-effector pose: no window → nearest neighbor by default
        "observation.state": {
            "dtype": "float32", "shape": (7,),
            "names": ["x", "y", "z", "roll", "pitch", "yaw", "gripper"],
            "fps": 30,
        },
        "action": {
            "dtype": "float32", "shape": (7,),
            "names": ["dx", "dy", "dz", "droll", "dpitch", "dyaw", "dgrip"],
            "fps": 30,
        },
    },
)

# recording: task = master clock, each frame is bounded by a task call
for i in range(300):
    # high-rate sensors: multiple writes per frame
    for _ in range(33):   # 1000Hz / 30fps ≈ 33
        ds.add_frame("observation.imu", imu_sample)
    for _ in range(8):    # 256Hz / 30fps ≈ 8
        ds.add_frame("observation.eeg", eeg_sample)

    # a task call = frame boundary, timestamp auto-generated
    ds.add_frame("task", "reach_target")

    # camera-rate sensors: once per frame
    ds.add_frame("observation.state", joint_state)
    ds.add_frame("action", action_cmd)
    ds.add_frame("observation.images.cam", camera_image)

ds.save_episode()
```

**Reading** — window features return all readings in the window, nearest features return a single vector:

```python
ds = HumancoinLeRobotDataset(repo_id="my_dataset", root="data/")
item = ds[10]

item["observation.imu"]             # tensor([33, 6]) — 33 readings in (-0.033, 0]
item["observation.imu_timestamps"]  # tensor([33])
item["observation.eeg"]             # tensor([8, 8])
item["observation.state"]           # tensor([7]) — nearest
item["observation.images.cam"]      # tensor([3, 480, 640]) — CHW
item["task"]                        # "reach_target"
```

```mermaid
sequenceDiagram
    participant User
    participant DS as Dataset
    participant PF as ParquetFeature<br/>(imu)
    participant VF as VideoFeature<br/>(cam)
    participant IX as MasterIndex

    loop each frame
        User->>DS: add_frame("imu", data)
        DS->>DS: _auto_timestamp("imu")<br/>ts = counter × 1/1000
        DS->>PF: add(data, ts)
        Note over PF: buffer.append((ts, data))

        User->>DS: add_frame("task", "reach")
        DS->>DS: ts = len(index) / fps
        DS->>IX: record_frame(ts, "reach")

        User->>DS: add_frame("cam", image)
        DS->>DS: _auto_timestamp("cam")<br/>ts = counter × 1/30
        DS->>VF: add(image, ts)
        Note over VF: save PNG + record timestamp
    end

    User->>DS: save_episode()
    DS->>IX: write(ep_idx) → master_index.parquet
    DS->>PF: save() → imu.parquet
    DS->>VF: save() → timestamp parquet + MP4
    DS->>DS: meta.save_episode()
    DS->>DS: _check_episode_alignment()
```

## 4. Demo & Visualization

### 4.1 Running the Demo

```bash
# write 3 episodes of simulated data
python -m examples.01write_simple

# read and inspect the data
python -m examples.01read_simple

# interactive visualizer
python -m scripts.visualize --episode 0

# web visualizer (browser UI, draggable time window)
python scripts/visualize_web.py [DATASET_ROOT]

# audio: microphone stored as WAV + timestamp parquet
python -m examples.06audio_example
```

The web visualizer supports optional zero-phase filtering of EEG/EMG strips: open the 滤波 panel on a strip and combine a high-pass / low-pass Butterworth with a comb notch at the line frequency and all its harmonics (EEG/EMG presets included). The API takes the same spec per feature: `/api/frame?...&filt=observation.eeg:hp=0.5,lp=40,notch=50`.

It also renders a 3D pose view in the SteamVR right-handed y-up world (`*_pose` 6-DoF features as coordinate triads + recent-history motion trails with time fading, fingertip end-effector paths from `hand_skeleton` FK; drag to rotate, wheel to zoom) and a color-coded electrode grid for static snapshot features such as `eeg_impedance` (quality-banded per channel). Pose rotation follows the recorder exactly (`R = Rz(yaw)·Ry(pitch)·Rx(roll)`, the inverse of the recorder's ZYX euler decomposition of the OpenVR matrix); hand skeletons attach via a constant per-hand Manus→tracker mount map (+z = finger, +y = back of hand, plus a −5cm offset along −y since the tracker sits above the back of the hand).

### 4.2 Demo 1: Writing (01write_simple.py)

The main write demo: a set of simulated sensor streams — 3 cameras + IMU + EEG + EMG + temperature — recording three 2-second episodes (`reach_target`, `pour_liquid`, `stack_blocks`):

| Sensor | Shape | Rate | Window |
|--------|------|------|--------|
| head_rgb | 480×640×3 | 30 fps | — (video) |
| left_wrist_rgb | 480×640×3 | 30 fps | — (video) |
| right_wrist_rgb | 480×640×3 | 30 fps | — (video) |
| imu | 6 (accel+gyro) | 1000 Hz | (-0.033, 0.0) |
| eeg | 8 (EEG channels) | 256 Hz | (-0.033, 0.0) |
| emg_left | 4 (muscle channels) | 100 Hz | (-0.033, 0.0) |
| emg_right | 4 (muscle channels) | 100 Hz | (-0.033, 0.0) |
| temperature | 1 | 10 Hz | — (nearest) |
| state | 7 (x,y,z,r,p,y,grip) | 30 Hz | — (nearest) |
| action | 7 (velocity commands) | 30 Hz | — (nearest) |

What it shows:

- `dtype: "video"` features auto-encode to MP4, exactly one frame per master frame
- high-rate sensors are written multiple times between two task calls; timestamps are auto-generated per feature as `timestamp_start + counter × 1/fps`
- each `save_episode()` writes `master_index.parquet` + per-feature parquets + videos, then runs the alignment checker (every feature should report OK)

```bash
python -m examples.01write_simple
```

### 4.3 Demo 2: Reading (01read_simple.py)

Reads the dataset written by Demo 1 and exercises every read path end to end:

- window features (`observation.imu`) return all readings in the window plus their timestamps
- nearest features (`observation.temperature`, `observation.state`, `action`) return a single vector
- videos are decoded back to CHW tensors
- `window_overrides` widen the IMU window at read time without touching the stored data
- prints the IMU segment sizes across frames — the first frames are empty because the IMU only comes online at `timestamp_start: 0.1`

```bash
python -m examples.01read_simple   # needs data/ from 01write_simple
```

### 4.4 Demo 3: Alignment Tolerance (02tolerance_example.py)

Shows the alignment checker catching a sensor that violates the alignment contract: the IMU is configured with `timestamp_start: 0.1` (it comes online 100 ms late) plus a tight `tolerance_s: 0.002`, so its samples are too far from the first master frames. After `save_episode()` the checker writes the offending frames to `meta/alignment_check.jsonl` and the demo prints the report.

Rule of thumb: `tolerance_s` must cover the distance from a master frame to the sensor's nearest sample — at most half its period (e.g. ≤ 50 ms for a 10 Hz sensor) — plus startup offset and jitter.

```bash
python -m examples.02tolerance_example
```

### 4.5 Demo 4: Window Modes (03window_example.py)

One episode, three aggregation modes side by side:

- `observation.imu` at 1000 Hz with `window: (-0.033, 0.0)` — range query, ~33 readings per frame
- `observation.finger_pose` at 120 Hz with `window: "interpolate"` — value interpolated at the frame timestamp
- `observation.temperature` at 10 Hz with no window — nearest neighbor

Then the dataset is reopened with `window_overrides` — a wider IMU window, finger pose switched to a range query — proving that the window is a read-time decision, not a property of the stored data.

```bash
python -m examples.03window_example
```

### 4.6 Demo 5: Delta Timestamps (04delta_example.py)

Multi-frame temporal queries with `delta_timestamps={"observation.state": [-0.1, 0.0, 0.1]}`: reading `ds[i]` returns state stacked over three timepoints (t-0.1, t, t+0.1) instead of one vector — the building block for delta-pose training and temporal augmentation.

```bash
python -m examples.04delta_example
```

### 4.7 Demo 6: Asynchronous Sensor Startup (05async_startup_example.py)

Real sensors don't come online at the same instant: after the process starts, each one returns its first frame at its own wall-clock time (IMU at 0.04 s, EMG at 0.12 s, EEG amplifier at 0.31 s, …). This demo simulates such streams and aligns them to the master clock **before** adding data:

1. **t_all_started** — the wall-clock moment every sensor has delivered its first frame (the max of all first-frame times).
2. **t = 0** — the master clock reference sensor's first frame at/after `t_all_started`; its earlier frames are discarded.
3. **timestamp_start** — for every other sensor, the delta between its next frame (first frame at/after t = 0) and the t = 0 frame; set it in the feature spec so the stored timestamps stay phase-aligned with the master clock.
4. **Record** everything from t = 0 onward. The episode ends when the master clock stops, but each sensor keeps streaming until its **next frame after the master's last one** — that trailing sample keeps the last frame's window full.

Simulated startup latencies:

| Sensor | Rate | First frame (wall clock) |
|--------|------|--------------------------|
| `observation.imu` | 1000 Hz | 0.04 s |
| `observation.emg` | 100 Hz | 0.12 s |
| `observation.state` (master clock reference) | 30 Hz | 0.18 s |
| `observation.temperature` | 10 Hz | 0.22 s |
| `action` | 30 Hz | 0.24 s |
| `observation.eeg` | 256 Hz | 0.31 s |

The demo computes `t_all_started = 0.31 s`, takes the master's next frame as t = 0, derives each sensor's `timestamp_start`, writes the dataset, verifies on disk that every feature's first stored timestamp equals its `timestamp_start` and that its last sample lands past the master stop, then reads back frame 0, a mid-episode frame, and the last frame.

Two consequences worth knowing:

- **Frame 0's backward window is empty.** Every sensor's first stored frame sits at its own `timestamp_start` > 0 and earlier data is discarded by design, so a window like `(-0.033, 0.0)` has nothing to return on the first frame(s).
- **Tolerance must cover half the sensor's period.** The alignment checker compares each master frame against the sensor's *nearest* sample, whose distance is at most half a period (a 10 Hz sensor can be up to 50 ms away) plus jitter — e.g. `tolerance_s: 0.06` for the 10 Hz temperature sensor here.

```bash
python -m examples.05async_startup_example
```

### 4.8 Demo 7: Audio (06audio_example.py)

One camera + state + a simulated microphone, showing the audio storage path end to end:

- `dtype: "audio"` features define `shape` as one chunk of samples `(n_samples,)` (mono) or `(n_samples, channels)` and require `sample_rate`
- one `add_frame` per master frame stores one chunk; `save_episode()` concatenates all chunks into one PCM16 WAV per episode under `audios/`, plus a timestamp parquet index (`timestamp | episode_index | num_samples`) next to the other per-feature parquets
- reading `ds[i]` in nearest mode resolves the chunk nearest the master timestamp through the parquet index and returns it whole — the atomic unit, analogous to one video frame; with a `window` (spec, `window_overrides`, or a direct `read(..., window=(start, end))` argument) it returns every sample whose time lies in `(t + start, t + end]`, sliced with sample precision across chunk boundaries and concatenated
- the alignment checker reports WAV integrity: chunk count vs index rows, sample count vs summed index

```bash
python -m examples.06audio_example
```

### 4.9 Visualizer Controls

**Visualizer** (`python -m scripts.visualize --episode 0`):

![Visualizer screenshot](examples/visualize.png)

```
space    — play / pause
← →      — step frame backward / forward
↑ ↓      — faster / slower playback
w s      — widen / shrink the window
q / Esc  — quit
```


## 5. Core Concepts

### 5.1 Per-Feature Independent Storage

Storage layout:

```
data/
└── chunk-000/
    └── episode_000000/
        ├── master_index.parquet                # timestamp | frame_index | task_index
        ├── observation.imu.parquet             # 1000Hz · 1980 rows · 6 cols
        ├── observation.eeg.parquet             # 256Hz · 512 rows · 8 cols
        ├── observation.state.parquet           # 30Hz · 60 rows · 7 cols
        └── observation.images.head_rgb.parquet # 30Hz · 60 rows · timestamps only

videos/
└── chunk-000/
    └── observation.images.head_rgb/
        └── episode_000000.mp4

audios/
└── chunk-000/
    └── observation.audio.mic/
        └── episode_000000.wav
```

Every non-video feature gets one parquet with a uniform column layout:

```
timestamp (float64) | episode_index (int64) | col_0 (float32) | ... | col_D (float32)
```

- camera-rate features (state, action): one row per frame, rows = frames
- high-rate features (imu, eeg): multiple samples between frames, rows = rate × duration
- camera features: timestamp parquet + MP4 video
- audio features: timestamp parquet (`timestamp | episode_index | num_samples`, one row per added chunk) + one WAV per episode; chunks are concatenated in timestamp order

### 5.2 Task as the Master Clock

The whole dataset is organized around **task calls as frame boundaries**: `add_frame("task", ...)` declares "the current frame ends here, the next one begins".

- **Frame** = all feature data written between two task calls. `add_frame("task", ...)` auto-generates the frame timestamp: `t = frame_index / fps` (override with `timestamp=` if needed)
- **Master clock** = the frame sequence. Reading `ds[i]` first looks up the i-th frame timestamp t in `master_index`; every feature is then aligned around t (window / nearest neighbor, see 5.3)
- **Each feature's timestamps are generated independently** of the master clock: one counter per feature, `ts = timestamp_start + counter × (1 / fps)` on each `add_frame`; counters reset after `save_episode()`
- **No master_feature to configure** — task is the master clock, and features of any sampling rate align to it

```python
# standard call order per frame: high-rate first → task bounds the frame → low-rate last
ds.add_frame("observation.imu", ...)   # high-rate sensor, written multiple times between frames
ds.add_frame("observation.eeg", ...)
ds.add_frame("task", "pick_up")        # ← frame boundary: frame_index++, master-clock timestamp
ds.add_frame("observation.state", ...) # camera-rate sensor, once per frame
ds.add_frame("observation.images.cam", image)
```

### 5.3 Window Parameter

The master-clock frame timestamp is the query anchor, but a feature's readings don't necessarily land on a frame — `window` defines "how to aggregate that feature's readings around the frame timestamp":

| `window` value | Behavior | Typical use |
|------------|------|---------|
| `None` (default) | nearest neighbor: return the single closest reading | camera-rate sensors (state, action); audio: one whole chunk |
| `"interpolate"` | linear interpolation at the query time | mid-rate sensors (finger pose) |
| `(start_s, end_s)` | range query: return all readings in `(t+start, t+end]` | high-rate sensors (IMU, EEG, EMG); audio: sample-precise slice + concat |

```python
# IMU at 1000Hz, 30fps master → ~33 readings per frame
"window": (-0.033, 0.0)   # (t-33ms, t]

# wider window for context
"window": (-0.066, 0.0)   # (t-66ms, t], ~66 readings

# override at read time
ds = HumancoinLeRobotDataset(
    repo_id="my_dataset", root="data/",
    window_overrides={"observation.imu": (-0.100, 0.0)},
)
```

**Readings returned per feature when querying t = 0.333s (frame 10):**

![Readings returned per feature when querying t = 0.333s (frame 10)](assets/window_query_en.png)

Note that temperature (10Hz) has no sample at 0.333 — its sampling grid is 0.3, 0.4, …, so nearest neighbor falls back to the reading at t=0.300.


## 6. API Reference

### Creating a Dataset

```python
ds = HumancoinLeRobotDataset.create(
    repo_id="my_dataset",
    fps=30,
    features={...},          # see Feature Definition Format below
)
```

| Parameter | Type | Default | Description |
|------|------|--------|------|
| `repo_id` | `str` | — | dataset name, directory name under `root` |
| `fps` | `float` | — | master-clock frame rate; frame timestamp = `frame_index / fps` |
| `features` | `dict[str, dict]` | — | feature definitions, format below |
| `root` | `str \| Path` | `~/.cache/huggingface/lerobot/<repo_id>` | storage root directory |
| `robot_type` | `str` | `None` | robot type, metadata only |
| `use_videos` | `bool` | `True` | whether `dtype="video"` features are encoded to MP4 |
| `image_writer_processes` | `int` | `0` | image writer subprocess count (0 = write in main process) |
| `image_writer_threads` | `int` | `0` | image writer thread count |
| `video_backend` | `str` | `None` | video backend (e.g. `"pyav"`), `None` auto-selects |
| `batch_encoding_size` | `int` | `1` | batch-encode videos every N episodes |
| `window_overrides` | `dict` | `None` | override each feature's window (same format as at read time) |

### Feature Definition Format

```python
features = {
    "feature_key": {
        "dtype": "video" | "audio" | "float32" | "int64",  # required
        "shape": (H, W, C) | (N,) | (D,),         # required
        "names": ["col_0", ...],                  # required
        # optional:
        "fps": 1000,              # sampling rate of this feature
        "window": (-0.033, 0.0),  # query window
        "timestamp_start": 0.1,   # timestamp start offset
        "tolerance_s": 0.002,     # alignment tolerance
        "sample_rate": 16000,     # audio only: sample rate of the WAV
    }
}
```

| Field | Type | Required | Description |
|------|------|------|------|
| `dtype` | `str` | ✓ | storage type: `"float32"` (continuous), `"int64"` (discrete/id), `"video"` (camera frames), `"audio"` (microphone chunks) |
| `shape` | `tuple` | ✓ | shape of a single reading: `(D,)` for numeric features, `(H, W, C)` for video, `(n_samples,)` or `(n_samples, channels)` for audio |
| `names` | `list[str]` | ✓ | column name per dimension (used as parquet column names) |
| `fps` | `float` | optional | sampling rate of this feature; timestamps are auto-generated as `timestamp_start + counter × 1/fps`; falls back to master-clock `fps` if omitted |
| `window` | `None \| "interpolate" \| (start, end)` | `None` | how readings are aggregated at query time (see 5.3) |
| `timestamp_start` | `float` | `0.0` | timestamp start offset, e.g. the sensor only starts recording at t=0.1s |
| `tolerance_s` | `float` | `None` | alignment tolerance: after `save_episode()`, per-frame nearest-reading gaps are checked and violations are logged to `meta/alignment_check.jsonl` |
| `sample_rate` | `int` | audio only | sample rate of the WAV file (e.g. 16000); required for `dtype="audio"` |

Conventions and tips:

- feature keys follow LeRobot naming and **must not contain `/`** (e.g. `observation.imu`, `action`)
- `dtype="video"` features only store a timestamp parquet; images are encoded as MP4 under `videos/`
- `dtype="audio"` features store one WAV per episode under `audios/` plus a timestamp parquet; each `add_frame` passes one chunk of float32 samples in [-1, 1] (`(n_samples,)` mono or `(n_samples, channels)`). Nearest read returns one whole chunk; a `window` `(start, end)` read returns the samples in `(t+start, t+end]` concatenated (timing contract: a chunk's first sample sits at its label, samples advance at exactly `1/sample_rate`, real dropouts shorten the segment rather than stretch it)
- always give high-rate features both `fps` and `window` — otherwise the nearest-neighbor query returns a single reading and the high-rate information is lost

### Writing

```python
ds.add_frame(key: str, value: np.ndarray | torch.Tensor | str,
             timestamp: float | None = None)
ds.save_episode()
```

- `key`: `"task"` or a defined feature key
- `value`: `np.ndarray` / `torch.Tensor` for numeric features (shape must match `shape`); `np.ndarray` (H, W, C) for video features; `str` task label for task
- `timestamp`: `None` auto-generates — task uses the master clock `len(records) / fps`, other features use their own counter `timestamp_start + counter × 1/fps`; pass a `float` to set it manually

A complete write loop:

```python
FPS = 30
imu_per_frame = 1000 // FPS   # 33
eeg_per_frame = 256 // FPS    # 8

for frame in range(60):                        # 2 seconds / 30fps
    for _ in range(imu_per_frame):             # high-rate: multiple writes per frame
        ds.add_frame("observation.imu", imu_sample)
    for _ in range(eeg_per_frame):
        ds.add_frame("observation.eeg", eeg_sample)

    ds.add_frame("task", "reach_target")       # ← frame boundary

    ds.add_frame("observation.state", joint_state)    # low-rate: once per frame
    ds.add_frame("action", action_cmd)
    ds.add_frame("observation.images.cam", camera_image)

ds.save_episode()    # flush all buffers, run alignment/video checks, reset counters
```

### Reading

```python
ds = HumancoinLeRobotDataset(
    repo_id="my_dataset", root="data/",
    episodes=[0, 2],                                    # load only episodes 0, 2
    image_transforms=my_transforms,                      # torchvision transforms
    delta_timestamps={"observation.state": [-0.1, 0, 0.1]},  # multi-frame query
    window_overrides={"observation.imu": (-0.066, 0)},   # override window
    video_backend="pyav",
)
item = ds[10]
```

| Parameter | Type | Default | Description |
|------|------|--------|------|
| `repo_id` | `str` | — | dataset name |
| `root` | `str \| Path` | `~/.cache/huggingface/lerobot/<repo_id>` | storage root directory |
| `episodes` | `list[int]` | `None` | load only the given episode indices |
| `image_transforms` | callable | `None` | image transforms (torchvision style) |
| `delta_timestamps` | `dict[str, list[float]]` | `None` | multi-frame query: returns `{key: master frame + each offset frame}`, e.g. `{"observation.state": [-0.1, 0, 0.1]}` |
| `tolerance_s` | `float` | `1e-4` | frame-alignment tolerance for `delta_timestamps` |
| `video_backend` | `str` | `None` | video decoding backend (e.g. `"pyav"`) |
| `batch_encoding_size` | `int` | `1` | (write side) video batch-encoding size |
| `window_overrides` | `dict` | `None` | override each feature's window at read time, e.g. `{"observation.imu": (-0.066, 0)}` |

Depth video reading: features with `info["video.is_depth_map"] = true` and HWC `shape = [H, W, 1]` are decoded directly with PyAV. Reads return native samples as a `(1, H, W)` tensor (e.g. `uint16` for `gray12le`), without RGB conversion or division by 255. Units and invalid values remain those of the stored data. This applies to reading; depth-specific encoding is not implemented.

### Return Structure

```python
item = ds[10]
# {
#     "timestamp": tensor(0.3333),
#     "frame_index": tensor(10),
#     "episode_index": tensor(0),
#     "index": tensor(10),
#     "task_index": tensor(0),
#     "task": "reach_target",
#
#     # window feature: all readings + timestamps
#     "observation.imu": tensor([33, 6]),
#     "observation.imu_timestamps": tensor([33]),
#
#     # nearest feature: a single vector
#     "observation.state": tensor([7]),
#     "observation.temperature": tensor([1]),
#
#     # video feature: CHW format
#     "observation.images.head_rgb": tensor([3, 480, 640]),
#
#     # audio feature: one chunk of samples, float32 in [-1, 1]
#     "observation.audio.mic": tensor([533]),
#
#     # audio feature with window (-0.033, 0): samples in (t-33ms, t], concatenated
#     "observation.audio.mic": tensor([528]),
# }
```

## 7. Architecture

```mermaid
flowchart TB
    subgraph Entry["entry layer"]
        DS["HumancoinLeRobotDataset<br/>dataset.py · orchestrator<br/>(inherits LeRobotDataset)"]
    end

    subgraph Core["core layer"]
        IX["MasterIndex<br/>index.py"]
        MD["HumancoinDatasetMetadata<br/>metadata.py"]
    end

    subgraph Features["feature layer"]
        PF["ParquetFeature<br/>parquet.py"]
        VF["VideoFeature<br/>video.py"]
        AuF["AudioFeature<br/>audio.py"]
    end

    subgraph Validate["validation layer"]
        CK["DatasetChecker<br/>checker.py"]
    end

    subgraph Store["storage layer"]
        PQ[("parquet files<br/>data/chunk-*/episode_*/")]
        MP4[("MP4 videos<br/>videos/chunk-*/")]
        WAV[("WAV audio<br/>audios/chunk-*/")]
        META[("metadata<br/>meta/")]
    end

    DS -->|"frame index I/O"| IX
    DS -->|"info.json / metadata"| MD
    DS -->|"write readings / window query"| PF
    DS -->|"write images / decode videos"| VF
    DS -->|"write chunks / read WAV"| AuF
    DS -->|"post-save validation"| CK
    PF -->|"save / read"| PQ
    VF -->|"encode MP4"| MP4
    VF -->|"timestamp table"| PQ
    AuF -->|"write WAV"| WAV
    AuF -->|"timestamp table"| PQ
    IX -->|"save"| PQ
    MD -->|"save"| META
    CK -->|"validation report"| META

    style DS fill:#e8eaf6,stroke:#4fc3f7,stroke-width:2px,color:#000
    style IX fill:#ffebee,stroke:#ef9a9a,color:#000
    style MD fill:#fafafa,stroke:#9e9e9e,color:#000
    style PF fill:#e8f5e9,stroke:#66bb6a,color:#000
    style VF fill:#f3e5f5,stroke:#ce93d8,color:#000
    style AuF fill:#e8f5e9,stroke:#66bb6a,color:#000
    style CK fill:#fff3e0,stroke:#ffcc80,color:#000
    style PQ fill:#fafafa,stroke:#9e9e9e,color:#000
    style MP4 fill:#fafafa,stroke:#9e9e9e,color:#000
    style WAV fill:#fafafa,stroke:#9e9e9e,color:#000
    style META fill:#fafafa,stroke:#9e9e9e,color:#000
```

**File layout:**

```
src/humancoin_lerobot/
├── dataset.py      (341 lines)  orchestrator — add_frame / save_episode / __getitem__
├── audio.py        (185 lines)  AudioFeature — buffer chunks, write WAV + timestamp parquet
├── parquet.py      (168 lines)  ParquetFeature — write buffer, save parquet, load, window query
├── video.py        (168 lines)  VideoFeature — save PNG, encode MP4, decode frames
├── index.py        (127 lines)  MasterIndex — frame index I/O
├── metadata.py     (123 lines)  HumancoinDatasetMetadata — info.json management
├── checker.py      (162 lines)  DatasetChecker — video frame count + audio integrity + timestamp alignment checks
└── utils.py        (236 lines)  constants + info.json creation/validation
```

**Design principles:**

1. **One object per feature**: `ParquetFeature`, `VideoFeature` and `AudioFeature` each encapsulate their buffer, file paths, and read/write logic. The Dataset is only an orchestrator.

2. **Symmetric read/write**: every Feature provides both `add/save` (write) and `load/query` (read).

3. **Deterministic paths**: file paths depend only on `episode_index` and `feature_key`; no lookups in `episodes.jsonl`.

4. **Inherit, don't rewrite**: inherit `LeRobotDataset` for `start_image_writer`, `_save_image` and other basics; only 4 core methods are overridden.

## 8. Comparison with Native LeRobotDataset

| Aspect | Native LeRobotDataset | humancoin_lerobot |
|------|-------------------|------------|
| Storage | 1 merged parquet | N per-feature parquets |
| Multi-rate sensors | pre-resampling required | native rates, independent storage |
| Time alignment | `tolerance_s` tolerance matching | window mechanism (range / interpolate / nearest) |
| Master clock | camera frame | `add_frame("task", ...)` calls |
| Video encoding | unified batch encoding in `save_episode` | independent per VideoFeature |
| Metadata | `episodes.jsonl` records everything | only basic frame info; paths computed deterministically |
| per-feature fps | not supported | independent `fps` per feature |
| per-feature tolerance | global `tolerance_s` | independent `tolerance_s` per feature |
| Audio storage | not supported | WAV per episode + timestamp parquet index, nearest-chunk read |
| Write API | `add_frame(frame_dict, task)` | `add_frame(key, value)` per field |
| Alignment check | none | automatic `alignment_check.jsonl` |
| Video check | none | automatic frame count/duration check |

## 9. Acknowledgements

This project builds on the following open-source projects and work:

- **[LeRobot](https://github.com/huggingface/lerobot)** — the base dataset framework. This project inherits the API design of `LeRobotDataset` and `LeRobotDatasetMetadata` and reuses its video encoding and image writing infrastructure. Thanks to the Hugging Face team for their contributions to the open-source robot learning ecosystem.

- **[PyArrow](https://arrow.apache.org/docs/python/)** — high-performance columnar storage, the basis of per-feature parquet I/O.

- **[PyAV](https://pyav.org/)** — Python bindings for FFmpeg, powering video encoding (AV1) and decoding.

- **[Matplotlib](https://matplotlib.org/)** — visualization.

- The [Neuropixels](https://www.neuropixels.org/) data format and the BIDS standard from the **[Allen Institute for Neural Dynamics](https://alleninstitute.org/division/neural-dynamics/)** — inspiration for organizing multimodal neural data.

Thanks to all colleagues for feedback and discussions.


## Building and checking a release

```bash
pip install -e ".[dev]"
python -m unittest discover -s tests -v
python -m build
python -m twine check dist/*
```

The wheel contains the `humancoin_lerobot` package and `humancoin-download` command. The repository's `scripts/` and `examples/` remain source tools. Private/gated downloads require both a valid token and access approval on the chosen platform; a token alone does not grant access. PyPI publishing is a separate step.
