Metadata-Version: 2.4
Name: macrodata-refiner
Version: 0.3.8
Summary: Refiner by Macrodata Labs, a data processing framework for Machine Learning large scale datasets
Author: Macrodata Labs
License-Expression: Apache-2.0
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aiohttp
Requires-Dist: cloudpickle==3.1.2
Requires-Dist: fsspec[http]
Requires-Dist: httpx
Requires-Dist: loguru
Requires-Dist: numpy
Requires-Dist: orjson
Requires-Dist: packaging
Requires-Dist: pyarrow
Requires-Dist: msgspec>=0.20.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: video
Requires-Dist: av; extra == "video"
Requires-Dist: pillow; extra == "video"
Provides-Extra: hf
Requires-Dist: huggingface-hub>=1.4.1; extra == "hf"
Requires-Dist: hf>=1.7.1; extra == "hf"
Provides-Extra: datasets
Requires-Dist: macrodata-refiner[hf]; extra == "datasets"
Requires-Dist: datasets>=3.0.0; extra == "datasets"
Provides-Extra: hand-tracking
Requires-Dist: macrodata-refiner[hf]; extra == "hand-tracking"
Requires-Dist: macrodata-refiner[video]; extra == "hand-tracking"
Requires-Dist: ego-vision[models]>=0.1.25; extra == "hand-tracking"
Provides-Extra: text
Requires-Dist: warcio; extra == "text"
Provides-Extra: hdf5
Requires-Dist: h5py; extra == "hdf5"
Provides-Extra: zarr
Requires-Dist: zarr<3,>=2.18; extra == "zarr"
Requires-Dist: numcodecs<0.16; extra == "zarr"
Provides-Extra: mcap
Requires-Dist: av; extra == "mcap"
Requires-Dist: mcap; extra == "mcap"
Requires-Dist: mcap-protobuf-support; extra == "mcap"
Requires-Dist: mcap-ros2-support; extra == "mcap"
Requires-Dist: pillow; extra == "mcap"
Provides-Extra: s3
Requires-Dist: s3fs; extra == "s3"
Provides-Extra: lance
Requires-Dist: pylance<11,>=4.0.1; extra == "lance"
Provides-Extra: gcs
Requires-Dist: gcsfs; extra == "gcs"
Provides-Extra: tensorflow
Requires-Dist: tensorflow; extra == "tensorflow"
Provides-Extra: tfds
Requires-Dist: macrodata-refiner[tensorflow]; extra == "tfds"
Requires-Dist: tensorflow-datasets; extra == "tfds"
Provides-Extra: testing
Requires-Dist: macrodata-refiner[all]; extra == "testing"
Requires-Dist: pytest>=8.0.0; extra == "testing"
Requires-Dist: pytest-cov>=5.0.0; extra == "testing"
Provides-Extra: all
Requires-Dist: macrodata-refiner[datasets]; extra == "all"
Requires-Dist: macrodata-refiner[hdf5]; extra == "all"
Requires-Dist: macrodata-refiner[hf]; extra == "all"
Requires-Dist: macrodata-refiner[mcap]; extra == "all"
Requires-Dist: macrodata-refiner[lance]; extra == "all"
Requires-Dist: macrodata-refiner[video]; extra == "all"
Requires-Dist: macrodata-refiner[zarr]; extra == "all"
Requires-Dist: macrodata-refiner[text]; extra == "all"
Requires-Dist: macrodata-refiner[s3]; extra == "all"
Requires-Dist: macrodata-refiner[gcs]; extra == "all"
Requires-Dist: macrodata-refiner[tfds]; extra == "all"
Dynamic: license-file

<p align="center">
  <img src="https://macrodata.co/logo.svg" alt="Macrodata" width="180">
</p>

<h1 align="center">Macrodata Refiner</h1>

Refiner is Macrodata's open-source Python library for reading, transforming, and
writing robotics datasets.

It provides one pipeline model for working with robot episodes, frames, videos,
metadata, and model-based processing. Use it to convert formats, transform data,
run inference, and write structured outputs on your own infrastructure.

This repository also includes open-source reference versions of some of the
pipelines we develop at Macrodata. They are useful starting points, but they are
not the full pipelines we adapt, evaluate, and run for customers. If you want to
see what those pipelines can do with your data,
[send us a representative sample](https://macrodata.co/contact).

## Quickstart

Install:

```bash
pip install macrodata-refiner
```

This gives you:

- the Python package as `refiner`
- the CLI as `macrodata`

Launch a local pipeline:

```python
import refiner as mdr

def add_preview(row):
    return row.update(
        preview=" ".join(row["text"].split()[:20]),
    )

(
    mdr.read_jsonl("input/*.jsonl")
    .filter(mdr.col("lang") == "en")
    .with_columns(
        text=mdr.col("text").str.strip(),
        text_len=mdr.col("text").str.len(),
    )
    .map(add_preview)
    .write_parquet("s3://my-bucket/english-cleanup/")
    .launch_local(
        name="english-cleanup",
        num_workers=2,
    )
)
```

## Batteries included

- a consistent row and episode model for robot trajectories, frames, videos,
  metadata, tasks, and statistics
- readers and writers for LeRobot, HDF5, Zarr, MCAP, Parquet, JSONL, and other
  common data formats
- composable transforms and model inference for converting and enriching data
- open-source reference operations for motion trimming, subtask annotation,
  reward scoring, and hand tracking
- access to storage backends supported by `fsspec`, including S3, GCP, and
  Hugging Face
- in-process debugging and local multi-worker execution

## Docs

Start here:

- [Docs index](docs/index.md)
- [Quickstart](docs/quickstart.md)
- [Running pipelines](docs/running-pipelines/index.md)

Build a dataset:

- [Reading data](docs/reading-data/index.md)
- [Episode data](docs/episode-data/index.md)
- [Transforms](docs/transforms/index.md)
- [Episode operations](docs/episode-operations/index.md)
- [Writing data](docs/writing-data/index.md)
- [Examples](docs/examples/index.md)
- [Reference](docs/reference/index.md)

## Community

- join the Macrodata Discord: https://discord.gg/S8kZtmBR2x
