Metadata-Version: 2.4
Name: macrodata-refiner
Version: 0.3.2
Summary: Refiner by Macrodata Labs, a data processing framework for Machine Learning large scale datasets
Author: Macrodata Labs
License-Expression: Apache-2.0
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: aiohttp
Requires-Dist: cloudpickle==3.1.2
Requires-Dist: fsspec[http]
Requires-Dist: httpx
Requires-Dist: loguru
Requires-Dist: numpy
Requires-Dist: orjson
Requires-Dist: packaging
Requires-Dist: pyarrow
Requires-Dist: msgspec>=0.20.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: video
Requires-Dist: av; extra == "video"
Requires-Dist: pillow; extra == "video"
Provides-Extra: hf
Requires-Dist: huggingface-hub>=1.4.1; extra == "hf"
Requires-Dist: hf>=1.7.1; extra == "hf"
Provides-Extra: datasets
Requires-Dist: macrodata-refiner[hf]; extra == "datasets"
Requires-Dist: datasets>=3.0.0; extra == "datasets"
Provides-Extra: hand-tracking
Requires-Dist: macrodata-refiner[hf]; extra == "hand-tracking"
Requires-Dist: macrodata-refiner[video]; extra == "hand-tracking"
Requires-Dist: ego-vision[models]>=0.1.25; extra == "hand-tracking"
Provides-Extra: text
Requires-Dist: warcio; extra == "text"
Provides-Extra: hdf5
Requires-Dist: h5py; extra == "hdf5"
Provides-Extra: zarr
Requires-Dist: zarr<3,>=2.18; extra == "zarr"
Requires-Dist: numcodecs<0.16; extra == "zarr"
Provides-Extra: mcap
Requires-Dist: av; extra == "mcap"
Requires-Dist: mcap; extra == "mcap"
Requires-Dist: mcap-protobuf-support; extra == "mcap"
Requires-Dist: mcap-ros2-support; extra == "mcap"
Requires-Dist: pillow; extra == "mcap"
Provides-Extra: s3
Requires-Dist: s3fs; extra == "s3"
Provides-Extra: gcs
Requires-Dist: gcsfs; extra == "gcs"
Provides-Extra: tensorflow
Requires-Dist: tensorflow; extra == "tensorflow"
Provides-Extra: tfds
Requires-Dist: macrodata-refiner[tensorflow]; extra == "tfds"
Requires-Dist: tensorflow-datasets; extra == "tfds"
Provides-Extra: testing
Requires-Dist: macrodata-refiner[all]; extra == "testing"
Requires-Dist: pytest>=8.0.0; extra == "testing"
Requires-Dist: pytest-cov>=5.0.0; extra == "testing"
Provides-Extra: all
Requires-Dist: macrodata-refiner[datasets]; extra == "all"
Requires-Dist: macrodata-refiner[hdf5]; extra == "all"
Requires-Dist: macrodata-refiner[hf]; extra == "all"
Requires-Dist: macrodata-refiner[mcap]; extra == "all"
Requires-Dist: macrodata-refiner[video]; extra == "all"
Requires-Dist: macrodata-refiner[zarr]; extra == "all"
Requires-Dist: macrodata-refiner[text]; extra == "all"
Requires-Dist: macrodata-refiner[s3]; extra == "all"
Requires-Dist: macrodata-refiner[gcs]; extra == "all"
Requires-Dist: macrodata-refiner[tfds]; extra == "all"
Dynamic: license-file

<p align="center">
  <img src="https://macrodata.co/logo.svg" alt="Macrodata" width="180">
</p>

<h1 align="center">Macrodata Refiner</h1>

Refiner is an open-source engine for turning raw robotics and multimodal data into **high-quality datasets** for model training.

It gives training-data teams one pipeline model for multimodal data, robotics
workflows, and model-based processing.

It also plugs into the Macrodata platform, which gives you visibility into what is happening to your data while pipelines run: job and shard lifecycle, logs, metrics, manifests, and pipeline behavior. The same code can run locally for development and then scale out through Macrodata's elastic serverless cloud.

## Quickstart

Install:

```bash
pip install macrodata-refiner
```

Create a Macrodata API key:

- https://macrodata.co/settings/api-keys

Log in:

```bash
macrodata login
```

### Cloud example

Launch a robotics pipeline on Macrodata Cloud.

This requires a valid API key.

```python
import refiner as mdr

(
    mdr.read_lerobot("hf://datasets/macrodata/aloha_static_battery_ep005_009")
    .map(
        mdr.robotics.motion_trim(
            threshold=0.001,
            pad_frames=5,
        )
    )
    .write_lerobot("hf://buckets/acme-robotics/aloha_motion")
    .launch_cloud(
        name="motion_trim",
        num_workers=4,
    )
)
```

Need cloud GPUs? See [Resources, GPUs, and Services](docs/running-pipelines/resources-gpus-and-services.md).

### Local example

Launch a local pipeline:

```python
import refiner as mdr

def add_preview(row):
    return row.update(
        preview=" ".join(row["text"].split()[:20]),
    )

(
    mdr.read_jsonl("input/*.jsonl")
    .filter(mdr.col("lang") == "en")
    .with_columns(
        text=mdr.col("text").str.strip(),
        text_len=mdr.col("text").str.len(),
    )
    .map(add_preview)
    .write_parquet("s3://my-bucket/english-cleanup/")
    .launch_local(
        name="english-cleanup",
        num_workers=2,
    )
)
```

`pip install` gives you:

- the Python package as `refiner`
- the CLI as `macrodata`

## Batteries included

- training-data-first pipeline primitives instead of generic ETL abstractions
- multimodal processing, with robotics support today
- built-in readers, transforms, sinks, and runtime machinery for common dataset work
- access to any storage backend supported by `fsspec` (S3, GCP, Hugging Face, etc.)
- local execution for development and elastic cloud execution for large runs
- built-in observability through the Macrodata platform for job state, logs, metrics, and manifests

## Docs

Start here:

- [Docs index](docs/index.md)
- [Quickstart](docs/quickstart.md)
- [Running pipelines](docs/running-pipelines/index.md)

Build a dataset:

- [Reading data](docs/reading-data/index.md)
- [Episode data](docs/episode-data/index.md)
- [Transforms](docs/transforms/index.md)
- [Episode operations](docs/episode-operations/index.md)
- [Writing data](docs/writing-data/index.md)
- [Examples](docs/examples/index.md)

Operate jobs:

- [Platform](docs/platform/index.md)
- [CLI](docs/cli/index.md)
- [Reference](docs/reference/index.md)

## Community

- join the Macrodata Discord: https://discord.gg/S8kZtmBR2x
