Metadata-Version: 2.4
Name: turboloader
Version: 2.27.0
Summary: High-performance ML data loading: C++20 core with SIMD-accelerated transforms, pipe-operator composition, TBL v2 format, and a PyTorch-compatible DataLoader.
Author: Arnav Jain
Maintainer: Arnav Jain
License: MIT
Project-URL: Homepage, https://github.com/ALJainProjects/TurboLoader
Project-URL: Documentation, https://github.com/ALJainProjects/TurboLoader/blob/main/README.md
Project-URL: Repository, https://github.com/ALJainProjects/TurboLoader
Project-URL: Bug Tracker, https://github.com/ALJainProjects/TurboLoader/issues
Project-URL: Changelog, https://github.com/ALJainProjects/TurboLoader/blob/main/CHANGELOG.md
Project-URL: Discussions, https://github.com/ALJainProjects/TurboLoader/discussions
Keywords: machine-learning,deep-learning,data-loading,pytorch,performance,simd,imagenet,computer-vision
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: AUTHORS.md
Requires-Dist: numpy>=1.19.0
Provides-Extra: torch
Requires-Dist: torch>=1.8.0; extra == "torch"
Provides-Extra: dev
Requires-Dist: pytest>=6.0; extra == "dev"
Requires-Dist: black>=21.0; extra == "dev"
Requires-Dist: flake8>=3.9; extra == "dev"
Requires-Dist: mypy>=0.900; extra == "dev"
Requires-Dist: build>=0.7.0; extra == "dev"
Requires-Dist: twine>=3.4.0; extra == "dev"
Provides-Extra: benchmarks
Requires-Dist: Pillow>=8.0.0; extra == "benchmarks"
Requires-Dist: tqdm>=4.60.0; extra == "benchmarks"
Requires-Dist: matplotlib>=3.3.0; extra == "benchmarks"
Requires-Dist: psutil>=5.8.0; extra == "benchmarks"
Provides-Extra: all
Requires-Dist: turboloader[benchmarks,dev]; extra == "all"
Dynamic: license-file

# TurboLoader

**Production-Ready ML Data Loading Library**

[![PyPI version](https://img.shields.io/pypi/v/turboloader.svg)](https://pypi.org/project/turboloader/)
[![Tests](https://github.com/ALJainProjects/TurboLoader/actions/workflows/test.yml/badge.svg)](https://github.com/ALJainProjects/TurboLoader/actions/workflows/test.yml)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![C++20](https://img.shields.io/badge/C%2B%2B-20-blue.svg)](https://en.wikipedia.org/wiki/C%2B%2B20)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

---

## Overview

TurboLoader is a high-performance data loading library for machine learning workflows. Built with C++20 and featuring Python bindings, it provides efficient data loading with SIMD-accelerated transforms, custom binary formats, and distributed training support.

### Core Features

- **Decoded Tensor Caching** - `FastDataLoader(..., cache_decoded=True)` keeps decoded arrays in RAM so later epochs skip decoding
- **Multiple Loader Types** - FastDataLoader, MemoryEfficientDataLoader, standard DataLoader
- **Distributed Training Support** - Multi-node data loading with deterministic sharding
- **SIMD-Accelerated Transforms** - 19 vectorized transforms using AVX2/AVX-512/NEON
- **TBL v2 Binary Format** - Custom format with LZ4 compression for reduced storage
- **Framework Integration** - Seamless support for PyTorch, TensorFlow, and JAX
- **Memory-Mapped I/O** - Zero-copy file access for improved throughput
- **Lock-Free Queues** - Concurrent data structures for efficient multi-threading
- **GPU JPEG Decoding** - Optional NVIDIA nvJPEG support for accelerated decoding

---

## Installation

### From PyPI (Recommended)

```bash
pip install turboloader
```

### From Source

```bash
git clone https://github.com/ALJainProjects/TurboLoader.git
cd TurboLoader
pip install -e .
```

### System Requirements

- **Python:** 3.10 or higher
- **Compiler:** C++20 capable (GCC 10+, Clang 12+, MSVC 19.29+)
- **OS:** macOS, Linux, Windows

#### Optional Dependencies

Install for enhanced performance:

```bash
# macOS
brew install jpeg-turbo libpng libwebp lz4

# Ubuntu/Debian
sudo apt-get install libjpeg-turbo8-dev libpng-dev libwebp-dev liblz4-dev
```

---

## Quick Start

### Basic Usage

```python
import turboloader

# Create DataLoader
loader = turboloader.DataLoader(
    'imagenet.tar',
    batch_size=128,
    num_workers=8
)

# Iterate over batches. Each sample is a dict:
#   {'image': np.ndarray (H, W, C), 'filename': str, 'index': int,
#    'width': int, 'height': int, 'channels': int}
for batch in loader:
    for sample in batch:
        image = sample['image']      # NumPy array (H, W, C)
        name = sample['filename']    # source path within the archive
        # Train your model...
```

> **Need (image, label) tuples like `torch.utils.data.DataLoader`?** Use
> `PyTorchCompatibleLoader`, which derives labels from the folder structure
> (ImageFolder-style). The base `DataLoader` does not attach labels.

### With Transforms

```python
import turboloader

# Create transforms
resize = turboloader.Resize(224, 224)
normalize = turboloader.ImageNetNormalize()
flip = turboloader.RandomHorizontalFlip(p=0.5)

# Apply transforms
loader = turboloader.DataLoader('data.tar', batch_size=64, num_workers=8)

for batch in loader:
    for sample in batch:
        img = sample['image']
        img = resize.apply(img)
        img = flip.apply(img)
        img = normalize.apply(img)
        # Ready for training
```

### PyTorch Integration

```python
import turboloader
import torch

loader = turboloader.DataLoader('imagenet.tar', batch_size=64, num_workers=8)

# Convert to PyTorch tensors
to_tensor = turboloader.ToTensor(
    format=turboloader.TensorFormat.PYTORCH_CHW
)

for batch in loader:
    images = []
    for sample in batch:
        img = to_tensor.apply(sample['image'])
        images.append(torch.from_numpy(img))

    batch_tensor = torch.stack(images)
    # Train model...
```

### Distributed Training

```python
import turboloader
import torch.distributed as dist

# Initialize distributed training
dist.init_process_group(backend='nccl')

# Create loader with distributed support
loader = turboloader.DataLoader(
    data_path="/data/imagenet.tar",
    batch_size=64,
    num_workers=4,
    shuffle=True,
    enable_distributed=True,
    world_rank=dist.get_rank(),
    world_size=dist.get_world_size(),
    drop_last=True
)

# Each rank automatically gets its shard
for batch in loader:
    # Your training code
    pass
```

---

## Transform Library

TurboLoader includes 24 transforms (19 per-image SIMD transforms + 5 batch
augmentations). The authoritative list is `turboloader.list_transforms()`.

### Core Transforms
- **Resize** - Bilinear/Bicubic/Lanczos interpolation
- **Normalize** - Mean/std normalization with SIMD
- **CenterCrop** - Center region extraction
- **RandomCrop** - Random crop with padding

### Augmentation Transforms
- **RandomHorizontalFlip** - SIMD horizontal flip
- **RandomVerticalFlip** - SIMD vertical flip
- **ColorJitter** - Brightness/contrast/saturation/hue
- **RandomRotation** - Arbitrary angle rotation
- **GaussianBlur** - Separable convolution
- **RandomErasing** - Cutout augmentation
- **Pad** - Border padding (CONSTANT/EDGE/REFLECT)

### Advanced Transforms
- **RandomPosterize** - Bit-depth reduction
- **RandomSolarize** - Threshold inversion
- **RandomPerspective** - Perspective warp
- **AutoAugment** - Learned policies (ImageNet/CIFAR10/SVHN)

### Batch Augmentations
- **MixUp**, **CutMix**, **Mosaic**, **RandAugment**, **GridMask**

### Tensor Conversion
- **ToTensor** - PyTorch CHW or TensorFlow HWC format

---

## TBL v2 Binary Format

TurboLoader includes a custom binary format optimized for ML workloads:

### Features
- LZ4 compression for reduced storage
- Memory-mapped access for fast loading
- O(1) random access via indexed structure
- Data integrity validation with CRC checksums
- Cached image dimensions for filtered loading

### Convert TAR to TBL

```python
import tarfile
import turboloader

writer = turboloader.TblWriterV2("/data/imagenet.tbl", enable_compression=True)

# The TAR archive is read with Python's stdlib (TurboLoader does not expose a
# standalone Python TarReader; the DataLoader reads TAR directly for training).
with tarfile.open("/data/imagenet.tar") as tar:
    for member in tar.getmembers():
        if not member.name.lower().endswith((".jpg", ".jpeg")):
            continue
        data = tar.extractfile(member).read()
        writer.add_sample(data=data, format=turboloader.SampleFormat.JPEG)

writer.finalize()
```

> For bulk conversion there is also a C++ CLI tool, `tools/tar_to_tbl_v2.cpp`.

---

## Documentation

### Getting Started
- **[Quick Start Notebook](https://github.com/ALJainProjects/TurboLoader/blob/main/examples/quickstart.ipynb)** - Interactive tutorial for beginners
- **[Installation Guide](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/installation.md)** - Detailed setup instructions
- **[Quick Start](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/quickstart.md)** - Getting started examples
- **[Troubleshooting Guide](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/TROUBLESHOOTING.md)** - Common issues and solutions

### API Documentation
- **[API Reference](https://github.com/ALJainProjects/TurboLoader/tree/main/docs/api)** - Complete API documentation
- **[Transforms API](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/api/transforms.md)** - All 19 transforms with examples

### Framework Integration
- **[PyTorch Integration Guide](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/guides/pytorch-integration.md)** - Complete PyTorch guide
- **[TensorFlow Integration Guide](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/guides/tensorflow-integration.md)** - Complete TensorFlow/Keras guide
- **[PyTorch Lightning Example](https://github.com/ALJainProjects/TurboLoader/blob/main/examples/pytorch_lightning_example.py)** - Production-ready Lightning integration
- **[Distributed Training (DDP)](https://github.com/ALJainProjects/TurboLoader/blob/main/examples/distributed_ddp.py)** - Multi-GPU PyTorch DDP example

### Examples
- **[ImageNet ResNet50 Training](https://github.com/ALJainProjects/TurboLoader/blob/main/examples/imagenet_resnet50.py)** - Complete training pipeline with AMP, checkpointing, TensorBoard
- **[Distributed Training](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/distributed.md)** - Multi-node setup guide

---

## Benchmarks

Measured on **Apple Silicon** over **Imagenette-160** (9,469 real ImageNet JPEGs →
resize 160×160 → ImageNet-normalize → batched CHW float32, batch 64). To control for
thermal throttling, every loader is built once, warmed up one epoch, then timed over
**5 interleaved rounds** (each loader runs once per round); the table reports the
median. Output is verified correct against torchvision (mean abs diff ≈ 0.04, bilinear
antialiasing only).

**Image — on-the-fly decode** (re-decode every epoch; for datasets too large to cache
or with per-epoch random augmentation):

| Loader | img/s (median) | vs tf.data |
|---|---:|---:|
| **TurboLoader `DataLoader`** (`output_format='pytorch'`, nw=6) | **~55,000** | **2.0×** |
| TensorFlow `tf.data` (AUTOTUNE) | ~27,300 | 1.00× |
| PyTorch `DataLoader` (PIL, 8 persistent workers) | ~20,500 | 0.75× |

**Image — cached** (decoded tensors held in RAM; both sides consume identically via
`np.sum`, i.e. delivered as numpy/torch-ready batches — the PyTorch use case):

| Loader | img/s (median) | vs tf.data.cache |
|---|---:|---:|
| **TurboLoader** (`cache_decoded=True`, prefetch) | **~67,000** | **1.9×** |
| TensorFlow `tf.data.cache()` (+ `.numpy()` materialize) | ~35,100 | 1.00× |

(For *TF-native* consumption that stays in tf tensors, `tf.data.cache()` is faster —
TurboLoader's cache win is for delivering numpy/torch batches.)

**LLM tokens** (real text, 55M-token memory-mapped corpus, `seq_len=1024`, next-token):

| Loader | sequences/s (median) |
|---|---:|
| **TurboLoader `TokenDataLoader`** | **~467,000** |
| numpy memmap idiom (nanoGPT `get_batch`) | ~251,000 |

**Transforms** (per-image throughput vs torchvision): Resize **2.7×**, ImageNetNormalize
**3.3×**, HFlip ~1.0×. For CenterCrop, torchvision returns a **lazy strided view** (moves
zero bytes); compared against TurboLoader's real contiguous crop that looks like 0.45×,
but when torchvision actually materializes the crop (`.contiguous()`, required before
batching/most ops) it drops to ~23k img/s and **TurboLoader's contiguous crop is ~6.8×
faster** (155k vs 23k). Like the cache, this is a lazy-vs-eager comparison; for the
realistic crop→batch path TurboLoader wins.

> Earlier drafts quoted single-run figures (~42k, "1.4×") and a "cached epoch" in the
> tens-of-millions img/s. Those were artifacts (thermal noise; a no-op loop over aliased
> cached arrays) and were replaced with the interleaved, identical-consumption medians
> above. Numbers are hardware-dependent — run `benchmarks/` yourself.

The fast path runs decode + resize + normalize + batch assembly in C++ across a thread
pool with zero Python per-sample work. Use it like this:

```python
loader = turboloader.DataLoader(
    'imagenet.tar', batch_size=64, num_workers=6,
    output_format='pytorch',          # (N, C, H, W) float32 array per batch
    image_size=160,                   # exact resize, done in C++
    transform=turboloader.ImageNetNormalize())
for epoch in range(epochs):           # re-iterable
    for images, meta in loader:       # images.shape == (64, 3, 160, 160)
        train_step(images)
```

Honest caveats:
- **Run it yourself** (`benchmarks/`) — results depend heavily on hardware, image size,
  and pipeline; Linux `fork`-based PyTorch workers shift the PyTorch numbers a lot.
- **Decode backend differs**: TurboLoader uses libjpeg-turbo; the PyTorch baseline uses PIL.
- The `output_format='dict'` path returns per-sample dicts and stacks in Python
  (GIL-bound), so it is much slower — use it only when you need per-sample metadata.

For **large source images**, the default path also wins: on 768×768 JPEGs resized to
160 it runs ~15,000 img/s — faster than even an expertly-tuned `tf.data` pipeline using
manual `decode_jpeg(ratio=...)` (~14,400) — because it picks the libjpeg-turbo DCT
scaled-decode factor automatically (you don't have to know to set `ratio`).

### Implementation notes
- **Direct-batch path** (`src/pipeline/direct_batch_loader.hpp`): the default fast path
  is FFCV/`tf.data`-style — a persistent thread pool reads JPEG bytes by index and
  decodes → resizes → normalizes **directly into the output batch buffer** in one
  parallel pass (no worker queue, no per-sample heap copy, no serial collection).
  Verified memory-safe and race-free (disjoint slot writes, const mmap reads, atomic
  cursor, per-thread decoders).
- **Automatic DCT scaled decode**: large JPEGs are decoded at the nearest libjpeg-turbo
  scale ≥ target, then finely resized — much faster than full-decode + resize.
- **Resize convention**: half-pixel centers (`align_corners=False`), matching
  PIL/OpenCV/PyTorch/TF (agrees with torchvision plain bilinear to ~0.4/255; the only
  remaining difference vs torchvision's default is its antialiasing low-pass filter).
- SIMD transforms (AVX2/AVX-512/NEON), libjpeg-turbo decode, lock-free SPSC queues
  (legacy/dict + remote path), persistent `std::thread` pool (`src/core/parallel_for.hpp`).
- The GIL is released during C++ processing.
- **OpenMP is opt-in** (`TURBOLOADER_ENABLE_OPENMP=1`); off by default because linking a
  second OpenMP runtime crashes alongside PyTorch on macOS — the thread pool replaces it.

---

## Beyond Images: Tokens & Arrays

TurboLoader also ships loaders for non-image modalities with the same ergonomics
(re-iterable, `shuffle`, `set_epoch`, batched arrays):

```python
# LLM pretraining: memory-mapped token stream -> (B, seq_len) next-token batches
loader = turboloader.TokenDataLoader('train.bin', seq_len=1024, batch_size=8,
                                     dtype='uint16', shuffle=True)
for x, y in loader:          # x, y: (8, 1024) int64; y is x shifted by one
    loss = model(x, y)

# Generic arrays/memmaps (embeddings, tabular features, labels, pre-tokenized data)
loader = turboloader.ArrayDataLoader(features, labels, batch_size=256, shuffle=True)
for xb, yb in loader:
    ...
```

`TokenDataLoader` uses a vectorized fancy-index gather over a `np.memmap` (so multi-GB
corpora stream without loading into RAM) and benchmarks ~1.9× the standard nanoGPT
`get_batch` idiom. The image pipeline (decode/transform/TBL) remains C++; these
modality loaders are NumPy-based and modality-agnostic.

All three modalities are also reachable from the **single `DataLoader` entry point**:

```python
turboloader.DataLoader('train.bin', modality='tokens', seq_len=1024, batch_size=8)
turboloader.DataLoader(arrays=[feats, labels], data_path=None, modality='array', batch_size=256)
turboloader.DataLoader('data.tar', image_size=160, output_format='pytorch')   # modality='image' (default)
```

### Wrap *any* Python dataset (`MapDataLoader`)

When your data doesn't fit the native paths, `MapDataLoader` batches **any** map-style
dataset — anything with `__len__` and `__getitem__(i)`, i.e. exactly the
`torch.utils.data.Dataset` protocol — so your loading/decoding/business logic can be
arbitrary Python:

```python
class MyDataset:
    def __len__(self): return len(self.records)
    def __getitem__(self, i):
        x = decode_however_you_like(self.records[i])   # any Python logic
        return x, self.labels[i]                       # (features, label)

# directly, or via the unified entry point with dataset=...
for xb, yb in turboloader.MapDataLoader(MyDataset(), batch_size=64, shuffle=True, num_workers=8):
    train_step(xb, yb)
```

It parallelizes `__getitem__` on a bounded thread pool with read-ahead and collates
(tuples/dicts/arrays, or a custom `collate_fn`). **Honest tradeoff:** because the
per-sample work runs in Python, this path is roughly PyTorch-`DataLoader` speed (and
GIL-bound for pure-Python CPU work — threads help most when `__getitem__` releases the
GIL, e.g. NumPy/PIL/file/network I/O). It's about *flexibility*, not the C++ fast path —
use the image/token/array loaders above when you want maximum throughput.

---

## Architecture

TurboLoader uses a multi-threaded pipeline architecture:

```
┌─────────────────────────────────────────────┐
│           Memory-Mapped Reader              │
│     (TAR/TBL v2 with zero-copy access)      │
└──────────────┬──────────────────────────────┘
               │
        ┌──────▼──────┐
        │Worker Pool  │
        │  (N threads)│
        ├─────────────┤
        │ Decode      │
        │ Transform   │
        │ Convert     │
        └──────┬──────┘
               │
        ┌──────▼──────────────┐
        │ Lock-Free Queue     │
        └──────┬──────────────┘
               │
        ┌──────▼──────┐
        │Python API   │
        └─────────────┘
```

### Key Components

- **Memory-Mapped I/O** - Zero-copy file access
- **Worker Thread Pool** - Parallel processing with per-thread decoders
- **SIMD Transforms** - Vectorized operations (AVX2/AVX-512/NEON)
- **Lock-Free Queues** - High-performance concurrent data structures

---

## License

TurboLoader is released under the MIT License.

---

## Citation

If you use TurboLoader in your research:

```bibtex
@software{turboloader,
  author = {Jain, Arnav},
  title = {TurboLoader: High-Performance ML Data Loading},
  year = {2026},
  version = {2.25.0},
  url = {https://github.com/ALJainProjects/TurboLoader}
}
```

---

## Support

- **Documentation:** [https://github.com/ALJainProjects/TurboLoader/tree/main/docs](https://github.com/ALJainProjects/TurboLoader/tree/main/docs)
- **Troubleshooting:** [https://github.com/ALJainProjects/TurboLoader/blob/main/docs/TROUBLESHOOTING.md](https://github.com/ALJainProjects/TurboLoader/blob/main/docs/TROUBLESHOOTING.md)
- **Verification Script:** Run `python scripts/verify_installation.py` to check your setup
- **Issues:** [GitHub Issues](https://github.com/ALJainProjects/TurboLoader/issues)
- **Discussions:** [GitHub Discussions](https://github.com/ALJainProjects/TurboLoader/discussions)
- **PyPI:** [https://pypi.org/project/turboloader/](https://pypi.org/project/turboloader/)

---

TurboLoader - High-performance ML data loading with a C++20 core and SIMD transforms.
