Metadata-Version: 2.4
Name: atdataset
Version: 0.2.1
Summary: Audio text dataset for PyTorch training based on webdataset.
Author: pkufool
License: Apache-2.0
Project-URL: Repository, https://github.com/pkufool/ATdataset
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: torch>=2.1.0
Requires-Dist: torchaudio>=2.1.0
Requires-Dist: webdataset>=0.2.86
Requires-Dist: soundfile>=0.13
Requires-Dist: numpy>=1.21
Requires-Dist: tqdm>=4.60
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest>=7.0; extra == "test"
Dynamic: license-file

# ATdataset

A streaming audio-text dataloader for PyTorch training, built on [WebDataset](https://github.com/webdataset/webdataset).

Designed for large-scale speech recognition training with multi-dataset muxing, dynamic batching, bucketing, and on-the-fly augmentation.

## Install

```bash
pip install atdataset
```

For development:

```bash
git clone https://github.com/pkufool/ATdataset.git
cd ATdataset
pip install -e ".[dev]"
```

## Quick Start

```python
from atdataset import ATDataloader

dl = ATDataloader(
    datasets="data/tars/train.lst",
    sample_rate=16000,
    max_duration=600.0,
)

for batch in dl:
    audio = batch["audio"]          # (B, T), padded
    audio_lens = batch["audio_lens"]  # (B,)
    feature = batch["feature"]      # (B, T, F), padded
    feature_lens = batch["feature_lens"]  # (B,)
    texts = batch["text"]           # list of str
    break
```

## CLI Tools

After installation, the `atdataset` command provides two subcommands:

```bash
# Build tar shards from a TSV manifest
atdataset build --input train.tsv --output-dir data/tars --num-tars 64

# Generate a .lst file from existing tar shards
atdataset gen_lst --audio-pattern 'data/tars/audios/*.tar' \
    --txt-dir data/tars/manifests --output data/tars/train.lst
```

Run `atdataset build --help` or `atdataset gen_lst --help` for full option details.

## Documentation

- [Design & Architecture](docs/design.md) — pipeline architecture, bucketing, muxing, epoch control
- [Usage Guide](docs/usage.md) — detailed examples for all features
- [Data Format](docs/data_format.md) — manifest, tar shard, and lst file formats

## License

Apache-2.0
