Metadata-Version: 2.4
Name: mavdt
Version: 0.1.1
Summary: Manifestro Audio Voice Dataset Toolkit — download, subset, and clean Mozilla Common Voice data
Project-URL: Homepage, https://github.com/Manifestro/mavdt
Project-URL: Repository, https://github.com/Manifestro/mavdt
Project-URL: Issues, https://github.com/Manifestro/mavdt/issues
Project-URL: PyPI, https://pypi.org/project/mavdt/
Author: Manifestro Inc.
License: MIT
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: requests>=2.31
Requires-Dist: scipy>=1.11
Requires-Dist: soundfile>=0.12
Requires-Dist: tqdm>=4.65
Provides-Extra: dev
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1; extra == 'dev'
Provides-Extra: viz
Requires-Dist: matplotlib>=3.7; extra == 'viz'
Requires-Dist: seaborn>=0.12; extra == 'viz'
Description-Content-Type: text/markdown

<div align="center">

# MAVDT

### Manifestro Audio Voice Dataset Toolkit

*Download, subset, and clean Mozilla Common Voice datasets — from 3-speaker test sets to 50+ hour corpora.*

[![PyPI](https://img.shields.io/pypi/v/mavdt?style=flat-square&logo=pypi&logoColor=white)](https://pypi.org/project/mavdt/)
[![Python](https://img.shields.io/badge/python-%3E%3D3.11-blue?style=flat-square&logo=python&logoColor=white)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green?style=flat-square)](LICENSE)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-orange?style=flat-square)](https://github.com/astral-sh/ruff)

</div>

---

## Overview

MAVDT is a modular CLI + Python toolkit that streamlines working with [Mozilla Common Voice](https://commonvoice.mozilla.org/) data. It covers the full preprocessing pipeline in three composable steps:

```
MDC API  ──►  mavdt-download  ──►  mavdt-select  ──►  mavdt-clean  ──►  Training-ready WAVs
```

Each step produces a **manifest** (`manifest.jsonl` or `manifest.csv`) so you always know exactly what's in your dataset and why.

---

## Features

| | Capability | Details |
|---|---|---|
| ⬇️ | **Download** | Fetch datasets via Mozilla Data Collective API with streaming and progress bars |
| ✂️ | **Subset** | Four selection modes — by speakers, all records, by duration, or random sample |
| 🔊 | **Clean** | Resample, mono-convert, peak/RMS normalize, filter by duration and loudness |
| 📋 | **Manifests** | Output `manifest.jsonl` and/or `manifest.csv` with full per-clip metadata |
| 🔁 | **Reproducible** | Seeded random selection, YAML config files, deterministic pipelines |
| 🐍 | **Dual interface** | CLI commands *and* importable Python API — use in shell or Jupyter notebooks |

---

## Quick Start

### 1. Install

```bash
pip install mavdt
```

### 2. Set your API key

```bash
export MAVDT_API_KEY="your_mdc_api_key"
```

> Get your key at [Mozilla Data Collective](https://mozilladatacollective.com/).

### 3. Run the pipeline

```bash
# Download dataset
mavdt-download --dataset_id cmndapwry02jnmh07dyo46mot --output_dir data/raw

# Select 3 female speakers, 5 clips each
mavdt-select \
  --input_dir data/raw/kk \
  --mode speakers \
  --num_speakers 3 \
  --max_clips_per_speaker 5 \
  --gender female \
  --output_dir data/selected

# Clean: 16 kHz · mono · peak-normalized
mavdt-clean \
  --input_dir data/selected \
  --output_dir data/cleaned \
  --target_sr 16000
```

That's it — `data/cleaned/` now contains ready-to-train WAV files and a `manifest.jsonl`.

---

## Installation

```bash
# From PyPI
pip install mavdt

# From source (for development)
git clone https://github.com/Manifestro/mavdt.git
cd mavdt
uv sync --extra dev
```

**Requirements:** Python ≥ 3.11

---

## CLI Reference

### `mavdt-download`

Fetch a Common Voice dataset from the Mozilla Data Collective.

```
mavdt-download --dataset_id ID [--api_key KEY] [--output_dir DIR] [--force] [--config FILE]
```

| Flag | Default | Description |
|---|---|---|
| `--dataset_id` | *required* | MDC dataset identifier |
| `--api_key` | `$MAVDT_API_KEY` | API Bearer token |
| `--output_dir` | `data/raw` | Destination directory |
| `--force` | `false` | Overwrite existing files |
| `--remove_archive` | `true` | Delete `.tar.gz` after extraction |
| `--config` | — | Path to YAML config file |

---

### `mavdt-select`

Subset a downloaded dataset using different selection strategies.

```
mavdt-select --input_dir DIR --output_dir DIR [options]
```

| Flag | Default | Description |
|---|---|---|
| `--input_dir` | *required* | Extracted dataset directory |
| `--output_dir` | *required* | Output directory |
| `--mode` | `speakers` | `speakers` · `all` · `length` · `random` |
| `--dataset_file` | `validated.tsv` | Source TSV file |
| `--num_speakers` | — | Number of speakers *(mode=speakers)* |
| `--max_clips_per_speaker` | — | Max clips per speaker |
| `--min_clips_per_speaker` | `1` | Min clips a speaker must have to qualify |
| `--max_total_clips` | — | Global clip limit *(mode=all / random)* |
| `--max_total_hours` | — | Global hours limit *(mode=all)* |
| `--min_duration_sec` | — | Min duration filter |
| `--max_duration_sec` | — | Max duration filter |
| `--gender` | — | `male` or `female` filter |
| `--flat` | `false` | Flat output (no per-speaker subdirs) |
| `--format` | `jsonl` | `jsonl` · `csv` · `both` |
| `--seed` | `42` | Random seed |
| `--config` | — | Path to YAML config file |

#### Selection modes

```bash
# speakers — N speakers, controlled clips per speaker
mavdt-select --mode speakers --num_speakers 10 --max_clips_per_speaker 20 --gender male

# all — every record, capped by hours
mavdt-select --mode all --max_total_hours 50

# length — only clips within a duration window
mavdt-select --mode length --min_duration_sec 2.0 --max_duration_sec 10.0

# random — N random clips, no speaker grouping
mavdt-select --mode random --max_total_clips 1000 --seed 7
```

---

### `mavdt-clean`

Resample, normalize, and filter audio clips.

```
mavdt-clean --input_dir DIR --output_dir DIR [options]
```

| Flag | Default | Description |
|---|---|---|
| `--input_dir` | *required* | Directory with manifest + audio |
| `--output_dir` | *required* | Output directory |
| `--target_sr` | `16000` | Target sample rate (Hz) |
| `--mono` / `--no-mono` | `mono` | Convert to mono |
| `--normalize` | `peak` | `peak` · `rms` · `none` |
| `--target_rms` | `0.1` | Target RMS value for rms normalization |
| `--min_duration_sec` | `1.0` | Drop clips shorter than this |
| `--max_duration_sec` | `30.0` | Drop clips longer than this |
| `--min_rms_db` | `-40.0` | Drop clips quieter than this (dB) |
| `--format` | `jsonl` | `jsonl` · `csv` · `both` |
| `--config` | — | Path to YAML config file |

---

## Configuration File

Every flag can be set in a YAML config. CLI arguments take precedence over file values.

```yaml
# examples/config_example.yaml

download:
  api_key: "YOUR_MDC_API_KEY"       # or set $MAVDT_API_KEY
  dataset_id: "cmndapwry02jnmh07dyo46mot"
  output_dir: "data/raw"
  force: false

select:
  input_dir: "data/raw/kk"
  mode: "speakers"
  output_dir: "data/selected"
  num_speakers: 3
  max_clips_per_speaker: 5
  gender: "female"
  seed: 42

clean:
  input_dir: "data/selected"
  output_dir: "data/cleaned"
  target_sr: 16000
  normalize: "peak"
  min_duration_sec: 1.0
  max_duration_sec: 30.0
  min_rms_db: -40.0
```

Pass it to any command:

```bash
mavdt-download --config examples/config_example.yaml
mavdt-select   --config examples/config_example.yaml
mavdt-clean    --config examples/config_example.yaml
```

---

## Python API

Use `mavdt` directly in scripts or Jupyter notebooks:

```python
from mavdt.download import download_dataset
from mavdt.select import run_selection
from mavdt.clean import clean_dataset

# Download
dataset_path = download_dataset(
    dataset_id="cmndapwry02jnmh07dyo46mot",
    api_key="YOUR_KEY",
    output_dir="data/raw",
)

# Subset — 3 female speakers, 5 clips each
selected_path = run_selection(
    input_dir="data/raw/kk",
    output_dir="data/selected",
    mode="speakers",
    num_speakers=3,
    max_clips_per_speaker=5,
    gender="female",
    seed=42,
)

# Clean — 16 kHz mono, peak-normalized, drop quiet clips
cleaned_path = clean_dataset(
    input_dir="data/selected",
    output_dir="data/cleaned",
    target_sr=16000,
    normalize="peak",
    min_rms_db=-40.0,
)
```

---

## Output Format

Every `mavdt-select` and `mavdt-clean` run produces audio files alongside a manifest.

### Directory layout

```
data/selected/
├── manifest.jsonl          # one JSON object per clip
├── speaker_001/
│   ├── common_voice_kk_1234.mp3
│   └── common_voice_kk_1235.mp3
├── speaker_002/
│   └── common_voice_kk_1236.mp3
└── ...
```

With `--flat`, all clips land directly in the output directory with no subdirectories.

### Manifest schema

```jsonc
// manifest.jsonl — one record per line
{
    "audio_path": "speaker_001/common_voice_kk_1234.wav",
    "client_id":  "a1b2c3d4e5f6",
    "sentence":   "Сәлем, әлем!",
    "duration":   3.45,
    "gender":     "female",
    "age":        "twenties",
    "up_votes":   2,
    "down_votes": 0,
    "locale":     "kk"
}
```

| Field | Type | Description |
|---|---|---|
| `audio_path` | `str` | Relative path from the manifest directory |
| `client_id` | `str` | Speaker identifier (hashed by Common Voice) |
| `sentence` | `str` | Transcription text |
| `duration` | `float` | Duration in seconds |
| `gender` | `str` | `male` · `female` · `nan` |
| `age` | `str` | Age bracket from Common Voice |
| `up_votes` | `int` | Community up-vote count |
| `down_votes` | `int` | Community down-vote count |
| `locale` | `str` | Language code (`kk`, `ru`, `en`, …) |

---

## Project Structure

```
mavdt/
├── src/mavdt/
│   ├── __init__.py
│   ├── download.py         # MDC API download & tar.gz extraction
│   ├── select.py           # Subsetting, sampling, manifest writing
│   ├── clean.py            # Resample, normalize, filter
│   ├── utils.py            # Logging, statistics, manifest I/O
│   └── config.py           # Dataclasses & YAML config loader
├── examples/
│   └── config_example.yaml
├── tests/
│   ├── test_download.py
│   ├── test_select.py
│   ├── test_clean.py
│   └── test_utils.py
└── pyproject.toml
```

---

## Contributing

Contributions are welcome!

```bash
# Set up dev environment
git clone https://github.com/Manifestro/mavdt.git
cd mavdt
uv sync --extra dev

# Run tests
uv run pytest

# Lint
uv run ruff check src/ tests/
```

Planned extensions (good first issues):
- `balance_dataset` — balance speakers / gender
- `split_train_val` — train / val / test splits
- `export_hf` — push to HuggingFace Hub
- `analyze_dataset` — deep statistics & plots

---

## License

[MIT](LICENSE) © 2026 Manifestro Inc.
