Metadata-Version: 2.4
Name: biasinear
Version: 0.2.0
Summary: BiasInEar: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations
Project-URL: Homepage, https://github.com/ntunlplab/BiasInEar
Project-URL: Repository, https://github.com/ntunlplab/BiasInEar
Project-URL: Paper, https://arxiv.org/abs/2602.01030
Project-URL: Dataset, https://huggingface.co/datasets/ntunlplab/BiasInEar
Author: Sheng-Lun Wei, Yu-Ling Liao, Yen-Hua Chang, Hen-Hsen Huang, Hsin-Hsi Chen
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: audio,bias,evaluation,fairness,multilingual,speech
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: numpy
Provides-Extra: all
Requires-Dist: datasets; extra == 'all'
Requires-Dist: google-genai; extra == 'all'
Requires-Dist: mistralai; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Requires-Dist: pydub; extra == 'all'
Requires-Dist: soundfile; extra == 'all'
Provides-Extra: audio
Requires-Dist: pydub; extra == 'audio'
Provides-Extra: data
Requires-Dist: datasets; extra == 'data'
Requires-Dist: soundfile; extra == 'data'
Provides-Extra: dev
Requires-Dist: datasets; extra == 'dev'
Requires-Dist: google-genai; extra == 'dev'
Requires-Dist: mistralai; extra == 'dev'
Requires-Dist: openai>=1.0; extra == 'dev'
Requires-Dist: pydub; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: soundfile; extra == 'dev'
Provides-Extra: gemini
Requires-Dist: google-genai; extra == 'gemini'
Provides-Extra: mistral
Requires-Dist: mistralai; extra == 'mistral'
Provides-Extra: nvidia
Requires-Dist: openai>=1.0; extra == 'nvidia'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Description-Content-Type: text/markdown

# BiasInEar

**Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations**

[![arXiv](https://img.shields.io/badge/arXiv-2602.01030-red)](https://arxiv.org/abs/2602.01030)
[![HuggingFace](https://img.shields.io/badge/HuggingFace-BiasInEar-yellow)](https://huggingface.co/datasets/ntunlplab/BiasInEar)
[![License](https://img.shields.io/badge/License-Apache%202.0-green)](LICENSE)

BiasInEar is a benchmark for evaluating speech bias in multilingual multimodal large language models (MLLMs). It provides 11,200 spoken multiple-choice questions across 3 languages, 7 accents, 2 genders, and 2 option orders.

## Installation

```bash
# Core (metrics only)
pip install biasinear

# With data loading
pip install biasinear[data]

# With audio utilities
pip install biasinear[audio]

# With a specific model provider
pip install biasinear[gemini]     # Google Gemini
pip install biasinear[openai]     # OpenAI
pip install biasinear[nvidia]     # NVIDIA Build
pip install biasinear[mistral]    # Mistral

# Everything
pip install biasinear[all]
```

Or with [uv](https://github.com/astral-sh/uv):

```bash
uv pip install biasinear[all]
```

## Model Providers

BiasInEar includes built-in support for several audio language model APIs:

| Provider | Class | Install | Default Model |
|----------|-------|---------|---------------|
| Google Gemini | `GeminiModel` | `pip install biasinear[gemini]` | `gemini-2.5-flash` |
| OpenAI | `OpenAIModel` | `pip install biasinear[openai]` | `gpt-4o-audio-preview` |
| NVIDIA Build | `NvidiaModel` | `pip install biasinear[nvidia]` | `google/gemma-3n-e4b-it` |
| Mistral | `MistralModel` | `pip install biasinear[mistral]` | `voxtral-small-2507` |

### API Keys

Set your API key as an environment variable:

```bash
export GEMINI_API_KEY="..."
export OPENAI_API_KEY="..."
export NVIDIA_API_KEY="..."
export MISTRAL_API_KEY="..."
```

Or pass directly when creating the model:

```python
from biasinear.models import GeminiModel
model = GeminiModel(api_key="your-api-key")
```

### Quick Example (Gemini)

```python
from biasinear.models import GeminiModel
from biasinear.utils import concat_audio

model = GeminiModel()  # uses GEMINI_API_KEY env var
combined = concat_audio(question=q_bytes, options=opt_bytes)
output = model.generate(combined)
print(output["answer"], output["raw_response"])
```

See [`examples/`](examples/) for complete provider scripts.

## Quick Start

### 1. Load Data

```python
from biasinear import load_dataset

# Load a specific config
dataset = load_dataset(config="en_Female")

# Load all configs merged
dataset = load_dataset()
```

### 2. Run Inference

```python
from biasinear.utils import concat_audio
from biasinear.models import BaseModel

# Implement your model by extending BaseModel
class MyModel(BaseModel):
    def generate(self, audio: bytes) -> dict:
        # Your API call here
        return {"answer": "A", "raw_response": "..."}

model = MyModel("my-model")
output = model.generate(audio_bytes)
```

### 3. Evaluate

```python
from biasinear import Evaluator

evaluator = Evaluator(
    predictions=["A", "B", "A", ...],
    references=["A", "A", "A", ...],
    question_ids=["q1", "q1", "q2", ...],
    groups={
        "language": ["en", "en", "zh", ...],
        "gender": ["Female", "Male", "Female", ...],
        "order": ["original", "reversed", "original", ...],
    },
)
results = evaluator.run()
# {
#     "accuracy": 0.75,
#     "entropy": {"mean": 0.32, "per_question": {...}},
#     "apes": {"language": 0.12, "gender": 0.03, "order": 0.15},
#     "fleiss_kappa": {"language": 0.65, "gender": 0.88, "order": 0.52},
# }
```

### Use Metrics Individually

```python
from biasinear import accuracy, question_entropy, apes, fleiss_kappa

acc = accuracy(predictions, references)
ent = question_entropy(["A", "A", "B", "C"], num_categories=4)
apes_val = apes([0.3, 0.5, 0.4])
kappa = fleiss_kappa(ratings_matrix)
```

## Metrics

| Metric | Description |
|--------|-------------|
| **Accuracy** | Standard MCQ correctness |
| **Question Entropy** | Prediction uncertainty across configurations |
| **APES** | Average Pairwise Entropy Shift across variable levels |
| **Fleiss' Kappa** | Inter-rater agreement across perturbations |

See the [paper](https://arxiv.org/abs/2602.01030) for details.

## Citation

```bibtex
@inproceedings{wei-etal-2026-biasinear,
  title={Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations},
  author={Wei, Sheng-Lun and Liao, Yu-Ling and Chang, Yen-Hua and Huang, Hen-Hsen and Chen, Hsin-Hsi},
  booktitle={Findings of the Association for Computational Linguistics: EACL 2026},
  year={2026},
  publisher={Association for Computational Linguistics}
}
```

## License

Apache License 2.0. See [LICENSE](LICENSE) for details.
