Metadata-Version: 2.1
Name: rag-consistency-scorer
Version: 0.1.0
Summary: MMD-based semantic consistency scoring for RAG pipeline outputs
License: MIT
Project-URL: Repository, https://github.com/laughinghugs/ragpulse
Project-URL: Issues, https://github.com/laughinghugs/ragpulse/issues
Keywords: rag,llm,evaluation,mmd,consistency,nlp
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: openai>=1.10
Provides-Extra: pca
Requires-Dist: scikit-learn>=1.3; extra == "pca"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: numpy>=1.24; extra == "dev"

# rag-consistency-scorer

**MMD-based semantic consistency scoring for RAG pipeline outputs.**

Measures whether a RAG pipeline produces semantically stable answers when asked the same question multiple times. Uses Maximum Mean Discrepancy (MMD) over atomic-fact embeddings — a statistically principled distributional test — rather than naive string similarity.

---

## Installation

```bash
pip install rag-consistency-scorer
```

Optional extras:

```bash
pip install "rag-consistency-scorer[pca]"   # enables PCA dimensionality reduction
pip install "rag-consistency-scorer[dev]"   # testing dependencies
```

---

## Quick start

```python
from rag_consistency_scorer import ConsistencyScorer

data = [
    {
        "query": "What is the boiling point of water?",
        "responses": [
            "Water boils at 100 degrees Celsius at sea level.",
            "The boiling point of water is 100°C at standard pressure.",
            "At 1 atm pressure, water reaches its boiling point at 100°C.",
            "Water transitions to steam at 100 degrees Celsius under normal conditions.",
            "The boiling point of water is 100°C, equivalent to 212°F.",
            "Under standard atmospheric conditions, water boils at 100°C.",
            "Water boils at 373 Kelvin, i.e. 100°C at sea level.",
            "Standard boiling point of water is 100°C at atmospheric pressure.",
        ],
    }
]

scorer = ConsistencyScorer()
results = scorer.score(data)

for r in results:
    print(r.query_id, r.stability_score)
    # → q_abc12345  0.921
```

No API key needed with default settings. The package uses a lightweight TF-IDF embedding fallback when no `embedding_config` is provided.

---

## Input format

`scorer.score()` accepts a **list of dicts**, each with:

| Key | Type | Required | Description |
|-----|------|----------|-------------|
| `query` | `str` | ✓ | The prompt / question sent to the RAG pipeline |
| `responses` | `List[str]` | ✓ | One answer string per run. **Minimum 2, recommended 8–15.** |
| `query_id` | `str` | – | Optional identifier. Auto-generated if omitted. |

```python
data = [
    {
        "query": "Who invented the telephone?",
        "query_id": "telephone_q1",
        "responses": [
            "Alexander Graham Bell invented the telephone in 1876.",
            # ... at least 7 more responses
        ]
    }
]
```

---

## Output format

Each call to `.score()` returns a `List[QueryScore]` — one per input query.

```python
result = results[0]

result.stability_score   # float in [0, 1]. 1 = perfectly stable.
result.mmd2_mean         # mean pairwise MMD² across all run pairs
result.mmd2_min          # min pairwise MMD²
result.mmd2_max          # max pairwise MMD²
result.sigma_used        # kernel bandwidth selected for this query
result.tau_used          # τ used in exp(-MMD²/τ) mapping
result.n_runs            # number of responses scored
result.query_id          # identifier
result.query             # original query string

# Per-pair statistical diagnostics
for pair in result.pairs:
    pair.run_i, pair.run_j   # which two runs
    pair.mmd2                # raw MMD² (may be slightly negative)
    pair.p_value             # permutation p-value (Phipson-Smyth corrected)
    pair.p_value_bh          # BH-FDR adjusted p-value
    pair.significant         # True if p_value_bh ≤ alpha

# Per-run outlier diagnostics (sorted most → least anomalous)
for run in result.runs:
    run.run_index         # index into original responses list
    run.outlier_score     # mean MMD² to all other runs (higher = more anomalous)
    run.z_score           # bootstrap z-score (> 2 is suspicious)
    run.n_facts           # number of atomic facts extracted
    run.extraction_source # "llm" or "heuristic"

# Serialise
import json
json.dumps(result.to_dict())
```

### Interpreting stability_score

| Score | Interpretation |
|-------|---------------|
| ≥ 0.80 | Stable — answers are semantically consistent |
| 0.60–0.79 | Watch — mild drift; monitor over time |
| 0.40–0.59 | Investigate — notable inconsistency |
| < 0.40 | Unstable — likely retrieval or generation failure |

*Thresholds depend on your `tau` setting. Run the calibration utilities to set domain-specific thresholds.*

---

## Configuration

```python
from rag_consistency_scorer import ConsistencyScorer, ScorerConfig

scorer = ConsistencyScorer(
    config=ScorerConfig(
        # --- Extraction ---
        extraction_mode="heuristic",   # or "llm"
        llm_config=None,               # required if extraction_mode="llm"

        # --- Embedding ---
        embedding_config=None,         # uses TF-IDF fallback if not set

        # --- Statistical ---
        tau=None,           # None = auto-calibrate from data (recommended)
        n_permutations=500, # higher → more accurate p-values, slower
        alpha=0.05,         # BH-FDR significance level
        bandwidth_candidates=10,
        seed=42,
    )
)
```

### LLM-backed extraction (OpenAI)

```python
scorer = ConsistencyScorer(
    config=ScorerConfig(
        extraction_mode="llm",
        llm_config={
            "api_key": "sk-...",
            "model": "gpt-4o-mini",
            "max_facts": 40,
            "heuristic_fallback": True,   # fall back silently on LLM error
        },
        embedding_config={
            "api_key": "sk-...",
            "model": "text-embedding-3-small",
        },
    )
)
```

### Azure OpenAI

```python
scorer = ConsistencyScorer(
    config=ScorerConfig(
        extraction_mode="llm",
        llm_config={
            "api_key": "your-azure-key",
            "model": "gpt-4o-mini",                         # deployment name
            "api_base": "https://your-resource.openai.azure.com/",
            "api_version": "2024-02-01",
        },
        embedding_config={
            "api_key": "your-azure-key",
            "model": "text-embedding-ada-002",              # deployment name
            "api_base": "https://your-resource.openai.azure.com/",
            "api_version": "2024-02-01",
        },
    )
)
```

---

## How it works

For each query the scorer:

1. **Extracts atomic facts** from each response (heuristic regex or LLM-based).
2. **Embeds** each fact into a vector space and L2-normalises.
3. **Selects σ** (kernel bandwidth) via power-guided selection on a log-uniform grid, replacing the naive median heuristic.
4. **Computes pairwise MMD²** (unbiased U-statistic) between every pair of runs.
5. **Runs permutation tests** with Phipson-Smyth corrected p-values.
6. **Applies BH-FDR correction** across all pairs.
7. **Detects outlier runs** via bootstrap z-scores.
8. **Maps to [0,1]**: `stability = exp(-max(0, MMD²_mean) / τ)`, with τ auto-calibrated from the data.

---

## Running tests

```bash
pip install pytest
pytest tests/ -v
```

---

## What this does NOT measure

- **Correctness** — a pipeline can be stably wrong.
- **Faithfulness** to retrieved context.
- **Logical contradiction** detection.

Use alongside correctness/faithfulness metrics (RAGAS, Giskard, etc.) for a complete RAG evaluation suite.

---

## License

MIT
