Metadata-Version: 2.4
Name: hallucination-gate
Version: 0.7.0
Summary: Conservative grounding gate for RAG and fine-tuned LLMs. Blocks answers that are not supported by caller-supplied evidence.
Author-email: Shreyas G <shreyas2559@gmail.com>
Maintainer-email: Shreyas G <shreyas2559@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/shrey315/hallucination-gate
Keywords: hallucination,rag,llm,bayesian,grounding,evaluation
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: MIT License
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pgmpy>=0.1.26
Requires-Dist: sentence-transformers>=3.0.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: uvicorn[standard]>=0.32.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: numpy>=1.26.0
Requires-Dist: pandas>=2.0
Requires-Dist: pypdf>=5.0.0
Requires-Dist: typer>=0.12.0
Requires-Dist: rich>=13.0
Requires-Dist: python-dotenv>=1.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: httpx>=0.27.0; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Provides-Extra: ocr
Requires-Dist: pillow>=10.0; extra == "ocr"
Requires-Dist: pytesseract>=0.3.10; extra == "ocr"
Requires-Dist: pdf2image>=1.17.0; extra == "ocr"
Requires-Dist: easyocr>=1.7.0; extra == "ocr"
Provides-Extra: vision
Requires-Dist: pillow>=10.0; extra == "vision"
Requires-Dist: pytesseract>=0.3.10; extra == "vision"
Requires-Dist: pdf2image>=1.17.0; extra == "vision"
Requires-Dist: easyocr>=1.7.0; extra == "vision"
Provides-Extra: api
Requires-Dist: fastapi>=0.115.0; extra == "api"
Requires-Dist: uvicorn[standard]>=0.32.0; extra == "api"
Dynamic: license-file

# hallucination-gate

[![PyPI](https://img.shields.io/pypi/v/hallucination-gate.svg)](https://pypi.org/project/hallucination-gate/)
[![Python](https://img.shields.io/pypi/pyversions/hallucination-gate.svg)](https://pypi.org/project/hallucination-gate/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![CI](https://github.com/shrey315/hallucination-gate/actions/workflows/ci.yml/badge.svg)](https://github.com/shrey315/hallucination-gate/actions/workflows/ci.yml)

**RAG evaluation + conservative release gate** for RAG and fine-tuned LLMs.

- **Eval:** RAGAS-class metrics (`faithfulness`, `answer_relevancy`, `context_precision`, `context_recall`) on **claim↔chunk** grounding — not whole-answer NLI only.
- **Gate:** pass / rewrite / abstain so production only ships supported text.

Author: **Shreyas G**.

## Install

```bash
pip install -U hallucination-gate
pip install "hallucination-gate[ocr]"   # optional OCR
```

## RAG eval (RAGAS replacement path)

```python
from hallucination_gate import RAGEval

evaler = RAGEval()  # neural; use_heuristic=True for CI
report = evaler.evaluate(
    [
        {
            "query": "What is the warranty?",
            "answer": "The Titan watch has a 2-year warranty.",
            "contexts": [
                "The Titan watch has a 2-year warranty covering defects.",
                "Shipping takes 3-5 days.",
            ],
            "ground_truth": "2-year warranty for manufacturing defects.",
            # optional labeled retrieval:
            # "relevant_contexts": ["The Titan watch has a 2-year warranty covering defects."],
        }
    ]
)
print(report.aggregate)
# {'faithfulness': ..., 'answer_relevancy': ..., 'context_precision': ..., ...}
report.to_json("report.json")
```

```bash
hallucination-gate eval-dataset samples.jsonl --out report.json
```

| Metric | How this package scores it |
|---|---|
| **faithfulness** | Fraction of answer claims supported by individual chunks (contradictions penalize) |
| **answer_relevancy** | Query↔answer embedding relevance |
| **context_precision** | Labeled `relevant_contexts` if provided; else claim-aligned chunk proxy |
| **context_recall** | Requires `ground_truth` — fraction of reference facts covered by contexts |
| **groundedness / hallucination_risk / release_safety** | BN posteriors from the same evidence stack |

**Why this beats typical RAGAS setups for grounding:** claim-level soft-OR against neighbors, false-release oriented gate, multimodal/OCR evidence, and a production `safe_answer` path — not only a mean score.

## Production gate

```python
from hallucination_gate import HallucinationGate, Evidence

gate = HallucinationGate()
result = gate.check(query, answer, context=retrieved_docs)
return result.text
```

```python
report = gate.evaluate(samples)  # same backends as the gate
```

## OCR

```python
from hallucination_gate import Evidence, ocr_available

ev = Evidence.from_image(path="warranty_card.jpg")
ev = Evidence.from_ocr(path="scanned_policy.pdf")
```

## Drawbacks (honest)

- **Latency & cost** — neural path adds inference time / GPU·CPU load per sample.
- **Over-refusal** — conservative gate can abstain on good extractive answers.
- **Only as good as evidence** — checks support, not world truth; bad retrieval still hurts.
- **Hard cases** — subtle math/code/reasoning can fool or over-block NLI.
- **Heuristic ≠ quality gate** — `use_heuristic=True` is for CI smoke, not calibrated faithfulness.
- **Ops surface** — HF downloads, torch/sentence-transformers weight, Windows symlink quirks.
- **Not magic** — still not a full substitute for domain-labeled regression + human review; it is a stronger *grounding-first* eval+gate stack than score-only RAGAS defaults.

## Eval (gate safety)

```bash
pip install -e ".[dev]"
set RAG_EVAL_HEURISTIC=1
pytest -q -m "not neural"
hallucination-gate eval-heldout
```

## License

MIT © Shreyas G
