Metadata-Version: 2.5
Name: ragwarden
Version: 0.3.0
Summary: An inline hallucination gate for production RAG pipelines: atomic-claim decomposition, a cost-tiered detection cascade, and a configurable decision policy.
Project-URL: Homepage, https://github.com/utsavopal/ragwarden
Project-URL: Documentation, https://utsavopal.github.io/ragwarden/
Project-URL: Repository, https://github.com/utsavopal/ragwarden
Project-URL: Changelog, https://github.com/utsavopal/ragwarden/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/utsavopal/ragwarden/issues
Author: RagWarden contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: faithfulness,grounding,guardrails,hallucination,llm,nli,rag
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: pysbd>=0.3.4
Requires-Dist: pyyaml>=6.0
Provides-Extra: all
Requires-Dist: chromadb>=0.5; extra == 'all'
Requires-Dist: datasets>=2.20; extra == 'all'
Requires-Dist: docling-core>=2.0; extra == 'all'
Requires-Dist: langchain-core>=0.3; extra == 'all'
Requires-Dist: lettucedetect>=0.1.4; extra == 'all'
Requires-Dist: llama-index-core>=0.11; extra == 'all'
Requires-Dist: opensearch-py>=2.4; extra == 'all'
Requires-Dist: opentelemetry-api>=1.25; extra == 'all'
Requires-Dist: opentelemetry-sdk>=1.25; extra == 'all'
Requires-Dist: protobuf>=4.0; extra == 'all'
Requires-Dist: sentencepiece>=0.2; extra == 'all'
Requires-Dist: torch>=2.1; extra == 'all'
Requires-Dist: transformers<5,>=4.40; extra == 'all'
Requires-Dist: transformers>=4.40; extra == 'all'
Provides-Extra: bench
Requires-Dist: datasets>=2.20; extra == 'bench'
Provides-Extra: chroma
Requires-Dist: chromadb>=0.5; extra == 'chroma'
Provides-Extra: docling
Requires-Dist: docling-core>=2.0; extra == 'docling'
Provides-Extra: hhem
Requires-Dist: torch>=2.1; extra == 'hhem'
Requires-Dist: transformers<5,>=4.40; extra == 'hhem'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.3; extra == 'langchain'
Provides-Extra: lettucedetect
Requires-Dist: lettucedetect>=0.1.4; extra == 'lettucedetect'
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.11; extra == 'llamaindex'
Provides-Extra: minicheck
Requires-Dist: torch>=2.1; extra == 'minicheck'
Requires-Dist: transformers>=4.40; extra == 'minicheck'
Provides-Extra: nli
Requires-Dist: protobuf>=4.0; extra == 'nli'
Requires-Dist: sentencepiece>=0.2; extra == 'nli'
Requires-Dist: torch>=2.1; extra == 'nli'
Requires-Dist: transformers>=4.40; extra == 'nli'
Provides-Extra: opensearch
Requires-Dist: opensearch-py>=2.4; extra == 'opensearch'
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.25; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.25; extra == 'otel'
Description-Content-Type: text/markdown

# RagWarden

> An inline hallucination gate for production RAG pipelines.

**RagWarden doesn't compete with RAGAS or UQLM — it's the runtime enforcement layer that uses
signals like theirs (and its own fast detectors) to make a bounded-latency, explainable decision
inline in production, with a policy engine enterprises can tune to their own risk tolerance.**

An answer generated by *any* RAG pipeline is broken into atomic, independently-checkable claims.
Each claim runs through a cascade of checks ordered cheapest-first:

1. **Tier 0 — retrieval heuristics** (deterministic, near-zero cost, always on)
2. **Tier 1 — claim-vs-evidence entailment** (fast encoder models: NLI, HHEM, LettuceDetect, MiniCheck)
3. **Tier 2 — uncertainty quantification** (consistency sampling; token log-probs if available)
4. **Tier 3 — LLM-as-judge** (structured chain-of-thought, budget-capped, only the ambiguous remainder)

Per-claim verdicts combine into a composite reliability score. A configurable **policy engine**
turns that score into an action — `ALLOW`, `REDACT_CLAIMS`, `RETRY`, `ABSTAIN`, or `ESCALATE` — and
every decision comes with a full explanation trail.

**`gate()` does not fail your request.** Every external dependency — a Tier-1 detector, your
`generate_fn`, your `judge_fn` — is fault-isolated and time-boxed; an unexpected failure anywhere in
the cascade degrades to a safe `ABSTAIN` instead of an unhandled exception. See
[Running in production](docs/production.md).

## Status

**v0.3.0** — all 13 build phases complete (contracts → cascade → policy → actions → adapters →
observability → docs → release pipeline → hardening), plus a production-hardening pass (fault
isolation, timeouts, an async API, startup warmup). Benchmark numbers are an honest baseline, not
yet competitive. On the path to a v1.0 that locks `ragwarden.contracts` under semver.

| Phase | Scope | State |
|---|---|---|
| 0 | Contracts + repo skeleton | done |
| 1 | Tier 0 heuristics + policy engine + `gate()` | done |
| 2 | Tier 1 NLI detector + real severity-weighted scoring | done |
| 3 | Benchmark harness (RAGTruth) + calibration | done |
| 4 | HHEM / LettuceDetect / MiniCheck detectors + ensembling | done |
| 5 | Tier 2 uncertainty quantification (consistency sampling) | done |
| 6 | Tier 3 LLM-as-judge + full budget-capped cascade | done |
| 7 | Actions (redact/retry/abstain/escalate) + `action_payload` | done |
| 8 | Retrieval adapters (OpenSearch, LangChain, Chroma, Docling, LlamaIndex) | done |
| 9 | Observability (OTel spans + versioned JSON logs) | done |
| 10 | MkDocs documentation site | done |
| 11 | Packaging, security, PyPI release readiness (Trusted Publishing, SBOM, pip-audit) | done |
| 12 | Hardening — frozen contracts, decisions documented, coverage audit, semver policy | done |

### Current baseline (honest, not competitive yet)

`ragwarden benchmark --dataset ragtruth --detector nli` on 150 shuffled RAGTruth test rows
(DeBERTa-v3-base NLI, default policy, Tiers 0–1 only):

| | Precision | Recall | F1 |
|---|---|---|---|
| Response-level (overall) | 0.42 | 0.98 | 0.59 |
| Data2txt | 0.73 | 1.00 | 0.85 |
| Summary | 0.28 | 0.93 | 0.43 |
| QA | 0.22 | 1.00 | 0.36 |

The gate currently **over-flags** (very high recall, low precision) — the expected over-abstention
baseline before Tier 2/3, calibration, and better claim decomposition land. Full run outputs live in
[`benchmarks/results/`](benchmarks/).

## Install

```bash
pip install ragwarden                 # core: light, no ML dependencies
pip install 'ragwarden[nli]'          # + DeBERTa-v3 NLI Tier-1 detector
pip install 'ragwarden[opensearch]'   # + OpenSearch adapter
pip install 'ragwarden[all]'          # everything (OSI-licensed extras)
```

The core install pulls only a YAML parser and a pure-Python sentence splitter — no `torch`,
`transformers`, or `numpy` for Tier-0-only usage.

## Quickstart

```python
from ragwarden import gate, Policy
from ragwarden.models import Chunk, Context, Answer

context = Context(
    query="When was the Eiffel Tower completed?",
    chunks=[Chunk(text="The Eiffel Tower was completed in 1889.", score=0.91, source_id="doc-1")],
    retrieval_method="hybrid",
)
answer = Answer(text="The Eiffel Tower was completed in 1889.")

result = gate(context, answer, policy=Policy.default())
print(result.action)  # GateAction.ALLOW
print(result.reliability_score)  # composite 0.0-1.0
print(result.explanation)  # human-readable decision trail
```

## How it relates to existing tools

RagWarden is **complementary, not competitive** — it is the runtime enforcement layer that turns
signals (its own, or others') into a bounded-latency, explainable inline decision.

| Tool | What it is | Relationship |
|---|---|---|
| **RAGAS**, **DeepEval** | Offline LLM-judge evaluators (faithfulness / groundedness metrics) for CI and batch analysis. | Different job. Their judge-call cost profile is unusable per-request; RagWarden can call a judge like theirs as its **Tier 3**, on the small ambiguous remainder only. |
| **UQLM** | Uncertainty-quantification toolkit for LLM outputs. | A signal source. RagWarden's Tier 2 is the same family of idea (consistency sampling); UQLM's methods can feed a custom detector. |
| **Vectara HHEM**, **LettuceDetect**, **MiniCheck**, **DeBERTa-v3 NLI** | Fast encoder grounding checkers — shipped as raw scorers. | **These are RagWarden's Tier-1 building blocks.** RagWarden adds decomposition, a cost-tiered cascade, and a decision policy around them. |
| **Guardrails AI**, **NeMo Guardrails** | Inline validator / rail frameworks with on-fail actions. | Closest OSS prior art for the policy engine. RagWarden is narrower and deeper: claim-level grounding verification with a tuned cascade, not a general rail system. |
| **AWS Bedrock Guardrails**, **Azure AI Content Safety**, **Google Vertex check-grounding** | Cloud inline grounding gates. | Validate the demand. All closed, vendor-locked, cloud-only. RagWarden is the open, pipeline-agnostic equivalent. |

## Honest limitations

- **A gate cannot fix bad retrieval.** If the wrong evidence was retrieved, no downstream check
  recovers the right answer — this bounds the whole system's ceiling.
- **Every detector has its own error rate.** The composite reliability score is a calibrated
  confidence estimate, not ground truth.
- **Over-aggressive gating trades hallucination risk for over-abstention.** Calibration against your
  own traffic (`ragwarden calibrate`) is the actual product experience, not optional polish.
- **Latency / cost budgets are real constraints.** The tiered design exists because "run an LLM
  judge on everything" is correct but unusable at production scale.
- **v0.1 claim decomposition is sentence splitting** — a known simplification vs. atomic-claim
  extraction; it bounds achievable span-level precision. See [design decisions](docs/decisions.md).

## License

Apache-2.0 (explicit patent grant — matches Transformers, TensorFlow, NeMo, RAGAS). See
[LICENSE](LICENSE).
