Metadata-Version: 2.4
Name: veralith
Version: 0.2.0
Summary: Hallucination diagnosis for RAG systems — Sufficiency, Faithfulness, Completeness verdicts plus rule-based remediation.
Author: Srijan Shekhar, Kaustav Dasgupta
License: MIT
Project-URL: Homepage, https://github.com/SrijanShekhar21/VeralithAI
Project-URL: Repository, https://github.com/SrijanShekhar21/VeralithAI
Project-URL: Issues, https://github.com/SrijanShekhar21/VeralithAI/issues
Keywords: rag,llm,evaluation,hallucination,openai,langchain,observability,agents,veralith
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openai>=1.40.0
Requires-Dist: pydantic>=2.6
Requires-Dist: python-dotenv>=1.0
Requires-Dist: tenacity>=8.2
Requires-Dist: tiktoken>=0.7.0
Requires-Dist: httpx>=0.27
Provides-Extra: langchain
Requires-Dist: langchain>=0.1.0; extra == "langchain"
Provides-Extra: sample
Requires-Dist: chromadb>=0.5.0; extra == "sample"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Dynamic: license-file

# Veralith

**Hallucination diagnosis for RAG systems.** Wrap one line around your retrieval-augmented pipeline and get structured reports on *what* failed and *how to fix it* — not just a single yes/no hallucination flag.

Veralith decomposes every `(query, context, response)` trace into atomic sub-questions and claims, runs three LLM-as-judge metrics over them (Sufficiency, Faithfulness, Completeness), and classifies the trace into one of six diagnostic cells with a concrete remediation suggestion.

> Status: alpha (0.1.x). Public API is stable; expect additions, not breaking changes.

---

## Why Veralith

A monolithic *"is this response hallucinated?"* judge is a smoke alarm — it can tell you something is wrong but not what or where. Veralith is a diagnostic dashboard:

- **Sufficiency** — was the *retrieval* adequate for each part of the query?
- **Faithfulness** — is each *claim* in the response grounded in the retrieved context?
- **Completeness** — does the response actually *answer* every part of the query?

Cross-tabulating these gives you a named failure mode (retrieval gap, intrinsic hallucination, padded answer, etc.) plus actionable fixes (lower temperature, bump retrieval-K, tighten generator prompt, ...) for every trace.

---

## Install

```bash
pip install veralith
```

Optional extras:

```bash
pip install "veralith[langchain]"   # LangChain auto-tracing adapter
pip install "veralith[dev]"          # pytest, ruff, build, twine (contributors)
```

Set your OpenAI key:

```bash
export OPENAI_API_KEY=sk-...
```

---

## 30-second quickstart

```python
import veralith

result = veralith.evaluate(
    query="What is a P/E ratio and what was Apple's P/E in 2023?",
    context=[
        "The price-to-earnings (P/E) ratio is computed by dividing a company's "
        "share price by its earnings per share."
    ],
    response=(
        "A P/E ratio divides share price by earnings per share. "
        "Apple's P/E in 2023 was 42.7."
    ),
    persist=False,
)

print(result.diagnosis.failure_cell.value)   # 'incomplete_ungrounded'
print(result.suggestion.title)               # 'Worst-case failure'
for action in result.suggestion.actions:
    print(" -", action)
```

You get back a typed `EvaluationResult` with per-claim verdicts, per-Qi sufficiency, a failure-cell diagnosis, and a concrete suggestion. Optionally persisted to a local SQLite database for later analysis.

---

## Integration patterns

### 1. Explicit one-liner — works with any RAG stack

```python
import veralith

def answer(query: str) -> str:
    chunks = my_retriever(query)
    response = my_generator(query, chunks)

    veralith.log(query=query, context=chunks, response=response)   # background eval
    return response
```

### 2. Decorator — zero code reshape

```python
import veralith

@veralith.trace
def my_rag(query: str):
    chunks = my_retriever(query)
    response = my_generator(query, chunks)
    return response, chunks   # the decorator captures (response, context)
```

### 3. Synchronous eval — full result inline

```python
result = veralith.evaluate(query, context, response, persist=False)
if result.diagnosis and result.diagnosis.failure_cell.value.endswith("ungrounded"):
    handle_hallucination(result.faithfulness)
```

### 4. LangChain — zero-code auto-tracing

```python
import veralith.adapters.langchain as adapter
adapter.install()

# every RetrievalQA.invoke() now auto-traces to Veralith
```

---

## What Veralith detects

Each evaluated trace lands in one of six cells from the cross-tab of Completeness × Faithfulness. The cell name follows the pattern `<completeness>_<faithfulness>`, so you can decode any cell without a lookup chart:

| | Grounded (every claim supported) | Ungrounded (some claim invented) |
|---|---|---|
| **Complete answer** | `complete_grounded` | `complete_ungrounded` |
| **Incomplete answer** | `incomplete_grounded` | `incomplete_ungrounded` |
| **Extra unrequested content** | `extra_grounded` | `extra_ungrounded` |

Read each cell as *"the response is `<X>` and the claims are `<Y>`."* So `incomplete_ungrounded` means the response *didn't cover everything asked AND some of what it did say is unsupported* — the worst-case trace.

Plus a per-trace Sufficiency level (HIGH/LOW), learned per knowledge base from the distribution of healthy traces. Together they drive a rule-based suggester that maps every diagnosis to a concrete remediation (lower temperature / bump K / tighten generator prompt / etc.).

---

## Configuration

Defaults work out of the box. Tunable via environment variables or `veralith.config.settings`:

| Variable | Default | Purpose |
|---|---|---|
| `OPENAI_API_KEY` | — | Required |
| `VERALITH_JUDGE_MODEL` | `gpt-4o` | Model for S/F/C judges |
| `VERALITH_DECOMPOSER_MODEL` | `gpt-4o-mini` | Model for query / response decomposition |
| `VERALITH_DB_PATH` | `veralith.db` | SQLite persistence path |

Each evaluation costs roughly 5 LLM calls (3 batched judges + 2 decomposition) — about $0.005 per trace on the default models. Cost is tracked per call via `veralith.observability.cost`.

---

## The result object

```python
class EvaluationResult:
    trace_id: int
    query: str
    sub_questions: list[SubQuestion]           # decomposed Q
    claims: list[Claim]                         # decomposed R
    sufficiency: list[SufficiencyJudgment]      # per-Qi verdicts
    faithfulness: list[FaithfulnessJudgment]    # per-Ri verdicts + grounding chunks
    completeness: CompletenessJudgment | None   # Ri ↔ Qi alignment
    diagnosis: Diagnosis | None                 # failure_cell + sufficiency level + counts
    suggestion: Suggestion                      # title + body + actionable steps
    created_at: datetime
    errors: dict[str, str]                       # any per-metric failures (D3)
    latency_ms: dict[str, float]                 # per-phase wall-clock timing
```

Every field is a typed Pydantic model.

---

## Roadmap

What's in 0.1:
- Three judges (Sufficiency, Faithfulness, Completeness) with batched LLM calls.
- Diagnostic classifier and rule-based suggester.
- Outcome-based threshold calibration per knowledge base.
- SDK: `log()`, `@trace`, LangChain adapter, background eval worker.
- SQLite persistence with self-healing migrations.
- Cost tracking with per-trace budget guard.
- CLI entry point.

On the roadmap:
- LLM-enriched trace-specific suggestions (`Suggestion.detailed_body`).
- Cross-trace pattern detection ("you keep hallucinating on time-sensitive queries").
- Additional judges (reasoning validity, temporal validity).
- More framework adapters (LlamaIndex, raw OpenAI tools).
- Hosted dashboard with multi-tenant projects.

---

## Authors

Srijan Shekhar and Kaustav Dasgupta.

## License

MIT — see [LICENSE](LICENSE).

## Links

- Source: https://github.com/SrijanShekhar21/VeralithAI
- Issues: https://github.com/SrijanShekhar21/VeralithAI/issues
