Metadata-Version: 2.4
Name: assevra
Version: 0.1.0
Summary: A reference implementation and methodology for measuring the reliability of LLM agents (the Assevra Reliability Scorecard).
Author: Veera Ravindra Divi
License: MIT
Project-URL: Homepage, https://github.com/assevra/assevra
Project-URL: Methodology, https://github.com/assevra/assevra/blob/main/METHODOLOGY.md
Keywords: llm,ai-agents,reliability,evaluation,llm-as-judge,grounding,pii,safety
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: pii
Requires-Dist: presidio-analyzer>=2.2; extra == "pii"
Requires-Dist: presidio-anonymizer>=2.2; extra == "pii"
Requires-Dist: spacy>=3.7; extra == "pii"
Provides-Extra: judge
Requires-Dist: anthropic>=0.40; extra == "judge"
Provides-Extra: all
Requires-Dist: presidio-analyzer>=2.2; extra == "all"
Requires-Dist: presidio-anonymizer>=2.2; extra == "all"
Requires-Dist: spacy>=3.7; extra == "all"
Requires-Dist: anthropic>=0.40; extra == "all"
Dynamic: license-file

# Assevra

[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

*Assevra* — from *asseverate*, to solemnly attest. A reference implementation
and a named methodology for measuring the reliability of LLM agents: the
**Assevra Reliability Scorecard**.

This is a personal open-source research project by **Veera Ravindra Divi**. It is
an open reference implementation and methodology for the research and engineering
community — not a product. The point is to make agent-reliability measurement
concrete, reproducible, and honest: every reliability claim is tied to a metric,
a threshold, and a confidence interval, and the scorecard states plainly what it
does *not* measure.

## The methodology in brief

An agent's reliability is reported across four independent dimensions, each
scored on a labeled dataset:

| Dimension | Question | Scoring | Threshold |
|---|---|---|---|
| **Grounding / faithfulness** | Is every claim traceable to the context? | LLM-as-judge | ≥ 0.90 |
| **Safety / refusal** | Does it refuse what it must (and answer what it should)? | LLM-as-judge* | 1.00 |
| **PII-leak** | Does it leak personal data outside sanctioned fields? | Deterministic | 1.00 |
| **Task-completion** | Are the required facts present in the output? | Deterministic | ≥ 0.90 |

<sub>*Safety falls back to a deterministic refusal heuristic when no judge is configured.</sub>

The verdict is a conjunction — the scorecard passes only if every scored
dimension passes. Two principles run through all of it: **deterministic before
judge** (you scan for a leaked SSN, you don't ask a model whether it leaked one),
and **report the interval, not just the mean** (every dimension carries a 95%
Wilson interval so nobody over-reads a small-sample move). The full specification
is in [METHODOLOGY.md](METHODOLOGY.md).

## Install

Requires Python 3.10+. The deterministic core (task-completion, the PII regex
fallback, the scorecard, and the CLI) has **no third-party dependencies**, so it
runs out of the box.

```bash
git clone https://github.com/assevra/assevra.git
cd assevra

# Core only — runs the deterministic dimensions immediately.
pip install -e .

# Full PII detector (Microsoft Presidio):
pip install -e ".[pii]"
python -m spacy download en_core_web_lg

# LLM-as-judge dimensions (grounding, safety):
pip install -e ".[judge]"
export ANTHROPIC_API_KEY=sk-...
```

The default judge is Anthropic Claude (`claude-opus-4-8` for highest agreement,
`claude-sonnet-5` for volume). The judge is pluggable and is **skipped, not
failed**, when no API key is set — so the scorecard still runs offline on the
deterministic dimensions.

## 60-second quickstart

```bash
python -m assevra run --dataset datasets/golden.jsonl
```

That scores the bundled illustrative dataset and writes three reports from the
same result: `scorecard.md`, `scorecard.json`, and a styled, self-contained
`scorecard.html` (inline CSS, no external assets — open it in any browser or
share it as-is). With no API key it runs the two deterministic dimensions plus
the safety heuristic, and skips grounding; set `ANTHROPIC_API_KEY` to enable the
judge dimensions. Add `--gate` to make the command exit non-zero when the
scorecard fails, so it can gate CI directly.

## An example scorecard

Running the quickstart offline produces output like this (deterministic
dimensions pass; grounding is skipped without a judge):

```
| Dimension       | Mode          | Score | 95% CI      | n | Threshold | Result  |
|-----------------|---------------|-------|-------------|---|-----------|---------|
| grounding       | llm-judge     |   —   |     —       | 0 |   0.90    | SKIPPED |
| safety          | deterministic | 1.000 | 0.438–1.000 | 3 |   1.00    | PASS    |
| pii             | deterministic | 1.000 | 0.438–1.000 | 3 |   1.00    | PASS    |
| task_completion | deterministic | 1.000 | 0.510–1.000 | 4 |   0.90    | PASS    |
```

For a fuller, worked example that reads like a real audit — with two of the four
dimensions failing — see
[examples/sample-scorecard.md](examples/sample-scorecard.md). For the rendered
HTML report, see
[examples/example-scorecard.html](examples/example-scorecard.html) (open it in a
browser).

## Honest scope

- **This is a reference implementation, not a certification.** A pass means the
  agent behaved on the dataset you gave it, not that it is safe.
- **The bundled dataset is illustrative.** `datasets/golden.jsonl` is ~13
  clearly-synthetic rows that prove the method runs. It does not characterize a
  production agent — real audits use larger, adversarial datasets.
- **Judge calibration is described, not automated.** A judge score is only
  trustworthy once you have shown judge-vs-human agreement on a labeled hold-out
  (see [METHODOLOGY.md §4](METHODOLOGY.md)). v0.1 documents that step; it does
  not perform it.
- **The scorers have real limits.** Task-completion checks fact presence, not
  phrasing. The regex PII fallback only sees hard-block entities — install the
  `pii` extra for the full detector.

The point of stating this here is that reliability claims are only as strong as
what they honestly exclude.

## How to cite

> Divi, Veera Ravindra. *Assevra: A Reliability Scorecard for LLM Agents*, v0.1,
> 2026. https://github.com/assevra/assevra

A [`CITATION.cff`](CITATION.cff) is included; GitHub renders a "Cite this
repository" button from it. When you report a number, say it was *measured with
Assevra v0.1*.

## License & contributing

MIT — see [LICENSE](LICENSE). Contributions are welcome; see
[CONTRIBUTING.md](CONTRIBUTING.md). Every scorer must ship with a definition, a
scoring method, a threshold, and a reported interval.
