Metadata-Version: 2.5
Name: sauron-verifier
Version: 1.0.1
Summary: Evidence-first agentic verifier for rubric-based evaluation.
Project-URL: Repository, https://github.com/rsanayei/Sauron
Project-URL: Issues, https://github.com/rsanayei/Sauron/issues
Project-URL: Paper, https://arxiv.org/abs/2610.17724
Project-URL: Blog, https://labs.scale.com
Author-email: Reza Sanayei <rsanayei@arizona.edu>
License-Expression: MIT
License-File: LICENSE
Keywords: agent-as-a-judge,agent-evaluation,agentic-verification,evidence-verification,llm-as-a-judge,rubric-based-evaluation
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: defusedxml>=0.7.1
Requires-Dist: openpyxl>=3.1.5
Requires-Dist: pydantic>=2.7
Requires-Dist: pypdf>=6.0
Requires-Dist: python-pptx>=1.0.2
Description-Content-Type: text/markdown

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rsanayei/Sauron/main/assets/animation/sauron-animated-dark.gif">
    <img alt="SAURON" src="https://raw.githubusercontent.com/rsanayei/Sauron/main/assets/animation/sauron-animated-light.gif" width="520">
  </picture>
</p>
<p align="center">
  <b>Evidence-first agentic verifier for rubric-based evaluation.</b>
</p>

<p align="center">
  <a href="https://arxiv.org/abs/2610.17724"><img src="https://img.shields.io/badge/Paper-arXiv-b31b1b?style=flat-square&logo=arxiv&logoColor=white" alt="Paper"></a>
  <a href="https://labs.scale.com"><img src="https://img.shields.io/badge/Blog-soon-orange?style=flat-square" alt="Blog"></a>
  <a href="LICENSE"><img src="https://img.shields.io/badge/License-MIT-blue?style=flat-square" alt="License: MIT"></a>
  <br>
  <a href="https://pypi.org/project/sauron-verifier/"><img src="https://img.shields.io/pypi/v/sauron-verifier?style=flat-square&logo=pypi&logoColor=white" alt="PyPI"></a>
  <a href="https://pypi.org/project/sauron-verifier/"><img src="https://img.shields.io/pypi/pyversions/sauron-verifier?style=flat-square" alt="Python versions"></a>
  <!-- Hidden until launch:
  <a href="https://pepy.tech/project/sauron-verifier"><img src="https://img.shields.io/pepy/dt/sauron-verifier?style=flat-square" alt="Downloads"></a>
  <a href="https://github.com/rsanayei/Sauron/stargazers"><img src="https://img.shields.io/github/stars/rsanayei/Sauron?style=flat-square&logo=github" alt="GitHub stars"></a>
  -->
</p>

<p align="center">
  <a href="#installation">Installation</a> •
  <a href="#quick-start">Quick start</a> •
  <a href="docs/architecture.md">How it works</a> •
  <a href="docs/configuration.md">Configuration</a> •
  <a href="#reference">Reference</a>
</p>

## What is Sauron?

Sauron evaluates agent runs against rubric criteria using the execution trace, final
environment state, and produced artifacts. It returns a verdict and an evidence receipt
for each criterion.

Sauron adaptively gathers information through bounded, read-only queries. An
investigator proposes claims, a separate verifier checks them against their sources,
and a finalizer assesses whether the verified claims support a verdict. This allows
Sauron to investigate long runs without loading the full trace and state into context.

Criteria that remain unresolved receive `not_awarded` with `evidence_missing=true` and
an explanation.

```
Rubric criteria
      │
      ▼
Cluster planner
      │
      ▼
Parallel investigation sessions ─── one per cluster
      │
      ├── Query trace, state, and artifacts
      ├── Propose claims → retrieve sources → verify
      └── Finalize each criterion using verified claims
      │
      ▼
Task-wide consistency check
      │
      ▼
Per-criterion verdicts and evidence receipts
      +
Run summary
```

## Installation

Requires Python 3.11 or newer.

```bash
pip install sauron-verifier
```

From source, with [uv](https://docs.astral.sh/uv/) (the supported development workflow):

```bash
git clone https://github.com/rsanayei/Sauron.git && cd Sauron
uv sync                      # creates .venv with the package and dev tools
uv run sauron --help
```

Sauron talks to any OpenAI-compatible chat-completions endpoint (a LiteLLM proxy, or a
provider that speaks that protocol). Put the endpoint in `.env` (copy `.env.example`):

```bash
LITELLM_BASE_URL=https://your-endpoint/v1
LITELLM_API_KEY=...
SAURON_MODEL=provider/your-model      # the investigator; or pass --model on each run
```

Model names are whatever that endpoint serves. The optional `--vision` surface
additionally needs `pdftoppm` (poppler) and, for PPTX/XLSX rendering, `soffice`
(LibreOffice) on the `PATH`.

## Quick start

```bash
# ten toy tasks with known expected verdicts; validates the whole loop end to end
sauron smoke --model provider/your-model --out results --name smoke

# a corpus slice through the built-in adapter (a task JSONL + per-task run.zip archives)
sauron run --model provider/your-model --data-root /path/to/corpus \
    --task-ids <task_id>,<task_id> --out results --name first_run

# a corpus through your own adapter (see docs/adapters.md)
sauron run --model provider/your-model --adapter path/to/my_adapter.py \
    --out results --name mine

# metrics for a finished run, against the gold labels its adapter attached
sauron score results/first_run
```

(From a source checkout, prefix each command with `uv run`.) Runs are resume-safe: a
task whose criterion records are already on disk is not re-judged. Add `--fresh` to
refuse resuming into an existing run directory, or `--dry-run` to resolve the
configuration and the slice without any model call.

## Configuration

A run is the shipped defaults plus explicit overrides. Every knob is listed by
`sauron defaults`; the fully resolved configuration is written to
`<out>/<name>/config.json`. You choose the investigator model; every other default is
the configuration of the reference evaluations:

| knob | default | what it does |
|---|---|---|
| `model` | none, you choose | the investigator: the model that runs each investigation session (`--model`, or `SAURON_MODEL` in `.env`) |
| `planner_model` | a light model | a cheaper model for the cluster planner, task gist and autoplan probe |
| `checker_model` | the investigator model | faithfulness and coverage checks; `--checker-model` selects another route |
| `budget_scale` | 1.0 | multiplier on every session cap; 1.0 is the standard budget the reference runs used |
| `cache_discount` | 0.1 | the token budget counts provider cache-hit input at this weight (billed cost and recorded usage are unaffected) |
| `coverage_check_min_criteria` | 8 | awards on quantified criteria (all, every, none, ...) in sessions of at least N criteria must pass an evidence-coverage check; 0 = off |
| `consistency_check` | on | task-wide re-check of verdict pairs that read one source fact with opposite polarity |
| `parallel_checks` | 4 | concurrent faithfulness checks for atoms proposed in one turn (latency only) |
| `gist` | on | one orientation pass per task, shared by its sessions |
| `autoplan` | `probe` | cluster cap, workers and singleton batching measured on the corpus; pin any of them with `--max-cluster-size` / `--cluster-workers` / `--singleton-batch` |
| `concurrency` | `fast` | shared scheduler preset (max 64 sessions in flight, max 16 tasks in parallel) |
| `mcp_config` | off | live-environment MCP servers as a read-only tool family (`--mcp-config servers.json`) |
| `vision` | off | rendered-page inspection of PDF/PPTX/XLSX/image artifacts and image-grounded atoms (`--vision`) |

See `docs/configuration.md` for every knob and the reasoning behind each default.

## What ships here

- `sauron.judges.SauronClusterJudge`: the verifier. Plans criteria into evidence
  clusters, runs one session per cluster, and finalizes each criterion under
  deterministic rules plus a conditional coverage check.
- `sauron.judges.SauronJudge`: a single-criterion verifier, usable standalone.
- `sauron.env`: bounded read-only views over a trace, a state snapshot or workspace
  directory, and produced artifacts (text, PDF, PPTX, DOCX, XLSX), plus a scratch
  sandbox for `run_python`. Optional surfaces, off by default: live-environment MCP
  servers and rendered-page vision.
- `sauron.verifier.evidence_store`: the evidence verifier (locator schema, source
  resolution, deterministic checks for mechanical claims, LLM faithfulness check).
- `sauron.adapters`: the `trajectories` adapter (a task JSONL plus per-task workspace
  archives), a `synthetic` suite, and a loader for external adapters.

## Development

```bash
uv sync --group dev
uv run ruff check .          # lint
uv run pytest                # offline contract tests; no network, no model calls
```

See `CONTRIBUTING.md` for the ground rules.

## Documentation

- `docs/architecture.md`: the verification loop, its invariants, and where each part
  lives in the code
- `docs/configuration.md`: every knob, its default, and where the default comes from
- `docs/adapters.md`: the `TaskBundle` contract and how to write an adapter
  (`examples/minimal_adapter.py` is a complete one)
- `docs/output.md`: the files a run writes and how to read them

## Reference

If you use Sauron in your research, please cite:

```bibtex
@misc{sanayei2026evidencefirst,
  title  = {Evidence-First Agentic Verification},
  author = {Sanayei, Reza and
            Dumitru, Razvan-Gabriel and
            Hou, Sunjie and
            Rezaei, MohammadHossein and
            Riaz, Haris and
            Vacareanu, Robert and
            Yang, Minglai and
            Liu, Ying and
            Surdeanu, Mihai and
            Zhang, Daniel Yue},
  year   = {2026},
  eprint = {2610.17724},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2610.17724}
}
```
