Metadata-Version: 2.5
Name: evalstone
Version: 0.1.0
Summary: The plug-and-play evals layer for LLM-based projects: datasets, checks, suites, runs and gates.
Project-URL: Homepage, https://evalstone.com
Project-URL: Documentation, https://evalstone.com/docs
Project-URL: Repository, https://github.com/marceloribeiro/evalstone
Author: Marcelo Ribeiro
License-Expression: MIT
License-File: LICENSE
Keywords: agents,evals,evaluation,llm,rag,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.12
Requires-Dist: httpx<1,>=0.28
Requires-Dist: jsonschema<5,>=4.26
Requires-Dist: pydantic<3,>=2.14
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: rich<16,>=15.0
Requires-Dist: typer<1,>=0.27
Provides-Extra: anthropic
Requires-Dist: anthropic<2,>=1.13; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: anthropic<2,>=1.13; extra == 'dev'
Requires-Dist: mcp<3,>=2.3; extra == 'dev'
Requires-Dist: pytest>=9.1; extra == 'dev'
Requires-Dist: respx>=0.23; extra == 'dev'
Requires-Dist: ruff>=0.17; extra == 'dev'
Provides-Extra: mcp
Requires-Dist: mcp<3,>=2.3; extra == 'mcp'
Description-Content-Type: text/markdown

# evalstone

The plug-and-play evals layer for LLM-based projects, as a Python library.

> **Status:** 0.1.0. Docs: [evalstone.com/docs/evalstone/library](https://evalstone.com/docs/evalstone/library).
> Source and samples: [github.com/marceloribeiro/evalstone](https://github.com/marceloribeiro/evalstone).

The library **computes**; it owns no database. A run lives in memory, prints a
table and writes a JSON report. Persistence happens through the Evalstone
center (PostgreSQL) when one is configured.

## Develop

```bash
just install     # venv + editable install with dev extras
just check       # ruff + pytest
just console     # REPL with `es` loaded
```

## Quick start

```bash
pip install evalstone
evalstone init my-evals                  # evalstone.yaml + a golden set + recorded outputs + a check
cd my-evals
evalstone run nightly                    # table + evalstone-report.json
evalstone gate nightly                   # exit 1 when the gate fails (CI)
```

A fuller example — a small retrieval bot with a gated suite — lives in the
repository under `samples/rag-bot` (`git clone` it, `pip install -e
python/evalstone`, then `evalstone run nightly` inside the sample).

Judges and prompt targets need a provider key in the environment
(`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, or any
OpenAI-compatible base URL such as OpenRouter or Ollama via `providers:`).
Without one, judge results are **errors, never passes**, and the gate fails.

## Human labels

```bash
evalstone run nightly --report r.json
evalstone label r.json --out evals/labels.jsonl      # p/f per output, or --scale 5
```

Labels are keyed by sample id **and** a hash of the output, so a changed answer
never inherits an old grade; outputs whose target errored are not offered for
grading. `--scale N` passes at N-1 (a 2-point scale at 2), as judges do. A
`human` check (`{type: human, labels: evals/labels.jsonl}`, path relative to
`evalstone.yaml`) scores from them; an unlabelled output is an error, never a
pass. Labels are also the ground truth judges get calibrated against (center).

## pytest

```python
from evalstone.pytest_plugin import suite


@suite("nightly")  # runs the suite; a failing gate fails the test
def test_support_bot(run):  # optional body sees the Run
    assert run.summary.cost_usd < 1.0
```

Fixtures `evalstone_run` / `evalstone_gate` / `evalstone_config`; options
`--evalstone-config`, `--evalstone-baseline` (a report, or a directory of them).
Each suite writes `evalstone-report.<suite>.json` beside the config; paths are
relative to the config's directory. A suite without a `gate:` fails the gated
forms. Works on test methods too. The plugin registers itself when evalstone is
installed, imports nothing of evalstone until a fixture runs, and pytest is not
a runtime dependency of the library.

## MCP server

```bash
pip install "evalstone[mcp]"
claude mcp add evalstone -- evalstone mcp      # run from the project directory
```

Tools: `list_suites`, `run_suite`, `judge_output`, `record_sample`, `compare_runs`,
`get_run`. The agent edits your prompt, runs the suite, reads the failures, and
compares runs — the gate stays yours. Report paths are relative to the directory
holding `evalstone.yaml` (pass `--config` to serve a project from elsewhere).

## Concepts

`Sample` → `Dataset` (content-hashed, so every run pins an exact version) ·
`Check` → `Score` (0..1 + pass/fail + reason; an error is never a pass) ·
`Suite` → `Run` → `Gate`. See the docs for the full vocabulary.
