Metadata-Version: 2.4
Name: evalshift
Version: 0.3.0
Summary: Run your prompts on two LLMs and find out, with statistical confidence, what regressed.
Project-URL: Homepage, https://github.com/babaliauskas/EvalShift
Project-URL: Documentation, https://github.com/babaliauskas/EvalShift#readme
Project-URL: Repository, https://github.com/babaliauskas/EvalShift
Project-URL: Issues, https://github.com/babaliauskas/EvalShift/issues
Project-URL: Changelog, https://github.com/babaliauskas/EvalShift/blob/main/CHANGELOG.md
Author: Lukas Babaliauskas
License: MIT
License-File: LICENSE
Keywords: anthropic,evaluation,gemini,llm,migration,openai,regression-testing
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.14
Requires-Dist: aiosqlite>=0.19
Requires-Dist: greenlet>=3.0
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3.1
Requires-Dist: jsonschema>=4.22
Requires-Dist: litellm>=1.40
Requires-Dist: numpy>=1.26
Requires-Dist: pydantic>=2.5
Requires-Dist: pygments>=2.18
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: scipy>=1.12
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: tiktoken>=0.7
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: mkdocs-material>=9.5; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pre-commit>=3.7; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: types-jsonschema>=4.22; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Description-Content-Type: text/markdown

# EvalShift

> Run your prompts on two LLMs and find out, with statistical confidence, what regressed.

[![CI](https://github.com/babaliauskas/EvalShift/actions/workflows/ci.yml/badge.svg)](https://github.com/babaliauskas/EvalShift/actions/workflows/ci.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python 3.14+](https://img.shields.io/badge/python-3.14%2B-blue.svg)](https://www.python.org/downloads/)
[![Status: alpha](https://img.shields.io/badge/status-alpha-orange.svg)](#status)

EvalShift is a local-first CLI that helps engineering teams migrate safely between
LLM versions (e.g. `claude-4.5-sonnet` → `claude-5-sonnet`). Point it at your
prompts and a golden suite of inputs; it runs both models, scores the outputs
with structural / semantic / LLM-as-judge evaluators, and produces a single-file
HTML report with **defensible statistics**: paired tests, Cohen's d, 95% CIs,
and Benjamini–Hochberg correction across every (prompt × evaluator × slice)
comparison.

## Status

**Alpha.** Every command in the pipeline is shipped and the test suite is at
95%+ coverage. APIs may still change as feedback comes in.

## Install

Requires Python 3.14+.

```bash
# Recommended
uv pip install evalshift     # or: pip install evalshift
```

From source (for contributors):

```bash
git clone https://github.com/babaliauskas/EvalShift.git
cd EvalShift
uv venv --python 3.14
source .venv/bin/activate
uv pip install -e ".[dev]"
```

## Quick start

```bash
# 1. Scaffold a starter project. Writes evalshift.yaml + prompts.py +
#    tools.yaml + a 40-row golden.jsonl for a customer-support agent.
mkdir my-eval && cd my-eval
evalshift init

# 2. Set whichever provider keys you'll use
export GOOGLE_API_KEY=...   # or ANTHROPIC_API_KEY / OPENAI_API_KEY

# 3. Run the whole pipeline in one command (doctor → run → evaluate
#    → analyze → report). Pass --open to launch the report.
evalshift all --yes --open
```

`evalshift all` drives the full five-stage pipeline under a single
Rich Live region — stacked status rows, an inline progress bar for
the run stage, and a final verdict block that tells you whether the
candidate is significantly better, regressed, or showed no
significant change.

If you want to drive each stage by hand (useful in CI, or when
re-running just one stage after fixing config):

```bash
evalshift doctor
evalshift run --yes
evalshift evaluate <run-id>
evalshift analyze <run-id>
evalshift report <run-id> --open
```

Every artefact lives under `.evalshift/runs/<run-id>/` — `state.json`,
`raw.jsonl`, `scores.jsonl`, `analysis.json`, `report.json`,
`report.html`. None of it leaves your machine.

## Agent migrations (v0.2)

Migrating an agent (a prompt that uses tools)? EvalShift v0.2 detects
regressions in *which* tools the new model calls, *what* arguments it
passes, and *how* it sequences them. The killer scenario: a routing
agent that silently stops calling `notify_security_team` after the
migration — text-only eval reports green, v0.2 marks it CRITICAL.

The default `evalshift init` scaffold *is* an agent project — six tools
plus a 40-row golden suite. Just run the quick-start above and the
tool-call evaluators kick in automatically.

See [`docs/agents.md`](docs/agents.md) for the full walkthrough and
the [`examples/agent/`](examples/agent/) directory for a runnable
customer-support example.

## What the report looks like

The HTML report (single file, no external assets, works offline) has:

* **Executive summary** — one row per prompt with a severity badge.
* **Per-prompt deep dive** — aggregate stats, per-slice breakdown,
  top-5 worst regressions side-by-side.
* **Methodology appendix** — every test, p-value, effect size, and
  CI is documented.

## Why local-first?

Your prompts and your suite never leave your machine. The only outbound calls
are to the LLM providers you configure (Anthropic, OpenAI, Google) using your
own API keys. There is no EvalShift cloud.

## Documentation

* [Getting started](docs/getting-started.md) — install + first run walkthrough
* [Configuration reference](docs/configuration.md) — every `evalshift.yaml` field
* [Evaluators](docs/evaluators.md) — when to use which family
* [Methodology](docs/methodology.md) — the statistical machinery
* [FAQ](docs/faq.md) — common questions
* [`MVP_TODO.md`](MVP_TODO.md) — the build checklist (every box ticked)

## Non-goals (for v0.1)

* Hosted backend / web UI
* Multi-criterion judge in a single call
* Custom evaluator plugin system
* Comparing more than 2 models in one run
* Auto-detection of LangChain / LlamaIndex prompt patterns

These are deferred to v0.2+; see the PDF spec in the repo for the
full deferred-features list.

## License

[MIT](LICENSE) — free for any use.
