Metadata-Version: 2.4
Name: evaling
Version: 0.1.0
Summary: Compare prompt variants and models from the command line
Project-URL: Homepage, https://github.com/amro/evaling
Project-URL: Repository, https://github.com/amro/evaling
Project-URL: Documentation, https://github.com/amro/evaling/tree/main/docs
Project-URL: Changelog, https://github.com/amro/evaling/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/amro/evaling/issues
Author: Amro Mousa
License: MIT License
        
        Copyright (c) 2026 Amro Mousa
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: eval,evaluation,llm,prompt,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: httpx>=0.28.1
Requires-Dist: jinja2>=3.1.6
Requires-Dist: jsonschema>=4.26.0
Requires-Dist: pydantic>=2.13.4
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: rich>=15.0.0
Provides-Extra: mcp
Requires-Dist: mcp>=1.28; extra == 'mcp'
Description-Content-Type: text/markdown

# evaling

[![CI](https://github.com/amro/evaling/actions/workflows/ci.yml/badge.svg)](https://github.com/amro/evaling/actions/workflows/ci.yml)

A command-line tool for comparing prompt variants and models, easily.

> **Status:** early development — **alpha testers wanted.** It works end to
> end and the test suite is thorough, but it's early: expect gaps where the
> tests don't reach, and expect the config format to shift a little before 1.0.
> If you try it, [open an issue](https://github.com/amro/evaling/issues) —
> rough edges, confusing output, and missing providers are all worth
> reporting.

## Why

When you're iterating on a prompt, you want fast answers to questions like:

- Which of these three phrasings performs best on my test cases?
- Does the cheaper model handle this prompt as well as the expensive one?
- Did my latest prompt tweak regress anything?

`evaling` runs your prompt variants against your chosen models over a set of test
cases, scores the outputs, and shows you a comparison — all from the terminal.

Answering those questions with evidence, rather than by re-reading a handful of
outputs, is usually what stands between a prompt change and shipping it. The
aim is to make that comparison cheap enough that you run it every time.

## What you can evaluate

**Anything you can call.** Anthropic and OpenAI directly; Ollama, vLLM, LM
Studio, OpenRouter, Gemini and Gemma through the OpenAI-compatible endpoint;
and anything else at all through the `command` provider, which pipes JSON to a
script — so an agent, a RAG pipeline, or a local binary is as evaluable as a
chat API.

**Text, and not only text.** Single prompts or scripted multi-turn
conversations. Images, PDFs, audio, and video attach to cases and are checked
against each model's declared capabilities before a request is sent.

**Against whatever "correct" means for you.** Exact match, substring, regex,
and JSON-schema validation for the mechanical parts; your own Python function
when the rule is real but not a pattern; an LLM judge with a rubric when the
question is genuinely a matter of judgment — and a meta-eval to check that
judge against labels you wrote yourself, before you trust it to gate anything.

**At the size you actually have.** A dozen cases inline in the config, a JSONL
or CSV dataset in git, or hundreds of thousands streamed from your own
warehouse or API through a [case source](docs/large-datasets.md) — Python you
write that evaling pages through. Memory is bounded by concurrency, not by
case count, so a 500,000-case run costs no more up front than a ten-case one.

**Including data you're not allowed to read.** For the narrower case where
the data is off-limits to humans, [no-look mode](docs/no-look.md) — *eyes-off*
evaluation, in privacy terms — keeps prompts, outputs, and attachments out of
every artifact: you get the scores, the data stays where it was.

**And it tells you whether the change helped.** Compare two runs cell by cell,
pin a baseline, and fail CI on a regression rather than on a hunch.

## Install

Needs Python 3.10+ and [uv](https://docs.astral.sh/uv/):

```sh
uv tool install evaling
evaling --version
```

Or from a checkout, which is also the contributor setup:

```sh
git clone https://github.com/amro/evaling && cd evaling
uv tool install .
```

Working on evaling itself? Use `uv sync` and `uv run evaling` instead, which
always reflects your working tree. See
[getting started](docs/getting-started.md).

## Quick taste

```sh
evaling init      # scaffold a working example (offline, mock provider)
evaling run       # run it: progress bar, summary matrix, pass/fail gate
evaling show latest --failures
evaling compare <run-a> <run-b>
evaling export latest --format md
```

See [getting started](docs/getting-started.md).

## Documentation

Full docs live in [`docs/`](docs/). New here? Start with the
**[tutorial](docs/tutorial.md)** — install through CI gating, with runnable
examples at every step.

- [Tutorial](docs/tutorial.md) — the complete walkthrough.
- [Getting started](docs/getting-started.md) — the short version.
- [CLI reference](docs/cli.md) — commands, flags, run references, exit codes.
- [Configuration](docs/configuration.md) — the `eval.yaml` reference, settings
  layering, and environment variables.
- [Prompts](docs/prompts.md) — variants, Jinja2 templating, multi-turn
  messages, multimodal inputs, and case datasets.
- [Scoring](docs/scoring.md) — scorers, scorecards, LLM judges, and CI
  thresholds.
- [Providers](docs/providers.md) — Anthropic, OpenAI, OpenAI-compatible (local
  models, Gemini, OpenRouter), the `command` provider, pricing, and adding your
  own.
- [Secrets](docs/secrets.md) — where API keys come from, and how they're kept
  out of git and out of your output.
- [Storage](docs/storage.md) — the run directory format, resuming interrupted
  runs, response caching, and programmatic access.
- [Large datasets](docs/large-datasets.md) — case sources: streaming cases from
  your own API or warehouse instead of a file.
- [No-look evals](docs/no-look.md) — evaluating data nobody is permitted to
  read.
- [CI recipes](docs/ci.md) — gating, baselines, cost ceilings, report
  artifacts.
- [Evaluating judges](docs/evaluating-judges.md) — calibrate autoraters against
  human labels (meta-evals).
- [MCP server](docs/mcp.md) — drive evaling from an agent for hands-off prompt
  iteration.
- [Python API](docs/python-api.md) — use evaling as a library.
- [Troubleshooting](docs/troubleshooting.md) — symptoms, causes, fixes.
- [Architecture](docs/architecture.md) — how it's built, and why.

## Development

Requires [uv](https://docs.astral.sh/uv/).

```sh
uv sync              # create the venv and install dependencies
uv run evaling       # run the CLI
uv run pytest        # run the test suite
uv run ruff check .  # lint
uv run ruff format . # format
```

Changes are documented in the [CHANGELOG](CHANGELOG.md).

Seven complete sample evals live in [`examples/`](examples/) — text and
multimodal, single- and multi-turn, a RAG pipeline behind the `command`
provider, and a no-look run over a paging source. The test suite runs them all
end to end on every commit, so they work with the current version.

See [CONTRIBUTING.md](CONTRIBUTING.md) before opening a pull request.

## How this was built

evaling was written with heavy use of AI coding tools — most of it with Claude
Code, working from requirements and design decisions recorded in
[REQUIREMENTS.md](REQUIREMENTS.md). Every change is reviewed before it lands,
carries tests, and has to pass the same CI as anything else: lint, formatting,
a dependency audit, and the full suite on Linux across Python 3.10–3.13, plus
macOS and Windows. The docs and the worked examples are
exercised by that suite too, so they cannot quietly drift from the code.

Worth saying plainly rather than leaving you to infer it from the commit log.

## License

[MIT](LICENSE); © 2026 Amro Mousa
