Metadata-Version: 2.4
Name: anyeval-cli
Version: 1.0.1
Summary: Check an evaluation repository against the AnyEval standard, and run it end to end.
License: Apache-2.0
Project-URL: Homepage, https://anyeval.com
Project-URL: Standard, https://anyeval.com/standard
Project-URL: Source, https://github.com/AnyEvalOrg/anyeval-app
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# anyeval

Check that an AI evaluation repository conforms to the [AnyEval standard](https://anyeval.com/standard),
and run it end to end to get real numbers.

```bash
pipx run anyeval-cli check .
```

## Why

A benchmark score is comparable to another score only when **five things match**: the
model, the provider that served it, the eval, the problem, and the harness. Published
numbers routinely state one and omit four, which makes them impossible to reproduce and
unsafe to compare — including for whoever published them.

The AnyEval standard is a repository layout and a manifest that make all five
recordable. This tool decides whether a repository conforms, so conformance is an
objective result rather than a review opinion.

## Commands

```bash
anyeval check .                  # 27 rules, pass/fail each, exit 1 on failure
anyeval check . --json           # machine-readable, for CI
anyeval spec                     # print the standard
anyeval run . --sample-id 3 --model qwen/qwen3-32b
```

### `check` never runs your code

It reads files and parses `run.py` with `ast`. It does not import or execute anything in
the repository, so it is safe to point at a stranger's checkout. The trade is stated
plainly: static analysis can confirm that `run.py` *declares* `--sample-id` and refuses
glob patterns, not that it honours them at runtime. That is what `run` is for.

### `run` does run your code

Running an eval is the only way to prove it works, so `anyeval run` executes the
repository's own `run.py` against a live model and prints the score, tokens, cost, and
which provider actually served the request.

It needs a [TrustedRouter](https://trustedrouter.com) key in `TRUSTEDROUTER_API_KEY`.
The run is billed to your account; this tool sends the key to nothing but TrustedRouter.

It refuses to start on a non-conforming repository unless you pass `--force`, because
the rules are what stop one problem becoming a full-dataset bill.

## The most expensive rule

`runner.rejects_globs` exists because Inspect matches sample ids with `fnmatch` and only
*warns* when a glob selects many samples — it errors solely when the filtered set is
empty. So `--sample-id '*'` silently runs the entire dataset while billing for one
problem. Every conforming `run.py` refuses glob metacharacters outright.

## Exit status

`0` when the repository conforms, `1` when it does not, `2` on a usage error. Suitable
for CI.

## Submitting an eval

Check it, run it, then email the repository URL to `anyeval-submission@trustedrouter.com`.
See [anyeval.com/submit](https://anyeval.com/submit).

You do not have to submit at all — the standard and this tool are open, and neither
depends on anyeval.com. A format only usable by its author is not a standard.

## Note on the package name

The `anyeval` name on PyPI belongs to an unrelated project, so this package is published
as **`anyeval-cli`**. It installs both an `anyeval` and an `anyeval-cli` command.

## Licence

Apache-2.0
