Metadata-Version: 2.4
Name: evalrouter
Version: 0.4.4a1
Summary: The EvalRouter command line and interactive terminal workspace
Project-URL: Homepage, https://evalrouter.ai
Project-URL: Documentation, https://evalrouter.ai/developers/docs/cli
License-Expression: LicenseRef-Kimpton-EvalRouter-SDK
License-File: LICENSE
Keywords: benchmarks,cli,evalrouter,llm,model-evaluation
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: <3.14,>=3.12
Requires-Dist: evalrouter-sdk==0.4.4a1
Requires-Dist: rich==15.0.0
Requires-Dist: textual==8.2.8
Provides-Extra: agents
Requires-Dist: evalrouter-sdk[agents]==0.4.4a1; extra == 'agents'
Provides-Extra: benchmark
Requires-Dist: evalrouter-sdk[benchmark]==0.4.4a1; extra == 'benchmark'
Provides-Extra: keyring
Requires-Dist: keyring==25.7.0; extra == 'keyring'
Provides-Extra: langsmith
Requires-Dist: evalrouter-sdk[langsmith]==0.4.4a1; extra == 'langsmith'
Provides-Extra: strands
Requires-Dist: evalrouter-sdk[strands]==0.4.4a1; extra == 'strands'
Provides-Extra: tracing
Requires-Dist: evalrouter-sdk[tracing]==0.4.4a1; extra == 'tracing'
Description-Content-Type: text/markdown

# EvalRouter CLI

Evaluate models in a full-screen terminal workspace or from scripts. Browse
benchmarks, choose models, start work using available wallet credit, follow runs,
compare or export their results, and author your own benchmarks. The CLI is
built on the [`evalrouter-sdk`](https://pypi.org/project/evalrouter-sdk/) Python SDK, which
it installs at exactly the same version.

Requires Python 3.12 or 3.13. Distributed under the proprietary Kimpton EvalRouter
SDK License, included as `LICENSE` in the package.

## Install and create your account

We recommend [uv](https://docs.astral.sh/uv/getting-started/installation/) to install the CLI in its own Python environment:

```sh
uv tool install --prerelease allow evalrouter
evalrouter
```

For this alpha, use `python -m pip install --pre evalrouter` in a Python 3.12 or 3.13 environment.
uv selects a compatible Python for the CLI and downloads one if needed. If
your shell cannot find `evalrouter`, run `uv tool update-shell` and restart your
terminal. Upgrade an existing installation with `evalrouter upgrade` (or
`evalrouter --upgrade`). Alpha, beta and rc installations include prereleases
automatically; stable installations stay on stable unless you pass `--pre`.
The command uses uv, pipx or pip in the current virtual environment. Source
installs, system Python and installer-pinned versions need their original installer.
Check the installed version with `evalrouter --version`. Use `evalrouter --help`
for explicit commands, or see the [CLI guide](https://evalrouter.ai/developers/docs/cli).

The interactive workspace checks for newer releases in the background and shows
an update notice when one is available. The check never installs an update or
delays opening the workspace. Set `EVALROUTER_NO_UPDATE_CHECK=1` to disable it.
Scripts, editable installs and explicitly pinned installations skip update checks.

### Complete command names with Tab

For zsh, add this line to `~/.zshrc` after your `compinit` setup, then start a new
terminal:

```sh
eval "$(evalrouter completion zsh)"
```

For bash, add this line to `~/.bashrc`:

```sh
eval "$(evalrouter completion bash)"
```

Type `evalrouter ru` and press Tab to complete `run`. With `evalrouter r`, Tab
shows matching commands such as `run`, `resume` and `results`. The shell setup
only completes top-level command names and makes no API request. It does not
change interactive prompts or run an evaluation.

This package was previously published as `kimpton-evalrouter-sdk`. The command
stays `evalrouter`. Use `from evalrouter import Client` for new integrations;
legacy `kimpton_evalrouter` imports remain compatible and refer to the same
public classes. To switch, run
`uv tool uninstall kimpton-evalrouter-sdk` (or
`pip uninstall -y kimpton-evalrouter-sdk evalrouter`) and install `evalrouter`.

The package supplies the `evalrouter` command; no repository checkout is needed.
[Create an EvalRouter account](https://evalrouter.ai/register), verify your email,
and follow the browser sign-in below.
Package installation does not grant account access or evaluation credit.

Sign in with your account:

```sh
evalrouter login
evalrouter whoami
```

Follow the browser sign-in; on a local computer the browser returns to the
waiting CLI. For a remote terminal, use `evalrouter login --no-browser` and
approve the code on a device with a browser. New accounts finish setup there.
The CLI selects your only workspace automatically, or lets you choose one.
Production is the default API; `evalrouter options` lists `--base-url` and the other
settings that apply to every command.
No API key or credential-storage setup is needed for a normal sign-in.
The optional `evalrouter[keyring]` extra enables a compatible OS keychain
backend; otherwise credentials use a private file readable only by your user.
In the workspace, **Account and workspace → Sign in** starts the same flow.
See the [sign-in guide](https://evalrouter.ai/developers/docs/cli-login) for
workspace switching, stored API keys and signing out.

CI and services can use `EVALROUTER_API_KEY` and `EVALROUTER_WORKSPACE_ID` supplied
privately by a secret manager. Never put credentials in arguments, URLs, request
files or logs.

## Interactive workspace

Run `evalrouter` without arguments in a terminal to open the full-screen
workspace. Choose **Account and workspace → Sign in** to connect your account.
Local sign-in returns from the browser to the waiting CLI; for a remote terminal,
use `evalrouter login --no-browser` and approve the code on a device with a browser.

**Run a benchmark** guides you through the benchmark, your selected compatible models, each
model's reasoning settings and coverage. Type to filter lists; Space or Enter
toggles a model and Tab continues. A supported native one-model evaluation
starts directly using available wallet credit. With several models or benchmarks,
the CLI previews each pair and starts one evaluation job for the compatible
pairs. Matched mode requires every pair to be compatible and runs the same
sampled tasks across models. No customer spending cap is set on new work.

Choose a suggested sample, **Custom sample size**, or **Full benchmark** for
coverage. A custom sample accepts 1 to 1,000 tasks, limited to the benchmark's
task count when known. Full benchmark includes every task.

Home includes **Run history**, **Run groups**, and **Results**, plus model and
benchmark browsing, billing and account settings. **Results** offers **Single
run**, **Group run**, then **Compare runs**. Group status shows each model's state;
open a member run, return to its group, or keep watching live progress.
While watching, choose **Open live run in browser** or **Open live group in
browser** with Enter, B or a click. Live status continues in the terminal.

Escape goes back; Left also goes back when you are not editing text. Backspace
edits text and filters. Back controls stay on Home at the root, and returning
to Home preserves its selection. Ctrl+Q or the **Exit** action exits. Leaving a
live run stops watching only: submitted server work continues. Use
PgUp/PgDn for long reviews and results, and F1 for keyboard help. Select output
with the mouse and press Ctrl+C to copy; with no selection, Ctrl+C goes back.
Password fields cannot be copied. `EVALROUTER_TUI=off` selects the older interactive
shell; explicit commands, plain output and JSON remain available.

## Run a model benchmark

For a shared native benchmark, the single-benchmark model picker lists the
routes accepted by that exact version's catalog checks. Its explanation names
fixed sampling or native continuation scoring requirements where applicable;
the listed models are alternatives. A named excluded route reports the API's
reason. Unavailable or unknown discovery remains explicit, and launch checks
current eligibility again. Private benchmarks and registry evals without this
scoped catalog check retain their qualification step before launch.

Sign in, choose a ready benchmark and compatible model, then run:

```sh
evalrouter login
evalrouter catalog
evalrouter catalog --models
evalrouter run BENCHMARK_SLUG --model MODEL_ID --sample 3
evalrouter run BENCHMARK_SLUG --model MODEL_ID --task TASK_ID --task OTHER_TASK_ID
```

Supported native single-model benchmarks start directly using available wallet
credit. Native matched comparisons start one run with the same sampled tasks
for every model. Separate mode starts one evaluation job with a run for each
compatible model–benchmark pair. Use `--pairs compatible` in scripts when you
intend to skip incompatible pairs. `--max-cost` is rejected for new evaluations.
Native actual-usage routes charge recorded usage, including
usage above a live-call hold for work already sent. A negative wallet stops new
dispatches until credit is added. Accepted historical quotes keep their terms.

For scripts, approve paid work explicitly and retain the operation identity:

```sh
evalrouter run BENCHMARK_SLUG --model MODEL_ID --sample 3 --yes --json
evalrouter run BENCHMARK_SLUG --model MODEL_A --model MODEL_B --mode matched --sample 3 --yes --json
evalrouter status
evalrouter wait RUN_ID
evalrouter results RUN_ID
evalrouter export RUN_ID --format json --output result.json
```

A sample is not a full-benchmark score. A local timeout or interrupted watcher
does not stop server work. Reuse the saved request identity when recovering an
uncertain submission; a new identity can start another evaluation.

Use repeatable `--task ID` flags to run exact task IDs. In an interactive run,
choose **Specific tasks** to browse the selected source. Required dependencies
are added automatically and included in the displayed task count. The service
checks direct task IDs during preflight before starting work.

Use `evalrouter --help` and each command's `--help` for supported options.
Account commands include `login`, `whoami`, `workspace` and `logout`;
`credits` reads balance and holds. Payments take place on the website.

## Author your own benchmark

`benchmark init`, `benchmark validate` and `benchmark bundle` run entirely on
your machine: no account, no network request, and nothing from your package is
imported or executed. The normative validator ships inside `evalrouter-sdk`, and
its one dependency (pydantic) is an optional extra. The base installation uses
httpx, Rich and Textual. With pip, use `pip install 'evalrouter[benchmark]'`:

```sh
uv tool install "evalrouter[benchmark]"
evalrouter benchmark init ./my-benchmark --namespace example --name my-benchmark
evalrouter benchmark validate ./my-benchmark
evalrouter benchmark bundle ./my-benchmark   # evalrouter-bundle-example-my-benchmark-<version>.zip
```

Without the extra these three commands stop before reading your package and
report `benchmark_extra_required` with the exact install string; `--help` and
every network command keep working. `init` writes an explicitly synthetic draft
and claims no license: replace the tasks, rights, references and limits before
submitting. Local validation is not platform admission — EvalRouter revalidates
every submission on its own servers.

Where private benchmark imports are enabled, `benchmark upload`, `import` and
`create` use workspace-private data with the service's built-in graders. These
are account-backed operations, separate from local package validation. See the
[private benchmark guide](https://evalrouter.ai/developers/docs/byob) for manifests,
supported formats, source management and permissions.

## Python SDK

Applications and scripts use the SDK directly: `python -m pip install --pre evalrouter-sdk`
installs the typed client, `evalrouter.agents`, trace capture and the LangSmith
connector without the terminal application. See the
[Python guide](https://evalrouter.ai/developers/docs/python).
