Metadata-Version: 2.4
Name: evalrouter-sdk
Version: 0.5.0
Summary: The EvalRouter Python SDK: typed client, agent evaluations and trace capture
Project-URL: Homepage, https://evalrouter.ai
Project-URL: Documentation, https://evalrouter.ai/developers/docs
Project-URL: Model API, https://evalrouter.ai/developers/docs/model-api
License-Expression: LicenseRef-Kimpton-EvalRouter-SDK
License-File: LICENSE
Keywords: benchmarks,evalrouter,llm,model-evaluation,sdk
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.12
Requires-Dist: httpx==0.28.1
Requires-Dist: pydantic==2.13.5
Provides-Extra: agents
Requires-Dist: agent-client-protocol==0.12.1; extra == 'agents'
Requires-Dist: pydantic==2.13.5; extra == 'agents'
Provides-Extra: benchmark
Requires-Dist: pydantic==2.13.5; extra == 'benchmark'
Provides-Extra: langsmith
Requires-Dist: langsmith==0.12.6; extra == 'langsmith'
Provides-Extra: strands
Requires-Dist: opentelemetry-api==1.45.0; extra == 'strands'
Requires-Dist: opentelemetry-sdk==1.45.0; extra == 'strands'
Requires-Dist: strands-agents==1.57.2; extra == 'strands'
Provides-Extra: tracing
Requires-Dist: opentelemetry-api==1.45.0; extra == 'tracing'
Requires-Dist: opentelemetry-sdk==1.45.0; extra == 'tracing'
Description-Content-Type: text/markdown

# EvalRouter Python SDK

Evaluate a model or an agent on a benchmark in one call. Requires Python 3.12 or
3.13. Distributed under the proprietary Kimpton EvalRouter SDK License, included
as `LICENSE` in the package.

```sh
python -m pip install evalrouter-sdk
```

The command line is a separate package: `uv tool install evalrouter`
(it depends on this SDK at the same version).

## Run a benchmark

```python
from evalrouter import EvalRouter

client = EvalRouter()  # EVALROUTER_API_KEY and EVALROUTER_BASE_URL from the environment
run = client.runs.create(benchmark="gsm8k", model="gpt-5", max_cost=5, wait=True)
print(run.score)
```

The same run from a terminal:

```sh
evalrouter run gsm8k --model gpt-5 --max-cost 5
evalrouter run gsm8k --model gpt-5 --max-cost 5 --dry-run   # the quote; nothing is charged
```

- **Names, not IDs.** `benchmark` and `model` take a slug or an unambiguous name
  (`"gsm8k"`, `"gsm8k@2"` to pin a version, `"gpt-5"`). Exact IDs work too.
- **Money is USD.** `max_cost=5` or `max_cost="0.25"` is the most the run may
  charge; the run is refused before it starts if its quote is higher. `max_cost`
  is optional: without it the run is bounded by your wallet balance. Your API
  key's spend ceiling bounds everything.
- **Waiting is one flag.** `wait=True` returns the finished run; without it,
  `run.wait()` later. `run.results()` reads the results, `run.score` is the
  headline metric, `run.cancel()` stops it and `run.export(format="csv")` exports it.
- **Look before you spend.** `dry_run=True` returns the quote (maximum charge and
  plan) and starts nothing; `client.runs.create(quote_id=quote.id)` starts it.
  Repeating a create never starts a second run.

Advanced quote fields are optional keywords: `coverage={"mode": "sample",
"sample_count": 50, "seed": 42}`, `concurrency`, `name`, `metadata`.
`AsyncEvalRouter` has the same calls, awaited.

## Evaluate an agent

The agent is one argument:

```python
client.runs.create(benchmark="gsm8k", model="gpt-5", agent="opencode")  # ACP Registry agent
client.runs.create(benchmark="gsm8k", model="gpt-5", agent="local:my-agent --acp")
client.runs.create(benchmark="gsm8k", agent="remote:my-a2a-agent")  # a connection's name
client.runs.create(
    benchmark="gsm8k", model="gpt-5", agent="opencode", hosted=True
)  # run hosted by EvalRouter
```

Registry and local agents run on your machine: EvalRouter delivers each task,
meters every model call through the run's gateway and grades the submission
with the benchmark's own grader. `evalrouter agents list` shows the registry
agents with their integrity: `pinned` (a binary with a published sha256),
`package-registry` (an exact npm or PyPI package) or `unverified-upstream`,
which runs only when you allow it (`agents.registry_agent("cursor",
allow_unverified=True)`, or `--allow-unverified` on the command line).
`registry_agent("opencode@1.18.34")` pins a version; without one the pinned
snapshot's version runs.

An agent written in Python is an `agents.Agent`:

```python
from evalrouter import EvalRouter, agents


class Solver(agents.Agent):
    name = "my solver"

    async def run(self, context):
        task = await context.task.get()
        reply = await context.gateway.chat(messages=task["inputs"]["messages"])
        await context.task.submit(reply["choices"][0]["message"]["content"])
        await context.task.done()


run = EvalRouter().runs.create(
    benchmark="gsm8k", model="gpt-5", agent=Solver(), max_cost=2, wait=True
)
```

For more control, `evalrouter.agents` has the step-by-step form:
`agents.quote(...)` returns an evaluation whose `start()` gives an `AgentRun`
with `events()`, `wait()` and `resume()`.

## Import LangSmith traces

The optional `langsmith` extra provides a local connector for completed agent
runs. Where your workspace has Traces, create an SDK destination connection in
**Traces → Connect agent**, then supply `LANGSMITH_API_KEY` and the connection's
EvalRouter environment variables through your secret manager. The source key
is never sent to EvalRouter.

```sh
python -m pip install 'evalrouter-sdk[langsmith]'
python -m evalrouter.langsmith projects
python -m evalrouter.langsmith sync --project 'YOUR_PROJECT_ID' \
  --state './.private/evalrouter-langsmith-state.json' --metadata-only --watch
```

The process must stay running for automatic imports. Its private checkpoint
resumes progress across restarts. Set `--since` to an ISO timestamp to choose
the initial import window. This explicit metadata-only mode omits source names,
inputs, outputs, error text, tags, events and arbitrary metadata. Hierarchy,
timing, error status and reported numeric usage remain. The compatibility default
captures bounded field-name-redacted content if `--metadata-only` is omitted.

Checkpoint version 2 binds the content policy and destination. Unknown
acknowledgments replay a private adjacent `.delivery.json` payload unchanged;
keep it with the checkpoint. A policy change on an existing checkpoint is
refused. Version 1 upgrades only in its original content-capture mode and cannot
restore payload bytes lost by the older connector. Finish unresolved deliveries
before using a new checkpoint and connection for a different policy. Only
activity recorded by LangSmith can be imported; two stable closed reads are
observed coverage, not proof that every upstream child exists.

## Live agent capture

Install with `python -m pip install 'evalrouter-sdk[tracing]'` in your agent's own environment.
Configure `EVALROUTER_BASE_URL`, `EVALROUTER_WORKSPACE_ID`,
`EVALROUTER_TRACE_CONNECTION_ID` and the restricted `EVALROUTER_TRACE_KEY` through
an authorized secret manager. Package installation and connection authorization are separate.

```python
import json
import os
from anthropic import Anthropic
from evalrouter.tracing import TraceCapture

# Also install anthropic, and supply ANTHROPIC_API_KEY privately.
model = Anthropic(max_retries=0)
capture = TraceCapture(capture_content=True)
with capture.agent("Support assistant") as root:
    root.input({"message": "Explain what an evaluation benchmark is."})
    with capture.model(os.environ["ANTHROPIC_MODEL"]) as span:
        answer = model.messages.create(
            model=os.environ["ANTHROPIC_MODEL"],
            max_tokens=256,
            messages=[{"role": "user", "content": "Explain what an evaluation benchmark is."}],
        )
        text = "".join(block.text for block in answer.content if block.type == "text")
        span.output({"answer": text})
    root.output({"answer": text})
receipt = capture.seal(root.trace_id)
print(json.dumps(receipt))
if receipt["state"] != "accepted" or not receipt["complete"] or not capture.close():
    raise SystemExit(1)
```

This example makes a real paid model request. Content capture is explicitly
enabled; use `capture_content=False` to omit input, output and error payloads.
Caller attributes still require your metadata policy.
Common secret field names are redacted, but secrets in arbitrary text cannot be
detected. Content capture defaults to true; select metadata-only explicitly.

Root exit and `flush()` do not seal. Join owned children, then `seal(root.trace_id)`
to acknowledge closure. `receipt(root.trace_id)` reports local delivery state,
counts, accepted server UUID and server completion, independently of evaluation
completion. Unknown acknowledgments retain identical batches for later
flush/seal/close on the same instance. Zero newly inserted spans on replay is
success. `close()` returns false while active, failed or unknown, and true after
acknowledged closure and shutdown. A context manager calls close, but does not
raise on a false result; inspect receipts explicitly. There is no background HTTP
export. Buffers are process-local; retry before exit. Each instance retains at
most 64 local trace receipts, 512 active spans and 16 MiB of pending records.
When another root needs room, the oldest successfully sealed idle receipt is
evicted. Save the receipt/server UUID if you need it later; `receipt(id)` raises
`ValueError` after eviction. Pending, unknown, active or dropped traces are never
evicted. If all 64 are unsettled, a new root is not recorded (its handle has no
trace ID and `failed_exports` increases); resolve delivery or use another instance.
Late activity inherited from an evicted trace is rejected, never made a new root.
Close attempts every retained trace within one aggregate deadline and leaves the
provider alive if any closure fails. Capture loss prevents successful seal. The optional OpenTelemetry dependency stays
isolated from the base HTTP client.

Create a connection under **Traces → Connect agent** where Traces is enabled for
your workspace. Detailed delivery receipts are described in the trace connection setup in your workspace.


## Attach existing OpenTelemetry instrumentation

Install the `evalrouter-sdk[tracing]` extra into your existing Python
3.12/3.13 agent environment. Pass the OTel SDK provider the framework actually
uses; do not replace its global provider. No agent inheritance or per-call
EvalRouter wrappers are required for supported spans:

```python
from evalrouter.tracing import TraceCapture

capture = TraceCapture(capture_content=False)
attachment = capture.attach(existing_provider, scope_names=("your.instrumentation",))
# Run the already-instrumented agent, then join all owned child work.
for trace_id in attachment.trace_ids:
    receipt = capture.seal(trace_id)
    print(receipt)  # IDs/counts/delivery state only; inspect accepted + complete.
# At application shutdown, check capture.close(); it never stops existing_provider.
```

Attachment exports metadata only, even when wrapper content capture is enabled.
It removes source names, resource attributes, events, prompts, tool arguments,
results and exception text. It retains original trace/span/parent IDs, closed
operation/kind/status enums and bounded numeric token counters. Source counters
are observations, not native usage charges. `evalrouter.otel.status=unset` means
unknown source success: wire `ok` only means no explicit source ERROR, never a
native grade or proof that a tool succeeded. Unknown/conflicting kinds are
counted in `attachment.unsupported_spans` and prevent successful sealing.
Provider flush queues records; it does not prove accepted ingestion or closure.

The optional `[strands]` extra pins Strands for applications that need it; the
base client has no Strands dependency. Attachment only sees selected spans that
the supplied provider actually records. It cannot recover unsampled or missing
work, or capture uninstrumented frameworks automatically. LangSmith project
imports and ACP evaluation execution are separate integrations.
