Metadata-Version: 2.4
Name: evalrouter-sdk
Version: 0.4.4a1
Summary: The EvalRouter Python SDK: typed client, agent evaluations and trace capture
Project-URL: Homepage, https://evalrouter.ai
Project-URL: Documentation, https://evalrouter.ai/developers/docs/python
Project-URL: Model API, https://evalrouter.ai/developers/docs/model-api
License-Expression: LicenseRef-Kimpton-EvalRouter-SDK
License-File: LICENSE
Keywords: benchmarks,evalrouter,llm,model-evaluation,sdk
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.12
Requires-Dist: httpx==0.28.1
Provides-Extra: agents
Requires-Dist: agent-client-protocol==0.12.1; extra == 'agents'
Requires-Dist: pydantic==2.13.5; extra == 'agents'
Provides-Extra: benchmark
Requires-Dist: pydantic==2.13.5; extra == 'benchmark'
Provides-Extra: langsmith
Requires-Dist: langsmith==0.12.6; extra == 'langsmith'
Provides-Extra: strands
Requires-Dist: opentelemetry-api==1.45.0; extra == 'strands'
Requires-Dist: opentelemetry-sdk==1.45.0; extra == 'strands'
Requires-Dist: strands-agents==1.57.2; extra == 'strands'
Provides-Extra: tracing
Requires-Dist: opentelemetry-api==1.45.0; extra == 'tracing'
Requires-Dist: opentelemetry-sdk==1.45.0; extra == 'tracing'
Description-Content-Type: text/markdown

# EvalRouter Python SDK

The programmatic interface to EvalRouter: a typed client for discovery, quotes,
evaluations and results, local waiting and exports, `evalrouter.agents` for
evaluating an agent you run, trace capture and the LangSmith connector.

Requires Python 3.12 or 3.13. Distributed under the proprietary Kimpton EvalRouter
SDK License, included as `LICENSE` in the package.

The SDK has no terminal interface. For the `evalrouter` command and its
interactive workspace, install [`evalrouter`](https://pypi.org/project/evalrouter/)
(`uv tool install --prerelease allow evalrouter`), which depends on this package at the
same version.

## Python client

Install this alpha with `python -m pip install --pre evalrouter-sdk` in your application's environment. The
import name is `evalrouter`; the client supports context-manager cleanup:

Canonical imports require a release containing this namespace. Older registry
releases through 0.3.24 use `kimpton_evalrouter`. Select alpha releases explicitly
to use the canonical `evalrouter` import and current tracing APIs.

```python
import os
from evalrouter import Client

with Client(
    base_url="https://api.evalrouter.ai",
    api_key=os.environ["EVALROUTER_API_KEY"],
    workspace_id=os.environ["EVALROUTER_WORKSPACE_ID"],
) as client:
    print(client.catalog.models())
```

Use `client.catalog`, `client.evals`, `client.quotes` and `client.runs` for
typed discovery, submission and result resources. Creating a quote starts no evaluation; submitting its accepted
ID can start paid work. Keep the quote ID and idempotency key before submission.
The [model API guide](https://evalrouter.ai/developers/docs/model-api) describes
request fields, limits and result contracts.

`client.catalog.resolve_benchmark(name)` and `resolve_model(name, benchmark=...)`
resolve the names the command line accepts, without prompting.
`client.runs.follow(run_id)` yields a run's events until it ends;
`client.evaluation_jobs.wait(job_id)` waits for a multi-model job; `report_url`
on runs and evaluation jobs returns the web report, where the charts are drawn.

Resources and imports are stable or preview. Preview ones (including
`evalrouter.agents` and `evalrouter.tracing`) are supported but may change in a
minor release. The [Python SDK guide](https://evalrouter.ai/developers/docs/python#stability-tiers)
lists each tier and the deprecation notes for entries that left stable.

## Evaluate an agent you run

`evalrouter.agents` evaluates an agent on EvalRouter tasks while the agent runs
on your machine. EvalRouter delivers each task, meters every model call through
the run's gateway (managed models only), grades the submission with the
benchmark's own grader and records the trajectory. It is available where your
workspace has agent evaluation enabled; otherwise quoting refuses with
`client_run_unavailable`.

The agent can be an ACP Registry agent (`registry_agent("opencode")`), any
local ACP command (`local_agent("my-agent --acp")`, which needs
`evalrouter-sdk[agents]`), or an agent written in Python:

```python
from evalrouter import Client, agents


class Solver(agents.Agent):
    name = "my solver"

    async def run(self, context):
        task = await context.task.task()
        reply = await context.gateway.chat(messages=task["messages"])
        await context.task.submit(reply["choices"][0]["message"]["content"])
        await context.task.done()


client = Client()
evaluation = agents.quote(
    client,
    agent=Solver(),
    model="ADMITTED_ROUTE_ID",
    tasks=agents.benchmark("eval://PROVIDER/NAME/VERSION"),
    coverage={"mode": "sample", "sample_count": 3, "seed": 42},
    budget_usd="2",
)
# Review evaluation.quote before starting; nothing is charged until start().
run = evaluation.start()
for event in run.events():  # or: async for event in run.events_async()
    print(event.type, event.task_ref, event.failure_code or "")
print(run.wait()["status"], run.results())
```

Tasks can also come from an environment version (`agents.environment(REF,
split="test")`), your private benchmarks (`agents.private_benchmark(ID)`) or a
suite. `context.task` offers the attempt's task tools (`call`, `list`,
`instructions`, `task`, `submit`, `done`); file tasks are written in
`context.workspace`. `context.gateway` is OpenAI chat-compatible
(`gateway.openai()` returns an `AsyncOpenAI` client when `openai` is installed).
Leaving the event loop early finishes the attempt in flight as an interrupted
controller; `agents.AgentRun.resume(client, RUN_ID, agent)` continues the run
with the same agent, and `run.cancel()` stops it. `run.costs()`,
`run.export("csv")`, `run.trajectories()` and `agents.compare(client, [run_a,
run_b])` read the finalized results. `agents.evaluate(...)` does all of this in
one call. Its result's `status` and `succeeded` report the run's outcome: a
failed, partial or abandoned run is returned with `succeeded` false, as
`runs.wait` returns it, not raised. The CLI's `evalrouter run --agent` uses the
same functions and exits 0 only for a completed run (1 when it failed, 4
otherwise).

## Import LangSmith traces

The optional `langsmith` extra provides a local connector for completed agent
runs. Where your workspace has Traces, create an SDK destination connection in
**Traces → Connect agent**, then supply `LANGSMITH_API_KEY` and the connection's
EvalRouter environment variables through your secret manager. The source key
is never sent to EvalRouter.

```sh
python -m pip install --pre 'evalrouter-sdk[langsmith]'
python -m evalrouter.langsmith projects
python -m evalrouter.langsmith sync --project 'YOUR_PROJECT_ID' \
  --state './.private/evalrouter-langsmith-state.json' --metadata-only --watch
```

The process must stay running for automatic imports. Its private checkpoint
resumes progress across restarts. Set `--since` to an ISO timestamp to choose
the initial import window. This explicit metadata-only mode omits source names,
inputs, outputs, error text, tags, events and arbitrary metadata. Hierarchy,
timing, error status and reported numeric usage remain. The compatibility default
captures bounded field-name-redacted content if `--metadata-only` is omitted.

Checkpoint version 2 binds the content policy and destination. Unknown
acknowledgments replay a private adjacent `.delivery.json` payload unchanged;
keep it with the checkpoint. A policy change on an existing checkpoint is
refused. Version 1 upgrades only in its original content-capture mode and cannot
restore payload bytes lost by the older connector. Finish unresolved deliveries
before using a new checkpoint and connection for a different policy. Only
activity recorded by LangSmith can be imported; two stable closed reads are
observed coverage, not proof that every upstream child exists.

## Live agent capture

Install the alpha with `python -m pip install --pre 'evalrouter-sdk[tracing]'` in your agent's own environment.
Configure `EVALROUTER_BASE_URL`, `EVALROUTER_WORKSPACE_ID`,
`EVALROUTER_TRACE_CONNECTION_ID` and the restricted `EVALROUTER_TRACE_KEY` through
an authorized secret manager. Install alpha releases explicitly; an ordinary install selects the stable release.

```python
import json
import os
from anthropic import Anthropic
from evalrouter.tracing import TraceCapture

# Also install anthropic, and supply ANTHROPIC_API_KEY privately.
model = Anthropic(max_retries=0)
capture = TraceCapture(capture_content=True)
with capture.agent("Support assistant") as root:
    root.input({"message": "Explain what an evaluation benchmark is."})
    with capture.model(os.environ["ANTHROPIC_MODEL"]) as span:
        answer = model.messages.create(
            model=os.environ["ANTHROPIC_MODEL"], max_tokens=256,
            messages=[{"role": "user", "content": "Explain what an evaluation benchmark is."}],
        )
        text = "".join(block.text for block in answer.content if block.type == "text")
        span.output({"answer": text})
    root.output({"answer": text})
receipt = capture.seal(root.trace_id)
print(json.dumps(receipt))
if receipt["state"] != "accepted" or not receipt["complete"] or not capture.close():
    raise SystemExit(1)
```

This example makes a real paid model request. Content capture is explicitly
enabled; use `capture_content=False` to omit input, output and error payloads.
Caller attributes still require your metadata policy.
Common secret field names are redacted, but secrets in arbitrary text cannot be
detected. Content capture defaults to true; select metadata-only explicitly.

Root exit and `flush()` do not seal. Join owned children, then `seal(root.trace_id)`
to acknowledge closure. `receipt(root.trace_id)` reports local delivery state,
counts, accepted server UUID and server completion, independently of evaluation
completion. Unknown acknowledgments retain identical batches for later
flush/seal/close on the same instance. Zero newly inserted spans on replay is
success. `close()` returns false while active, failed or unknown, and true after
acknowledged closure and shutdown. A context manager calls close, but does not
raise on a false result; inspect receipts explicitly. There is no background HTTP
export. Buffers are process-local; retry before exit. Each instance retains at
most 64 local trace receipts, 512 active spans and 16 MiB of pending records.
When another root needs room, the oldest successfully sealed idle receipt is
evicted. Save the receipt/server UUID if you need it later; `receipt(id)` raises
`ValueError` after eviction. Pending, unknown, active or dropped traces are never
evicted. If all 64 are unsettled, a new root is not recorded (its handle has no
trace ID and `failed_exports` increases); resolve delivery or use another instance.
Late activity inherited from an evicted trace is rejected, never made a new root.
Close attempts every retained trace within one aggregate deadline and leaves the
provider alive if any closure fails. Capture loss prevents successful seal. The optional OpenTelemetry dependency stays
isolated from the base HTTP client.

Create a connection under **Traces → Connect agent** where Traces is enabled for
your workspace. Detailed delivery receipts are described in the trace connection setup in your workspace.


## Attach existing OpenTelemetry instrumentation

Install the alpha `evalrouter-sdk[tracing]` extra into your existing Python
3.12/3.13 agent environment. Pass the OTel SDK provider the framework actually
uses; do not replace its global provider. No agent inheritance or per-call
EvalRouter wrappers are required for supported spans:

```python
from evalrouter.tracing import TraceCapture

capture = TraceCapture(capture_content=False)
attachment = capture.attach(existing_provider, scope_names=("your.instrumentation",))
# Run the already-instrumented agent, then join all owned child work.
for trace_id in attachment.trace_ids:
    receipt = capture.seal(trace_id)
    print(receipt)  # IDs/counts/delivery state only; inspect accepted + complete.
# At application shutdown, check capture.close(); it never stops existing_provider.
```

Attachment exports metadata only, even when wrapper content capture is enabled.
It removes source names, resource attributes, events, prompts, tool arguments,
results and exception text. It retains original trace/span/parent IDs, closed
operation/kind/status enums and bounded numeric token counters. Source counters
are observations, not native usage charges. `evalrouter.otel.status=unset` means
unknown source success: wire `ok` only means no explicit source ERROR, never a
native grade or proof that a tool succeeded. Unknown/conflicting kinds are
counted in `attachment.unsupported_spans` and prevent successful sealing.
Provider flush queues records; it does not prove accepted ingestion or closure.

The optional `[strands]` extra pins Strands for applications that need it; the
base client has no Strands dependency. Attachment only sees selected spans that
the supplied provider actually records. It cannot recover unsampled or missing
work, or capture uninstrumented frameworks automatically. LangSmith project
imports and ACP evaluation execution are separate integrations.

## Native multi-model client requests

Preview the matrix before starting paid work. Save the request and idempotency
key so an uncertain reply can be recovered without starting another job.

```python
from evalrouter import Client

with Client() as client:
    request = {
        "name": "Model comparison",
        "candidates": [
            {"model": {"kind": "managed", "route_id": "YOUR_FIRST_ROUTE"}},
            {"model": {"kind": "managed", "route_id": "YOUR_SECOND_ROUTE"}},
        ],
        "evaluations": [{
            "selection": {"profile_ids": ["YOUR_EXACT_NATIVE_PROFILE"]},
            "coverage": {"mode": "sample", "sample_count": 1, "seed": 42},
        }],
    }
    preview = client.evaluation_jobs.preflight(request)
    if preview["supported"]:
        job = client.evaluation_jobs.start(
            {**request, "preflight_digest": preview["digest"]},
            idempotency_key="persist-this-job-request-key",
        )
        print(job["id"])
```

The job's selected models or agents and benchmarks stay together in the results
board. Each runnable pair has its own run and receipt. Charges draw from the
workspace wallet as work proceeds. Existing capped quotes keep their terms.
