# Autobench Full Documentation

> Complete, LLM-readable documentation for Autobench: a YAML-first framework for semantic, replayable benchmark evidence.

Canonical documentation: https://vcoderun.github.io/autobench/

This file follows the site navigation order. Each section includes the canonical page URL followed by its complete Markdown source.

---

## Home

Canonical page: https://vcoderun.github.io/autobench/

# Autobench

**Build benchmarks once. Keep the evidence, semantics, lineage, and replay.**

Autobench is a YAML-first Python framework for evaluating AI and non-AI systems. It replaces
one-off benchmark runners with a reusable runtime that expands datasets across variants, executes
application-owned tasks, collects semantic evidence, evaluates outcomes, and writes immutable run
records.

```bash
uv add autobench
autobench validate autobench.yaml
autobench run autobench.yaml --record runs/latest
```

The same recorded experiment can then be inspected without executing the application again:

```bash
autobench replay runs/latest
autobench report runs/latest
autobench compare runs/latest --baseline current --candidate proposed
autobench export runs/latest --format csv --path analysis/runs.csv
```

## The Framework Loop

```text
BenchmarkSpec
  Dataset[Case] x Variant[Factor]
    -> task(ctx, case)
    -> observations + ABP trace + artifacts + asset versions
    -> scorers + per-run derivation
  -> cross-run derivation + policies
  -> immutable RunRecords
  -> replay + Rich reports + comparison + exports + optimization feedback
```

The task is the only application-specific part. Autobench owns the repeated infrastructure around
it: matrix planning, context propagation, instrumentation, scoring, derivation, persistence,
reporting, and replay.

## Why Semantic Evidence Matters

Raw names such as `prompt_tokens`, `input_tokens`, `accuracy`, and `answer_quality` are local
conventions. Autobench observations can also declare stable meaning:

```text
llm.tokens.input
llm.tokens.output
llm.model.name
quality.correctness
time.latency
money.cost
agent.tool.argument.correctness
```

That semantic layer lets reports, pricing derivation, policy checks, and optimization systems use
evidence from different applications without guessing what every local metric name means.

## What You Can Benchmark

Autobench is optimized for AI systems but does not require one:

| System | Cases | Variants | Evidence |
| --- | --- | --- | --- |
| LLM application | prompts and expected answers | model, prompt, temperature | quality, tokens, latency, cost |
| Agent | user goals and expected actions | instructions, tools, model | action selection, arguments, sequence, completion |
| Search or retrieval | queries and relevant items | index, reranker, limits | recall, precision, latency |
| Service/API | requests and expected responses | release, configuration | correctness, errors, throughput, SLA |
| Algorithm | input fixtures | implementation | correctness, repeated timings, speedup |
| Data pipeline | source batches | parser or policy | coverage, validity, loss, runtime |

See [Use Cases](use-cases.md) for complete patterns.

## Core Capabilities

| Area | Included |
| --- | --- |
| Definition | Human-readable YAML DSL, typed Python builder, JSON Schema completion |
| Data | Inline/file/glob datasets, defaults, attachments, generated and production cases |
| Execution | Sync/async tasks, deterministic matrices, bounded concurrency, failure isolation |
| Evidence | Semantic observations, checks, events, artifacts, measurements, ABP traces |
| Evaluation | Built-in and custom scorers, expected actions, policies, metric packs |
| Derivation | Token cost, tiered pricing, paired baselines, comparison verdicts |
| Instrumentation | Manual spans, method instrumentation, Pydantic AI, OpenAI, Agents, HTTPX |
| Lineage | Explicit and automatic prompt/tool/schema/capability/agent asset versioning |
| Persistence | Immutable YAML records, source hashes, environment metadata, portable artifacts |
| Analysis | Replay, Rich reports, leaderboards, matrices, distributions, comparisons, exports |

## Choose A Starting Point

| Goal | Read |
| --- | --- |
| Install the right extras | [Installation](installation.md) |
| Run a complete benchmark | [First Benchmark](getting-started.md) |
| Find a pattern for your system | [Use Cases](use-cases.md) |
| Understand ownership and data flow | [Architecture](architecture.md) |
| Author the full DSL | [YAML Spec](yaml-spec.md) |
| Compose benchmarks in Python | [Python API](python-api.md) |
| Instrument an existing SDK application | [Native Instrumentation](native-instrumentation.md) |
| Collect prompt/tool/schema lineage automatically | [Automatic Asset Discovery](automatic-asset-discovery.md) |
| Inspect all shipped features | [Capability Map](capabilities.md) |

## Project Boundaries

Autobench records and evaluates evidence. It does not own your application, make causal claims from
confounded runs, keep provider pricing permanently current, or choose an optimization algorithm.
Those boundaries keep the core usable for arbitrary systems while allowing pydantic-gepa,
autoptimize, or another consumer to build on stable experiment records.

---

## Installation

Canonical page: https://vcoderun.github.io/autobench/installation/

# Installation

Autobench supports Python 3.11 through 3.14. The base package includes the benchmark DSL, runtime,
semantic evidence models, evaluation, recording, replay, reports, CLI, and manual ABP spans.

## Base Package

=== "uv"

    ```bash
    uv add autobench
    ```

=== "pip"

    ```bash
    python -m pip install autobench
    ```

Verify the installation:

```bash
autobench --help
python -c "import autobench; print(autobench.__version__)"
```

## Optional SDK Integrations

Native ABP instrumentors are optional so a generic benchmark does not install AI SDKs.

```bash
uv add 'autobench[instrumentation]'
```

The instrumentation extra supplies the supported Pydantic AI, OpenAI Python, and HTTPX integration
environment. OpenAI Agents support has its own extra:

```bash
uv add 'autobench[openai-agents]'
```

Inspect what the current environment can instrument:

```bash
autobench instrumentation doctor
```

The command reports compatibility rather than failing because an optional SDK is absent.

## Development Checkout

From the repository root:

```bash
uv sync --extra dev --extra instrumentation --extra openai-agents
make prod
```

Useful targets:

| Command | Purpose |
| --- | --- |
| `make tests` | Test suite with source line and branch coverage |
| `make check` | Ruff, ty, and basedpyright |
| `make docs` | LLM bundles and strict Zensical build |
| `make examples` | Offline end-to-end example matrix |
| `make prod` | Full supported-Python and release quality gates |
| `make pre-commit` | Repository-wide hooks |

## Editor Setup For YAML

Every exported Autobench YAML document starts with a `yaml-language-server` schema directive.
Versioned schemas are shipped under `schemas/<autobench-version>/` and installed to the user schema
directory when the schema helpers run.

For a repository-local benchmark:

```yaml
# yaml-language-server: $schema=./schemas/0.2.0/benchmark_schema.json
benchmark:
  smoke-test:
    cases: []
```

Use the schema matching the Autobench version that validates and executes the file. This provides
completion for scorer variants, policy operators, instrumentation settings, report configuration,
and semantic registry entries.

## Credential Handling

Autobench itself does not require model credentials. Live examples read provider configuration from
the relevant SDK environment. For example:

```bash
export OPENROUTER_API_KEY=...
export OPENROUTER_MODEL=openrouter:openai/gpt-5.6-luna
```

Do not place credentials in benchmark specs, cases, artifacts, or capture policies. Use the
[capture policy](automatic-asset-discovery.md#privacy-and-capture-policy) to prevent sensitive SDK
inputs from being retained.

## Next Step

Continue with [First Benchmark](getting-started.md), which creates a task, dataset, variant matrix,
score, record, report, comparison, and export.

---

## First Benchmark

Canonical page: https://vcoderun.github.io/autobench/getting-started/

# First Benchmark

This guide builds a complete deterministic benchmark. It compares two text transformations, scores
their outputs, records every case and variant, and replays the result.

## Project Layout

```text
text-benchmark/
  autobench.yaml
  benchmark_task.py
```

The task remains ordinary application code. The YAML file describes how Autobench should execute
and evaluate it.

## Write The Task

Create `benchmark_task.py`:

```python
from __future__ import annotations

from typing import Literal

from pydantic import BaseModel, TypeAdapter

from autobench import Case, RunContext

Transform = Literal["upper", "title_upper"]
TRANSFORM = TypeAdapter(Transform)


class TextInput(BaseModel):
    text: str


class TextOutput(BaseModel):
    text: str


def run(ctx: RunContext, case: Case) -> TextOutput:
    sample = TextInput.model_validate(case.input)
    transform = TRANSFORM.validate_python(ctx.factor("transform"))

    with ctx.span(
        "transform_text",
        kind="workflow",
        input=sample.model_dump(),
        attributes={"transform": transform},
    ) as span:
        text = sample.text.upper()
        if transform == "title_upper":
            text = sample.text.title().upper()
        output = TextOutput(text=text)
        span.set_output(output.model_dump())
        return output
```

The required task signature is `task(ctx, case)`: `RunContext` is always first and `Case` is always
second. Sync and async functions are both supported. Span duration is measured by Autobench.

## Define The Benchmark

Create `autobench.yaml`:

```yaml
# yaml-language-server: $schema=./schemas/0.2.0/benchmark_schema.json
benchmark:
  text-transform:
    description: Compare deterministic text transformations.
    cases:
      - id: greeting
        input:
          text: hello autobench
        expected:
          text: HELLO AUTOBENCH
      - id: whitespace
        input:
          text: release ready
        expected:
          text: RELEASE READY
    run:
      python: benchmark_task:run
    variants:
      current:
        label: Current implementation
        factors:
          transform: upper
      proposed:
        label: Proposed implementation
        factors:
          transform:
            value: title_upper
            optimize: true
    score:
      exact_text:
        exact:
          actual: output.text
          expected: case.expected.text
        semantic: quality.correctness
        goal: maximize
        role: objective
    report:
      leaderboard:
        show:
          correctness:
            metric: quality.correctness
            aggregate: ratio_true
      matrix:
        metric: quality.correctness
      compare:
        current -> proposed:
          show:
            correctness:
              metric: quality.correctness
              aggregate: ratio_true
```

This produces four runs: two cases multiplied by two variants.

## Validate Before Running

From `text-benchmark/`:

```bash
autobench validate autobench.yaml
```

Validation parses the DSL, resolves the task and source files relative to the spec, loads external
datasets and pricing files, verifies unique IDs, and displays the planned matrix. It does not call
the task.

## Run And Record

```bash
autobench run autobench.yaml --record runs/text-transform
```

Autobench renders Rich terminal tables and writes:

```text
runs/text-transform/
  experiment.yaml
  summary.yaml
  cases/
    greeting/current/run.yaml
    greeting/proposed/run.yaml
    whitespace/current/run.yaml
    whitespace/proposed/run.yaml
  artifacts/
```

The actual run filenames use stable run IDs inside the case and variant directories. The records
include the case snapshot, factors, output, observations, score, trace, source hashes, environment,
and status.

## Replay And Analyze

```bash
autobench replay runs/text-transform
autobench report runs/text-transform
autobench compare runs/text-transform --baseline current --candidate proposed
```

These commands load recorded evidence. They do not import `benchmark_task.py` and do not execute the
subject again. Comparison reports factor changes and metric deltas but does not claim that a
confounded difference is causal.

## Export A Projection

```bash
autobench export runs/text-transform \
  --format yaml \
  --path analysis/text-transform.yaml

autobench export runs/text-transform \
  --format csv \
  --path analysis/text-transform.csv
```

Terminal output stays human-oriented and uses Rich tables. YAML, CSV, and Markdown are file export
formats.

## Add Runtime Evidence

Tasks can emit evidence that is not part of the return value:

```python
ctx.metric(
    "characters",
    len(output.text),
    semantic_type="text.characters",
    unit="count",
)
ctx.check("not_empty", bool(output.text), reason="The transformed text must not be empty.")
ctx.artifact("output", output.model_dump(), media_type="application/yaml")
```

Use scores for evaluation results, observations for runtime facts, and artifacts for payloads that
must remain inspectable.

## Run Concurrently

```bash
autobench run autobench.yaml \
  --concurrency 4 \
  --record runs/text-transform-concurrent
```

The matrix order and run IDs remain deterministic. ABP context is task-local, so concurrent runs do
not share parent spans or evidence.

## Next Steps

- Move cases to a file: [Datasets And Variants](datasets-and-variants.md)
- Add quality, cost, and policy gates: [Scoring And Derivation](scoring-and-derivation.md)
- Instrument an SDK automatically: [Native Instrumentation](native-instrumentation.md)
- Track prompt and tool versions: [Automatic Asset Discovery](automatic-asset-discovery.md)
- Select a complete pattern: [Use Cases](use-cases.md)

---

## Use Cases

Canonical page: https://vcoderun.github.io/autobench/use-cases/

# Use Cases

The same Autobench runtime supports deterministic functions, services, LLM applications, agents,
and performance experiments. The patterns below show where domain code ends and framework
infrastructure begins.

## Choose A Pattern

| Need | Core primitives |
| --- | --- |
| Compare implementations | cases, variants, exact/pass scorers, comparison |
| Measure noisy performance | `measure_callable`, sample artifacts, paired baseline |
| Compare LLM quality and cost | semantic token metrics, pricing derivation, policies |
| Evaluate agent behavior | ABP tool spans, expected actions, span selectors |
| Instrument an existing AI app | `instrument_all()`, native SDK instrumentors |
| Track prompts/tools/schemas | explicit tracking or automatic asset discovery |
| Turn production failures into regressions | `ProductionSample`, sampling policy, reviewed cases |
| Feed an optimizer | objectives, constraints, factors, asset versions, feedback records |

## Application Regression Benchmark

Use a file-backed dataset when the benchmark is a maintained regression suite:

```yaml
# yaml-language-server: $schema=./schemas/0.2.0/benchmark_schema.json
benchmark:
  support-routing:
    dataset:
      source: file://datasets/tickets.yaml
      version: "2026-08-06"
      defaults:
        tags: [regression]
    run:
      python: benchmark_tasks:route_ticket
    variants:
      production:
        factors:
          routing_profile: v3
      candidate:
        factors:
          routing_profile:
            value: v4
            optimize: true
    score:
      route:
        exact:
          actual: output.queue
          expected: case.expected.queue
        semantic: quality.correctness
        goal: maximize
        role: objective
      handled:
        pass: output.handled
        semantic: result.success
        role: constraint
```

Keep routing logic in `benchmark_tasks.py`. Autobench handles matrix expansion, status isolation,
score projection, and comparison. This pattern also fits parsers, validators, ranking functions,
API clients, and data transformations.

## Repeated Performance Measurement

Do not hand-roll warmup, repetition budgets, percentiles, or sample artifacts:

```python
from autobench import Case, RunContext, Semantic, measure_callable


def run(ctx: RunContext, case: Case) -> dict[str, bool]:
    values = list(case.input["values"])
    target = int(case.input["target"])
    strategy = str(ctx.factor("strategy"))

    def execute() -> None:
        if strategy == "linear":
            target in values
        else:
            target in set(values)

    measurement = measure_callable(
        execute,
        warmup=3,
        repetitions=25,
        max_seconds=2.0,
    )
    ctx.record_measurement(
        "lookup_latency",
        measurement,
        semantic_type=Semantic.TIME_LATENCY,
        include_samples_artifact=True,
    )
    return {"found": target in values}
```

Derive candidate speedup only after both matched runs exist:

```yaml
post_derive:
  - kind: paired_baseline
    baseline_variant: linear
    match_on: case_id
    metric: time.latency
    formula: baseline_over_candidate
    include_baseline: true
    output:
      name: speedup
      semantic_type: performance.speedup
      unit: ratio
      direction: maximize
      role: objective
```

Correctness should remain a constraint. A faster wrong implementation is not a successful
candidate.

## LLM Quality, Usage, And Cost

Instrumentors or tasks record usage as semantic observations:

```python
ctx.metric(
    "input_tokens",
    usage.input_tokens,
    semantic_type="llm.tokens.input",
    unit="token",
)
ctx.metric(
    "output_tokens",
    usage.output_tokens,
    semantic_type="llm.tokens.output",
    unit="token",
)
ctx.factor_observation("model", model_name, semantic_type="llm.model.name")
ctx.factor_observation("provider", provider, semantic_type="llm.provider")
```

Cost remains a derivation instead of being hard-coded into Autobench instrumentation:

```yaml
derive:
  - kind: token_cost
    pricing: file://pricing/models.yaml
    output:
      name: request_cost
      semantic_type: money.cost
      unit: usd
      direction: minimize
      role: constraint
policies:
  - name: quality-floor
    metric: quality.correctness
    must_greater_equal: 0.9
  - name: per-request-budget
    metric: money.cost
    must_less_equal: 0.01
```

The pricing file can normalize provider-specific model identifiers, aliases, cache prices, and
tiered input/output rates. Price sources are convenience adapters into this format; Autobench does
not become a live pricing service.

## Existing Pydantic AI Application

For a Pydantic AI application, automatic instrumentation removes task-level telemetry:

```python
from autobench import Benchmark, Case, ExactScorer, Semantic

benchmark = (
    Benchmark("support-agent")
    .dataset(
        [
            Case(
                id="order-status",
                input="Where is order A-42?",
                expected={"status": "delayed"},
            )
        ]
    )
    .variants(
        [
            {
                "id": "luna",
                "factors": {
                    "model": "openrouter:openai/gpt-5.6-luna",
                },
            }
        ]
    )
    .task("support_benchmark:run")
    .scoring(
        [
            ExactScorer(
                name="status",
                actual="output.status",
                expected="case.expected.status",
                semantic_type=Semantic.QUALITY_CORRECTNESS,
            )
        ]
    )
    .instrument_all()
)
```

The task can contain only the agent call. Compatible instrumentors collect Pydantic AI agent/model/
tool/validation activity, the OpenAI-compatible client layer, and HTTPX transport evidence. The run
also receives automatically discovered prompt, tool, output-schema, capability, and agent versions
when those values cross supported SDK boundaries.

Use `exclude={"httpx"}` to avoid transport spans or select a narrower asset family:

```python
benchmark.instrument_all(
    exclude={"httpx"},
    assets={
        "representations": ["definition", "effective"],
        "include": ["prompt", "tool", "output_schema"],
    },
)
```

## Agent Tool Selection And Arguments

Agent evaluation should use execution evidence, not only final text. Declare expected actions in
the case:

```yaml
cases:
  - id: refund-order
    input:
      message: Refund order A-42
    expected:
      actions:
        - tool: lookup_order
          args:
            order_id: A-42
          order: 1
        - tool: issue_refund
          args:
            order_id: A-42
          order: 2
```

Then score the tool spans:

```yaml
score:
  tool_selection:
    expected_action:
      metric: selection
      span:
        kind: tool
    semantic: agent.tool.selection.correctness
    goal: maximize
    role: objective
  tool_arguments:
    expected_action:
      metric: arguments
      span:
        kind: tool
    semantic: agent.tool.argument.correctness
    goal: maximize
    role: objective
  tool_sequence:
    expected_action:
      metric: sequence
      span:
        kind: tool
    semantic: agent.tool.sequence.correctness
    goal: maximize
    role: constraint
```

This works with manually recorded tool spans and native SDK traces. It does not require an LLM judge
for deterministic action contracts.

## Custom SDK Without Application Changes

When an SDK is not built in, instrument a stable method and declare both evidence and assets:

```python
from autobench import (
    InstrumentAssetSpec,
    InstrumentMetricSpec,
    Semantic,
    SpanKind,
    instrument_method,
)

instrument_method(
    WorkflowClient,
    "execute",
    span="workflow.execute",
    span_kind=SpanKind.WORKFLOW,
    metrics=[
        InstrumentMetricSpec(
            name="confidence",
            semantic_type=Semantic.QUALITY_SCORE,
            value_path="result.confidence",
        ),
    ],
    assets=[
        InstrumentAssetSpec(
            kind="prompt",
            local_id="instructions",
            value_path="kwargs.instructions",
            name="routing_instructions",
        ),
        InstrumentAssetSpec(
            kind="tool",
            local_id="tools",
            value_path="kwargs.tools",
            many=True,
        ),
        InstrumentAssetSpec(
            kind="output_schema",
            local_id="output",
            value_path="kwargs.output_type",
            name="routing_output",
        ),
    ],
)
```

Serializable configurations use `value_path` or an import target. Typed Python integrations may use
`value_factory` for extraction that cannot be represented as a path. Keep domain computation in the
application; instrumentation should describe stable boundaries and evidence extraction.

## Production Failures As Regression Cases

Convert selected production samples into cases without coupling the benchmark to a production
database:

```python
from autobench import (
    ProductionSample,
    SampleReason,
    SamplingPolicy,
    samples_to_cases,
)

samples = [
    ProductionSample(
        id="trace-1842",
        input={"message": "Refund order A-42"},
        expected={"route": "billing"},
        reason=SampleReason.FAILURE_ONLY,
        privacy_tags=("customer_text",),
    )
]

cases = samples_to_cases(
    samples,
    policy=SamplingPolicy(
        reasons=(SampleReason.FAILURE_ONLY,),
        max_samples=100,
    ),
)
```

Review state, source reason, timestamp, trace identity, and privacy tags become metadata. Promote
reviewed cases into a versioned YAML dataset before using them as a release gate.

## Synthetic Case Generation With Provenance

Autobench does not own a model-based generator, but it preserves generated-data lineage:

```python
from autobench import Case, generated_batch_from_cases

batch = generated_batch_from_cases(
    [Case(id="edge-1", input={"message": "..."})],
    generator_asset_version="prompt.generate_cases@82ab39",
    model_provider="openrouter",
    model_name="openai/gpt-5.6-luna",
)
```

Generated cases remain candidates until reviewed. This keeps generator behavior and benchmark truth
from collapsing into the same untracked process.

## CI Regression Gate

A typical CI job validates, executes, stores artifacts, and checks policy state:

```bash
set -e
autobench validate benchmarks/release.yaml
autobench run benchmarks/release.yaml \
  --concurrency 4 \
  --record artifacts/autobench-release
autobench report artifacts/autobench-release
autobench export artifacts/autobench-release \
  --format csv \
  --path artifacts/autobench-runs.csv
```

Persist the whole record directory, not only the CSV. The CSV is a projection; the immutable YAML
records and artifacts contain replay, lineage, source, and diagnostic evidence.

## Optimization Handoff

Autobench marks metrics by role and direction:

- objective: improve this metric;
- constraint: do not violate this boundary;
- diagnostic: explain behavior without becoming an objective.

Factors can set `optimize: true`, and tracked assets identify the exact prompt/tool/schema versions
used. Convert a recorded run into compact feedback:

```python
from pathlib import Path

from autobench import build_optimization_feedback_input, load_run_record

record = load_run_record(
    Path("runs/latest/cases/refund-order/candidate/run.yaml"),
    root_dir=Path("runs/latest"),
)
feedback = build_optimization_feedback_input(record)
```

An optimizer should propose candidates and run controlled validation experiments. Autobench supplies
evidence and comparison; it does not claim that independently best assets can be mixed safely.

## Replay-Only Analysis

Recorded evidence supports analysis in an environment without the application or provider SDKs:

```python
from pathlib import Path

from autobench import build_report, replay_experiment

experiment = replay_experiment(Path("runs/latest"))
report = build_report(experiment)
```

This is the correct boundary for dashboards, offline reports, audits, post-hoc extraction, and
optimizer data ingestion.

---

## Example Projects

Canonical page: https://vcoderun.github.io/autobench/examples/

# Example Projects

The repository examples use the public Autobench runtime. They are ordered by the amount of
framework surface they demonstrate, not by whether the subject is AI-based.

## Offline Release Matrix

These examples are credential-free and run in `make examples`:

| Example | Subject | Main features |
| --- | --- | --- |
| `minimal` | text transformation | inline cases, variants, exact score, matrix, comparison |
| `basic` | support routing | file dataset, spans, checks, artifacts, Rich reports |
| `mid` | response generation | semantic usage, pricing, cost, policies, distributions |
| `advanced` | search implementations | repeated samples, noise, paired speedup |
| `abp_manual` | ticket router | manual span plus method instrumentation |
| `abp_concurrent` | async workers | task-local trace context and concurrent runs |
| `automatic_assets` | Pydantic AI and custom SDK | automatic behavioral asset lineage |

Run all offline examples:

```bash
make examples
```

## Minimal: Learn The Matrix

```bash
autobench validate examples/minimal/autobench.yaml
autobench run examples/minimal/autobench.yaml --record /tmp/autobench-minimal
autobench replay /tmp/autobench-minimal
```

Read `examples/minimal/autobench.yaml` together with `minimal_benchmark.py`. This is the shortest
complete `case x variant -> task -> score -> record -> report` implementation.

## Basic: Application Evidence

```bash
autobench run examples/basic/autobench.yaml --record /tmp/autobench-basic
```

The task validates typed input, reads a factor, opens a workflow span, stores its output as an
artifact, and lets declarative scorers evaluate correctness and handling. The candidate fixes a
known routing failure, so the case matrix and comparison contain a visible behavioral delta.

## Mid: Quality, Cost, And Constraints

```bash
autobench run examples/mid/autobench.yaml --record /tmp/autobench-mid
```

This example records input/output tokens and latency, resolves a local model pricing table, derives
`money.cost`, checks success and cost policies, and configures leaderboard and distribution views.
It is the best starting point for an LLM benchmark that already has a task implementation.

## Advanced: Measurement And Paired Baselines

```bash
autobench run examples/advanced/autobench.yaml --record /tmp/autobench-advanced
```

The task uses `measure_callable()` and `ctx.record_measurement()` instead of custom timing loops.
The post-deriver matches runs by case and computes candidate speedup against the baseline while
correctness remains a constraint.

## Pydantic AI: Live Layered Instrumentation

```bash
uv sync --extra instrumentation
export OPENROUTER_API_KEY=...
export OPENROUTER_MODEL=openrouter:openai/gpt-5.6-luna
uv run python examples/pydantic_ai/openrouter_instrument_all.py \
  --record /tmp/autobench-openrouter
```

The program makes a real OpenRouter request through Pydantic AI, uses a tool, streams structured
output, and calls `Benchmark.instrument_all()`. The task has no manual metrics, spans, or tracking
decorators. Autobench collects layered Pydantic AI, OpenAI client, and HTTPX evidence plus prompt,
tool, output-schema, and agent versions.

Inspect it afterward:

```bash
autobench instrumentation trace /tmp/autobench-openrouter
autobench replay /tmp/autobench-openrouter
```

`examples/pydantic_ai/agent_benchmark.py` is provider-neutral and accepts any configured Pydantic AI
model identifier through `PYDANTIC_AI_MODEL`.

## Automatic Asset Discovery

```bash
uv run python examples/automatic_assets/pydantic_ai_discovery.py \
  --record /tmp/autobench-pydantic-assets

uv run python examples/automatic_assets/custom_sdk_discovery.py \
  --record /tmp/autobench-custom-assets
```

Both are offline. The first uses a real Pydantic AI `Agent`, `AbstractCapability`, tool, and Pydantic
output model with no explicit tracking. The second adds prompt, tools, and output-schema extraction
to an arbitrary method with `InstrumentAssetSpec`.

## ABP Manual And Concurrent

```bash
autobench run examples/abp_manual/autobench.yaml --record /tmp/abp-manual
autobench run examples/abp_concurrent/autobench.yaml \
  --concurrency 2 \
  --record /tmp/abp-concurrent
```

Use the manual example to learn `RunContext.span()` and `instrument_method()`. Use the concurrent
example to inspect sibling span parentage and task-local context under async execution.

## OpenAI Streaming

```bash
uv sync --extra instrumentation
uv run python examples/abp_openai/run_openai_streaming.py
```

This uses the official OpenAI client and a real streaming parser over an offline HTTPX mock
transport. It demonstrates first-chunk and stream-completion evidence without network access.

## OpenAI Agents

```bash
uv sync --extra openai-agents
uv run python examples/abp_openai_agents/run_openai_agents.py
```

The example sends real OpenAI Agents workflow/function/custom trace events through the Autobench
trace processor. It requires no model request.

## Replay And Extraction

```bash
uv run python examples/abp_replay/replay_and_extract.py /tmp/recorded-experiment
```

The script loads records without provider SDKs and creates extraction-derived records with explicit
parent lineage.

## CodeMode: Migrating A Real Benchmark Runner

```bash
export OPENROUTER_API_KEY=...
uv run python examples/codemode/run_benchmark.py --only parse_cron \
  --record /tmp/autobench-codemode
```

The CodeMode example replaces a bespoke benchmark script with cases, model-pair factors, a task,
semantic coverage/success/latency evidence, generated-spec artifacts, recording, and reports. Its
task still owns Vowel CodeMode calls; Autobench remains generic. The external CodeMode runtime and
network credentials are required.

## What To Copy

Copy the pattern, not generated run directories:

- task signature and typed input/output from `minimal` or `basic`;
- pricing and policies from `mid`;
- measurement and paired comparison from `advanced`;
- automatic SDK setup from `pydantic_ai`;
- custom instrumentation from `automatic_assets`;
- replay processing from `abp_replay`.

For combinations not represented by one project, use [Use Cases](use-cases.md) and the
[Capability Map](capabilities.md).

---

## Architecture

Canonical page: https://vcoderun.github.io/autobench/architecture/

# Architecture

Autobench separates application execution from experiment infrastructure. This is the central
design constraint: the framework can benchmark any system because it does not own the system.

## Layered Model

| Layer | Owns | Does not own |
| --- | --- | --- |
| Definition | benchmark, dataset, variants, evaluation and report configuration | application implementation |
| Runtime | matrix planning, context, task invocation, concurrency, statuses | provider event loops or business orchestration |
| ABP | trace context, signals, spans, measurements, capture, source provenance | OpenTelemetry or hosted trace storage |
| Evaluation | scorers, derivation, policies, paired comparisons | domain truth that only the application can supply |
| Tracking | behavioral asset identity, versions, representations, diffs, uses | source control or deployment promotion |
| Records | immutable run and experiment evidence, artifacts, source hashes | mutable operational databases |
| Reports | semantic projections, aggregation, comparison and exports | causal inference from uncontrolled changes |

## One Canonical Spec

YAML and the Python builder converge on `BenchmarkSpec`:

```text
YAML DSL -----------+
                    +--> BenchmarkSpec --> BenchmarkPlan --> ExperimentResult
Benchmark builder --+
```

The builder is ergonomic composition; it is not a second runtime. `Benchmark.to_spec()` returns the
same model loaded by `load_benchmark_spec()`.

## Execution Lifecycle

For each case x variant pair, Autobench:

1. creates a stable run ID and `RunContext`;
2. activates task-local ABP context;
3. invokes `task(ctx, case)` synchronously or asynchronously;
4. preserves evidence even when the task fails;
5. evaluates built-in and Python scorers;
6. projects scores into semantic observations;
7. derives per-run metrics such as token cost;
8. finalizes status and trace state.

After all runs finish, it applies cross-run derivation and policy evaluation. Recording then
materializes immutable YAML records and referenced artifacts.

## Evidence Model

`Observation` is the common query and aggregation unit. Its local `name` explains the metric in the
application; `semantic_type` explains what the value means across applications. Source and
provenance distinguish task observations, scores, derived values, and trace extraction.

ABP preserves richer execution evidence as an ordered signal stream and a materialized `Trace`.
Useful trace values can be extracted into observations without discarding their span or source-map
lineage.

```text
SDK call
  -> native instrumentor
  -> ABP signals
  -> canonical Trace
  -> semantic extraction
  -> Observation
  -> report / policy / optimizer
```

## Definition And Effective Assets

Behavioral components have two useful representations:

- **definition**: what application code configured, such as a prompt template or Python tool;
- **effective**: what an SDK sent to a model or downstream system after normalization.

Automatic asset discovery can record both and link their versions. This makes lineage explain not
only that a tool changed, but also how its model-facing schema changed.

## Immutability And Replay

A `RunRecord` is evidence, not a cache entry. Replay never mutates it and never silently executes
the task. Rescoring, recanonicalization, or trace extraction creates new derived records with parent
lineage.

This supports three distinct workflows:

- **report replay**: render new views over unchanged evidence;
- **evidence replay**: run a versioned extractor or canonicalizer over stored ABP data;
- **execution rerun**: intentionally execute a new experiment against the current application.

## Extension Seams

Choose the narrowest seam that owns the behavior:

| Need | Extension |
| --- | --- |
| Call application code | Python task |
| Evaluate domain output | Python scorer |
| Compute from same-run observations | Deriver |
| Compare matched runs | Post-deriver |
| Enforce an acceptance rule | Policy |
| Collect a stable SDK boundary | Instrumentor |
| Map vendor fields to semantics | Source map / extractor |
| Add domain defaults | Metric pack |
| Version a behavioral component | Tracking or asset discovery |

Application-specific logic belongs in tasks and scorers. Generic SDK behavior belongs in an
instrumentor. This prevents core Autobench from accumulating one-off integrations disguised as
framework concepts.

## Optimization Boundary

Autobench produces optimization-grade evidence: objectives, constraints, diagnostics, factors,
asset versions, candidate feedback, and replayable run lineage. It deliberately does not select
mutation strategies or promote candidates. Consumers such as pydantic-gepa and autoptimize can use
the records without Autobench becoming coupled to one optimizer.

---

## Core Concepts

Canonical page: https://vcoderun.github.io/autobench/concepts/

# Core Concepts

Autobench models a benchmark as a deterministic experiment over cases and variants. The concepts
below appear in both the YAML DSL and Python API.

## BenchmarkSpec

The canonical definition of one benchmark. It contains metadata, capture policy, dataset, task,
variants, scoring, derivation, policies, instrumentation, report configuration, and a semantic
registry.

## Case And Dataset

A `Case` is one input and its optional expectation:

```python
from autobench import Case

case = Case(
    id="refund-request",
    input={"message": "Refund order 42"},
    expected={"route": "billing"},
    tags=["regression", "routing"],
    metadata={"language": "en"},
)
```

A `DatasetSpec` adds identity, version, defaults, source provenance, and attachments to a case
collection.

## Variant And Factor

A variant is one concrete configuration of the subject. Factors are independent variables such as
model, prompt version, implementation strategy, or feature flag.

```text
case: refund-request
variant: candidate
factors: model=gpt-x, prompt=refund-v4, temperature=0
```

Autobench records factors and their semantics. It does not assume that changing several factors at
once proves which one caused a metric delta.

## Task

The task adapts a case and variant to the system being benchmarked:

```python
def run(ctx, case):
    model = ctx.factor("model")
    return application.execute(case.input, model=model)
```

The task owns application calls. Autobench owns invocation, timing context, evidence preservation,
and status classification.

## Observation

An `Observation` is a typed fact produced during a run. It has a kind, name, value, optional
semantic type and unit, source, role, direction, and provenance.

Kinds include metrics, factors, events, diagnostics, outcomes, checks, and artifacts. Sources let
projection distinguish a task-emitted metric from a scorer or derived value with the same semantic
type.

## Semantic Type

A semantic type is a stable string such as `quality.correctness`, `llm.tokens.input`, or
`time.latency`. It lets generic components consume meaning rather than application-local names.

The registry carries aliases, parent relationships, aggregation hints, cardinality, privacy, and
stability metadata. Applications may extend it without replacing built-ins.

## Score

A `ScoreRecord` is evaluator output. Scores can be objectives, constraints, or diagnostics and can
include reasons, errors, and selected span provenance. They are projected into observations with
score precedence for reporting and policies.

## Derivation

A deriver computes a metric from evidence:

- per-run derivation uses one run, such as tokens + model pricing -> cost;
- post-derivation uses the experiment, such as matched baseline/candidate latency -> speedup.

Derived observations preserve their input references and source.

## Policy

A policy is a pass/fail requirement over semantic metrics. Operators are explicit fields such as
`must_equal`, `must_greater_equal`, `must_less_equal`, and `must_be_between`. Policies affect
evaluation status without hiding the underlying metric.

## ABP Trace

The Autobench Protocol (ABP) is the native execution evidence model. Instrumentors and manual spans
emit immutable signals that materialize into a trace containing spans, measurements, events, links,
references, errors, stream state, and diagnostics.

ABP is not an OpenTelemetry wrapper. Optional bridges may export it later, but Autobench controls
its semantic, replay, accounting, and optimization contracts.

## Tracked Asset

A tracked asset is a behavioral component whose exact version matters to a run: prompt, tool,
output schema, type, capability, agent, guardrail, handoff, policy, toolset, or arbitrary config.

Assets have stable logical IDs and content-addressed versions. Their history records normalized
state, source hashes, parent versions, changed fields, and diffs. An `AssetUse` binds the version and
representation actually used to a run and optional span.

## RunRecord And ExperimentRecord

`RunRecord` is the immutable evidence for one case x variant execution. `ExperimentRecord` describes
the plan, source hashes, environment, report configuration, run paths, and aggregate statuses for
the whole matrix.

Records are human-readable YAML views backed by strict typed models and versioned JSON Schemas.

## Report

A report is a replay-time projection, not the source of truth. Leaderboards, case matrices,
comparisons, distributions, and run tables all derive from RunRecords. New report configuration can
therefore analyze existing evidence without running the subject.

## Optimization Feedback

Feedback records compact failed scores, policy violations, task and span errors, factors, and asset
versions. They give optimizers structured evidence without forcing them to scrape terminal output or
infer semantics from metric names.

---

## Datasets And Variants

Canonical page: https://vcoderun.github.io/autobench/datasets-and-variants/

# Datasets And Variants

Autobench expands a dataset against variants to create a deterministic run matrix. Cases describe
what is evaluated; variants describe what changes.

## Cases

Each `Case` has a stable ID and may carry arbitrary input, expected output, metadata, tags, and
attachments:

```python
from autobench import Case

case = Case(
    id="refund-request",
    input={"message": "I need a refund for order 42"},
    expected={"route": "billing", "priority": "normal"},
    metadata={"tenant": "demo"},
    tags=["routing", "smoke"],
)
```

Inputs and expected values are intentionally generic. They can be strings, mappings, Pydantic
models serialized by the task, structured multimodal references, or domain-specific payloads.
Attachments use `ArtifactRef` values when a case depends on external material.

## Dataset Sources

Cases may be authored inline:

```yaml
dataset:
  version: v1
  cases:
    - id: refund-request
      input:
        message: I need a refund
      expected:
        route: billing
```

Or loaded relative to the benchmark file:

```yaml
dataset:
  source: file://datasets/cases.yaml
  version: v1
```

File-backed datasets use the same DSL representation. Glob-backed sources can combine separate
case files while duplicate case IDs remain validation errors.

## Case Defaults

Defaults reduce repeated metadata without hiding the final case payload:

```yaml
dataset:
  defaults:
    metadata:
      locale: en-US
    tags: [regression]
  cases:
    - id: ticket-1
      input: {message: Reset my password}
      tags: [authentication]
```

Mapping values are merged, tags are deduplicated, and explicit scalar case values override
defaults.

## Variants And Factors

A variant is one concrete factor set:

```yaml
variants:
  baseline:
    label: Current production route
    factors:
      model:
        value: openrouter:openai/gpt-5.6-luna
        semantic: llm.model.name
        optimize: true
      prompt_version:
        value: route-v3
        semantic: prompt.version
        optimize: true
      temperature: 0
```

`value` is the runtime value. `semantic` tells downstream consumers what the factor means.
`optimize` is a hint that the factor is a candidate optimization axis; Autobench records it but
does not choose search strategies.

The Python form is equivalent:

```python
from autobench import FactorValue, Semantic, Variant

variant = Variant(
    id="baseline",
    label="Current production route",
    factors=[
        FactorValue(
            name="model",
            value="openrouter:openai/gpt-5.6-luna",
            semantic_type=Semantic.LLM_MODEL_NAME,
            optimize=True,
        ),
        FactorValue(name="temperature", value=0),
    ],
)
```

Tasks read factors through `ctx.factor(name)`. Factors are also copied into RunRecords and report
variant-configuration tables.

## Generated And Production Cases

The data helpers preserve where generated examples came from:

- `ProductionSample` models a source sample and review state.
- `sample_to_case` and `samples_to_cases` convert samples without losing provenance.
- `mark_generated_case` records generation metadata.
- `generated_batch_from_cases` creates a `GeneratedCaseBatch` with generator, model, and source
  details.

This layer is intentionally not a synthetic-data generator. It defines the evidence contract so a
generator, production sampler, or review system can supply cases consistently.

## Identity And Reproducibility

- Case IDs and variant IDs must be unique.
- Dataset content hashes depend on normalized content rather than filesystem location.
- Matrix order is deterministic.
- Run IDs are stable for a given plan position, case, and variant.
- Dataset version may also be emitted as `dataset.version` semantic evidence.

---

## Tasks And Runtime

Canonical page: https://vcoderun.github.io/autobench/tasks-and-runtime/

# Tasks And Runtime

The task is the only application-specific execution boundary required by Autobench. It receives a
runtime context and a case, invokes the subject, records evidence, and returns the output that
scorers evaluate.

## Task Contract

```python
from autobench import Case, RunContext


def run_case(ctx: RunContext, case: Case) -> dict[str, object]:
    model = ctx.factor("model")
    result = call_application(case.input, model=model)
    ctx.outcome(result.ok)
    return {"answer": result.answer, "ok": result.ok}
```

The positional contract is always `task(ctx, case)`. Tasks may be synchronous or asynchronous:

```python
async def run_case(ctx: RunContext, case: Case) -> dict[str, object]:
    result = await call_application(case.input)
    return {"answer": result.answer}
```

YAML resolves the callable relative to the benchmark file before falling back to import paths:

```yaml
run:
  python: benchmark_tasks:run_case
```

## RunContext

`RunContext` owns evidence for one case x variant run:

| Method | Use |
| --- | --- |
| `factor(name)` | Read a variant factor |
| `span(...)` | Open a timed nested operation |
| `metric(...)` / `metrics(...)` | Record one or many metrics |
| `factor_observation(...)` | Record a runtime-discovered factor |
| `event(...)` | Record a discrete event |
| `diagnostic(...)` | Record non-objective diagnostic evidence |
| `outcome(...)` | Record semantic run success |
| `check(...)` | Record a boolean correctness check with an optional reason |
| `record_measurement(...)` | Record summary statistics and optional sample artifact |
| `artifact(...)` | Attach a structured or file-like payload |
| `error(...)` | Attach a structured error without losing collected evidence |
| `attach_tracked_asset(...)` | Bind a tracked asset version to the run |

Context evidence remains available even when the task raises. The runtime captures the exception,
preserves observations and artifacts already emitted, and records a structured error.

## Matrix Execution

`build_benchmark_plan` validates and counts the matrix before execution. `expand_matrix` produces
one `MatrixRunSpec` per case x variant pair. The CLI renders the same plan during validation.

```bash
autobench validate autobench.yaml
autobench run autobench.yaml --concurrency 4 --record runs/latest
```

Concurrency bounds the number of active runs. Result ordering stays deterministic even when task
completion order differs.

## Failure And Status Model

Autobench separates three status layers:

- `TaskStatus`: whether application execution completed, failed, or was skipped.
- `EvaluationStatus`: whether scoring and constraints completed.
- `RunStatus`: final passed, failed, errored, or skipped state.

This distinction prevents a policy failure from looking like an application exception and lets
reports separate execution reliability from evaluation quality.

## Progress Events

`ProgressEvent` and `ProgressEventKind` provide typed lifecycle notifications. Known event fields
remain stable while event-specific data is carried in the payload. This is the extension surface
for terminal progress, service runners, and future UIs without coupling the core runtime to one
frontend.

## Python Builder

The builder compiles to the same `BenchmarkSpec` used by YAML:

```python
from autobench import Benchmark, Case, FactorValue, PassFailScorer, Semantic, Variant

result = (
    Benchmark("routing")
    .dataset([Case(id="refund", input={"message": "Refund order 42"})])
    .variants(
        [
            Variant(
                id="baseline",
                factors=[FactorValue(name="route", value="v1")],
            )
        ]
    )
    .task("benchmark_tasks:run_case")
    .scoring(
        [
            PassFailScorer(
                name="success",
                path="output.ok",
                semantic_type=Semantic.RESULT_SUCCESS,
            )
        ]
    )
    .run()
)
```

Use YAML for portable benchmark definitions and the builder when a Python application needs to
compose specs programmatically. Both execute through the same planner and runtime.

---

## Observations And Semantics

Canonical page: https://vcoderun.github.io/autobench/observations-and-semantics/

# Observations And Semantics

An observation is Autobench's atomic evidence unit. Raw names remain useful to humans, while
semantic types make evidence portable across applications, reports, and optimizers.

## Observation Model

An `Observation` carries:

- stable ID and local name
- kind: metric, factor, event, diagnostic, or artifact
- value and optional unit
- semantic type
- optimization direction and role
- source and optional span ID
- tags, case ID, and variant ID

```python
from autobench import Direction, ObservationRole, Semantic

ctx.metric(
    "answer_accuracy",
    0.94,
    semantic_type=Semantic.QUALITY_CORRECTNESS,
    direction=Direction.MAXIMIZE,
    role=ObservationRole.OBJECTIVE,
)
```

The local name may be `answer_accuracy`, `judge_score`, or `coverage`; the semantic type tells the
framework whether those values share meaning.

## Built-In Semantic Families

| Family | Examples |
| --- | --- |
| LLM | `llm.tokens.input`, `llm.tokens.output`, `llm.request.count`, `llm.model.requested`, `llm.model.response`, `llm.provider.name` |
| Cost | `money.cost`, `serving.cost`, `optimization.cost`, `lifetime.cost` |
| Time | `time.latency`, `time.first_chunk`, `time.critical_path` |
| Result | `result.success` |
| Quality | `quality.score`, `quality.correctness`, `coverage.ratio` |
| Agent | task completion, plan quality/adherence, step efficiency, tool selection/arguments/sequence, output correctness |
| Assets | `prompt.version`, `agent.tool.version`, `agent.version`, `dataset.version` |
| Operations | count, maximum depth/fan-out, incomplete work, parallelism, retries, recovered retries, first-attempt success |
| Workflow | validation failures, approval count/wait, tool-call success/failure, message growth, evidence-reference counts |

`Semantic` exposes completion-friendly constants. `SemanticType` remains extensible so domain
metrics can use names such as `retrieval.recall` or `business.conversion`.

## Registry And Aliases

`SemanticRegistry` stores definitions, aliases, and parent relationships. A custom registry can be
embedded in a benchmark spec and is merged with built-ins:

```yaml
semantic_registry:
  version: 1
  types:
    business.conversion:
      description: Whether the workflow produced a qualified conversion.
      parent: result.success
      unit: boolean
  aliases:
    conversion: business.conversion
```

Parent relationships let a query request a broad semantic category while preserving specific
metrics. Aliases prevent local naming differences from fragmenting evidence.

## Roles And Directions

Roles describe how a metric participates in evaluation:

- objective: something to optimize
- constraint: something that must remain acceptable
- diagnostic: explanatory evidence

Directions are `maximize` or `minimize`. Factors, events, and artifacts cannot declare an
optimization direction because they are not outcomes.

## Sources And Projection

The same semantic metric can be emitted by a task, scorer, deriver, policy, or adapter. Raw
observations are never discarded. Projection chooses a canonical value using explicit source
priority and ABP accounting scope. A derived aggregate summary is preferred to same-source direct
measurements for single-value reporting, while direct observations remain queryable. Logical
operation IDs correlate equivalent framework/client evidence; equal-priority disagreements are
marked ambiguous instead of silently picking one.

Use `ObservationQuery` for raw or projected lookup and `filter_observations` for selectors such as
semantic type, role, source, or span.

```python
from autobench import ObservationQuery

query = ObservationQuery(observations=list(result.observations))
costs = query.values("money.cost", projected=False)
```

Reports, policies, and derivation use this semantic projection layer rather than relying on local
metric names.

## Metric Packs

A `MetricPack` bundles reusable semantic defaults without forcing every metric into core:

- semantic registry additions
- scorer factory references
- default report metrics
- feedback extractors

Built-in packs cover `agentic`, `structured_output`, `llm_usage`, and `performance`. Applications
can register their own packs through `MetricPackRegistry` while keeping the RunRecord contract
unchanged.

---

## Scoring And Derivation

Canonical page: https://vcoderun.github.io/autobench/scoring-and-derivation/

# Scoring And Derivation

Scorers evaluate one run. Derivers compute new metrics from collected evidence. Post-derivers work
across runs after the complete experiment exists. Policies turn semantic metrics into explicit
requirements.

## Scoring Contract

Every scorer declares:

- a local score name
- semantic type
- optional unit
- optimization direction
- role: objective, constraint, or diagnostic
- whether scorer failure is optional

Scores are stored as `ScoreRecord` values and projected into observations with score-source
precedence. The original task observations remain available.

## Output Metric

Project an output value directly:

```yaml
score:
  coverage:
    value: output.coverage
    semantic: coverage.ratio
    goal: maximize
    role: objective
```

Use this when the task already computes a trustworthy metric.

## Pass/Fail

```yaml
score:
  success:
    pass: output.ok
    semantic: result.success
    role: constraint
```

The path must resolve to a boolean-like success value.

## Exact Match

```yaml
score:
  route_correctness:
    exact:
      actual: output.queue
      expected: case.expected.queue
    semantic: quality.correctness
    goal: maximize
```

Paths can address `output`, `case.input`, `case.expected`, factors, and structured values.

## Schema Validation

`SchemaScorer` validates a selected output path against a JSON Schema mapping. It is appropriate
for contracts where structural validity is separate from domain correctness.

```python
from autobench import SchemaScorer, Semantic

scorer = SchemaScorer(
    name="output_schema",
    path="output",
    schema={
        "type": "object",
        "required": ["customer_name", "id"],
        "properties": {
            "customer_name": {"type": "string"},
            "id": {"type": "string"},
        },
    },
    semantic_type=Semantic.AGENT_OUTPUT_STRUCTURE_VALIDITY,
)
```

## Python Scorers

Custom scorers receive `ScoringCall`, not loose callback dictionaries:

```python
from autobench import ScoreRecord, ScoringCall


def field_accuracy(call: ScoringCall) -> ScoreRecord:
    expected = call.case.expected
    output = call.output
    fields = ("name", "id", "pocket_id")
    matches = sum(output[field] == expected[field] for field in fields)
    return ScoreRecord(
        name="field_accuracy",
        semantic_type="quality.field_accuracy",
        value=matches / len(fields),
    )
```

`ScoringCall` exposes the case, variant, task output/result, observations, spans, and selected spans.
Python scorers may be sync or async. Optional scorers record errors without failing the run.

## Expected Actions

`ExpectedActionScorer` deterministically evaluates action/tool selection, arguments, or ordered
sequence from spans. See [Agentic Evaluation](agentic-evaluation.md).

## Dotted Paths

`resolve_dotted_path` is the shared structured-path resolver used by built-in scorers. Missing
paths produce explicit scorer errors instead of silently returning `None`.

## Per-Run Derivation

`derive` runs after task observations and scores are available for one run. `TokenCostDeriver` is
the built-in per-run deriver.

```yaml
derive:
  - kind: token_cost
    pricing: file://pricing/models.yaml
    output:
      name: request_cost
      semantic_type: money.cost
      unit: usd
      direction: minimize
      role: constraint
```

By default it reads:

- `llm.tokens.input`
- `llm.tokens.output`
- `llm.provider`
- `llm.model.name`

Input semantics and output metadata can be overridden through `TokenCostInputs` and
`DerivedMetricOutput` in the Python API.

Unknown models, missing usage, missing rates, and ambiguous inputs produce diagnostics; Autobench
does not invent a zero cost.

## Pricing DSL

Pricing is normalized into a `PricingTable` keyed by stable model IDs. Provider-specific aliases
allow input forms such as `provider:model`, `provider/model`, or application-specific model slugs
to resolve to the same entry.

```yaml
pricing:
  version: 1
  provider: openai
  models:
    openai/gpt-demo:
      aliases: [openai:gpt-demo, gpt-demo]
      input:
        unit: mtok
        price_per_million_tokens: 1.0
      output:
        unit: mtok
        tiers:
          - up_to_tokens: 100000
            price_per_million_tokens: 4.0
          - price_per_million_tokens: 6.0
      cache_read:
        unit: mtok
        price_per_million_tokens: 0.1
```

Supported fields include input, output, cache-read, and cache-write prices plus token-count tiers.
`StaticPriceSource`, `LLMPricesSource`, and `GenAIPricesSource` only import external price data into
this model. They do not make an external catalog authoritative at runtime.

## Paired Baseline Post-Derivation

`post_derive` has access to the full experiment:

```yaml
post_derive:
  - kind: paired_baseline
    baseline_variant: baseline
    match_on:
      - kind: case_id
      - kind: factor
        name: workload.size
    metric: time.latency
    formula: baseline_over_candidate
    include_baseline: true
    output:
      name: speedup
      semantic_type: performance.speedup
      unit: ratio
      direction: maximize
      role: objective
```

Formulas:

- `baseline_over_candidate`
- `candidate_over_baseline`
- `candidate_minus_baseline`
- `baseline_minus_candidate`
- `percent_change_from_baseline`

Matching supports case IDs and factor keys. Missing matches, nonnumeric metrics, absent metrics, and
zero division can be skipped or recorded as diagnostics.

Relative-noise thresholds and `ComparisonVerdictSpec` can emit improved, regressed, unchanged, or
inconclusive verdicts. These are controlled comparisons, not automatic causal claims.

## Policies

Policies evaluate projected semantic values and append `PolicyResult` evidence:

```yaml
policies:
  - name: request-must-succeed
    metric: result.success
    must_equal: true
  - name: cost-cap
    metric: money.cost
    must_less_equal: 0.001
  - name: acceptable-latency
    metric: time.latency
    must_between:
      min: 0
      max: 500
      inclusive: true
```

Each policy declares exactly one requirement:

- `must_equal` / `must_not_equal`
- `must_greater` / `must_greater_equal`
- `must_less` / `must_less_equal`
- `must_in` / `must_not_in`
- `must_between`

A failed constraint can change final run status while preserving the successful task output and
all evidence that explains the decision.

## Repeated Measurement

`measure_callable` avoids repeating warmup and sampling loops in benchmark tasks:

```python
from autobench import MeasurementBudget, measure_callable

measurement = measure_callable(
    lambda: search(case.input["items"], case.input["query"]),
    budget=MeasurementBudget(warmup=3, repetitions=20, max_seconds=2.0),
)
ctx.record_measurement("search", measurement)
```

`Measurement` includes samples, count, min, max, mean, median, p95, standard deviation, and relative
noise. A custom timer can measure accelerators or remote systems without adding domain-specific
logic to Autobench.

---

## Agentic Evaluation

Canonical page: https://vcoderun.github.io/autobench/agentic-evaluation/

# Agentic Evaluation

Autobench evaluates agents as traced systems rather than treating only the final text as evidence.
The same primitives also work for workflow engines, retrievers, and tool-using applications.

## Record Agent Behavior

```python
from autobench import Semantic, SpanKind


def run_case(ctx, case):
    with ctx.span("support_agent", kind=SpanKind.AGENT, input=case.input) as agent:
        with ctx.span(
            "lookup_user",
            kind=SpanKind.TOOL,
            input={"user_id": case.input["user_id"]},
        ) as tool:
            profile = lookup_user(case.input["user_id"])
            tool.set_output(profile)

        answer = compose_answer(profile, case.input["message"])
        agent.set_output(answer)
        agent.metric(
            "task_completed",
            True,
            semantic_type=Semantic.AGENT_TASK_COMPLETION,
        )
        return answer
```

Spans preserve selection, arguments, output, order, duration, errors, tags, and hierarchy.

## Declare Expected Actions

Cases can use generic `actions` or the tool-oriented `tool_calls` compatibility shape:

```yaml
cases:
  - id: refund
    input:
      user_id: u1
      message: Refund order 42
    expected:
      actions:
        - id: lookup
          kind: tool
          target: lookup_user
          input:
            user_id: u1
          order: 1
          required: true
```

Expected input matching is subset-based, so a tool may receive additional nonessential arguments.
Actions may also declare expected output, tolerance metadata, optional status, and explicit order.

## Score Selection, Arguments, And Sequence

```yaml
score:
  tool_selection:
    expected_action:
      metric: selection
      observed_kind: tool
      span:
        kind: tool
    semantic: agent.tool.selection.correctness
    goal: maximize

  tool_arguments:
    expected_action:
      metric: arguments
      observed_kind: tool
      span:
        kind: tool
    semantic: agent.tool.argument.correctness
    goal: maximize

  tool_sequence:
    expected_action:
      metric: sequence
      observed_kind: tool
      span:
        kind: tool
    semantic: agent.tool.sequence.correctness
    goal: maximize
```

These scorers are deterministic and do not require an LLM judge. They produce normal scores and
semantic observations, so policies and reports consume them like any other metric.

## Span Selection

`SpanSelector` filters spans by:

- kind
- name
- tags
- nested path
- emitted semantic type

Selectors can be composed with positive and negative report/evaluation filters. A scorer receives
the selected spans through `ScoringCall`, allowing custom component-level evaluators without
parsing raw traces.

## Agentic Semantic Types

Built-in semantics include:

- task completion and goal accuracy
- plan quality and plan adherence
- step efficiency and orchestration quality
- tool name and version
- tool selection, argument, and sequence correctness
- tool-call quality
- output correctness and structure validity
- agent version and serving volume

Applications may add more specific child semantics through the registry.

## Metric Packs

The `agentic` metric pack contributes standard semantic definitions and report defaults. Metric
packs are optional: they provide conventions, not a required agent SDK. A custom agent runtime can
emit the same evidence through spans or a trace adapter.

## Optimization Feedback

`build_feedback_records` compacts run evidence into one record per case. It captures:

- score and evaluator reasons
- task, scorer, policy, and span errors
- `failure_category` only when a failure exists
- factor values and tracked asset versions
- selected observations and trace context

`build_optimization_feedback_input` packages those records with benchmark identity and semantic
context. pydantic-gepa or autoptimize can consume this structured evidence without scraping Rich
tables or replay YAML.

Autobench reports association and comparison evidence; it does not claim causal attribution when
multiple factors changed together. Controlled experiment planning belongs to the optimizer layer.

---

## Recording And Reporting

Canonical page: https://vcoderun.github.io/autobench/recording-and-reporting/

# Recording And Reporting

Recording turns an in-memory experiment into portable, immutable evidence. Replay and analysis use
those records without executing the application again.

## Record Layout

```bash
autobench run autobench.yaml --record runs/support-routing
```

The directory contains:

```text
runs/support-routing/
  experiment.yaml
  summary.yaml
  cases/<case-id>/<variant-id>/run.yaml
  artifacts/...
```

Paths are stable and artifact references are relative so the directory can be moved or archived.
Recording is append-only: an existing run payload is never silently replaced.

## RunRecord

One `RunRecord` represents one case x variant execution:

- record, run, experiment, benchmark, case, and variant IDs
- final, task, and evaluation statuses
- complete case snapshot and task output
- observations and scores
- canonical ABP trace, including signals, span graph, measurements, events, links, references,
  diagnostics, and instrumentation scope provenance
- ABP protocol and semantic registry versions
- legacy span tree for records created before canonical trace storage
- materialized artifacts
- factors and tracked asset versions
- extraction and source-map replay lineage
- structured errors

The YAML view groups the data for people rather than dumping internal Pydantic fields. A schema
header points editors to the versioned Autobench JSON schema.

Small traces remain inline in `run.yaml`. Larger traces are written to
`artifacts/<run-id>/trace.yaml`; the RunRecord keeps a relative `ArtifactRef` and a compact trace
summary. Trace artifacts have their own versioned JSON Schema header and load back into the same
typed `Trace` model.

## ExperimentRecord

The experiment-level record stores:

- benchmark plan and counts
- captured environment metadata
- semantic registry
- report configuration
- normalized benchmark snapshot and hash
- hashes of resolved specs, datasets, pricing files, tasks, and scorer modules
- relative run paths and status counts

This is enough to explain what was planned, which files defined it, and where every run record
lives.

## Environment And Source Identity

`capture_environment` records reproducibility metadata such as Python, platform, package, and
working-environment details. `collect_benchmark_source_files` resolves benchmark dependencies and
records content hashes.

Source paths are stored portably when possible. Missing optional source files do not erase a run;
recording captures what was resolvable at execution time.

## Artifacts

`ctx.artifact(name, value)` adds an `ArtifactRef`. During recording, supported values are
materialized under `artifacts/` and the RunRecord keeps the relative path, media type, and tags.

Use artifacts for:

- generated specs and prompts
- traces too large for `run.yaml`
- measurement samples
- model responses and structured debug payloads
- Markdown or text reports produced by the subject

Artifact path collisions and attempts to overwrite existing payloads are recording errors.

## Replay

```bash
autobench replay runs/support-routing
```

Replay loads `ExperimentRecord` and every `RunRecord` into an `ExperimentResult`. It deliberately
does not import task or scorer modules, call models, or mutate the original directory.

This enables:

- offline report regeneration
- new exports from old evidence
- baseline/candidate comparison after execution
- future rescoring into a separate derived experiment
- optimization systems consuming stable records

Autobench distinguishes three replay modes:

- **report replay** reads stored observations without re-extracting evidence
- **extraction replay** runs a typed `TraceExtractor` against the immutable ABP trace and creates a
  derived RunRecord
- **canonicalization replay** applies newer source maps to retained source snapshots and creates a
  separate derived RunRecord

Derived records point to the original `run_id`, identify the extractor or source-map versions, and
retain the source protocol and semantic registry versions. The original record and trace bytes are
never rewritten. Replay resolves trace artifacts only inside the experiment directory and imports
neither application task modules nor optional SDK integrations.

The default `SignalExtractor` reconstructs canonical observations from stored ABP measurements and
events. `SpanExtractor` derives generic topology and workflow evidence, while `UsageExtractor`
owns LLM request/token/model accounting. `CompositeExtractor` can run them as one versioned replay
processor. Custom extractors implement the typed `TraceExtractor` interface and return
observations, diagnostics, and evidence references without mutating the trace.

When a newer version of the same extractor is replayed, its observations replace the older
version's observations in the new derived record. The previous derived record remains the lineage
parent, so extractor evolution is auditable without mixing two versions of one derived metric.

## Rich Reports

```bash
autobench report runs/support-routing
```

The terminal report can include:

- experiment overview and status counts
- variant configuration table with factor values
- semantic leaderboards
- per-run metric tables grouped by semantic family
- case x variant matrices
- baseline/candidate factor and metric deltas
- metric distributions

Reports use projected semantic metrics. They do not depend on application-specific local names.

## Report Configuration

```yaml
report:
  leaderboard:
    show:
      accuracy:
        metric: quality.correctness
        aggregate: ratio_true
      total_cost:
        metric: money.cost
        aggregate: sum
      p95_latency:
        metric: time.latency
        aggregate: p95
  matrix:
    metric: quality.correctness
  compare:
    baseline -> candidate:
      show:
        accuracy:
          metric: quality.correctness
          aggregate: ratio_true
  distributions:
    - name: request_latency
      semantic_type: time.latency
      summaries: [min, median, p95, max]
```

Aggregation functions include count, mean, sum, min, max, median, p95, standard deviation,
geometric mean, and boolean true ratio.

## Comparison Semantics

```bash
autobench compare runs/support-routing --baseline baseline --candidate candidate
```

Comparison pairs runs by case, displays changed factors, aggregates requested semantic metrics, and
sets `confounded=true` when multiple relevant factors changed. It reports association and deltas;
it does not claim which factor caused the result.

Use paired-baseline post-derivation when a per-run derived metric such as speedup must be written
back into candidate evidence.

## Exports

```bash
autobench export runs/support-routing --format yaml --path report.yaml
autobench export runs/support-routing --format csv --path runs.csv
autobench export runs/support-routing --format markdown --path report.md
```

- YAML is a human-readable summary projection.
- CSV is a flat run-and-metric table for analysis tools.
- Markdown is a portable rendered report.

The CLI always writes the requested file and then renders a Rich preview. Machine exports never
replace immutable source RunRecords.

---

## Asset Tracking

Canonical page: https://vcoderun.github.io/autobench/asset-tracking/

# Asset Tracking

Benchmarks need to know which prompt, tool, schema, or configuration produced each result.
Autobench tracking assigns content-derived versions, captures structured metadata, persists history,
and binds exact asset versions to RunRecords.

Supported SDK instrumentors can discover these assets without decorators. Use explicit tracking on
this page when the application owns a better logical identity or when the component never crosses an
instrumented boundary. See [Automatic Asset Discovery](automatic-asset-discovery.md) for unannotated
Pydantic AI, OpenAI, OpenAI Agents, capability scopes, privacy, and custom SDK extraction.

## Prompts And Text Assets

Track inline text:

```python
from autobench import track

SYSTEM_PROMPT = track.prompt(
    name="support_system_prompt",
    text="Route the request to billing, account, or technical support.",
)
```

Or load it from a file:

```python
SYSTEM_PROMPT = track.prompt(
    name="support_system_prompt",
    source="prompts/support.md",
)
```

`TrackedPrompt.raw` returns the text, and `str(SYSTEM_PROMPT)` provides the same value for APIs that
expect a string. File-backed prompts retain their source path and source hash.

## Tools

`@track.tool` preserves the callable's exact signature and return type while collecting tool
metadata:

```python
from typing import Literal

from autobench import track


@track.tool
def route_ticket(
    queue: Literal["billing", "account", "technical"],
    priority: int = 1,
) -> bool:
    """Route a ticket to a support queue."""
    return priority > 0
```

The resulting `ToolAsset` records:

- qualified name and docstring
- parameter names, kinds, annotations, defaults, and requirements
- return annotation
- source path and source hash
- structured parameter schema
- semantic type and version lineage

Annotations are normalized by structure rather than alias spelling. If the contents of a
`Literal`, union, generic, model, or referenced type change, the asset hash changes even when the
alias name stays the same.

## Pydantic Models, Dataclasses, And Classes

```python
from dataclasses import dataclass
from typing import Literal

from autobench import track
from pydantic import BaseModel, Field


@track.type
class Car(BaseModel):
    make: Literal["audi", "bmw", "mercedes"]
    model: str = Field(examples=["a3", "320i"])
    year: int = Field(gt=0)


@track.dataclass(frozen=True, slots=True)
class CarRequest:
    make: Literal["audi", "bmw", "mercedes"]
    model: str
    year: int
```

Pydantic models are hashed from normalized JSON Schema plus source identity. Standard dataclasses
use dataclass field definitions and resolved annotations. Other typed classes use resolved class
annotations, inspectable signatures, and source hashes.

`TypeAsset` and `FieldAsset` preserve field names, resolved annotations, descriptions, examples,
aliases, defaults, required state, and relevant constraints.

## Composing Another Class Decorator

When `@track.type` above a class-transforming decorator gives poor type-checker inference, use
`track.decorate_type`:

```python
from dataclasses import dataclass

from autobench import track


@track.decorate_type(dataclass, frozen=True, slots=True)
class Request:
    value: str
```

The decorator and its normalized arguments are stored as asset metadata. `track.dataclass(...)` is
the typed convenience form for the standard dataclass decorator.

## Arbitrary Assets

Use `track.asset` for configurations, policies, routing tables, or other application components:

```python
@track.asset(kind="routing_policy", name="enterprise_routing")
def route_policy(ticket):
    return "priority" if ticket["enterprise"] else "standard"
```

The decorator returns the original object unchanged. Callables use source and signature metadata;
manual `version`, `hash`, `source_path`, `parent_version`, and metadata values are available when
automatic identity is not enough.

## Versions, Diffs, And Persistence

`TrackingRegistry` keeps current assets and version history in memory during execution. Persist it
with:

```python
from pathlib import Path

from autobench import track

track.write_assets(Path(".autobench/assets"))
```

The YAML history contains an index plus one file per asset. Every new version links to its parent
when available and stores a human-readable diff from the previous serialized state. Source changes,
schema changes, decorator options, and metadata changes therefore remain reviewable.

## Binding Assets To Runs

```python
def run_case(ctx, case):
    ctx.attach_tracked_asset(SYSTEM_PROMPT)
    ctx.attach_tracked_asset(route_ticket)
    ctx.attach_tracked_asset(Car)
    return execute(case.input)
```

The exact `AssetVersion` values are copied into the RunRecord. Reports and optimization feedback
can then relate metric changes to prompt, tool, or output-schema versions without guessing from
source control state.

---

## Automatic Asset Discovery

Canonical page: https://vcoderun.github.io/autobench/automatic-asset-discovery/

# Automatic Asset Discovery

Automatic asset discovery turns SDK-visible prompts, tools, output schemas, capabilities, agents,
guardrails, handoffs, and policies into versioned benchmark evidence. It is part of the common ABP
instrumentation runtime, not a Pydantic AI-specific tracking mode.

The application does not need `@track.prompt`, `@track.tool`, or `@track.type` when a supported
instrumentor can already see the behavioral component at a stable SDK boundary. Explicit tracking
still composes with discovery when the application owns a better identity, semantic type, source
path, or parent version.

## Quick Start

Install the SDK integrations used by the application:

```bash
pip install 'autobench[instrumentation]'
```

Then enable compatible integrations for the benchmark:

```python
from autobench import Benchmark

benchmark = Benchmark("support-agent").instrument_all()
```

While a benchmark run is active, Autobench now:

1. observes definitions at supported framework and client surfaces;
2. normalizes them into SDK-independent asset candidates;
3. resolves stable logical identity and aliases;
4. computes behavioral content versions;
5. attaches exact asset uses to the owning run and span;
6. persists referenced histories when the experiment is recorded.

Calls made outside an active Autobench run remain unchanged and produce no discovery evidence.

## What Counts As An Asset

An asset is a versionable component whose content or behavior can change benchmark outcomes.

| Observed value | Autobench treatment |
| --- | --- |
| static or callable instructions | prompt definition |
| rendered system/developer instructions | effective prompt |
| function, hosted, native, or MCP tool | tool definition or effective tool schema |
| Pydantic type, dataclass, or JSON Schema | output schema |
| Pydantic AI capability | scoped composite asset |
| OpenAI Agents guardrail or handoff | guardrail or handoff asset |
| routing, retry, output, or tool-use configuration | policy asset |
| composed agent or toolset | composite asset with child locators |
| model/provider/settings | factor or configuration evidence |
| user input, output, message history, tool arguments/results | evidence, not an asset |
| tokens, cost, latency, quality | metric |

Kinds are open strings. Autobench does not require every domain to fit an AI-only taxonomy.

## Definitions And Effective Representations

Definitions and model-facing values answer different questions:

```text
definition
  Which source component did the application declare?

effective
  Which resolved representation did this operation actually use?
```

A dynamic instruction callable is a definition asset. Calling it during the SDK's normal lifecycle
may produce an effective prompt for one run. Autobench records both and links the effective
`AssetUse.definition_asset_id` and `definition_version` to the source definition. Case interpolation
therefore does not create a fake source edit for every input.

Autobench discovery never invokes instruction callbacks, tools, validators, or guardrails merely to
inspect them. It observes declaration values or values already resolved by the actual SDK lifecycle.

## Pydantic AI Without Tracking Decorators

This agent has no explicit Autobench tracking:

```python
from pydantic import BaseModel, Field
from pydantic_ai import Agent


class SupportAnswer(BaseModel):
    answer: str = Field(description="A grounded support answer.")
    queue: str


def lookup_policy(topic: str) -> str:
    """Return the active support policy."""
    return f"policy for {topic}"


agent = Agent(
    model,
    name="support-router",
    output_type=SupportAnswer,
    instructions="Use the policy tool before routing.",
    tools=[lookup_policy],
)
```

With `instrument_all()`, a run discovers the agent composite, prompt, function tool, toolset, output
schema, final request instructions, effective tool definitions, and validated output schema. The
plain Python function and Pydantic type keep their source-aware identities; no wrapper replaces
their signatures or results.

The complete offline example uses Pydantic AI's real `Agent` lifecycle and `TestModel`:

```bash
uv run python examples/automatic_assets/pydantic_ai_discovery.py \
  --record /tmp/autobench-pydantic-assets
```

## Capability Scopes

Pydantic AI capabilities are first-class scopes because multiple capabilities can expose an
`instructions` or `search` component with the same local name:

```python
from pydantic_ai.capabilities import AbstractCapability


class RetrievalCapability(AbstractCapability[None]):
    id = "retrieval"

    def get_instructions(self) -> str:
        return "Ground answers in retrieved evidence."
```

The local and global locators are both retained:

```text
retrieval:prompt:instructions
pydantic_ai:retrieval:prompt:instructions
pydantic_ai:retrieval:capability:self
```

Capability identity uses a non-empty `id`, then `get_serialization_name()`, then the stable module
and qualified class name. A shared explicitly tracked component keeps one logical version while its
uses retain each capability alias and provenance.

## OpenAI Client And OpenAI Agents

The OpenAI client instrumentor discovers provider-facing assets:

- Chat Completions system/developer messages, tools, legacy functions, and response format;
- Responses instructions, managed prompt references, tools, text format, and output schema;
- Pydantic output types passed through structured parsing surfaces.

These are effective representations because the client sees the final request rather than the
framework's source declaration. Embedding inputs remain evidence and produce no asset.

The OpenAI Agents instrumentor discovers public Agent and Runner definitions:

- instructions and prompt references;
- function, hosted, MCP, computer, and agent tools;
- output schema;
- input/output/tool guardrails;
- handoffs and routing/tool-use policy;
- the composed agent.

It does not run callback instructions, guardrails, handoffs, or tools for discovery. Native trace
processing and public Runner resolution are combined without changing callback counts.

HTTPX intentionally discovers no semantic assets. A transport request lacks ownership context and
may contain secrets. It remains transport evidence while framework/client layers provide the
authoritative asset projections.

## Selecting Asset Families

Definition and effective discovery are enabled by default for semantic instrumentors. Restrict the
surface from Python:

```python
from autobench import AssetDiscoverySettings, AssetRepresentation, Benchmark

benchmark = Benchmark("support-agent").instrument_all(
    assets=AssetDiscoverySettings(
        representations=(
            AssetRepresentation.DEFINITION,
            AssetRepresentation.EFFECTIVE,
        ),
        include=("prompt", "tool", "output_schema", "capability"),
    )
)
```

Or use the YAML DSL:

```yaml
# yaml-language-server: $schema=schemas/0.2.0/benchmark_schema.json
benchmark:
  support-agent:
    instrumentation:
      all:
        assets:
          discover: true
          representations: [definition, effective]
          include: [prompt, tool, output_schema, capability]
```

An explicit integration can use a different filter and overrides automatic selection:

```yaml
instrumentation:
  all:
    exclude: [openai]
  openai:
    assets:
      representations: [effective]
      include: [prompt, tool, output_schema]
```

Set `discover: false` to keep spans and metrics from that instrumentor while disabling only its
asset discovery.

## Privacy And Capture Policy

Asset content passes through the same `CapturePolicy` as ABP evidence before it reaches the
registry. Configure one policy for every run in the benchmark:

```python
from autobench import Benchmark, CaptureLevel, CapturePolicy

benchmark = Benchmark("private-agent").capture(
    CapturePolicy.hashed(
        semantic_overrides={
            "tool": CaptureLevel.FULL,
            "output_schema": CaptureLevel.FULL,
        },
        deny_paths=("assets.*:prompt:private_notes",),
    )
)
```

The equivalent YAML is validated and completed by the versioned schema:

```yaml
benchmark:
  private-agent:
    capture:
      default_level: hash
      use_semantic_defaults: false
      semantic_overrides:
        tool: full
        output_schema: full
      deny_paths:
        - assets.*:prompt:private_notes
```

Capture levels are `none`, `metadata`, `hash`, `redacted`, and `full`. Sensitive prompt content at
the default metadata level is upgraded to a hash. Public definitions at metadata level may retain
full normalized content. Secret field names are always filtered by the capture normalizer.

The private content fingerprint still drives behavioral versioning. Changing capture from hash to
full does not create a false asset version. Omitted values retain an omission marker and digest;
large allowed values can be stored as bounded artifact references.

## Explicit Tracking Composition

Explicit tracking wins when Autobench observes the exact Python target:

```python
from autobench import track


@track.tool(name="knowledge_search")
def search(query: str) -> list[str]:
    """Search the approved knowledge base."""
    return backend.search(query)
```

If Pydantic AI or another instrumented SDK receives `search`, automatic discovery reuses the
explicit `ToolAsset` identity and version. SDK and capability locators become aliases; Autobench
does not create a second source version. The final provider schema remains a linked effective asset.

Use explicit tracking when the application needs:

- a domain-owned ID or name;
- a manually supplied source path or parent version;
- custom semantic classification;
- lineage before an instrumented run exists;
- a component that no SDK boundary exposes.

## Custom SDK Discovery

`InstrumentAssetSpec` adds the same lineage to arbitrary methods without modifying the target SDK:

```python
from autobench import InstrumentAssetSpec, SpanKind, instrument_method

handle = instrument_method(
    WorkflowClient,
    "execute",
    span="workflow_client.execute",
    span_kind=SpanKind.WORKFLOW,
    assets=[
        InstrumentAssetSpec(
            kind="prompt",
            local_id="instructions",
            value_path="kwargs.instructions",
        ),
        InstrumentAssetSpec(
            kind="tool",
            local_id="tools",
            value_path="kwargs.tools",
            many=True,
        ),
        InstrumentAssetSpec(
            kind="output_schema",
            local_id="output",
            value_path="kwargs.output_type",
        ),
    ],
)

try:
    benchmark.run()
finally:
    handle.close()
```

`value_path` traverses trusted call arguments, mappings, attributes, results, and zero-argument
accessors. Python integrations can use a typed `value_factory`. Serializable integration settings
can use `extractor_target="package.extractors:extract_assets"`; Autobench imports the callable and
passes its `InstrumentCall`. It never evaluates expression strings.

Set `many=True` when the extracted value is a sequence or mapping of independent assets. Each use
is attached to the method span. Extraction failures become run errors or instrumentation
diagnostics without replacing the SDK method's result or exception.

Run the complete offline custom SDK example:

```bash
uv run python examples/automatic_assets/custom_sdk_discovery.py \
  --record /tmp/autobench-custom-assets
```

## Identity, Aliases, And Cross-Layer Correlation

Autobench resolves identity conservatively:

1. the exact explicitly tracked Python target;
2. an explicit asset ID;
3. the target's stable module and qualified name;
4. a previously registered source locator or alias;
5. the SDK, scope, kind, and local ID locator.

Content equality alone does not merge implementations. Two tools can expose the same schema while
running different code. When one framework definition has a unique matching client projection,
Autobench links source and effective forms. Multiple possible definitions produce an
`asset_correlation_ambiguous` diagnostic instead of a silent merge.

Repeated observations of the same asset/version/source/span are deduplicated. Observations on
different spans remain separate `AssetUse` evidence because they prove separate participation.

## Persistence And Replay

`record_experiment(...)` persists only assets referenced by the experiment:

```text
recording/
  experiment.yaml
  cases/
    <case>/<variant>/run.yaml
  assets/
    index.yaml
    <safe-asset-id>.yaml
```

Each asset history contains its current definition, immutable versions, parent links, changed
paths, and a readable diff. Writes are atomic and protected by a file lock. Independent workers
merge existing versions, locators, and aliases instead of replacing each other's histories.

Every run record contains:

- `assets`: exact `AssetVersion` references used by the run;
- `asset_uses`: representation, source locator, scope, span, provenance, aliases, and source link;
- ABP references on the participating spans;
- capture and conflict diagnostics in the materialized trace.

Replay loads those values without importing Pydantic AI, OpenAI, OpenAI Agents, or the application
task module. Rescoring and report replay never mutate the original asset history.

## Compatibility And Failure Behavior

| Integration | Discovery | Representations | Default asset families |
| --- | --- | --- | --- |
| Pydantic AI `>=2.22,<2.23` | yes | definition + effective | agent, capability, prompt, tool, toolset, output schema, policy |
| OpenAI Python `>=2.52,<2.53` | yes | effective | prompt, tool, output schema |
| OpenAI Agents `>=0.19.2,<0.20` | yes | definition | agent, prompt, tool, output schema, guardrail, handoff, policy, toolset |
| HTTPX `>=0.28,<0.29` | no | none | transport evidence only |

Run `autobench instrumentation doctor` to inspect installed compatibility and declared asset
families. Unsupported versions are not patched silently.

Discovery failure is non-fatal by default. Autobench emits typed diagnostics for normalization,
correlation, callback, capture, or persistence problems while preserving the host call's return
value, exception identity, streaming lifecycle, and callback count.

## Choosing The Right Surface

Use automatic discovery for SDK-visible behavioral components. Use explicit tracking for
application-owned identity and components that never cross an SDK boundary. Use
`InstrumentAssetSpec` for a custom SDK or framework. Keep user inputs and outputs as evidence,
models as factors, and measured outcomes as metrics.

That separation makes the resulting RunRecords suitable for reporting today and controlled
candidate optimization later without coupling Autobench core to one AI framework.

---

## Protocol And Traces

Canonical page: https://vcoderun.github.io/autobench/instrumentation-and-traces/

# Instrumentation And Traces

Autobench supports four collection styles that can be mixed in one run:

1. Explicit `RunContext` and `Span` calls inside a task.
2. Lightweight method instrumentation for existing application classes.
3. Trace-envelope adapters for an external agent or workflow runtime.
4. Native Pydantic AI, OpenAI, OpenAI Agents, and HTTPX instrumentors configured from Python or
   YAML.

OpenTelemetry is not a core dependency. Future OTLP bridges can export Autobench spans, but the
evidence model remains owned by Autobench.

See [Native Instrumentation](native-instrumentation.md) for the typed fluent API, YAML DSL,
compatibility doctor, privacy defaults, layered traces, and provider examples.

ABP is the native collection protocol underneath these APIs. It owns signal ordering, task-local
context, capture policy, instrumentation scope, trace materialization, and compatibility
diagnostics. Instrumentors emit ABP evidence directly; they do not create OpenTelemetry spans and
then convert them back into Autobench records.

## Manual Spans

```python
from autobench import DurationMetricSpec, Semantic, SpanKind


def run_case(ctx, case):
    with ctx.span(
        "support_agent",
        kind=SpanKind.AGENT,
        input=case.input,
        duration_metric=DurationMetricSpec(
            name="agent_latency",
            semantic_type=Semantic.TIME_LATENCY,
            unit="ms",
        ),
    ) as agent:
        result = call_agent(case.input)
        agent.set_output(result)
        agent.outcome(result["ok"])
        return result
```

Span duration is calculated when the context manager closes. Nested spans preserve parent-child
relationships and retain evidence emitted before an exception.

## Span Kinds

`SpanKind` includes:

- agent
- LLM
- tool
- retriever
- parser
- workflow
- custom

Kinds are semantic selectors, not restrictions. A domain can use custom kinds and tags while
generic agentic scorers continue selecting standard spans.

## Method Instrumentation

`instrument_method` is the high-level helper for one class method. It records evidence only while
a `RunContext` is active:

```python
from autobench import InstrumentMetricSpec, Semantic, instrument_method

handle = instrument_method(
    SearchClient,
    "search",
    span="search.request",
    metrics=[
        InstrumentMetricSpec(
            name="result_count",
            semantic_type="retrieval.result_count",
            value_factory=lambda call: len(call.result),
        ),
        InstrumentMetricSpec(
            name="request_count",
            semantic_type="llm.requests",
            value_path="result.usage.requests",
        ),
    ],
)

try:
    run_benchmark()
finally:
    handle.close()
```

Instrumentation supports:

- instance, static, class, and inherited methods;
- synchronous and asynchronous calls;
- iterators and generators, including `send`, `throw`, and early close;
- asynchronous iterators and generators, including `asend`, `athrow`, and `aclose`;
- synchronous and asynchronous context managers.

The wrapper preserves the original descriptor, callable signature, return value, exception
identity, and lazy streaming behavior. A stream span ends when the stream actually completes,
fails, times out, or closes, so its duration is not merely the time required to construct an
iterator.

`value_factory` is the typed Python extraction seam. It receives an `InstrumentCall` containing
the bound instance, arguments, result, error, stream item count, and last stream item. `value_path`
is the declarative alternative for trusted attribute, mapping, and zero-argument accessor paths.
Autobench does not execute arbitrary YAML expressions.

Extraction and lifecycle callback errors are recorded as evidence or compatibility diagnostics.
They do not replace the application's result or exception.

The returned `InstrumentationHandle` is also a context manager and restores the original method on
close.

## Scoped Suppression

Instrumentation can be suppressed for the current task without changing global process state:

```python
from autobench import suppress_instrumentation

with suppress_instrumentation("search.client"):
    result = client.search("internal health check")
```

Suppression keys can identify an instrumentor or an operation family. Unrelated instrumentors stay
active, nested scopes compose, and context tokens are reset even when application code raises. An
empty `suppress_instrumentation()` scope suppresses all ABP instrumentation in the current task.

## Native Instrumentors

Reusable SDK integrations implement the `Instrumentor` contract:

```python
from autobench import (
    Compatibility,
    InstrumentationHandle,
    InstrumentationRuntime,
    InstrumentorInfo,
)
from autobench.protocol import AbstractionLayer, CaptureMechanism


class ClientInstrumentor:
    info = InstrumentorInfo(
        id="example.client",
        version="1.0.0",
        target_distribution="example-client",
        supported_versions=">=2,<3",
        mechanism=CaptureMechanism.HOOK,
        layer=AbstractionLayer.CLIENT,
        span_kinds=("client.request",),
        semantic_families=("request", "response"),
    )

    def check(self) -> Compatibility:
        return Compatibility.compatible()

    def install(self, runtime: InstrumentationRuntime) -> InstrumentationHandle:
        unsubscribe = register_native_callback(...)
        return InstrumentationHandle(unsubscribe, info=self.info)
```

Install instrumentors directly through one manager when building a custom integration:

```python
from autobench import InstrumentationManager

with InstrumentationManager() as manager:
    compatibility = manager.check(ClientInstrumentor())
    if compatibility.installable:
        manager.install(ClientInstrumentor())
        run_benchmark()
```

`InstrumentorInfo` declares stable identity, target package and version range, mechanism, layer,
semantic families, source convention, optional dependencies, and sync/async/streaming/native-hook
capabilities. `Compatibility` distinguishes compatible, degraded, unavailable, unsupported, and
conflicting installations. Missing or incompatible optional dependencies degrade only the feature
that needs them; a missing required target package prevents installation.

Installing the same instrumentor version twice increments an owner reference count instead of
installing duplicate hooks. Closing the final handle unregisters native callbacks or restores the
exact patched descriptor. Competing owners can instrument the same method independently, while an
external wrapper replacement produces a conflict diagnostic instead of being overwritten.

Mechanisms should be selected in this order:

1. stable native processor or callback;
2. stable native wrapper/decorator extension point;
3. public method patch;
4. explicitly version-pinned private method patch;
5. unsupported with a compatibility diagnostic.

Application benchmarks normally use the higher-level lifecycle owner instead:

```python
from autobench import Benchmark, HTTPXInstrumentation, OpenAIInstrumentation

benchmark = Benchmark("chat").instrument(
    OpenAIInstrumentation(),
    HTTPXInstrumentation(),
)
result = benchmark.run()
```

`Benchmark.instrument(...)` installs configured and custom instrumentors before any matrix item,
keeps them active through concurrent runs and streams, and closes them after execution.

## Trace Envelopes

Adapters can normalize a completed external trace into `TraceEnvelope`:

```python
from autobench import TraceEnvelope, attach_trace

trace = TraceEnvelope(
    trace_id="trace-42",
    name="checkout-agent",
    input={"cart_id": "c1"},
    output={"status": "complete"},
    spans=tuple(converted_spans),
    attributes={"framework": "custom-agent-runtime"},
)

attach_trace(ctx, trace)
```

`attach_trace` preserves spans and errors and projects known usage, model, provider, duration, and
outcome fields into semantic observations. Large native trace payloads should be written as an
artifact and referenced by `raw_artifact`.

## Pydantic AI Usage

Install the optional native instrumentor when the application uses Pydantic AI:

```bash
pip install 'autobench[pydantic-ai]'
```

The integration uses Pydantic AI's public capability hooks and only injects its
capability while an Autobench run is active:

```python
from autobench import Benchmark, PydanticAIInstrumentation

experiment = benchmark.instrument(PydanticAIInstrumentation()).run()
```

No manual span or metric calls are required. The instrumentor captures:

- agent runs and streamed execution;
- model requests, requested and response model identities, providers, and direct usage;
- tool argument validation, execution, retry, failure, approval, and deferred control flow;
- structured-output validation;
- first-chunk latency, partial streams, failures, and normal completion;
- tracked prompt, tool, and output-schema versions;
- multimodal metadata, with binary references only when the capture policy requests full content.

The instrumentor composes with user event handlers and Pydantic AI's own
`Instrumentation` capability. It does not configure, replace, or require
OpenTelemetry. Autobench 0.2.x pins the public integration seam
to Pydantic AI 2.22.x; `InstrumentationManager.check()` reports incompatible
versions before installing hooks.

Application outputs and exceptions are passed through unchanged. Aggregate agent
usage and direct model usage retain distinct accounting scopes, and cost remains a
downstream derivation. Replaying recorded ABP evidence does not require Pydantic AI
to be installed.

See the [live Pydantic AI example](examples.md#pydantic-ai-live-layered-instrumentation) for a tool-using,
structured-output, streaming benchmark with a retry path.

### Usage Bridge

Pydantic AI usage can be normalized without importing Pydantic AI into core:

```python
from autobench import PydanticAIUsage, record_pydantic_ai_usage

record_pydantic_ai_usage(
    ctx,
    PydanticAIUsage(
        requests=1,
        input_tokens=420,
        output_tokens=83,
        model_name="gemini-3-flash-preview",
        provider="openrouter",
    ),
)
```

The bridge emits canonical LLM token, model, and provider observations that pricing derivation and
reports can consume.

## Trace Extraction And Accounting

Instrumentors record immutable facts. Extractors turn a completed ABP trace into semantic
observations without mutating that trace:

```python
from autobench import (
    CompositeExtractor,
    SignalExtractor,
    SpanExtractor,
    UsageExtractor,
    replay_extraction,
)

extractor = CompositeExtractor(
    SignalExtractor(),
    SpanExtractor(),
    UsageExtractor(),
)
derived_record = replay_extraction(record, extractor)
```

The extractors have separate ownership:

- `SignalExtractor` reconstructs measurements and events and preserves their accounting scope,
  abstraction layer, logical operation ID, and instrumentor identity.
- `SpanExtractor` derives generic operation counts, direct durations, maximum depth and fan-out,
  critical-path makespan, parallelism, incomplete work, retry/recovery, validation, approval,
  tool-call, message-growth, and reference evidence.
- `UsageExtractor` derives LLM request, token, requested-model, response-model, and provider
  evidence. It never derives cost.

Every extractor has a stable name and version. Replay records both in extraction evidence and
RunRecord lineage. Replaying a newer version replaces observations owned by the older version in
the derived record; the parent record remains unchanged.

### Direct And Aggregate Evidence

ABP keeps all raw measurements but prevents framework/client nesting from inflating totals:

1. Aggregate parent measurements are never added to direct child measurements.
2. Usage totals select one abstraction layer per semantic, preferring client evidence before
   framework, application, and transport evidence.
3. Equivalent direct operations with a shared logical operation ID are counted once.
4. Equal equivalent values are deduplicated. Conflicting values require a unique explicit
   authority; unresolved conflicts produce `ambiguous_direct_measurement` and are excluded from
   the derived total.
5. Aggregate values are retained as validation evidence. A disagreement with the direct total
   produces `aggregate_measurement_mismatch`.
6. Requested and response model identities remain separate factors.

Reports and `ObservationQuery.first_exact()` prefer an accounting-safe aggregate summary over
same-source per-operation direct evidence. Raw and projected queries can still inspect every
underlying observation.

Graph timing uses monotonic timestamps only. `time.critical_path` is the observed trace makespan,
and `operation.parallelism` is completed leaf work divided by that makespan. Invalid or partial
clock evidence is retained through diagnostics rather than repaired with wall-clock subtraction.

## Adapter Boundary

Core instrumentation intentionally does not know Pydantic AI, OpenAI Agents, LangChain, DSPy, or
OpenTelemetry internals. An integration should:

1. Collect from the framework's stable hooks.
2. Convert native calls or traces into Autobench spans and observations.
3. Store large raw payloads as artifacts.
4. Keep native dependencies optional.

This boundary lets applications use existing instrumentation while RunRecords remain portable.

---

## Native Instrumentation

Canonical page: https://vcoderun.github.io/autobench/native-instrumentation/

# Native Instrumentation

Autobench native instrumentors collect ABP traces from supported SDKs without task-level
`ctx.span()` or `ctx.metric()` calls. They are optional adapters around public hooks or pinned,
reviewed patch points. Core benchmark, record, replay, and report imports do not require any of the
instrumented SDKs.

## Install

Install one integration or the complete set:

```bash
pip install 'autobench[pydantic-ai]'
pip install 'autobench[openai]'
pip install 'autobench[openai-agents]'
pip install 'autobench[httpx]'
pip install 'autobench[instrumentation]'
```

The integration registry is lazy. Loading a YAML spec, replaying evidence, or running
`autobench instrumentation doctor` does not import an SDK that is not installed.

## Automatic Discovery

Use `instrument_all()` when the benchmark should activate every built-in integration that is
installed and compatible in the current environment:

```python
from autobench import Benchmark

benchmark = Benchmark("support-agent").instrument_all()
```

Semantic instrumentors also discover SDK-visible behavioral assets by default. The application does
not need tracking decorators for prompts, tools, output schemas, capabilities, agents, guardrails,
handoffs, or policies already visible at those boundaries. See
[Automatic Asset Discovery](automatic-asset-discovery.md) for identity, source/effective links,
privacy, persistence, and custom SDK extraction.

Unavailable or unsupported integrations are skipped by default and recorded on each run as
`instrumentation.skipped` diagnostic evidence. Use `strict=True` when the environment must support
the complete selected set:

```python
benchmark = Benchmark("support-agent").instrument_all(
    exclude={"httpx"},
    strict=True,
)
```

Explicit settings take precedence over discovery, including an explicit `false`. A custom runtime
instrumentor with the same instrumentor ID also takes precedence, so automatic discovery does not
install a duplicate. Calling `instrument_all()` again replaces the previous automatic settings.

### Live OpenRouter Trace

The live Pydantic AI example exercises automatic discovery across all three active layers:

```bash
uv sync --extra instrumentation
export OPENROUTER_API_KEY=...
export OPENROUTER_MODEL=openrouter:openai/gpt-5.6-luna
uv run python examples/pydantic_ai/openrouter_instrument_all.py \
  --record /tmp/autobench-openrouter
```

The benchmark itself only opts in once:

```python
benchmark = Benchmark("openrouter-shopping-agent").instrument_all()
```

The example uses plain instructions, a plain tool function, and an undecorated Pydantic output type.
Automatic discovery adds Pydantic AI, OpenAI client, and HTTPX transport instrumentors. One real
request therefore records agent, model, tool, output
validation, stream, client request, and transport spans with their native parentage. It also records
model identity, token usage, durations, HTTP method/host/path/status, score observations, asset
versions, capture diagnostics, and replayable source provenance. HTTP bodies and credentials remain
redacted by the default capture policy.

The full source is `examples/pydantic_ai/openrouter_instrument_all.py`. It deliberately contains no
manual `ctx.span()` or `ctx.metric()` calls so the resulting record demonstrates native collection
rather than hand-authored benchmark telemetry.

## YAML

Instrumentation belongs to the named benchmark:

```yaml
# yaml-language-server: $schema=schemas/0.2.0/benchmark_schema.json
benchmark:
  support-agent:
    dataset:
      source: file://datasets/cases.yaml
    run:
      python: support_benchmark:run
    variants:
      baseline:
        factors:
          model.name: openrouter:openai/gpt-5.6-luna
    instrumentation:
      all:
        exclude: [httpx]
        strict: false
        assets:
          representations: [definition, effective]
          include: [prompt, tool, output_schema, capability]
      pydantic_ai: {}
      openai: {}
      httpx:
        capture:
          path: hash
          request_headers: [x-request-id]
          response_headers: [x-request-id]
          request_body: false
          response_body: false
          max_body_bytes: 65536
```

Use `false` to retain a known integration in a shared spec without installing it:

```yaml
instrumentation:
  openai_agents: false
```

Unknown integration names and unknown settings fail validation. YAML never evaluates Python
expressions.

The `all` block follows the same precedence rules as the Python builder. In this example HTTPX is
excluded from discovery but its explicit capture settings still install it; all explicit entries
remain authoritative.

## Python

The fluent API accepts typed, serializable settings:

```python
from autobench import (
    Benchmark,
    HTTPXCaptureSettings,
    HTTPXInstrumentation,
    OpenAIInstrumentation,
)

benchmark = Benchmark("streaming-chat").instrument(
    OpenAIInstrumentation(),
    HTTPXInstrumentation(
        capture=HTTPXCaptureSettings(
            path="hash",
            response_headers=("x-request-id",),
        )
    ),
)
experiment = benchmark.run()
```

It also accepts a custom `Instrumentor` instance. Runtime instances are installed for the whole
benchmark matrix and closed even when execution fails. They are intentionally not serialized into
the YAML spec:

```python
benchmark.instrument(MyNativeInstrumentor(settings))
```

Duplicate instrumentor IDs are rejected before hooks are installed. This avoids ambiguous
ownership when a typed setting and a custom instance configure the same integration.

## Built-In Integrations

| Integration | Layer | Collection seam | Evidence |
| --- | --- | --- | --- |
| Pydantic AI | framework | public agent capability | spans plus agent/capability/prompt/tool/toolset/output-schema lineage |
| OpenAI Python | client | reviewed public client methods and stream types | spans plus effective prompt/tool/output-schema lineage |
| OpenAI Agents | framework | native trace processor and public Runner surface | spans plus agent/prompt/tool/output-schema/guardrail/handoff/policy lineage |
| HTTPX | transport | public transport methods | request method/host/path policy, status, selected headers, body metadata, stream lifecycle |

Run compatibility diagnostics before a benchmark:

```bash
autobench instrumentation doctor
```

The Rich output shows availability, installed version, supported range, abstraction layer,
mechanism, sync/async/streaming capabilities, span kinds, semantic families, capture defaults, and
degradation diagnostics.

## Layered Traces

Instrumentors compose instead of flattening one another. A Pydantic AI request using the OpenAI
client over HTTPX can produce this parent chain:

```text
task
  agent
    llm framework operation
      OpenAI client operation
        HTTP request
```

Transport spans do not emit token or cost usage. Framework aggregate usage and client direct usage
retain different accounting scopes. Trace extraction selects one authoritative direct layer and
keeps aggregate values as validation evidence, so enabling HTTPX cannot inflate LLM totals.

## Streaming Lifecycle

A stream span does not end when an SDK returns an iterator. It remains open until the stream:

- completes normally;
- raises;
- is cancelled;
- is explicitly closed early;
- is abandoned when the instrumentor manager closes.

ABP records first-chunk evidence, item/chunk counts, partial state, and the final end reason. Native
items, exceptions, iterator methods, and context-manager behavior pass through unchanged.

## HTTP Privacy Defaults

HTTPX capture defaults are deliberately conservative:

- query-free path hash, not the raw path;
- no request or response headers unless named;
- authorization, cookies, API keys, tokens, passwords, and secrets always redacted;
- no request or response body capture;
- bounded capture when bodies are explicitly enabled;
- binary bodies represented by metadata and a digest, not embedded bytes.

`path: full` is an explicit opt-in. Query strings and URL user information are not recorded by the
path setting. Capture policies apply before evidence reaches a RunRecord.

## Trace Diagnostics

Every native span records an `InstrumentationScope`: instrumentor and target versions, mechanism,
abstraction layer, and source convention. Source facts can be retained alongside canonical
Autobench semantic attributes. Unsupported library versions fail installation instead of silently
patching an unknown lifecycle.

Inspect recorded trace shape without importing task modules or optional SDKs:

```bash
autobench instrumentation trace runs/support-agent/exp_...
```

The command reports per-case span/root counts, partial traces, diagnostics, span-kind totals, and
instrumentor composition.

## Replay Without SDKs

RunRecords contain materialized ABP traces, not live provider objects. A reporting or optimization
worker can replay and re-extract evidence with only Autobench installed:

```python
from autobench import CompositeExtractor, SignalExtractor, SpanExtractor, UsageExtractor
from autobench.records.replay import load_run_record, replay_extraction

record = load_run_record(path, root_dir=run_dir)
derived = replay_extraction(
    record,
    CompositeExtractor(SignalExtractor(), SpanExtractor(), UsageExtractor()),
)
```

Extraction creates a derived record with lineage; it never mutates the original record.

## ABP And OpenTelemetry

ABP is not an OpenTelemetry wrapper and has no OTel dependency. It is Autobench's evidence protocol
for benchmark execution, semantic measurements, accounting scope, partial streams, replay, and
optimization lineage. Native instrumentors use the same kinds of stable SDK hooks that mature OTel
instrumentations validate, but emit ABP directly.

A future bridge can export ABP spans to OTLP systems such as Logfire or Datadog. That bridge will be
an adapter: ABP remains the source evidence model, and importing Autobench will not require an OTel
SDK or collector.

## Protocol Stability

ABP protocol version `1` is the initial public serialized contract. Autobench `0.2.x` preserves the
meaning of its signal, trace, scope, provenance, and accounting fields. Readers retain unknown
additive data through extension maps, while a breaking wire-format change requires a new protocol
version. Instrumentor patch points are compatibility-gated separately because provider SDK
lifecycles can change independently of ABP.

## Examples

- `examples/abp_manual`: explicit workflow spans plus method instrumentation.
- `examples/abp_concurrent`: concurrent sibling operations with task-local parentage.
- `examples/pydantic_ai`: tool use, retry, streaming, and structured output; OpenAI models add
  OpenAI and HTTPX layers.
- `examples/abp_openai`: offline official OpenAI streaming over an HTTPX mock transport.
- `examples/abp_openai_agents`: offline native OpenAI Agents trace-processor workflow.
- `examples/abp_replay`: trace extraction from recorded evidence without importing provider SDKs.

---

## Compatibility Contract

Canonical page: https://vcoderun.github.io/autobench/abp-compatibility/

# ABP Compatibility Contract

This page freezes the observable behavior preserved while the Autobench
Instrumentation Protocol (ABP) replaces legacy span and instrumentation
internals. Phases 1 through 8 now satisfy this contract. The complete public
instrumentation guide is in [Instrumentation And Traces](instrumentation-and-traces.md).

## Compatibility Boundary

The following top-level imports remain available while ABP is introduced:

```python
from autobench import (
    ArtifactRef,
    AssetVersion,
    DurationMetricSpec,
    ErrorRecord,
    InstrumentationHandle,
    InstrumentCall,
    InstrumentFactorSpec,
    InstrumentMetricSpec,
    Observation,
    RunContext,
    RunRecord,
    Span,
    SpanKind,
    SpanRecord,
    TraceEnvelope,
    attach_trace,
    get_active_run_context,
    instrument_method,
    trace_to_observations,
)
```

ABP may move implementations into new packages, but these imports and their
current behavior remain compatibility facades until a separately announced
deprecation cycle.

### Manual spans

Existing manual spans preserve these guarantees:

- `ctx.span(...)` is a synchronous context manager;
- nested spans receive the active span as `parent_id`;
- every completed span has UTC start/end timestamps and a non-negative
  monotonic duration;
- a configured duration metric is linked to the span;
- metrics, factors, events, errors, and artifacts retain their span link;
- exceptions are recorded and then propagated;
- `Span.set_output`, `Span.set_attribute`, and `Span.set_usage` continue to
  update the recorded span;
- entering instrumentation without an active run context remains a no-op;
- closing the final `InstrumentationHandle` restores the original descriptor.

The tests in `tests/test_abp_compatibility.py` are the executable form of this
contract.

### Stored evidence

Legacy model-shaped RunRecord and TraceEnvelope YAML remains loadable. ABP
will add protocol data additively and preserve the existing `spans` input
during migration. Replay must not require the task module or an optional
instrumented SDK.

The frozen legacy examples are:

- `tests/fixtures/abp/legacy_run_record.yaml`
- `tests/fixtures/abp/legacy_trace_envelope.yaml`

## Concurrency Regression Contract

`RunContext` now uses task-local ABP context. The concurrency migration is
covered by passing regression tests for all of these cases:

1. concurrent sibling spans under one parent both point to that parent;
2. a nested task inherits the parent active at task creation;
3. completing one sibling does not change the other sibling's active parent;
4. out-of-order completion does not corrupt later parent selection;
5. cancellation closes only the cancelled branch and restores its context;
6. separate RunContexts never share active spans.

The old mutable-stack reproduction is retained only in design history; it is
not the current runtime behavior.

## Canonical Trace Decision

ABP will have one canonical immutable `Trace` model. `TraceEnvelope` does not
have behavior that justifies a second trace representation, so it will become
a compatibility name for `Trace` rather than a parallel model. Existing
`TraceEnvelope(...)`, `attach_trace(...)`, and `trace_to_observations(...)`
callers continue to work.

This avoids conversion drift between manually attached traces and traces
materialized from native ABP signals.

## Package Shape

ABP code is introduced only when its phase needs it. Empty placeholder modules
are not created.

```text
autobench/
  protocol/
    ids.py
    values.py
    signals.py
    traces.py
    context.py
    capture.py
    collector.py
  instrumentation/
    models.py
    manager.py
    patching.py
    streaming.py
    pydantic_ai.py
    openai.py
    openai_agents.py
    httpx.py
```

Small modules are combined when separation would only create navigation cost.
Existing unrelated modules are not moved as part of ABP.

## Optional Integration Targets

The initial integration extras are reserved as follows:

| Extra | Research baseline | First implementation phase |
| --- | ---: | ---: |
| `autobench[pydantic-ai]` | Pydantic AI 2.22.0 | 10 |
| `autobench[openai]` | OpenAI Python 2.52.0 | 11 |
| `autobench[openai-agents]` | OpenAI Agents 0.19.2 | 11 |
| `autobench[httpx]` | HTTPX 0.28.1 | 12 |
| `autobench[instrumentation]` | all integrations above | 13 |

These versions are the public-API research baseline captured on 2026-08-03,
not a compatibility claim. Dependency metadata is added only when each native
instrumentor and its version matrix exist. Autobench core remains free of
these dependencies.

## Manual Span Performance Baseline

The baseline measures a minimal completed manual span with no observations,
artifacts, or errors. Each repeat creates one RunContext and records 10,000
spans. Timing uses `timeit.repeat`; duration comes from the host monotonic
clock. The benchmark does not enforce a CI latency threshold because shared CI
timing is not stable.

Reproduce it with:

```bash
uv run python scripts/benchmark_spans.py --iterations 10000 --repeats 7
```

Baseline captured before ABP runtime changes:

| Field | Value |
| --- | ---: |
| Date | 2026-08-03 |
| Python | 3.11.13 |
| Platform | macOS 26.1 arm64 |
| Minimum | 3,227.4 ns/span |
| Median | 3,324.8 ns/span |

The raw per-repeat values were `3467.0`, `3308.4`, `3299.3`, `3394.4`,
`3394.3`, `3324.8`, and `3227.4` ns/span. Later phases compare using the same
script and workload; they do not compare unrelated machine results.

Release measurements captured after ABP materialization on the same host:

| Workload | Measurement |
| --- | ---: |
| Manual ABP span | 28,836.7 ns/span median |
| HTTPX baseline request | 35,346.2 ns/request median |
| Instrumented HTTPX request | 224,263.6 ns/request median |
| HTTPX instrumentation overhead | 188,917.4 ns/request median |
| 10,000 x 32-byte HTTP stream | 2,618.0 ns/chunk median |
| Long-stream peak allocation | 30,918 bytes |

The long-stream result is about `3.1` peak allocated bytes per emitted chunk, which confirms the
instrumentor does not retain chunk payloads as the stream grows. These numbers characterize this
machine and are not release thresholds. Reproduce transport and stream measurements with:

```bash
uv run python scripts/benchmark_spans.py --httpx --iterations 1000 --repeats 7
uv run python scripts/benchmark_spans.py --httpx-stream --chunks 10000 --chunk-size 32 --repeats 7
```

---

## YAML Spec

Canonical page: https://vcoderun.github.io/autobench/yaml-spec/

# YAML Spec

Autobench is YAML-first. Python builders compile to the same internal `BenchmarkSpec`.
Every YAML file written by Autobench includes a `yaml-language-server` schema header that points
to the versioned schema cache under `~/.autobench/<version>/schemas/`.

## Authoring Sections

The authoring DSL places the benchmark ID under `benchmark` and keeps all behavior inside that
named benchmark:

| Section | Required | Purpose |
| --- | --- | --- |
| `description` | No | Human-readable benchmark intent |
| `dataset` | Yes | Inline or file-backed cases, defaults, version, and metadata |
| `run` | For execution | Python task target |
| `variants` | Yes | Named factor combinations |
| `score` | No | Built-in or Python scorers |
| `derive` | No | Per-run semantic derivation such as token cost |
| `post_derive` | No | Cross-run derivation such as paired baseline |
| `policies` | No | Semantic metric constraints |
| `report` | No | Leaderboard, matrix, comparisons, and distributions |
| `semantic_registry` | No | Custom semantic definitions and aliases |

## Complete Authoring Example

```yaml
# yaml-language-server: $schema=./schemas/0.2.0/benchmark_schema.json
benchmark:
  support-routing:
    description: Compare current and candidate routing behavior.
    dataset:
      source: file://datasets/cases.yaml
      version: v2
      defaults:
        tags: [regression]
    run:
      python: benchmark_tasks:run_case
    variants:
      baseline:
        factors:
          model:
            value: openrouter:openai/gpt-5.6-luna
            semantic: llm.model.name
          prompt_version:
            value: route-v3
            semantic: prompt.version
            optimize: true
      candidate:
        factors:
          model:
            value: openrouter:openai/gpt-5.6-luna
            semantic: llm.model.name
          prompt_version:
            value: route-v4
            semantic: prompt.version
            optimize: true
    score:
      route_correctness:
        exact:
          actual: output.route
          expected: case.expected.route
        semantic: quality.correctness
        goal: maximize
        role: objective
      success:
        pass: output.ok
        semantic: result.success
        role: constraint
    derive:
      - kind: token_cost
        pricing: file://pricing/models.yaml
        output:
          name: request_cost
          semantic_type: money.cost
          unit: usd
          direction: minimize
          role: constraint
    policies:
      - name: must-succeed
        metric: result.success
        must_equal: true
    report:
      leaderboard:
        show:
          accuracy:
            metric: quality.correctness
            aggregate: ratio_true
          total_cost:
            metric: money.cost
            aggregate: sum
      matrix:
        metric: quality.correctness
      compare:
        baseline -> candidate:
          show:
            accuracy:
              metric: quality.correctness
              aggregate: ratio_true
```

## Resolution Rules

- File references resolve relative to the benchmark YAML.
- Python targets use `module:callable` and receive inferred search paths from the spec directory.
- Duplicate case and variant IDs are validation errors.
- A nonempty runnable matrix requires a task.
- Scorer definitions must select exactly one scoring action.
- Remote file references are rejected; price-source URL loading is an explicit integration API.
- Custom semantics should be declared in the semantic registry.

## Shape

```yaml
benchmark:
  support-routing:
    description: Deterministic support routing benchmark.
    dataset:
      source: file://datasets/cases.yaml
      defaults:
        metadata:
          owner: docs
    run:
      python: app.benchmarks.support:run_ticket_case
    variants:
      route_v1:
        factors:
          prompt_version:
            value: route-v1
            semantic: prompt.version
            optimize: true
          routing_profile: baseline
    score:
      routing_correctness:
        exact:
          actual: output.queue
          expected: case.expected.queue
        semantic: quality.correctness
      tool_arguments:
        expected_action:
          metric: arguments
          observed_kind: tool
        span:
          kind: tool
        semantic: agent.tool.argument.correctness
    report:
      leaderboard:
        show:
          pass_rate:
            metric: result.success
            aggregate: ratio_true
```

## Exported Benchmark YAML

When Autobench renders a benchmark spec back to YAML, it uses a DSL-like shape instead of a raw
model dump:

```yaml
benchmark:
  support-routing:
    description: Route support tickets.
    dataset:
      source: datasets/cases.yaml
      cases:
        - id: ticket_1
          input:
            subject: Refund
    run:
      python: app.benchmarks.support:run_ticket_case
    variants:
      route_v1:
        factors:
          prompt_version:
            value: route-v1
            semantic: prompt.version
            optimize: true
          routing_profile: baseline
    score:
      success:
        pass: output.matched
        semantic: result.success
        goal: maximize
    report:
      leaderboard:
        show:
          pass_rate:
            metric: result.success
            aggregate: ratio_true
```

## Notes

- `dataset.source` supports local `file://` references and globs.
- task targets use `module:function`.
- variant factors accept either mapping or list form.
- YAML does not execute inline expressions.
- importable code hooks such as Python scorers remain explicit dotted targets.
- `score.<name>.span` can target component spans by kind, name, tag, path, or semantic type.
- `expected_action` scores compare `case.expected.actions` or `case.expected.tool_calls` with observed spans.

## Native Instrumentation

The optional `instrumentation` section installs ABP SDK integrations for the complete benchmark
matrix:

```yaml
benchmark:
  support-agent:
    instrumentation:
      all:
        exclude: [httpx]
        strict: false
        assets:
          discover: true
          representations: [definition, effective]
          include: [prompt, tool, output_schema, capability]
      pydantic_ai: {}
      openai: {}
      openai_agents: false
      httpx:
        capture:
          path: hash
          request_headers: [x-request-id]
          response_headers: [x-request-id]
          request_body: false
          response_body: false
          max_body_bytes: 65536
```

`all` discovers every installed, compatible built-in integration. Missing integrations are skipped
and recorded as run diagnostics unless `strict: true` is set. `exclude` accepts `pydantic_ai`,
`openai`, `openai_agents`, and `httpx`. An explicit entry, including `false`, overrides discovery;
the explicit HTTPX block above therefore remains enabled despite the discovery exclusion.

`{}` selects privacy-safe defaults. `false` disables a known integration. Unknown integration
names, settings, exclusions, or HTTP capture modes are validation errors. Optional SDKs are
imported only when their enabled integration is resolved for execution. Replay never resolves this
section.

The versioned `benchmark_schema.json` describes this surface, so YAML language servers complete
integration names and capture settings. See [Native Instrumentation](native-instrumentation.md) for
the lifecycle and privacy contract.

## Capture Policy

The benchmark-level `capture` section controls ABP evidence and discovered asset content for every
case/variant run:

```yaml
benchmark:
  private-agent:
    capture:
      default_level: hash
      use_semantic_defaults: false
      semantic_overrides:
        tool: full
        output_schema: full
      deny_paths:
        - assets.*:prompt:private_notes
```

Supported levels are `none`, `metadata`, `hash`, `redacted`, and `full`. Other fields include
semantic/path allow and deny lists, secret names, inline/artifact limits, collection/string/depth
limits, binary retention, and source-attribute retention. Unknown fields or levels fail validation.
See [Automatic Asset Discovery](automatic-asset-discovery.md#privacy-and-capture-policy) for the
content/version behavior.

## Safe Extensibility

YAML is intended to be shareable and replayable. For that reason:

- file references are resolved relative to the spec path
- remote URLs are rejected
- inline Python expressions are not part of the YAML surface

## Exported Run Record YAML

Run records are the immutable per-case/per-variant evidence files used by replay. The trace signal
objects below are abridged; recorded files retain their timestamps, sequence IDs, execution
references, scope provenance, and captured attributes:

```yaml
record:
  type: run
  version: 4

protocol:
  name: abp
  version: 1
  semantic_registry: 1

run:
  id: run_ticket_1_route_v1
  experiment: exp_support_routing_20260507T120000Z
  benchmark: support-routing
  case: ticket_1
  variant: route_v1
  status: passed
  outcome:
    evaluation: passed
    task: passed

case:
  id: ticket_1
  input:
    subject: Refund
  expected:
    queue: billing

variant:
  id: route_v1
  factors:
    prompt_version:
      value: route-v1
      semantic: prompt.version
      optimize: true

scores:
  routing_correctness:
    value: true
    semantic: quality.correctness
    role: objective

metrics:
  measurements:
    routing_correctness:
      id: observation_1
      name: routing_correctness
      kind: metric
      value: true
      semantic: quality.correctness
  diagnostics:
    latency_ms:
      value: 12.4
      semantic: time.latency
      unit: ms

trace:
  protocol: abp
  protocol_version: 1
  trace_id: 70d8f4b6742d412a85cb7a198db07fe1
  execution:
    benchmark_id: support-routing
    experiment_id: exp_support_routing_20260507T120000Z
    run_id: run_ticket_1_route_v1
    case_id: ticket_1
    variant_id: route_v1
  root_span_ids: [3f2f6c57b9f56a11]
  spans:
    - span_id: 3f2f6c57b9f56a11
      operation: benchmark.run
      kind: task
      scope:
        instrumentor_name: autobench.manual
        instrumentor_version: 0.2.0
        package_name: autobench
        package_version: 0.2.0
        mechanism: manual
        layer: application
      status: ok
      end_reason: completed
      measurements: []
      events: []
      links: []
      references: []
      partial: false
  links: []
  references: []
  diagnostics: []
  signals:
    - type: span_start
      protocol: abp
      protocol_version: 1
      span_id: 3f2f6c57b9f56a11
      operation: benchmark.run
      kind: task
    - type: span_end
      protocol: abp
      protocol_version: 1
      span_id: 3f2f6c57b9f56a11
      status: ok
      reason: completed
  partial: false

spans:
  call_router:
    kind: workflow
    started_at: "2026-05-07T12:00:00Z"
    duration: 0.0124
    attributes:
      component: router
  lookup_user:
    kind: tool
    parent: call_router
    input:
      user_id: u1
    output:
      tier: gold
    duration: 0.004

artifacts:
  generated_spec:
    media: application/x-yaml
    path: artifacts/run_ticket_1_route_v1/generated_spec.yaml

assets:
  prompt.router:
    version: 7c91d4d7b1af

output:
  queue: billing
```

When the serialized ABP trace exceeds the inline limit, the same section becomes a compact summary
and artifact reference:

```yaml
trace:
  id: 70d8f4b6742d412a85cb7a198db07fe1
  partial: false
  spans: 7
  signals: 31
  artifact:
    id: abp_trace
    name: ABP trace
    media: application/vnd.autobench.abp-trace+yaml
    path: artifacts/run_ticket_1_route_v1/trace.yaml
```

## Exported Dataset YAML

Dataset exports use a DSL-like shape instead of raw model dumps:

```yaml
record:
  type: dataset
  version: 1

dataset:
  id: tickets
  version: v1
  metadata:
    owner: support
  defaults:
    tags: [smoke]
  cases:
    - id: ticket_1
      input:
        subject: Refund
```

## Exported Semantic Registry YAML

Semantic registry exports use stable type ids with compact metadata:

```yaml
record:
  type: semantic_registry
  version: 1

semantic_registry:
  version: 1
  aliases:
    quality.answer: quality.score
  types:
    money.cost:
      unit: usd
      shape: number
    serving.cost:
      parent: money.cost
      unit: usd
      shape: number
```

## Exported Pricing YAML

Pricing tables are helper data, not a required runtime dependency. They keep provider/model aliases
and tiered token prices readable:

```yaml
record:
  type: pricing
  version: 1

pricing:
  provider: openrouter
  source: genai-prices
  updated_at: "2026-05-07"
  models:
    google/gemini-3-flash-preview:
      name: Gemini 3 Flash Preview
      aliases:
        - google:gemini-3-flash-preview
        - openrouter/google/gemini-3-flash-preview
      input:
        unit: mtok
        price: 0.3
        tiers:
          - up_to: 1000000
            price: 0.3
          - price: 0.6
      output:
        unit: mtok
        price: 2.5
      cache_read:
        unit: mtok
        price: 0.03
```

## Exported Report YAML

Report exports keep the summary under a single `report:` body:

```yaml
record:
  type: report
  version: 1

report:
  benchmark: support-routing
  experiment: exp_support_routing_20260507T120000Z
  runs: 6
  status:
    passed: 5
    failed: 1
  variants:
    baseline:
      factors:
        model.name: openrouter:openai/gpt-5.6-luna
  leaderboard:
    baseline:
      runs: 2
      metrics:
        avg_coverage: 0.82
  cases:
    ticket_1:
      baseline:
        status: passed
        metrics:
          coverage (coverage.ratio): 0.8
  matrix:
    metric: coverage.ratio
    cases:
      ticket_1:
        baseline: 0.8
  compare:
    baseline -> candidate:
      runs: 2
      confounded: true
  distributions:
    cost_distribution:
      semantic: money.cost
      variants:
        baseline: [0.01, 0.02]
```

## Exported Experiment YAML

Experiment records keep replay data structured, but the outer shape stays readable:

```yaml
record:
  type: experiment
  version: 4

experiment:
  id: exp_support_routing_20260507T120000Z
  benchmark: support-routing

benchmark:
  id: support-routing
  dataset:
    id: tickets
    version: v1
    hash: 9b5d...
  cases:
    - ticket_1
    - ticket_2
  counts:
    cases: 2
    variants: 3
    runs: 6
  warnings: []
  spec:
    hash: a13c...
    snapshot:
      benchmark:
        id: support-routing

runs:
  count: 6
  passed: 5
  failed: 1
  errored: 0
  skipped: 0
  paths:
    - cases/ticket_1/route_v1/run.yaml

files:
  /abs/path/autobench.yaml: 3c4d...

environment:
  python: "3.11.13"
  platform: macOS-15.5-arm64-arm-64bit
  cwd: /workspace/autobench

semantic_registry:
  version: 1
  aliases:
    quality.answer: quality.score
  types:
    money.cost:
      unit: usd
      shape: number
```

## Exported Artifact YAML

Artifacts are split into metadata and payload files. Text payloads stay as text. Structured payloads
are wrapped so they remain recognizable YAML records:

```yaml
record:
  type: artifact
  version: 1

artifact:
  id: trace
  name: trace
  media_type: application/x-yaml
  span_id: call_router
  payload: artifacts/run_ticket_1_route_v1/trace.yaml
```

```yaml
record:
  type: artifact_payload
  version: 1

artifact:
  id: trace
  name: trace
  media_type: application/x-yaml

payload:
  steps:
    - tool: route_ticket
      arguments:
        queue: billing
```

## Exported Asset YAML

Tracked assets are stored as a readable index plus per-asset history files:

```yaml
record:
  type: asset_index
  version: 1

assets:
  tool.create_car:
    kind: tool
    name: create_car
    semantic: agent.tool
    current_version: 7c91d4d7b1af
    file: tool_create_car.yaml
```

```yaml
record:
  type: asset
  version: 1

asset:
  id: tool.create_car
  kind: tool
  name: create_car
  semantic: agent.tool
  current_version: 7c91d4d7b1af
  doc: Create a new car instance.
  params:
    make:
      type: Literal["audi", "bmw", "mercedes"]
      required: true
    model:
      type: str
      required: true
  returns:
    type: Car
    asset_id: type.Car

versions:
  - version: 15aa0dbceb02
    state:
      kind: tool
      name: create_car
      params:
        make:
          type: Literal["audi", "bmw", "mercedes"]
          required: true
    hashes:
      content: ...
    changes:
      fields: [initial]
  - version: 7c91d4d7b1af
    parent: 15aa0dbceb02
    state:
      kind: tool
      name: create_car
      params:
        make:
          type: Literal["audi", "bmw", "mercedes"]
          required: true
    hashes:
      content: ...
      source: ...
    source:
      path: ./vsh.py
    changes:
      fields:
        - params.year.type
      diff: |
        --- 15aa0dbceb02
        +++ 7c91d4d7b1af
        @@ ...
```

---

## Python API

Canonical page: https://vcoderun.github.io/autobench/python-api/

# Python API

Autobench exposes the same runtime through a fluent builder, typed specification models, and
lower-level extension seams. Use the highest-level surface that can express the benchmark clearly.

## Surface Selection

| Surface | Use it when |
| --- | --- |
| `Benchmark` | Application code composes a benchmark dynamically |
| `BenchmarkSpec` | You need the complete typed configuration surface |
| YAML + `load_benchmark_spec` | Humans or agents author portable benchmark definitions |
| Runtime/evaluation functions | You are building an adapter, service, or custom runner |

All three authoring paths execute through `run_benchmark_spec()`.

## Fluent Builder

```python
from autobench import (
    Benchmark,
    Case,
    Direction,
    ExactScorer,
    FactorValue,
    ObservationRole,
    PassFailScorer,
    Semantic,
    Variant,
)

benchmark = (
    Benchmark("builder-demo")
    .description("Compare current and candidate behavior.")
    .dataset(
        [
            Case(
                id="refund",
                input={"message": "Refund order 42"},
                expected={"route": "billing"},
            )
        ],
        dataset_id="routing-regressions",
        version="v3",
    )
    .variants(
        [
            Variant(
                id="current",
                factors=[FactorValue(name="routing_profile", value="v3")],
            ),
            {
                "id": "candidate",
                "factors": {
                    "routing_profile": {
                        "value": "v4",
                        "optimize": True,
                    }
                },
            },
        ]
    )
    .task("my_app.benchmarks:run_case")
    .scoring(
        [
            ExactScorer(
                name="route",
                actual="output.route",
                expected="case.expected.route",
                semantic_type=Semantic.QUALITY_CORRECTNESS,
                direction=Direction.MAXIMIZE,
                role=ObservationRole.OBJECTIVE,
            ),
            PassFailScorer(
                name="success",
                path="output.success",
                semantic_type=Semantic.RESULT_SUCCESS,
                role=ObservationRole.CONSTRAINT,
            ),
        ]
    )
)

result = benchmark.run(experiment_id="routing-candidate-42", concurrency_limit=4)
```

### Builder Methods

| Method | Configures |
| --- | --- |
| `description(value)` | Benchmark description |
| `capture(policy)` | ABP and asset capture policy |
| `dataset(...)` | Inline cases or a typed dataset source |
| `variants(items)` | Typed variants or normalized dictionaries |
| `task(target, kind="python")` | Task target |
| `scoring(items)` | Built-in or Python scorer specs |
| `derive(items)` | Per-run derivers |
| `instrument(*items)` | Typed built-ins or runtime custom instrumentors |
| `instrument_all(...)` | Compatible built-in discovery |
| `to_spec()` | Canonical `BenchmarkSpec` |
| `run(...)` / `run_async(...)` | Sync or async execution |

Post-derivation, policies, report views, and custom semantic registries currently live on the full
`BenchmarkSpec`. Extend the compiled spec rather than inventing builder-only state:

```python
import asyncio

from autobench import PolicySpec, run_benchmark_spec

spec = benchmark.to_spec().model_copy(
    update={
        "policies": [
            PolicySpec(
                name="quality-floor",
                metric=Semantic.QUALITY_CORRECTNESS,
                must_greater_equal=0.9,
            )
        ]
    }
)
result = asyncio.run(run_benchmark_spec(spec, concurrency_limit=4))
```

## Task Contract

```python
from autobench import Case, RunContext


def run_case(ctx: RunContext, case: Case) -> Result:
    ...
```

`ctx` is always first and `case` is always second. A task may be sync or async and may return any
serializable result. A Pydantic model is useful because scorers can resolve output fields reliably.

The runtime resolves `module:function` targets relative to the benchmark file before falling back to
normal Python import paths.

## RunContext

`RunContext` owns evidence for one case x variant run:

| Method | Purpose |
| --- | --- |
| `factor(name)` | Read a configured factor value |
| `span(...)` | Time and nest an operation |
| `metric(...)` / `metrics(...)` | Record numeric, boolean, or structured metrics |
| `factor_observation(...)` | Record a factor discovered at runtime |
| `event(...)` | Record a discrete event |
| `diagnostic(...)` | Record non-objective evidence |
| `outcome(...)` | Record semantic success |
| `check(...)` | Record a correctness constraint and reason |
| `record_measurement(...)` | Record summaries plus optional raw samples |
| `artifact(...)` | Attach a payload |
| `error(...)` | Preserve a structured error |
| `attach_tracked_asset(...)` | Bind an explicit tracked asset version |

Evidence emitted before an exception remains in the failed run.

## Load And Run YAML

```python
import asyncio
from pathlib import Path

from autobench import load_benchmark_spec, run_benchmark_path, run_benchmark_spec

path = Path("benchmarks/routing.yaml")
spec = load_benchmark_spec(path)

sync_result = run_benchmark_path(
    path,
    experiment_id="routing-42",
    concurrency_limit=4,
)

async_result = asyncio.run(
    run_benchmark_spec(
        spec,
        experiment_id="routing-43",
        concurrency_limit=4,
    )
)
```

Loading resolves dataset, pricing, task, and Python scorer references relative to the YAML file.

## Record And Replay

```python
from pathlib import Path

from autobench import (
    collect_benchmark_source_files,
    record_experiment,
    replay_experiment,
)

record_dir = Path("runs/routing-42")
record = record_experiment(
    async_result,
    record_dir,
    source_files=list(collect_benchmark_source_files(path)),
    path_root=Path.cwd(),
)
replayed = replay_experiment(record_dir)
```

`record_experiment()` refuses to overwrite an existing experiment. Use a new directory for every
execution. Referenced tracked-asset histories and large trace artifacts are persisted automatically.

Load one exact record when building an audit or optimizer adapter:

```python
from autobench import load_experiment_record, load_run_record

experiment = load_experiment_record(record_dir)
run = load_run_record(record_dir / experiment.run_paths[0], root_dir=record_dir)
```

## Reports And Exports

```python
from pathlib import Path

from autobench import (
    build_report,
    compare_variants,
    export_markdown_report,
    export_runs_csv,
    export_summary_yaml,
)

report = build_report(replayed)
comparison = compare_variants(
    replayed,
    baseline="current",
    candidate="candidate",
)

export_summary_yaml(replayed, Path("analysis/summary.yaml"))
export_runs_csv(replayed, Path("analysis/runs.csv"))
export_markdown_report(replayed, Path("analysis/report.md"))
```

`build_leaderboard`, `build_case_matrix`, `build_metric_distribution`, and
`build_run_metric_rows` expose individual projections.

## Native Instrumentation

```python
from autobench import Benchmark

benchmark = Benchmark("agent").instrument_all(
    exclude={"httpx"},
    strict=False,
    assets={
        "representations": ["definition", "effective"],
        "include": ["prompt", "tool", "output_schema"],
    },
)
```

Unavailable integrations become diagnostic observations. `strict=True` instead requires every
selected integration to be compatible.

Use typed settings for explicit control:

```python
from autobench import HTTPXCaptureSettings, HTTPXInstrumentation, OpenAIInstrumentation

benchmark.instrument(
    OpenAIInstrumentation(),
    HTTPXInstrumentation(
        capture=HTTPXCaptureSettings(
            path="hash",
            response_headers=("x-request-id",),
        )
    ),
)
```

Explicit settings override automatic discovery, including `enabled=False`. A custom runtime
`Instrumentor` can also be passed to `instrument()` and remains Python-only.

## Explicit Tracking

```python
from autobench import track

SYSTEM_PROMPT = track.prompt(
    name="support_system",
    source="prompts/support.md",
)


@track.tool
def lookup_order(order_id: str) -> dict[str, str]:
    """Return the current order status."""
    ...
```

`track.prompt`, `track.tool`, `track.type`, `track.dataclass`, and `track.asset` register exact
versions. `track.write_assets(path)` writes DSL-shaped history files. Native discovery can attach
unadorned SDK-visible components to runs.

## Production And Generated Cases

```python
from autobench import (
    SamplingPolicy,
    generated_batch_from_cases,
    samples_to_cases,
)

review_cases = samples_to_cases(production_samples, policy=SamplingPolicy(max_samples=50))
generated = generated_batch_from_cases(
    synthetic_cases,
    generator_asset_version="prompt.generator@v4",
    model_provider="openrouter",
    model_name="openai/gpt-5.6-luna",
)
```

These helpers normalize provenance; they do not own production querying or case generation.

## Extension Rules

- Put subject execution in a task.
- Put domain judgment in a Python scorer.
- Use a deriver for same-run computations and a post-deriver for matched runs.
- Use a policy for acceptance boundaries.
- Use an instrumentor for a stable SDK boundary.
- Use source maps and extractors for external field normalization.
- Use metric packs for reusable domain defaults.
- Never mutate recorded evidence; create a derived record or a new experiment.

See [API Reference](api-reference.md) for generated signatures and model fields.

---

## CLI

Canonical page: https://vcoderun.github.io/autobench/cli/

# CLI

The CLI is human-first. Commands render Rich panels and tables; YAML, CSV, and Markdown are explicit
file exports instead of raw terminal output.

## Command Map

| Command | Executes subject? | Input | Purpose |
| --- | --- | --- | --- |
| `validate` | No | benchmark YAML | Resolve and validate the planned matrix |
| `run` | Yes | benchmark YAML | Execute, record, and render an experiment |
| `replay` | No | record directory | Reconstruct recorded results |
| `report` | No | record directory | Render configured analysis views |
| `compare` | No | record directory | Compare two variants without causal claims |
| `export` | No | record directory | Write a YAML, CSV, or Markdown projection |
| `instrumentation doctor` | No | environment | Inspect integration compatibility |
| `instrumentation trace` | No | record directory | Summarize ABP traces and diagnostics |

## Validate

```bash
autobench validate benchmarks/routing.yaml
```

Validation parses the DSL, loads file or glob datasets, resolves pricing and task sources relative
to the spec, checks duplicate IDs and runnable requirements, and displays case, variant, and run
counts. It does not invoke the task.

## Run

```bash
autobench run benchmarks/routing.yaml \
  --concurrency 4 \
  --record runs/routing-42
```

Options:

| Option | Meaning |
| --- | --- |
| `--concurrency INTEGER` | Maximum active runs; default and minimum are `1` |
| `--record DIRECTORY` | Write immutable evidence to this new directory |
| `--no-record` | Execute and display without persistence |

Without either recording flag, Autobench creates
`.autobench/<spec-stem>/<experiment-id>/`. `--record` and `--no-record` are mutually exclusive.

The CLI records the benchmark file and resolved referenced-source hashes so evidence can explain
what was executed.

## Replay

```bash
autobench replay runs/routing-42
```

Replay imports neither the task nor optional provider SDKs. It reconstructs normal result models
from `experiment.yaml`, per-run records, and referenced artifacts.

## Report

```bash
autobench report runs/routing-42
```

The report includes experiment status, variant configuration, leaderboard values, run metrics, case
matrix, configured comparisons, and distributions. Missing metrics remain visible rather than being
silently converted to zero.

## Compare

```bash
autobench compare runs/routing-42 \
  --baseline current \
  --candidate candidate
```

Both IDs must exist. The view shows changed factors, aggregate metric values and deltas, paired run
count, and whether several factors changed. `confounded=true` is a warning against causal
attribution, not a failed comparison.

## Export

```bash
autobench export runs/routing-42 \
  --format yaml \
  --path analysis/routing-summary.yaml

autobench export runs/routing-42 \
  --format csv \
  --path analysis/routing-runs.csv

autobench export runs/routing-42 \
  --format markdown \
  --path analysis/routing-report.md
```

`--format` is required and accepts `yaml`, `csv`, or `markdown`. `--path` is also required. YAML
exports include a versioned schema header; CSV is a run-level projection; Markdown is a portable
report. The complete evidence remains the record directory.

## Instrumentation Doctor

```bash
autobench instrumentation doctor
```

The compatibility table shows distribution and version state, supported range, mechanism,
abstraction layer, sync/async/streaming support, asset discovery, capture defaults, optional extra,
and diagnostics for every built-in integration.

Use it before enabling `strict=True` or when an SDK upgrade stops producing evidence.

## Trace Inspection

```bash
autobench instrumentation trace runs/routing-42
```

This is replay-only. It summarizes span roots, kinds, instrumentors, partial state, and protocol
diagnostics without loading the original SDK.

## Exit And Failure Behavior

- Invalid YAML, unresolved sources, schema errors, recording collisions, and missing records exit
  nonzero.
- YAML failures include the file and source location when available.
- Task failures are isolated to their run; already collected evidence is preserved.
- An experiment can finish with passed, failed, errored, and skipped runs. Inspect status tables and
  policies rather than assuming process completion means every run passed.
- Replay and reporting never fall back to live execution.

## CI Workflow

```bash
set -e
autobench validate benchmarks/release.yaml
autobench run benchmarks/release.yaml \
  --concurrency 4 \
  --record artifacts/autobench
autobench report artifacts/autobench
autobench export artifacts/autobench \
  --format csv \
  --path artifacts/autobench-runs.csv
```

Upload the entire `artifacts/autobench` directory so replay, traces, asset histories, and source
lineage remain available.

---

## Capability Map

Canonical page: https://vcoderun.github.io/autobench/capabilities/

# Capability Map

This page is the inventory of what Autobench owns today. Every public feature belongs to one of
the layers below; application-specific behavior stays in tasks, scorers, adapters, and metric
packs.

## End-To-End Lifecycle

```text
BenchmarkSpec
  -> Dataset x Variants
  -> BenchmarkPlan
  -> Task(ctx, case)
  -> Observations + Spans + Artifacts + Errors
  -> Scores + Derived Metrics + Policies
  -> Cross-run Derivation
  -> Immutable RunRecord / ExperimentRecord
  -> Replay -> Report -> Compare -> Export -> Optimization Feedback
```

The same lifecycle is available through the YAML DSL, Python models, the `Benchmark` builder, and
the CLI. YAML is the portable authoring format; Python remains the extension surface for
application execution and custom evaluation logic.

## Definition And Data

| Capability | What it provides |
| --- | --- |
| `BenchmarkSpec` | Validated benchmark metadata, dataset, task, variants, scoring, derivation, policies, and reports |
| Dataset | Inline cases, file-backed datasets, glob-backed case files, defaults, tags, metadata, attachments, and versions |
| Cases | Arbitrary input and expected payloads with stable IDs and artifact references |
| Variants | Named factor combinations with labels, semantic types, and `optimize` hints |
| Generated cases | Production-sample conversion, provenance, review status, reasons, and generation batches |
| YAML schemas | Versioned JSON schemas and `yaml-language-server` headers for completion and validation |
| Source discovery | Hash collection for specs, datasets, pricing files, task modules, and scorer modules |

See [Datasets And Variants](datasets-and-variants.md) and [YAML Spec](yaml-spec.md).

## Planning And Execution

| Capability | What it provides |
| --- | --- |
| Matrix planning | Deterministic case x variant expansion and stable run IDs |
| Task runtime | Sync and async Python callables with `ctx` first and `case` second |
| Concurrency | Bounded async execution while preserving deterministic result ordering |
| Failure isolation | One task, scorer, derivation, or policy failure does not erase other runs |
| Progress events | Typed lifecycle events for runners and future UI integrations |
| Optional Pydantic Evals bridge | Internal conversion to Pydantic Evals-compatible case and dataset payloads |

See [Tasks And Runtime](tasks-and-runtime.md).

## Evidence Collection

| Capability | What it provides |
| --- | --- |
| Observations | Metrics, factors, events, diagnostics, artifacts, roles, units, directions, tags, and sources |
| Semantic registry | Canonical semantic types, aliases, parent relationships, and custom extensions |
| Projection | Source precedence and duplicate detection for one canonical metric view |
| Context spans | Nested agent, LLM, tool, retriever, parser, workflow, and custom spans |
| Automatic duration | Span timing and optional duration metrics owned by the runtime |
| Artifacts | Structured values and files materialized outside the main record payload |
| Errors | Structured task, scorer, trace, and policy errors with traceback capture |
| Measurement | Warmup, repetitions, time budgets, samples, median, p95, standard deviation, and noise |

See [Observations And Semantics](observations-and-semantics.md) and
[Instrumentation And Traces](instrumentation-and-traces.md).

Native Pydantic AI, OpenAI, OpenAI Agents, and HTTPX integrations can be selected through typed
Python settings or the YAML `instrumentation` section. They emit ABP directly, compose across
framework/client/transport layers, preserve streaming lifecycle, and remain optional for replay.
See [Native Instrumentation](native-instrumentation.md).

Semantic instrumentors automatically discover SDK-visible prompt, tool, output-schema, capability,
agent, guardrail, handoff, policy, and toolset versions. Definition/effective relationships,
capability scopes, aliases, privacy-controlled content, and span-local `AssetUse` evidence survive
recording and replay. HTTPX remains transport evidence and performs no semantic asset inference.
See [Automatic Asset Discovery](automatic-asset-discovery.md).

## Scoring And Constraints

| Scorer | Purpose |
| --- | --- |
| `output` | Project an output path into a semantic score |
| `pass_fail` | Turn a boolean output path into a pass/fail score |
| `exact` | Compare actual and expected paths |
| `schema` | Validate output against a schema/model |
| `python` | Run a sync or async custom scorer using `ScoringCall` |
| `expected_action` | Evaluate action/tool selection, arguments, or sequence from spans |

Scores declare semantic type, unit, direction, role, and optional failure behavior. Policies add
typed requirements including equality, membership, numeric bounds, and inclusive ranges.

See [Scoring And Derivation](scoring-and-derivation.md) and
[Agentic Evaluation](agentic-evaluation.md).

## Derivation And Cost

| Capability | What it provides |
| --- | --- |
| Token cost | Derive `money.cost` from input/output tokens and normalized model/provider factors |
| Pricing DSL | Static YAML pricing, aliases, provider maps, cache prices, token tiers, and model normalization |
| Price sources | Optional llm-prices and genai-prices importers that normalize external data into `PricingTable` |
| Paired baseline | Per-case or factor-matched speedup, delta, percent change, diagnostics, and verdicts |
| Comparison classifier | Improved, regressed, unchanged, or inconclusive outcomes with relative noise thresholds |

External price sources are convenience importers, not runtime dependencies or Autobench's source
of truth. A local pricing YAML remains fully supported.

## Agentic Evidence

Autobench records agent behavior without requiring OpenTelemetry:

- typed trace envelopes and nested span records
- expected tool/action selection, argument, and sequence checks
- span selectors by kind, name, tag, path, or semantic type
- Pydantic AI usage normalization
- metric packs for agentic, structured-output, LLM-usage, and performance defaults
- compact feedback records for optimization systems

See [Agentic Evaluation](agentic-evaluation.md).

## Asset Lineage

The tracking registry understands:

- text prompts from inline text or files
- arbitrary assets and configuration values
- callable tools, signatures, parameters, docs, and return types
- Pydantic models, standard dataclasses, and typed classes
- field names, annotations, descriptions, aliases, defaults, requirements, constraints, and examples
- source hashes, structured-schema hashes, versions, parent versions, and diffs
- persistent human-readable YAML asset histories
- automatic SDK-boundary discovery without tracking decorators
- source/effective representation links, capability scopes, provenance, and cross-layer aliases
- automatic experiment persistence and replayable span-local asset uses

Decorators preserve the original callable or class type so tracking does not degrade static
typing. See [Asset Tracking](asset-tracking.md) and
[Automatic Asset Discovery](automatic-asset-discovery.md).

## Records, Replay, And Analysis

| Capability | What it provides |
| --- | --- |
| `RunRecord` | Immutable case x variant evidence including output, scores, observations, spans, factors, assets, artifacts, and errors |
| `ExperimentRecord` | Plan, environment, semantic registry, report config, source hashes, and run paths |
| Replay | Load records without importing task or scorer modules |
| Rich terminal reports | Status, variant configuration, leaderboard, run metrics, case matrix, comparisons, and distributions |
| Exports | Human-readable YAML summary, CSV run projection, and Markdown report |
| Optimization feedback | Failure category, score, reasons, factors, asset versions, and selected evidence |

See [Recording And Reporting](recording-and-reporting.md).

## Ownership Boundaries

Autobench deliberately does not own:

- application or model execution
- hosted tracing or observability storage
- model-specific pricing as an always-current service
- causal claims from confounded comparisons
- optimizer search strategies or candidate promotion
- large catalogs of domain-specific LLM judges

Tasks and adapters own application execution. Optional integrations may import traces, pricing, or
evaluator results, but the core contract remains semantic, generic, and replayable.

---

## API Reference

Canonical page: https://vcoderun.github.io/autobench/api-reference/

# API Reference

This reference is generated from the installed public `autobench` package. The root package is the
supported import surface; subpackages organize implementation and extension areas.

## Public Package

::: autobench
    options:
      members: true
      members_order: source
      show_root_heading: true
      show_source: true
      show_signature_annotations: true
      separate_signature: true
      heading_level: 3

## Public Areas

| Area | Representative symbols |
| --- | --- |
| Definition | `Benchmark`, `BenchmarkSpec`, `TaskSpec`, `load_benchmark_spec` |
| Data | `Case`, `DatasetSpec`, `Variant`, `FactorValue`, production/generated helpers |
| Runtime | `RunContext`, `Span`, `ExperimentResult`, `run_benchmark_spec` |
| Semantics | `Observation`, `Semantic`, `SemanticRegistry`, queries and projection |
| Evaluation | scorers, derivers, policies, expected actions, measurement, feedback |
| Protocol | ABP signals, traces, capture, emitter, collector and context |
| Instrumentation | settings, manager, instrumentors, method instrumentation, diagnostics |
| Tracking | `track`, asset models, discovery candidates, registry and history views |
| Records | `RunRecord`, `ExperimentRecord`, recording and replay helpers |
| Reports | report models, builders, Rich renderers and exporters |

Prefer root imports for application code:

```python
from autobench import Benchmark, Case, RunContext, Semantic
```

Import a submodule when implementing an extension against that subsystem, such as a custom native
instrumentor or source-map adapter.

---

## Troubleshooting

Canonical page: https://vcoderun.github.io/autobench/troubleshooting/

# Troubleshooting

Start with the narrowest command that can identify the failing layer.

## Task Module Cannot Be Imported

```text
Could not import task module 'benchmarks.tasks'
```

Task targets must use `module:function`, not a file path:

```yaml
run:
  python: benchmark_task:run
```

Autobench first uses normal Python imports, then searches relative to the benchmark spec. Common
fixes:

- place `benchmark_task.py` next to `autobench.yaml` and use `benchmark_task:run`;
- for a package, ensure package directories have the expected Python import structure;
- do not include `.py` in the target;
- run `autobench validate path/to/autobench.yaml` from any directory to test resolution.

The function must accept `(ctx, case)` in that order.

## YAML Validates In One Editor But Not In Autobench

The schema directive improves editor completion; Autobench's installed Pydantic models remain the
runtime authority. Match the schema version to the installed package:

```bash
python -c "import autobench; print(autobench.__version__)"
```

```yaml
# yaml-language-server: $schema=./schemas/0.2.0/benchmark_schema.json
```

Run `autobench validate` and use its file/line diagnostics. Unknown scorer, policy,
instrumentation, and capture fields are rejected intentionally.

## Dataset File Is Not Found

`file://` references and glob patterns resolve relative to the benchmark YAML, not the shell's
current directory:

```yaml
dataset:
  source: file://datasets/cases.yaml
```

For a glob:

```yaml
dataset:
  source: file://datasets/cases/*.yaml
```

An unmatched glob is an error. Dataset files can contain a dataset DSL document, a case list, or a
single case mapping.

## Record Directory Already Exists

Autobench records are immutable. `record_experiment()` and `autobench run --record` refuse to
overwrite an existing experiment:

```text
Experiment record already exists
```

Use a new directory or remove/archive the old directory explicitly outside Autobench. Do not merge
unrelated experiments by copying run files together.

## A Metric Is Missing From Reports

Reports query semantic types, not only local names. Check:

1. the task/scorer/deriver emitted the observation;
2. `semantic_type` matches the report metric;
3. the value is numeric or boolean for the selected aggregate;
4. the selected span/query is not filtering it out;
5. source precedence did not intentionally select a score or derived observation instead.

Inspect the per-run YAML or use Python:

```python
from autobench import ObservationQuery

query = ObservationQuery(observations=run.task_result.observations)
matches = query.exact("money.cost")
```

Missing cost is not converted to zero. Ensure token, model, provider, and pricing inputs are all
available to the token-cost deriver.

## Paired Baseline Does Not Produce A Value

The baseline and candidate must match on the configured key, normally `case_id`, and both must have
the source metric. Check:

- `baseline_variant` exactly matches a variant ID;
- both runs emit the same semantic metric;
- the metric unit is compatible;
- `match_on` identifies a unique counterpart;
- the configured missing policy is appropriate.

Use a case matrix for the source metric before debugging the formula.

## `instrument_all()` Records Skipped Integrations

This is normal when optional SDKs are not installed. Automatic discovery records
`instrumentation.skipped` diagnostics and continues by default.

```bash
autobench instrumentation doctor
```

Install the relevant extra, remove the integration from `exclude`, or use `strict=True` when absence
must fail the benchmark.

Explicit `enabled: false` wins over automatic discovery. A custom runtime instrumentor with the
same ID also prevents a duplicate built-in installation.

## No Automatic Assets Appear

Automatic discovery only observes values that cross a supported instrumented SDK boundary while a
benchmark run is active. Check:

- the corresponding instrumentor is compatible and installed;
- asset discovery is enabled;
- `include` contains the expected family;
- `representations` includes `definition` or `effective` as needed;
- the SDK call occurs inside the task;
- capture policy does not reduce the asset below the expected content level.

Run the offline `examples/automatic_assets/` programs to separate environment issues from
application behavior.

## Duplicate Or Conflicting Instrumentation

Autobench prevents unsafe double patching. Do not install two instrumentors with the same ID or
instrument the same owner/method with incompatible specs. Prefer one of:

- automatic discovery only;
- explicit typed settings only;
- a custom runtime instrumentor that owns the same ID.

`InstrumentationConflictError` and patch diagnostics identify the owner and method involved.

## Trace Is Partial

A partial trace can be valid evidence. It may result from cancellation, an interrupted stream, a
task exception, or unmatched start/end signals. Inspect:

```bash
autobench instrumentation trace runs/example
```

ABP materialization keeps completed spans and diagnostics instead of dropping the trace. Accounting
extractors avoid double counting aggregate and leaf usage even when evidence is incomplete.

## Captured Content Is Redacted Or Hashed

Capture is privacy-first. A value's retained representation is controlled by the benchmark
`CapturePolicy`, semantic defaults, path rules, and SDK-specific HTTP settings.

Use `CapturePolicy.full()` only for controlled local evidence. Prefer targeted semantic overrides:

```python
from autobench import CaptureLevel, CapturePolicy

policy = CapturePolicy.hashed(
    semantic_overrides={"output_schema": CaptureLevel.FULL},
)
```

Secret names, denied paths, truncation limits, and binary rules still apply.

## Replay Needs An Optional SDK

It should not. `replay`, `report`, `compare`, `export`, and `instrumentation trace` are designed to
load records without benchmark or provider imports. If replay fails, verify that:

- `experiment.yaml` and every path in `runs.paths` exist;
- trace artifact paths remain inside the experiment directory;
- referenced artifacts were copied with the records;
- the record version is supported.

Do not solve a missing artifact by re-executing the benchmark implicitly.

## Python Type Errors In Tasks

Autobench keeps case input and factor values generic because applications define their schemas.
Validate at the task boundary:

```python
from pydantic import BaseModel, TypeAdapter

request = Request.model_validate(case.input)
mode = TypeAdapter(Mode).validate_python(ctx.factor("mode"))
```

This gives application-specific errors without weakening Autobench's public types.

## Get A Reproducible Diagnostic Bundle

For a bug report, include:

```bash
autobench --help
autobench instrumentation doctor
autobench validate path/to/autobench.yaml
```

Also include the package version, Python version, failing record directory when it contains no
sensitive data, and the smallest benchmark/task that reproduces the problem. Review capture policy
before sharing records.

---

## Development

Canonical page: https://vcoderun.github.io/autobench/development/

# Development

Autobench uses `uv` for dependency management and exposes stable repository operations through
the Makefile.

## Environment

```bash
uv sync --extra dev
```

The committed lock file is the reproducible dependency contract used by CI.

## Quality Gates

```bash
make format
make prod
make pre-commit
```

`make prod` runs the test suite, enforces `100%` line and branch coverage, checks formatting and
typing, builds the documentation, validates Python 3.11 through 3.13, and executes the offline
examples end to end.

## Documentation

The site uses Zensical's modern theme while retaining `mkdocs.yml` as the supported migration
configuration format.

```bash
make docs
make docs-serve
```

Pushes to `main` build the site in strict mode. The workflow stores generated files in the
`gh-pages` branch and deploys the same artifact through GitHub Pages Actions.

## Release Artifacts

```bash
make build
```

The build produces a wheel and source distribution under `dist/`. Generated documentation,
benchmark runs, internal planning files, references, and agent instructions are excluded from the
published package.

---

## 0.2.0

Canonical page: https://vcoderun.github.io/autobench/release-notes/0.2.0/

# 0.2.0

Autobench `0.2.0` introduces the Autobench Instrumentation Protocol (ABP) and native evidence
collection for supported AI and HTTP SDKs.

## Included

- immutable ABP signals, traces, scopes, links, references, and protocol diagnostics
- task-local trace context with correct concurrent parentage and partial-run preservation
- privacy-first capture policy with redaction, truncation, hashing, and artifact references
- versioned semantic source maps and accounting-safe trace extraction
- native Pydantic AI, OpenAI Python, OpenAI Agents, and HTTPX instrumentors
- automatic prompt, tool, output-schema, capability, agent, guardrail, handoff, policy, and toolset
  discovery at supported semantic SDK boundaries
- definition/effective asset links, capability scopes, cross-layer aliases, and conservative
  duplicate correlation
- automatic experiment asset persistence with atomic worker-safe history merges and replayed
  `AssetUse` lineage
- `InstrumentAssetSpec` for custom SDK asset extraction without tracking decorators
- benchmark-level typed/YAML capture policy for privacy-controlled evidence and asset persistence
- sync, async, iterator, context-manager, and streaming lifecycle preservation
- typed fluent and YAML instrumentation configuration with versioned schema completion
- `autobench instrumentation doctor` compatibility diagnostics
- `autobench instrumentation trace` replay-only trace summaries
- real offline instrumentation, layering, and replay/extraction examples
- Python 3.11, 3.12, 3.13, and 3.14 quality matrix
- built-wheel/no-extras and target-library compatibility gates

## Compatibility

Existing `0.1.0` benchmark specs and RunRecords remain loadable. ABP evidence is additive: manual
spans and method instrumentation now materialize through the same protocol used by native
instrumentors. Replaying ABP records does not import optional provider SDKs.

## Intentionally Deferred

- OTLP and vendor exporters
- distributed context propagation
- import-hook auto-instrumentation
- execution cassette replay
- visualization

ABP protocol version `1` is the initial public protocol. Autobench `0.2.x` will preserve its
serialized meaning; additive fields remain forward-compatible through extension maps. A breaking
wire-format change requires a new ABP protocol version.

---

## 0.1.0

Canonical page: https://vcoderun.github.io/autobench/release-notes/0.1.0/

# 0.1.0

Autobench `0.1.0` is the first release-shaped core.

## Included

- YAML-first benchmark specs
- deterministic task runtime
- semantic observations and projection
- scoring, derivation, post-derivation, and policies
- immutable YAML recording and replay
- Markdown, YAML, and CSV reporting
- offline minimal, basic, mid, and advanced examples
- real optional CodeMode dogfood integration
- portable CLI source provenance
- Python 3.11, 3.12, and 3.13 quality matrix

## Intentionally Not Included

- autoptimize orchestration
- GEPA integration
- OpenTelemetry bridge
- hosted dashboard features
- distributed execution
- full Pydantic Evals dataset/evaluator execution

`PydanticEvalsBridge` in this release is an optional payload and availability bridge. It does not
claim to execute Pydantic Evals datasets. The full internal evaluation runtime remains a later
integration milestone.
