Metadata-Version: 2.5
Name: kaisen-evalbridge
Version: 0.1.0
Summary: Connect agents to rigorous evaluations effortlessly
Author: EvalBridge Contributors
License-Expression: MIT
Keywords: agents,evaluation,llm,pydantic,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: bcrypt>=4.2
Requires-Dist: email-validator>=2.2
Requires-Dist: fastapi>=0.115
Requires-Dist: pydantic<3,>=2.10
Requires-Dist: pyjwt>=2.9
Requires-Dist: python-dotenv>=1.0
Requires-Dist: uvicorn[standard]>=0.32
Provides-Extra: dev
Requires-Dist: httpx>=0.28; extra == 'dev'
Requires-Dist: mypy>=1.14; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: ruff>=0.9; extra == 'dev'
Description-Content-Type: text/markdown

# EvalBridge

**Connect agents to rigorous evaluations effortlessly**

EvalBridge is a production-grade Python library for **contract-first evaluation** of LLM agents. Run contract-first eval scenarios against a user-supplied Python agent while capturing tool traces, validating structured outputs, scoring with typed rubrics, clustering failures by cause, and producing a release scorecard with actionable diffs. You define scenarios with Pydantic models, plug in any agent that implements a small turn protocol, run under fixed budgets and guardrails, and export JUnit/JSON/Markdown/HTML for CI and review.

- **PyPI distribution:** set via `PYPI_PACKAGE_NAME` (default build name in this repo: `kaisen-evalbridge`)
- **Import package:** `kaisen_evalbridge`
- **CLI:** `kaisen-evalbridge` (entry point from `kaisen_evalbridge.cli:main`)
- **Python:** 3.11+

## Features

### Scenario DSL (code-first)

Define eval cases in Python with typed inputs, expected structured outputs (Pydantic `BaseModel`), and attached rubrics.

**What it does:** Each `Scenario` carries an `id`, `name`, optional `description`, `inputs`, `expected`, plus lists of `DeterministicRubric` and `LLMRubric` instances. Output typing is inferred from the expected model for harness validation.

**How to use it:** Call `Scenario.define(...)` with your input/expected model instances and rubrics. Use the scenario as the first argument to `ContractHarness.run(scenario, agent)`.

```python
from pydantic import BaseModel
from kaisen_evalbridge import Scenario, DeterministicRubric

class TaskIn(BaseModel):
    task: str

class TaskOut(BaseModel):
    result: str

scenario = Scenario.define(
    id="task-1",
    name="Complete task",
    inputs=TaskIn(task="run"),
    expected=TaskOut(result="done"),
    deterministic_rubrics=[
        DeterministicRubric(
            id="match",
            description="Exact result",
            evaluate=lambda output, expected: output.result == expected.result,
        ),
    ],
)
```

### Contract execution harness

Run agents under comparable limits: max turns, retries, and tool guardrails.

**What it does:** `ContractHarness` loops agent turns until the agent marks `done=True`, validates each turn’s `structured_output` against the scenario output type, enforces `GuardrailConfig` (allowed/blocked tools, max tool calls per turn), scores rubrics on success, and returns a `HarnessResult` with status `passed`, `failed`, `guardrail`, `validation`, or `budget_exhausted`. Traces are recorded automatically.

**How to use it:** Implement the `Agent` protocol (`run_turn(inputs, turn_index, history) -> AgentTurnResult`). Construct `ContractHarness(budget=ExecutionBudget(...), guardrails=GuardrailConfig(...))` and call `run(scenario, agent)`.

```python
from kaisen_evalbridge import AgentTurnResult, ContractHarness, ExecutionBudget

class MyAgent:
    def run_turn(self, *, inputs, turn_index, history):
        return AgentTurnResult(
            structured_output=TaskOut(result="done"),
            done=True,
        )

result = ContractHarness(budget=ExecutionBudget(max_turns=5, max_retries=1)).run(
    scenario, MyAgent()
)
```

### Trace capture

Record prompts, messages, tool calls, structured outputs, and per-step validation.

**What it does:** The harness uses `TraceRecorder` internally. Each step stores optional `prompt`, `ChatMessage` list, `ToolCallRecord` list, serialized structured output, and `ValidationResult`. The finished `Trace` is on `HarnessResult.trace`. You can also use `TraceRecorder` directly for custom instrumentation.

**How to use it:** Inspect `result.trace.steps` after a harness run, or build traces manually with `TraceRecorder(scenario_id).record_step(...)` and `finish()`.

### Typed scoring pipeline

Aggregate schema compliance, rubric outcomes, and optional custom validators.

**What it does:** `ScoringPipeline.score(result)` / `score_many(results)` produce `ScenarioScoreReport` with `passed`, `schema_compliance`, `mean_rubric_score`, and `rubric_scores`. `DeterministicValidator` runs pure functions over `HarnessResult`. `rubric_scores_to_json` serializes rubric scores for artifacts.

**How to use it:** Pass harness results to `ScoringPipeline().score_many(results)` or rely on `build_release_scorecard`, which uses the pipeline internally.

### Rubrics (deterministic and LLM-graded)

Attach checkers that emit typed `RubricScore` JSON (`rubric_id`, `kind`, `passed`, `score`, `details`).

**What it does:** `DeterministicRubric` runs a Python callable (`evaluate(output, expected)`). `LLMRubric` delegates to your `LLMGrader.grade(...)` implementation. The harness runs all scenario rubrics after a valid structured output is produced.

**How to use it:** Add rubrics on `Scenario.define(..., deterministic_rubrics=[...], llm_rubrics=[...])`, or materialize them from golden manifests via `GoldenSetRegistry`.

### Failure clustering

Group failed runs into root-cause categories using rules and trace-derived features.

**What it does:** `extract_trace_features(result)` builds compact features (status, step counts, validation failures, tool usage, guardrail flag). `cluster_failures(results, rules)` returns ordered `FailureCluster` lists with counts and example scenario IDs. `default_cluster_rules()` covers guardrail, schema, budget, and rubric failures.

**How to use it:** Pass custom `ClusterRule` lists to `cluster_failures` or to `build_release_scorecard(..., cluster_rules=...)`.

### Release scorecard

Compact per-scenario and aggregate metrics plus top failure clusters.

**What it does:** `build_release_scorecard(results)` returns `ReleaseScorecard` with `per_scenario` entries (pass flag, schema compliance, rubric scores), `aggregate` metrics (pass rate, mean schema compliance, mean rubric score, rubric pass rate), and `top_failure_clusters`.

**How to use it:** After a batch harness run, `scorecard = build_release_scorecard(results)`; read `scorecard.aggregate.pass_rate` and `scorecard.top_failure_clusters` for release review.

### Regression gating

Configurable CI thresholds with structured violations.

**What it does:** `evaluate_regression_gate(results=..., thresholds=RegressionThresholds(...))` checks minimum pass rate, mean schema compliance, per-rubric minimum scores, and optionally blocks **new** high-severity clusters compared to a baseline set. Returns `RegressionGateResult` with `passed` and `violations` (`GateViolation` with codes such as `pass_rate`, `schema_compliance`, `rubric_score`, `new_high_severity_cluster`).

**How to use it:** In CI, fail the job when `not gate.passed`; attach `gate.violations` to logs. Include the gate in exports via `build_eval_artifacts(..., regression_gate=gate)`.

### Golden-set management as code

Version eval cases as JSON manifests plus a runtime registry.

**What it does:** `GoldenSetManifest` / `GoldenCaseSpec` / `RubricConfigSpec` describe cases (model names, input/expected dicts, rubric configs). `load_golden_manifest(path)` and `save_golden_manifest(manifest, path)` round-trip JSON. `GoldenSetRegistry` maps model names and checker/grader refs to Python types and callables. `materialize_scenario` / `materialize_golden_set` produce runnable `Scenario` objects.

**How to use it:** Commit manifests beside tests (see `tests/fixtures/golden_manifest.json`), register Pydantic models and checker functions, then `scenarios = materialize_golden_set(manifest, registry)` and run the harness on each.

### Near-miss explorer

Find similar past failures using embeddings over run summaries (local, optional store).

**What it does:** `summarize_run(result)` builds a text summary from scenario id, status, and trace features. `NearMissExplorer` with `LocalNearMissStore` indexes runs via `HashedEmbeddingProvider` (deterministic local vectors, no external API). `find_near_misses(result, top_k=...)` returns `NearMissMatch` rows with cosine similarity. Optional persistence: `LocalNearMissStore(directory=...)` loads/saves cases on disk.

**How to use it:** After failures, `explorer.index_result(past_result)` for historical runs; on a new failure, `explorer.find_near_misses(failed_result)` to surface comparable cases.

### Artifact export

JUnit for CI, JSON for machines, Markdown/HTML for humans.

**What it does:** `build_eval_artifacts(results, ...)` builds an `EvalArtifactDocument` (scorecard, per-scenario summaries, optional regression gate) plus `json_text`, `junit_xml`, `markdown`, and `html`. `write_eval_artifacts(artifacts, directory)` writes `{prefix}.json`, `.xml`, `.md`, and `.html` files.

**How to use it:** `artifacts = build_eval_artifacts(results, regression_gate=gate)` then `write_eval_artifacts(artifacts, "./reports")` or publish `artifacts.junit_xml` in your CI test reporter.

### CLI

Minimal command-line entry for version and help.

**What it does:** Prints distribution name and version, or usage help.

**How to use it:** `kaisen-evalbridge --version` · `kaisen-evalbridge` (help)

### Package metadata

Resolve distribution name from the environment at runtime and build time.

**What it does:** `get_pypi_package_name()` / `get_import_package_name()` read `PYPI_PACKAGE_NAME`. `package_info()` returns `PackageInfo` (distribution name, import name, version).

**How to use it:** `from kaisen_evalbridge import package_info` for tooling and the CLI prog name.

## Pages

EvalBridge is delivered as a **Python library**, not a hosted web application. There are **no frontend routes** or authenticated UI in this repository.

| Location | What you see |
|----------|----------------|
| *(no web app)* | All product workflows are code-first: define scenarios, run `ContractHarness`, export artifacts from your scripts or CI. |
| `frontend/public/logo.svg` | Brand logo asset (primary `#0056a6`, accent `#ff8c00`). |
| `frontend/public/favicon.svg` | Favicon asset for future or external docs sites. |

## API

There is **no HTTP backend** in this repo—no REST paths under `/api`. Integration is through the **public Python API** exported from `kaisen_evalbridge` (and the CLI below).

### Configuration and metadata

| Entry point | Purpose |
|-------------|---------|
| `get_pypi_package_name(required=True)` | Read PyPI/wheel distribution name from `PYPI_PACKAGE_NAME`. |
| `get_import_package_name(required=True)` | Derive Python import name (dashes → underscores). |
| `package_info()` | Return `PackageInfo` (distribution name, import name, version). |
| `__version__` | Package version string. |

### Scenarios and rubrics

| Entry point | Purpose |
|-------------|---------|
| `Scenario.define(...)` | Create a typed scenario with inputs, expected output, and rubrics. |
| `DeterministicRubric` / `.run(output=, expected=)` | Deterministic rubric definition and execution. |
| `LLMRubric` / `.run(output=, expected=)` | LLM rubric using your `LLMGrader` implementation. |
| `RubricScore` | Typed rubric result model (JSON-serializable). |

### Agent and harness

| Entry point | Purpose |
|-------------|---------|
| `Agent` (protocol) | Contract for user agents (`run_turn`). |
| `AgentTurnResult` | Per-turn prompt, messages, tool calls, structured output, `done`. |
| `ContractHarness.run(scenario, agent)` | Execute scenario with trace capture and rubric scoring. |
| `ExecutionBudget` | `max_turns`, `max_retries`. |
| `GuardrailConfig` | Allowed/blocked tools, max tool calls per turn. |
| `HarnessResult` | Run outcome, status, trace, rubric scores, errors. |

### Trace

| Entry point | Purpose |
|-------------|---------|
| `TraceRecorder` | Record steps and finish a `Trace`. |
| `Trace`, `TraceStep` | Scenario-level trace container and step records. |
| `ChatMessage`, `MessageRole` | Message model for traces and turns. |
| `ToolCallRecord` | Tool name and arguments on a step. |
| `ValidationResult` | Per-step schema validation pass/fail and errors. |

### Scoring

| Entry point | Purpose |
|-------------|---------|
| `ScoringPipeline.score(result)` | Score one harness result. |
| `ScoringPipeline.score_many(results)` | Score a batch. |
| `DeterministicValidator` / `.run(result)` | Custom deterministic validator over a run. |
| `rubric_scores_to_json(scores)` | Serialize rubric scores to JSON-friendly dicts. |

### Clustering and scorecard

| Entry point | Purpose |
|-------------|---------|
| `extract_trace_features(result)` | Features for clustering rules. |
| `cluster_failures(results, rules)` | Group failures into `FailureCluster` list. |
| `ClusterRule`, `FailureCluster`, `TraceFeatures` | Rule and result models. |
| `default_cluster_rules()` | Built-in guardrail/schema/budget/rubric rules. |
| `build_release_scorecard(results, ...)` | Build `ReleaseScorecard` with aggregates and top clusters. |

### Regression gating

| Entry point | Purpose |
|-------------|---------|
| `RegressionThresholds` | Min pass rate, schema compliance, per-rubric mins, new high-severity cluster blocking. |
| `evaluate_regression_gate(...)` | Evaluate thresholds; return `RegressionGateResult`. |
| `GateViolation`, `RegressionGateResult` | Violation detail and overall pass/fail. |

### Golden set

| Entry point | Purpose |
|-------------|---------|
| `load_golden_manifest(path)` | Load JSON manifest. |
| `save_golden_manifest(manifest, path)` | Write JSON manifest. |
| `GoldenSetRegistry` | Register Pydantic models, checkers, graders by name. |
| `materialize_scenario(spec, registry)` | One `Scenario` from a case spec. |
| `materialize_golden_set(manifest, registry)` | All scenarios from a manifest. |
| `scenario_to_golden_spec(scenario, ...)` | Export a scenario to a serializable spec. |

### Near-miss explorer

| Entry point | Purpose |
|-------------|---------|
| `summarize_run(result)` | Text summary for embedding. |
| `cosine_similarity(a, b)` | Similarity helper for vectors. |
| `HashedEmbeddingProvider` | Local deterministic embeddings. |
| `LocalNearMissStore` | In-memory or on-disk case store. |
| `NearMissExplorer.index_result(result)` | Index a run for later search. |
| `NearMissExplorer.find_near_misses(result, top_k=5)` | Retrieve similar indexed cases. |

### Artifacts

| Entry point | Purpose |
|-------------|---------|
| `build_eval_artifacts(results, ...)` | Build full `EvalArtifacts` bundle. |
| `export_json(document)` | JSON string from `EvalArtifactDocument`. |
| `export_junit_xml(results, suite_name=...)` | JUnit XML string. |
| `export_markdown_report(document, ...)` | Markdown report string. |
| `export_html_report(document, ...)` | HTML report string. |
| `write_eval_artifacts(artifacts, directory, prefix=...)` | Write all formats to disk. |

### Command-line interface

| Invocation | Purpose |
|------------|---------|
| `kaisen-evalbridge --version` | Print `{distribution_name} {version}` and exit 0. |
| `kaisen-evalbridge` | Print help and exit 0. |

## Configuration

| Variable | Required | Purpose |
|----------|----------|---------|
| `PYPI_PACKAGE_NAME` | Yes (runtime and build) | PyPI/wheel distribution name. Must match `[project].name` in `pyproject.toml` when building. If unset, `get_pypi_package_name()` and package builds raise a clear error. Copy from `.env.example`; do not commit `.env`. |

**Third-party API keys:** None are required by EvalBridge itself. LLM rubrics use your own `LLMGrader` implementation (you supply credentials in your agent or grader code). The near-miss store uses `HashedEmbeddingProvider` by default (offline); swap in another `EmbeddingProvider` if you connect an external embedding API from your application code.

## Run it locally

From a clone, using [uv](https://docs.astral.sh/uv/):

```bash
export PYPI_PACKAGE_NAME=kaisen-evalbridge
uv run python scripts/ensure_pyproject_name.py
uv sync
```

Use the library in Python (set `PYPI_PACKAGE_NAME` in the environment or before import):

```python
import os
os.environ.setdefault("PYPI_PACKAGE_NAME", "kaisen-evalbridge")

from kaisen_evalbridge import ContractHarness, Scenario
# ... define models, scenario, agent, then harness.run(...)
```

CLI:

```bash
export PYPI_PACKAGE_NAME=kaisen-evalbridge
uv run kaisen-evalbridge --version
```

**Test:**

```bash
export PYPI_PACKAGE_NAME=kaisen-evalbridge
uv run pytest
```

**Lint, typecheck, and build** (as in project automation):

```bash
export PYPI_PACKAGE_NAME=kaisen-evalbridge
uv run ruff check .
uv run mypy src
uv build
```

Install from PyPI (when published) with `pip install <your PYPI_PACKAGE_NAME value>`; development installs use `uv sync` as above.

## Architecture

| Layer | Technology |
|-------|------------|
| Frontend | — (brand SVG assets only under `frontend/public/`) |
| Backend / API server | — (library-only; no HTTP service) |
| Database | — (optional local JSON files for `LocalNearMissStore` when a directory is configured) |

**Repository layout**

| Path | Role |
|------|------|
| `src/kaisen_evalbridge/` | Library source: harness, scenarios, rubrics, trace, scoring, clustering, scorecard, regression, golden manifests, near-miss store, artifacts, config, CLI. |
| `tests/` | Pytest suite and golden manifest fixture. |
| `scripts/ensure_pyproject_name.py` | Syncs `pyproject.toml` project name with `PYPI_PACKAGE_NAME`. |
| `hatch_build.py` | Hatch metadata hook enforcing name match at build time. |
| `pyproject.toml` / `uv.lock` | Project metadata and locked dependencies (`pydantic` runtime; pytest, ruff, mypy for dev). |
| `frontend/public/` | Static logo and favicon. |

**How components connect**

Your application (or CI job) imports `kaisen_evalbridge`, defines Pydantic contracts and scenarios, and passes an `Agent` instance into `ContractHarness.run`. The harness calls your agent each turn, validates structured outputs, enforces guardrails, and fills a `Trace`. Rubric scores attach to `HarnessResult`. Downstream modules consume the same results: `ScoringPipeline` and `build_release_scorecard` for metrics, `cluster_failures` for root-cause groups, `evaluate_regression_gate` for CI policy, `build_eval_artifacts` / `write_eval_artifacts` for reports, and `NearMissExplorer` for similarity over past runs. There is no separate frontend or backend process—everything runs in-process in Python.

```mermaid
flowchart LR
  subgraph your_code [Your code]
    Agent[Agent.run_turn]
    Scenarios[Scenario / golden manifest]
  end
  subgraph evalbridge [kaisen_evalbridge]
    Harness[ContractHarness]
    Trace[TraceRecorder]
    Score[ScoringPipeline / scorecard]
    Gate[evaluate_regression_gate]
    Artifacts[build_eval_artifacts]
    NearMiss[NearMissExplorer]
  end
  Scenarios --> Harness
  Agent --> Harness
  Harness --> Trace
  Harness --> Score
  Score --> Gate
  Score --> Artifacts
  Harness --> NearMiss
```

## License

MIT
