Metadata-Version: 2.4
Name: evalgate-sdk
Version: 3.5.1
Summary: EvalGate Python SDK — CI for AI behavior. Traces, evaluations, assertions, and regression gates for LLM apps.
Project-URL: Homepage, https://evalgate.com
Project-URL: Documentation, https://github.com/evalgate/ai-evaluation-platform#readme
Project-URL: Repository, https://github.com/evalgate/ai-evaluation-platform
Project-URL: Issues, https://github.com/evalgate/ai-evaluation-platform/issues
Project-URL: Changelog, https://github.com/evalgate/ai-evaluation-platform/blob/main/src/packages/sdk-python/CHANGELOG.md
Author-email: EvalGate <team@evalgate.com>
License-Expression: MIT
Keywords: ai,anthropic,assertions,ci,evaluation,llm,monitoring,observability,openai,regression,testing,tracing,workflow
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx<1,>=0.27
Requires-Dist: pydantic<3,>=2.0
Requires-Dist: rich>=13
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: anthropic>=0.20; extra == 'all'
Requires-Dist: langchain-core>=0.2; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Requires-Dist: rich>=13; extra == 'all'
Requires-Dist: typer>=0.12; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.20; extra == 'anthropic'
Provides-Extra: cli
Requires-Dist: rich>=13; extra == 'cli'
Requires-Dist: typer>=0.12; extra == 'cli'
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: rich>=13; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: typer>=0.12; extra == 'dev'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.2; extra == 'langchain'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Description-Content-Type: text/markdown

# evalgate-sdk

[![PyPI](https://img.shields.io/pypi/v/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
[![Python](https://img.shields.io/pypi/pyversions/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Typed](https://img.shields.io/badge/typing-typed-blue)](https://peps.python.org/pep-0561/)

`evalgate-sdk` brings EvalGate's evaluation control plane to Python.

Use it to run assertions, trace agent workflows, inspect platform runs, orchestrate judges, and participate in the same closed-loop evaluation workflow as the TypeScript SDK and the web app.

## Install

```bash
pip install evalgate-sdk
```

Optional extras:

```bash
pip install "evalgate-sdk[openai]"
pip install "evalgate-sdk[anthropic]"
pip install "evalgate-sdk[langchain]"
pip install "evalgate-sdk[all]"
```

## Quick Start

### Local Assertions

```python
from evalgate_sdk import expect

result = expect("The capital of France is Paris.").to_contain("Paris")
print(result.passed)
```

### Platform Client

```python
from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import CreateTraceParams

client = AIEvalClient(api_key="sk-...")

trace = await client.traces.create(
    CreateTraceParams(
        name="support-run",
        input="Cancel my subscription",
        output="I've canceled your plan effective today.",
    )
)

print(trace.id)
```

## Closed-Loop Workflow

The Python package participates in the same EvalGate loop:

```text
trace -> cluster -> synthesize -> gate -> review -> auto -> ship
```

Representative CLI commands:

```bash
evalgate gate
evalgate cluster --run .evalgate/runs/latest.json
evalgate synthesize --dataset .evalgate/golden/labeled.jsonl
evalgate judge registry
evalgate judge presets
```

## Web App Sync

The Python CLI uses the same sync contracts as the TypeScript SDK. Local labeled
JSONL can be pushed into the web app, web-labeled cases can be merged back into
the same file, and prompt files can round-trip with stale active-version
protection.

```bash
# Push local labels into the web app and pull web labels back into JSONL
evalgate sync \
  --project-key support-agent \
  --labeled-dataset .evalgate/golden/labeled.jsonl \
  --pull-labels

# Pull the active web prompt into a local file
evalgate sync \
  --project-key support-agent \
  --prompt-id prompt_support \
  --target-path prompts/support.md \
  --pull-prompt

# Push an edited local prompt as a new web prompt version
evalgate sync \
  --project-key support-agent \
  --prompt-id prompt_support \
  --target-path prompts/support.md \
  --push-prompt
```

## Judge Orchestration

The Python SDK supports the same core judge model as the platform:

- registry-backed judge selection
- saved judge configs
- multi-judge aggregation
- per-run and per-case judge inspection
- structured reasoning and signal breakdowns

### CLI

```bash
evalgate judge registry
evalgate judge presets

evalgate judge test \
  --provider openai \
  --model gpt-5.2-chat-latest \
  --judge openai:gpt-5.2-chat-latest \
  --judge anthropic:claude-sonnet-4-20250514 \
  --aggregation weighted \
  --prompt-template "Return strict JSON with score, passed, reasoning, and signals." \
  --input "Cancel my subscription" \
  --output "I've canceled your plan effective today."
```

### Client

```python
from evalgate_sdk.types import TestLLMJudgeConfigParams

registry = await client.llm_judge.list_registry()
presets = await client.llm_judge.list_presets()

result = await client.llm_judge.test_config(
    TestLLMJudgeConfigParams(
        provider="openai",
        model="gpt-5.2-chat-latest",
        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
        judges=[
            {
                "id": "primary",
                "type": "llm",
                "provider": "openai",
                "model": "gpt-5.2-chat-latest",
                "weight": 0.6,
            },
            {
                "id": "backup",
                "type": "llm",
                "provider": "anthropic",
                "model": "claude-sonnet-4-20250514",
                "weight": 0.4,
            },
        ],
        aggregation="weighted",
        input="Cancel my subscription",
        output="I've canceled your plan effective today.",
        behavior="tool_use",
        task_type="support",
    )
)

print(result.result.score, result.result.reasoning)
```

## Enterprise Controls

The Python SDK respects the same platform guardrails:

- provider allowlists
- pre-send PII checks for outbound judge/provider calls
- org-scoped policy
- shared config precedence with the API and web app

## Surface Parity

The Python SDK now covers the primary EvalGate workflows used in the product:

- traces and runs
- clustering and synthesis
- gate/check/review loops
- judge registry, presets, configs, results, and comparisons
- workflow and control-plane inspection commands

TypeScript still has a few extra convenience wrappers, but the core control-plane model is shared.

For broader product context, see:

- [Root README](../../../README.md)
- [Integration reference](../../app/docs/integration/page.tsx)
- [SDK parity guide](../../app/docs/sdk-parity/page.tsx)
