Metadata-Version: 2.5
Name: evalgate-sdk
Version: 3.13.0
Summary: EvalGate Python SDK — CI for application behavior. Health checks, traces, evaluations, assertions, and regression gates for AI and non-AI apps.
Project-URL: Homepage, https://evalgate.com
Project-URL: Documentation, https://www.evalgate.com/docs/sdk/python
Project-URL: Issues, https://www.evalgate.com/contact
Project-URL: Changelog, https://www.evalgate.com/changelog
Author-email: EvalGate <team@evalgate.com>
License-Expression: MIT
Keywords: ai,anthropic,application-health,application-quality,assertions,ci,evaluation,llm,monitoring,observability,openai,regression,testing,tracing,workflow
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: cryptography<51,>=50.0.2
Requires-Dist: httpx<1,>=0.27
Requires-Dist: pydantic<3,>=2.4
Requires-Dist: rich>=13
Requires-Dist: tomli>=2; python_version < '3.11'
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: anthropic>=0.20; extra == 'all'
Requires-Dist: langchain-core>=0.2; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.20; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Requires-Dist: rich>=13; extra == 'dev'
Requires-Dist: ruff==0.12.5; extra == 'dev'
Requires-Dist: typer>=0.12; extra == 'dev'
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.2; extra == 'langchain'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Description-Content-Type: text/markdown

# evalgate-sdk

[![PyPI](https://img.shields.io/pypi/v/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
[![Python](https://img.shields.io/pypi/pyversions/evalgate-sdk)](https://pypi.org/project/evalgate-sdk/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Typed](https://img.shields.io/badge/typing-typed-blue)](https://peps.python.org/pep-0561/)

`evalgate-sdk` helps Python teams decide whether an AI change is safe to ship.
It combines local assertions and release gating, Python client/library surfaces,
CLI workflows, optional hosted collaboration and evidence, and CI integration.

## Install

```bash
pip install evalgate-sdk
```

Optional extras:

```bash
pip install "evalgate-sdk[openai]"
pip install "evalgate-sdk[anthropic]"
pip install "evalgate-sdk[langchain]"
pip install "evalgate-sdk[all]"
```

`all` installs the OpenAI, Anthropic, LangChain, and CLI dependencies. The
CrewAI and AutoGen adapters do not install those frameworks; install the
framework distribution used by your application separately.

Navigate by product outcome instead of memorizing the flat command catalog:

```bash
evalgate capabilities
evalgate capabilities playground
evalgate capabilities --format json
```

The exported `PRODUCT_CAPABILITY_CONTRACT` is generated from the same versioned
workflow map used by the Python and TypeScript CLIs.

## Availability

The lifecycle below describes the checked-in source contract. PyPI publication
is separately governed and the registry can lag the source tree; run
`evalgate --version` and check the public feature-status page before relying on
a source-only command. Use `evalgate capabilities --format json` for the
installed package's machine-readable workflow and exit-code contract.

## Start in a cloud-agent sandbox — no account or browser required

```bash
evalgate init --local --format json
evalgate init --local --apply --format json
evalgate baseline update
evalgate gate
```

Here, local names the execution/evidence boundary for the current checkout, not
a requirement that a person be on a laptop. It requires no account, API key,
organization, GitHub remote, browser, interactive TTY, or network call. It does
not invent hosted identity. Review and commit durable
configuration/spec files before accepting the baseline; generated reports and
run artifacts follow managed ignore rules. `--preview` never writes or prompts,
and JSON/non-interactive execution remains preview-only without explicit
`--apply`.

The daily loop is `change -> evalgate gate -> PASS / WARN / BLOCK + evidence`.
Use `evalgate status --json` to read local readiness independently from hosted
identity and cloud evidence. Status exit zero means the document was produced,
not that every readiness dimension passed.

## Add hosted collaboration

```bash
evalgate login
evalgate link
evalgate status
```

Current source stores the revocable human CLI session outside the repository in the operating-system
user configuration directory: `%APPDATA%\evalgate\config.json` on Windows,
`~/Library/Application Support/evalgate/config.json` on macOS, and
`${XDG_CONFIG_HOME:-~/.config}/evalgate/config.json` on Linux. An absolute
`EVALGATE_CONFIG_HOME` can override that directory for managed environments.
`link` establishes the durable organization, GitHub repository, and selected
repository-relative root, then pins the cloud activation snapshot to the exact
local commit. It writes only non-secret version 2 context to
`.evalgate/project.json`. Branch and commit movement do not unlink the project.
`status` reports durable link, checkout relation, cloud-snapshot currency, and
local/cloud readiness separately. A checkout two commits behind `main` remains
linked and locally gate-ready; cloud intake may target that exact SHA when GitHub
proves it remains reachable. EvalGate never silently substitutes the remote
head. Repository files
cannot provide an API key or redirect a saved/environment credential to another
API origin. Alternate origins require HTTPS except localhost and loopback
development. CI should inject a separately managed `EVALGATE_API_KEY` through
its secret store; linking never creates or replaces that automation credential.
An API key authorizes automation against an already established hosted context;
it is not a bootstrap shortcut for human login, repository linking, or local
initialization.

Version 3.7.9 source briefly used `.evalgate/config.json`. Current source ignores
that legacy file: run `evalgate login`, delete it, and rotate the key if it
may have entered Git history.

### Hosted local gate setup

```bash
evalgate init
evalgate baseline update
```

Interactive `init` previews the scaffold and asks before writing. JSON and
non-interactive invocations remain preview-only; explicit `init --preview` never
prompts or writes, and automation must pass `evalgate init --apply` explicitly.
A valid safe version 2 project context is required for hosted initialization,
but commit drift does not invalidate it and linked projects may initialize
offline. Then use `evalgate gate` before shipping a change. Linking never mints
an automation credential. Existing local specs, configuration, and baselines
remain in place when a local project is later linked.

## Migrating from 3.8.x source

Existing local project state remains usable; do not reinitialize or replace a
valid baseline just to adopt the new lifecycle.

| 3.8.x | Current path |
| --- | --- |
| `auth status` | `whoami` |
| `auth configure --api-key ...` | Human: `login`; automation: environment credential with an existing hosted context; local: `init --local` |
| `setup` | `login`, then `link` |
| `setup status` | `status` |
| `start` | No direct alias: use `init`/`init --local` for setup, `gate` for a decision, or `watch` for continuous reruns |
| `verify` | `status`, then `gate` |

Removed commands provide targeted guidance rather than reviving repository
secret storage.

## Optional product understanding (Experimental current source)

`understand` is optional and is not required for initialization, baseline
acceptance, or a release decision:

```bash
evalgate understand --format json
evalgate understand --apply
```

It reads bounded local evidence and produces provisional product context without
network/model calls, application execution, baseline mutation, or promotion.

### On-demand application health

The same local path works for ordinary applications such as APIs, web apps,
webhook consumers, background jobs, and CLI tools. Create a small
`evalgate.application.json` manifest with safe HTTP or command checks and run:

```bash
evalgate health --config evalgate.application.json --format github
```

The report measures bounded latency and errors sequentially, redacts failure
text, and does not keep a collector, daemon, or cloud connection alive. The
reviewed `evalgate init` scaffold installs the `evalgate-application-health`
repository Skill so a coding agent can infer checks and prepare a reviewed PR
rationale only when called.

### Local Assertions

```python
from evalgate_sdk import expect

result = expect("The capital of France is Paris.").to_contain("Paris")
print(result.passed)
```

### Platform Client

```python
import asyncio

from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import CreateTraceParams

async def main() -> None:
    async with AIEvalClient(api_key="sk-...") as client:
        trace = await client.traces.create(
            CreateTraceParams(
                name="support-run",
                input="Cancel my subscription",
                output="I've canceled your plan effective today.",
            )
        )

        print(trace.id)

asyncio.run(main())
```

## Closed-Loop Workflow

The Python package participates in the same EvalGate loop:

```text
trace -> cluster -> synthesize -> gate -> review -> auto -> ship
```

Representative CLI commands:

```bash
evalgate gate
evalgate cluster --run .evalgate/runs/latest.json
evalgate synthesize --dataset .evalgate/golden/labeled.jsonl
evalgate auto run --objective tone_mismatch --budget 3
evalgate auto daemon --objective tone_mismatch --cycles 5
evalgate judge registry
evalgate judge presets
```

### Connect a coding client

Install EvalGate's repository Skill using the client-native layout where one is
known. Preview is the default; `--apply` is the only write authority:

```bash
evalgate connect --client codex
evalgate connect --client codex --apply
```

Use `claude-code`, `cursor`, `cowork`, or `generic` for other clients. Setup
writes local client files after the repository's signed-in EvalGate install.
The generated `.evalgate/agent-clients.json` records
the metrics-only export path and next capture/sync commands; EvalGate does not
read provider transcripts or start a collector.

## Web App Sync

The Python CLI uses the same sync contracts as the TypeScript SDK. Local labeled
JSONL can be pushed into the web app, web-labeled cases can be merged back into
the same file, and prompt files can round-trip with stale active-version
protection.

```bash
# Push local labels into the web app and pull web labels back into JSONL
evalgate sync \
  --project-key support-agent \
  --labeled-dataset .evalgate/golden/labeled.jsonl \
  --pull-labels

# Pull the active web prompt into a local file
evalgate sync \
  --project-key support-agent \
  --prompt-id prompt_support \
  --target-path prompts/support.md \
  --pull-prompt

# Push an edited local prompt as a new web prompt version
evalgate sync \
  --project-key support-agent \
  --prompt-id prompt_support \
  --target-path prompts/support.md \
  --push-prompt
```

## Product Workflows

The purpose-built client and CLI cover the same governed workflows as the web
app, including pack installation, exact-commit repository intelligence,
Playground case authoring, Prompt Hub, and Dataset Hub.

```bash
evalgate repo repositories
evalgate repo scan --repository 42 --head-sha <sha>
evalgate repo ask --repository 42 --question "What AI exists?"
evalgate packs list --domain coding
evalgate packs install coding-agent-release-safety
evalgate playground from-trace --evaluation 7 --trace 123
evalgate prompts --help
evalgate datasets --help
```

```python
repositories = await client.repository_intelligence.list_repositories()
scan = await client.repository_intelligence.start_scan(
    42,
    idempotency_key="release-42",
    head_sha="a" * 40,
)
answer = await client.repository_intelligence.answer_question(
    42,
    question_text="What models, agents, tools, and evals exist?",
    graph_version_id=scan["scanRun"]["graphVersionId"],
)
packs = await client.eval_packs.list(domain="coding")
```

Repository scanning reads a bounded immutable GitHub tree and does not execute
repository code. Source detections and pack recommendations include evidence
locators and completeness state; they are not represented as runtime proof.

## Judge Orchestration

The Python SDK supports the same core judge model as the platform:

- registry-backed judge selection
- saved judge configs
- multi-judge aggregation
- per-run and per-case judge inspection
- structured reasoning and signal breakdowns

### CLI

```bash
evalgate judge registry
evalgate judge presets

evalgate judge test \
  --provider openai \
  --model gpt-5.2-chat-latest \
  --judge openai:gpt-5.2-chat-latest \
  --judge anthropic:claude-sonnet-4-20250514 \
  --aggregation weighted \
  --prompt-template "Return strict JSON with score, passed, reasoning, and signals." \
  --input "Cancel my subscription" \
  --output "I've canceled your plan effective today."
```

### Client

```python
from evalgate_sdk.types import TestLLMJudgeConfigParams

registry = await client.llm_judge.list_registry()
presets = await client.llm_judge.list_presets()

result = await client.llm_judge.test_config(
    TestLLMJudgeConfigParams(
        provider="openai",
        model="gpt-5.2-chat-latest",
        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
        judges=[
            {
                "id": "primary",
                "type": "llm",
                "provider": "openai",
                "model": "gpt-5.2-chat-latest",
                "weight": 0.6,
            },
            {
                "id": "backup",
                "type": "llm",
                "provider": "anthropic",
                "model": "claude-sonnet-4-20250514",
                "weight": 0.4,
            },
        ],
        aggregation="weighted",
        input="Cancel my subscription",
        output="I've canceled your plan effective today.",
        behavior="tool_use",
        task_type="support",
    )
)

print(result.result.score, result.result.reasoning)
```

## Enterprise Controls

The Python SDK respects the same platform guardrails:

- provider allowlists
- pre-send PII checks for outbound judge/provider calls
- org-scoped policy
- shared config precedence with the API and web app

## Cross-Language Contract

The generated `OpenApiClient` is the cross-language operation surface. It
validates stable operation IDs, required path/query/header parameters, required
bodies, idempotency keys, cancellation, and bounded pagination from the same
OpenAPI registry as TypeScript and the CLI.

```python
from evalgate_sdk.openapi_client import OpenApiClient

async with OpenApiClient(api_key="sk-...") as api:
    versions = await api.request(
        "get_evaluations_id_prompts_promptId_versions",
        path={"id": 42, "promptId": "support/primary"},
    )
```

Purpose-built Python and TypeScript clients are intentionally documented
separately and are not assumed to be one-for-one. See the generated parity
report for the structural operation inventory.

For broader product context, see:

- [Root README](../../../README.md)
- [Python SDK guide](../../../mintlify/sdk/python.mdx)
- [Generated SDK parity report](../../../docs/sdk-parity.md)
- [Feature availability](../../../mintlify/platform/feature-status.mdx)
