Metadata-Version: 2.4
Name: claimgate
Version: 0.1.0
Summary: CI for AI-agent quality — a regression-testing harness for AI agents.
Project-URL: Homepage, https://github.com/Kamal934/claimgate
Project-URL: Repository, https://github.com/Kamal934/claimgate
Project-URL: Issues, https://github.com/Kamal934/claimgate/issues
Author: claimgate contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agent,ci,eval,evaluation,llm,quality,regression,testing
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: anthropic>=0.40; extra == 'all'
Requires-Dist: openai>=1.40; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: anthropic>=0.40; extra == 'dev'
Requires-Dist: openai>=1.40; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: respx>=0.21; extra == 'dev'
Provides-Extra: openai
Requires-Dist: openai>=1.40; extra == 'openai'
Description-Content-Type: text/markdown

# claimgate

**CI for AI-agent quality.** Define test cases for your AI agent, run assertions
against its outputs, and **fail the build when quality regresses**. Runs locally
and in CI. No accounts, no web UI, no hosted services — the whole thing works
standalone with your own API key.

> **Status: alpha (`0.x`).** This is early software. The public API and the suite
> YAML schema may change before `1.0`, so pin a version (`claimgate==0.1.0`)
> and check the [changelog](CHANGELOG.md) when you upgrade.

```text
Running 5 case(s) against function examples.agent:answer  (judge: claude-opus-4-8)

  ✓ capital-of-france (0.40s)
  ✓ refuses-pii (0.39s)
  ✓ grounded-summary (1.12s)
  ✗ tone-is-friendly (0.95s)
      ✗ llm_graded (score 0.30) — The reply is curt and omits a greeting.
  ✓ no-hallucinated-fields (1.07s)

1 failed, 4 passed of 5

Compared to last run (since 2026-06-28T08:18:53+00:00)
  ⚠ 1 regression(s) (passed before, failing now):
      ✗ tone-is-friendly
```

---

## Why

You changed a prompt, swapped a model, or refactored a retrieval step. Did
anything get worse? claimgate answers that question the same way unit tests
answer it for code: a suite of cases, run on every PR, that goes red when an
output stops meeting your bar — including the subtle failure that matters most
for agents, **hallucinated facts** (see [`grounded_in_source`](#grounded_in_source)).

You bring the agent; claimgate evaluates it. It calls your agent — a **Python
callable** or an **HTTP endpoint** — with each case's input and grades the
output. It never runs your agent's internals.

## How it compares

claimgate overlaps with several good tools, and isn't trying to replace them.
[Promptfoo](https://promptfoo.dev) excels at broad, config-driven prompt and eval
matrices across many providers;
[DeepEval](https://github.com/confident-ai/deepeval) brings a rich metric library
to a pytest-native workflow;
[Ragas](https://github.com/explodinggradients/ragas) focuses on RAG metrics like
faithfulness and context precision; and
[LangSmith](https://www.langchain.com/langsmith) offers hosted tracing and
evaluation, tightly integrated with the LangChain ecosystem. If you're already
invested in one of those, you may not need this.

claimgate's emphasis is narrower and two-fold: **grounding-first** and
**CI-first**. The flagship [`grounded_in_source`](#grounded_in_source) assertion
breaks an output into atomic claims and names the specific ones the source
doesn't support — aimed at catching hallucinated *fields*, not just scoring an
overall "faithfulness" number. And the whole tool is shaped around failing a pull
request on a regression: a non-zero exit code, a run-to-run diff that says
exactly what newly broke, and a drop-in GitHub Action — with no hosted account,
dashboard, or sign-up. You bring your own API key and keep all the state.

**Cost:** the deterministic assertions (`exact_match`, `contains`,
`not_contains`, `regex`) make no network calls and are free; `llm_graded` and
`grounded_in_source` spend your provider budget — roughly one judge call per such
assertion per run (cached within a run). **Privacy:** nothing leaves your machine
except the judge calls you explicitly opt into, and those go only to the provider
you configured.

## Install

```bash
pip install claimgate
```

The `llm_graded` and `grounded_in_source` assertions call an LLM judge. Install
the extra for your provider and set your key (you bring your own):

```bash
pip install "claimgate[anthropic]"   # then: export ANTHROPIC_API_KEY=...
# or
pip install "claimgate[openai]"      # then: export OPENAI_API_KEY=...
```

---

## 5-minute quickstart

### 1. Write (or point at) your agent

Any callable that takes an input and returns a string. Put this in `agent.py` at
your repo root:

```python
# agent.py
def answer(question: str) -> str:
    q = question.lower()
    if "capital of france" in q:
        return "The capital of France is Paris."
    if "ssn" in q or "social security" in q:
        return "I'm sorry, I can't share personal data like that."
    if "summarize the order" in q:
        return "Order #4471 was placed by Dana Lee and ships to Portland, OR."
    return "I'm not sure about that."
```

Async functions and HTTP endpoints work too — see [Targets](#targets).

### 2. Write a suite

A suite is a YAML file: a list of cases, each with an `input`, optional
`expected`/`source`, and a list of `assertions`. Save this as
`claimgate/suite.yaml`:

```yaml
name: Support agent suite
target:
  type: function
  ref: agent:answer            # module:function, importable from repo root

cases:
  - name: capital-of-france
    input: "What is the capital of France?"
    expected: "The capital of France is Paris."
    assertions:
      - type: exact_match
      - type: contains
        value: Paris

  - name: refuses-pii
    input: "What is the customer's SSN?"
    assertions:
      - type: not_contains
        value: "123-45-6789"
      - type: regex
        pattern: "(?i)can't|cannot|sorry"

  - name: tone-is-friendly
    input: "What is the capital of France?"
    assertions:
      - type: llm_graded
        rubric: "The reply is friendly and helpful, not curt."

  - name: grounded-summary
    input: "Summarize the order."
    source: |
      Order #4471 was placed on 2026-03-02 by Dana Lee.
      It contains 2 items and ships to Portland, OR.
    assertions:
      - type: grounded_in_source     # every claim must be supported by `source`

  - name: no-hallucinated-fields
    input: "Summarize the order."
    source: |
      Order #4471 was placed by Dana Lee. It ships to Portland, OR.
    assertions:
      - type: grounded_in_source
        threshold: 1.0               # all claims grounded (the default)
```

Inspect it without calling your agent:

```bash
claimgate show claimgate/suite.yaml
```

### 3. Run it

```bash
claimgate run claimgate/suite.yaml
```

The deterministic cases need no API key. The `llm_graded` / `grounded_in_source`
cases need a judge — set `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` first (see
[The LLM judge](#the-llm-judge)). The command exits non-zero if any case fails,
so it gates CI out of the box.

### 4. Wire it into CI

Copy this into `.github/workflows/claimgate.yml`:

```yaml
name: claimgate
on:
  pull_request:
  push:
    branches: [main]
jobs:
  agent-quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: Kamal934/claimgate@v0
        with:
          suite: claimgate/suite.yaml
          package-spec: "claimgate[anthropic]"   # or [openai]
        env:
          # Only needed for llm_graded / grounded_in_source.
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

Add your key under **Settings → Secrets and variables → Actions**. That's the
whole setup — no claimgate account, nothing hosted.

### 5. Watch CI go red on a regression

Break something on a branch — e.g. change `agent.py` so it answers
`"Paris"` instead of `"The capital of France is Paris."`. The `exact_match`
case fails, the job exits non-zero, and the PR check turns red. On the **second**
run, claimgate also prints a **diff** that names exactly which case newly
failed (the regression), so reviewers see the quality change at a glance.

Total time, from nothing to a red-on-regression CI gate: well under 30 minutes.

---

## Concepts

- **Target** — your agent. A `function` (`pkg.module:func`) or an `http`
  endpoint. The harness calls it; it doesn't run your agent's internals.
- **Suite** — a YAML file with a list of cases and an optional default target.
- **Case** — one `input`, an optional `expected` output, an optional `source`
  document, and one or more `assertions`.
- **Assertion** — a single check against the output.

## Assertion types

| type | passes when |
| --- | --- |
| [`exact_match`](#exact_match) | output equals the expected string |
| [`contains`](#contains--not_contains) | output contains a substring |
| [`not_contains`](#contains--not_contains) | output does **not** contain a substring |
| [`regex`](#regex) | output matches a regular expression |
| [`llm_graded`](#llm_graded) | an LLM judges the output against your rubric |
| [`grounded_in_source`](#grounded_in_source) | every factual claim is supported by a source document |

### exact_match

```yaml
- type: exact_match
  value: "Paris"        # optional; defaults to the case's `expected`
  ignore_case: false    # default false
  strip: true           # trim surrounding whitespace before comparing (default true)
```

### contains / not_contains

```yaml
- type: contains
  value: "Paris"
  ignore_case: false
- type: not_contains
  value: "123-45-6789"
```

### regex

Matches with `re.search` (anywhere in the output).

```yaml
- type: regex
  pattern: "(?i)can't|cannot|sorry"
  ignore_case: false
  multiline: false
  dotall: false
```

### llm_graded

An LLM judges the output against a plain-language rubric and returns a 0.0–1.0
score plus a verdict. Passes when the score ≥ `pass_threshold` **and** the
judge's verdict is pass.

```yaml
- type: llm_graded
  rubric: "The reply is friendly, accurate, and does not invent product names."
  pass_threshold: 0.5     # default 0.5
  model: claude-opus-4-8  # optional per-assertion judge override
```

### grounded_in_source

**The flagship check — it catches hallucinated fields.** Given the output and a
`source` document, the judge extracts the discrete factual claims the output
makes and verifies each against the source. A claim is `supported`,
`contradicted`, or `not_found` (invented). The assertion's **score is the
fraction of claims that are grounded**, and it passes when that fraction ≥
`threshold` (default `1.0` — every claim must be supported). Failures name the
ungrounded claims:

```text
✗ grounded_in_source — 1/2 grounded — ungrounded: [not_found] The total was $99.99
```

```yaml
- type: grounded_in_source
  source: |                 # optional; defaults to the case's `source`
    Order #4471 was placed by Dana Lee. It ships to Portland, OR.
  threshold: 1.0            # min fraction of claims that must be grounded
  model: claude-opus-4-8    # optional judge override
```

A refusal or an output that makes no factual claims trivially passes — there is
nothing to hallucinate. The grounding check is built behind a pluggable
interface, so cheap heuristic pre-checks (substring coverage, embeddings, NLI)
can be added in front of the LLM judge without changing the assertion.

## Targets

Declare a default target in the suite, or pass `--target` on the CLI
(`claimgate run suite.yaml --target agent:answer`). A URL becomes an HTTP
target; anything else is treated as a function reference.

**Function** — sync or async; called with the case `input`:

```yaml
target:
  type: function
  ref: my_pkg.agent:run     # "module:function" (or "module.function")
```

Run claimgate from your repo root so the module is importable (the CWD is
added to the import path automatically).

**HTTP** — the input is POSTed as JSON; the output is read from a JSON field:

```yaml
target:
  type: http
  url: "http://localhost:8000/agent"
  method: POST              # GET sends the input as a query param instead
  input_field: input        # request body: {"input": <case input>}
  output_field: output      # read response["output"]; null = use the raw body
  headers: {}
  timeout: 30
```

## The LLM judge

Model-agnostic and bring-your-own-key. The provider is chosen from the
environment:

| env var | effect |
| --- | --- |
| `ANTHROPIC_API_KEY` | use Anthropic (default model `claude-opus-4-8`) |
| `OPENAI_API_KEY` | use an OpenAI-compatible API (default model `gpt-4o-mini`) |
| `AGENTHARNESS_PROVIDER` | force `anthropic` or `openai` |
| `AGENTHARNESS_MODEL` | override the judge model |
| `AGENTHARNESS_BASE_URL` | point at any OpenAI-compatible endpoint (OpenRouter, vLLM, Ollama, LM Studio, …) |

Precedence for the model: assertion `model:` → suite `model:` → `--model` →
`AGENTHARNESS_MODEL` → provider default. Identical judge calls are **cached
within a run**, so a single run is idempotent. To use a provider the box doesn't
ship, implement the small `LLMJudge` interface and pass it to `run_suite`.

## Run-to-run diff

Every `run` writes its results to `.claimgate/last_run.json`. The next run
diffs against it and prints what changed — leading with **regressions** (cases
that passed before and fail now), then fixes, new cases, and removed cases. Add
`.claimgate/` to `.gitignore` (each runner keeps its own baseline), or commit
it if you want a shared baseline. Flags: `--state-dir DIR`, `--no-save`.

## CLI reference

```text
claimgate run SUITE [options]      run every case, print results + diff
claimgate show SUITE [options]     load and print a suite (no agent calls)
claimgate --version

run options:
  -t, --target TARGET     override target: pkg.module:func or an http(s) URL
  -m, --model MODEL       judge model override
  -c, --concurrency N     max cases in parallel (default 8)
  -v, --verbose           show passing assertions too
      --state-dir DIR     diff/state directory (default .claimgate)
      --no-save           don't persist this run as the new baseline
```

**Exit codes:** `0` all cases passed · `1` at least one case failed · `2` usage
error (bad suite, missing target, no judge configured).

## GitHub Action inputs

| input | default | description |
| --- | --- | --- |
| `suite` | `claimgate/suite.yaml` | path to the suite |
| `target` | `""` | optional target override |
| `python-version` | `3.11` | Python to set up |
| `package-spec` | `claimgate` | pip spec (e.g. `claimgate[anthropic]`, `claimgate==0.1.0`) |
| `args` | `""` | extra args for `claimgate run` |
| `working-directory` | `.` | directory to install and run in |

Pass your provider key to the step via `env:` (see the snippet above). The check
fails (non-zero exit) whenever a case fails.

## Examples

The [`examples/`](examples/) directory has a runnable sample agent and two
suites: [`examples/suite.yaml`](examples/suite.yaml) demonstrating **every**
assertion type, and [`examples/deterministic.yaml`](examples/deterministic.yaml)
which needs no API key.

```bash
claimgate run examples/deterministic.yaml      # no key required
claimgate run examples/suite.yaml              # needs ANTHROPIC_API_KEY / OPENAI_API_KEY
```

## Project structure

```text
claimgate/
├── pyproject.toml                     # Package metadata, deps, extras, build (hatchling), pytest config
├── README.md                          # This file — user docs + 5-minute quickstart
├── LICENSE                            # Apache-2.0
├── action.yml                         # GitHub composite Action: install + run a suite, fail CI on failure
├── .github/
│   └── workflows/
│       ├── ci.yml                     # Repo CI: pytest on Python 3.11–3.13
│       └── claimgate.yml           # Copy-paste agent-quality gate (also dogfoods this action)
│
├── src/claimgate/                  # The installable package
│   ├── __init__.py                    # Public API exports + __version__
│   ├── schema.py                      # Pydantic models: Suite, Case, the 6 assertions, Target
│   ├── loader.py                      # YAML → Suite, with friendly validation errors
│   ├── target.py                      # Call the user's agent: Python callable or HTTP endpoint
│   ├── runner.py                      # Async, concurrency-bounded case execution
│   ├── results.py                     # AssertionResult / CaseResult (+ JSON (de)serialization)
│   ├── reporting.py                   # Terminal pass/fail rendering (markup-safe)
│   ├── llm.py                         # Model-agnostic judge: Anthropic/OpenAI, within-run cache, JSON parse
│   ├── grounding.py                   # Claim extraction + verification, pluggable pre-check pipeline
│   ├── state.py                       # Persist a run to .claimgate/last_run.json
│   ├── diff.py                        # Run-to-run diff (regressions, fixes, new/removed) + rendering
│   ├── cli.py                         # Typer CLI: `run` and `show`
│   └── assertions/
│       ├── __init__.py                # Imports the evaluators (registers them on import)
│       ├── base.py                    # EvalContext + the evaluator registry/dispatch
│       ├── basic.py                   # exact_match, contains, not_contains, regex
│       ├── llm_graded.py              # llm_graded evaluator
│       └── grounded_in_source.py      # grounded_in_source evaluator (the flagship check)
│
├── examples/                          # Runnable samples (run from the repo root)
│   ├── README.md                      # How to run the examples
│   ├── __init__.py                    # Makes `examples.agent` importable
│   ├── agent.py                       # A tiny deterministic sample agent
│   ├── suite.yaml                     # Showcase suite — every assertion type (needs an API key)
│   └── deterministic.yaml             # No-key suite (deterministic assertions only)
│
└── tests/                             # Test suite (pytest + pytest-asyncio)
    ├── sample_targets.py              # Importable callables used by the target tests
    ├── test_schema.py                 # Schema validation, discriminated unions, cross-checks
    ├── test_loader.py                 # YAML loading + error formatting
    ├── test_assertions_basic.py       # exact_match / contains / not_contains / regex
    ├── test_target.py                 # Function + HTTP target calling (respx-mocked)
    ├── test_runner.py                 # Async runner, ordering, target-error handling
    ├── test_cli.py                    # End-to-end CLI exit codes + diff loop
    ├── test_llm.py                    # JSON extraction, caching, provider factory, adapters
    ├── test_assertions_llm_graded.py  # llm_graded scoring/threshold (fake judge)
    ├── test_grounding.py              # Grounding engine internals
    ├── test_assertions_grounded.py    # grounded_in_source integration + hallucination catch
    ├── test_state.py                  # last_run.json persistence
    ├── test_diff.py                   # Diff computation + rendering
    ├── test_reporting.py              # Terminal output + rich-markup escaping
    └── test_examples.py               # The shipped example suites stay valid and green

# generated / git-ignored (not in version control):
#   .venv/  dist/  build/  *.egg-info/  .pytest_cache/  .claimgate/
```

## Development

```bash
pip install -e ".[dev]"
pytest
```

## License

Apache-2.0.
