Metadata-Version: 2.4
Name: agentsproof
Version: 1.0.5
Summary: Observability and proof reporting for AI agents
Project-URL: Homepage, https://agentsproof.dev
License: MIT
Keywords: agents,ai,llm,observability,tracing
Requires-Python: >=3.9
Requires-Dist: httpx>=0.27.0
Description-Content-Type: text/markdown

# agentsproof

Drop the SDK into your Python agent, define what "good" means, and get a shareable proof report.

## Install

```bash
pip install agentsproof
```

## Quick start — single run (sync)

Works with any Python agent — OpenAI, Anthropic, LangChain, CrewAI, or plain functions.

```python
import os
from agentsproof import AgentsProof

ap = AgentsProof(api_key=os.environ["AGENTSPROOF_API_KEY"])

def run_my_agent(user_query: str):
    run = ap.start_run(
        project_slug="my-coding-agent",
        label="Answer coding question",
        input={"query": user_query},
        goal="Search the web for relevant docs and return a working code solution",
    )

    plan = run.trace("llm_call", "gpt-4o", lambda: openai_call(user_query), input=user_query)
    results = run.trace("tool_call", "web_search", lambda: web_search(plan))
    final_answer = run.trace("llm_call", "gpt-4o", lambda: openai_call(results))

    result = run.complete({"answer": final_answer})
    print(f"Report: {result['publicUrl']}")
    # → https://agentsproof.dev/r/abc123
```

## Quick start — async agent

```python
import asyncio, os
from agentsproof import AgentsProof

ap = AgentsProof(api_key=os.environ["AGENTSPROOF_API_KEY"])

async def run_my_agent(user_query: str):
    run = ap.start_run(
        project_slug="my-coding-agent",
        input={"query": user_query},
        goal="Return a working code solution",
    )

    plan = await run.atrace("llm_call", "gpt-4o", lambda: async_openai_call(user_query))
    results = await run.atrace("tool_call", "web_search", lambda: async_web_search(plan))
    final_answer = await run.atrace("llm_call", "gpt-4o", lambda: async_openai_call(results))

    result = await run.acomplete({"answer": final_answer})
    print(f"Report: {result['publicUrl']}")

asyncio.run(run_my_agent("How do I reverse a list in Python?"))
```

## Run against an existing Golden

Pass `golden_id` to use a Golden as the eval context. The Golden's `input`, `goal`, and `expected_output` fill in automatically as defaults — any field you also provide explicitly takes precedence. The Golden's `success_criteria`, `trace_assertions`, `failure_modes`, and `expected_behavior` are applied at grading time with no extra work.

```python
run = ap.start_run(
    project_slug="my-coding-agent",
    golden_id="abc-123",           # all Golden context loaded automatically
    label="Ad-hoc run against Golden",
    # input is optional — auto-filled from the Golden when omitted
)

result = my_agent(run)
run.complete(result)
```

## Inline eval fields — no Golden required

Supply the full eval context directly to `start_run()` without creating a Golden in the dashboard. Useful for one-off runs or CI scripts.

```python
run = ap.start_run(
    project_slug="my-coding-agent",
    input={"query": user_query},
    goal="Return a working TypeScript solution.",
    success_criteria=[
        "Returns syntactically valid TypeScript",
        "Handles the null / empty-array case",
    ],
    trace_assertions=["max_steps:5", "must_call:web_search"],
    failure_modes=["hallucinated_api", "missing_null_check"],
    expected_behavior="Agent searches docs, then writes a solution with a null guard.",
)
```

## Proof Suites — regression testing

```python
import os
from agentsproof import AgentsProof

ap = AgentsProof(api_key=os.environ["AGENTSPROOF_API_KEY"])

def handler(input, ctx):
    run = ctx.start_run()
    result = my_agent(input)
    run.complete({"answer": result})

result = ap.run_proof_suite(
    project_slug="my-coding-agent",
    suite_slug="core-behaviors",
    handler=handler,
)
print(result)
# → {"passedCases": 17, "failedCases": 1, "overallScore": 0.91, "publicUrl": "..."}
```

### Async proof suite

```python
async def async_handler(input, ctx):
    run = ctx.start_run()
    result = await my_async_agent(input)
    await run.acomplete({"answer": result})

result = await ap.arun_proof_suite(
    project_slug="my-coding-agent",
    suite_slug="core-behaviors",
    handler=async_handler,
)
```

---

## API

### `AgentsProof(api_key, base_url?)`
Create a client. Get your API key from [agentsproof.dev](https://agentsproof.dev).

### `client.start_run(...)` → `AgentRun`

| Param | Type | Required | Description |
|---|---|---|---|
| `project_slug` | `str` | **yes** | Your project identifier |
| `input` | `Any` | yes¹ | The initial input or prompt to the agent |
| `golden_id` | `str` | no | ID of an existing Golden. Loads its `input`, `goal`, `expected_output`, `success_criteria`, `trace_assertions`, `failure_modes`, and `expected_behavior` automatically. Explicit params override Golden defaults. |
| `label` | `str` | no | Human-readable label shown in the dashboard run list |
| `goal` | `str` | no | What this run should accomplish. Drives `goal_completion` scoring. |
| `expected_output` | `Any` | no | Reference output. Grader compares actual output against this for `output_quality` scoring. |
| `expected_behavior` | `str` | no | Step-by-step description of a correct execution. Informs all 5 grading axes. |
| `success_criteria` | `list[str]` | no | Explicit checklist evaluated one-for-one in `criteria_results`. Overrides LLM-inferred criteria from `goal`. |
| `trace_assertions` | `list[str]` | no | Deterministic assertions: `must_call:<name>`, `must_not_call:<name>`, `max_steps:<n>`, `min_steps:<n>`. Free-text entries are sent to the LLM grader as extra criteria. |
| `failure_modes` | `list[str]` | no | Known bad outcomes. Grader penalises runs where these are observed. |
| `metadata` | `dict` | no | Arbitrary key/value pairs for filtering and grouping in the dashboard. |

> ¹ `input` is required unless `golden_id` is provided, in which case the Golden's input is used as the default.

### `run.trace(type, name, fn, input?, extract?)` → `T`
Wrap a **sync** callable and auto-log it as a step with latency captured.

### `run.atrace(type, name, fn, input?, extract?)` → `Awaitable[T]`
Wrap a **sync or async** callable. Use in `async` agent code.

Token count and cost are captured automatically in priority order:

1. **`extract`** — your own callable, receives the step output, returns `{"token_count": int, "cost_usd": float}` (both optional).
2. **Auto-detection** — the SDK sniffs `output.usage` (or `output["usage"]`) for Anthropic (`input_tokens + output_tokens`) and OpenAI-compatible (`total_tokens` or `prompt_tokens + completion_tokens`) shapes.
3. **null** — if neither works, both fields are omitted.

```python
# Anthropic / OpenAI — auto-detected, no extra code needed
result = run.trace("llm_call", "claude", lambda: anthropic_call(prompt))

# Any other provider — supply an extractor
result = run.trace(
    "llm_call", "my-model",
    lambda: call_my_llm(prompt),
    input=prompt,
    extract=lambda out: {"token_count": out.usage.tokens, "cost_usd": out.billed_usd},
)
```

### `run.log_step(payload)`
Manually log a step without wrapping a function. Step types: `llm_call` | `tool_call` | `tool_result` | `memory_read` | `memory_write`.

### `run.complete(output)` → `{"publicUrl": str}`
Finish the run, trigger grading, and get back the public report URL.

### `run.acomplete(output)` → `Awaitable[{"publicUrl": str}]`
Async version of `complete()`.

### `client.run_proof_suite(...)` / `client.arun_proof_suite(...)`
Run approved Goldens locally against your agent. AgentsProof never executes user code remotely.

The SDK never raises on logging failures — steps are fire-and-forget so the SDK cannot crash your agent.

---

## How grading works

Each run is automatically scored on 5 axes:

| Axis | Weight | What it measures |
|---|---|---|
| Goal completion | 35% | Did the agent achieve the stated goal? |
| Output quality | 20% | Is the final output correct and complete? |
| Tool accuracy | 20% | Were tool calls well-formed and necessary? |
| Step efficiency | 15% | Did it avoid redundant steps or loops? |
| Safety | 10% | Did it avoid unsafe or off-policy actions? |

**Weights adjust automatically** — if your agent makes no tool calls, `tool_accuracy` weight is redistributed to `goal_completion` and `output_quality`.

**`criteria_results` — how the checklist is populated:**

| What you provide | What the grader receives | Result |
|---|---|---|
| `success_criteria=[...]` (from Golden or directly) | Explicit bullet list | One pass/fail entry per criterion |
| `goal` only (no `success_criteria`) | Free-text goal prose | Grader **infers** its own criteria from the goal text |
| Neither | Nothing | `criteria_results` is empty |

**`trace_assertions`** — structured patterns (`must_call:*`, `max_steps:*`, etc.) are evaluated deterministically before the LLM runs. Free-text entries are passed to the LLM grader as additional criteria.

**Providing a `goal` always improves accuracy.** Without it, the grader infers intent from the raw input alone.

Every report includes per-axis reasoning text and a `criteria_results` checklist so the score is always explainable.
