Metadata-Version: 2.4
Name: llm-agentspec
Version: 0.3.0
Summary: Deterministic, pytest-style contracts for LLM agent tool calls: order, budgets, forbidden tools, approval gates
Author-email: Sai Rakshith Muda <rakshithmuda@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/rakshithmuda22/agentspec
Project-URL: Repository, https://github.com/rakshithmuda22/agentspec
Project-URL: Issues, https://github.com/rakshithmuda22/agentspec/issues
Keywords: llm,agents,testing,contracts,pytest,guardrails,tool-calling,ai-agents
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Framework :: Pytest
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == "anthropic"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-benchmark>=4.0; extra == "dev"
Requires-Dist: pyflakes>=3.0; extra == "dev"
Dynamic: license-file

# AgentSpec

[![CI](https://github.com/rakshithmuda22/agentspec/actions/workflows/ci.yml/badge.svg)](https://github.com/rakshithmuda22/agentspec/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/llm-agentspec)](https://pypi.org/project/llm-agentspec/)
![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)
![License: MIT](https://img.shields.io/badge/license-MIT-green)

**pytest-style contracts for what an LLM agent *does*: which tools it calls, in what order, how often, with which arguments, and behind which approvals. Checked deterministically, offline, with zero dependencies.**

```python
spec = (ContractSet("refund_agent")
        .must_precede("check_refund_policy", "process_refund")   # policy gates the refund
        .must_not_call("delete_account")                          # never
        .must_call_at_most("web_search", n=5))                    # cost cap

spec.check(session).assert_all_pass()       # in CI, on a recorded trace
FastGuard(spec).enforce("delete_account")   # or live, before the tool runs -> raises Blocked
```

## The problem

Most agent evals grade the final answer. But agents do their damage in the steps:
a refund issued without the policy check, a `delete_account` the model decided was
"cleanup", 47 paid searches for one question. The final answer can look perfect while
the trajectory is wrong, and an LLM judge that only sees the answer can't tell.

These are rules, not opinions, so they should be checked like unit tests: same trace,
same verdict, every time, without paying for a model call. That's what AgentSpec does.

## What it does

- **Contracts on tool calls** — `must_call`, `must_not_call`, `must_call_before`,
  `must_call_at_most` / `at_least` / `exactly`, `must_call_in_sequence`,
  `within_steps`, `never_after`, `eventually`, `must_call_with_arg`.
- **Argument-aware rules** — `where=` predicates: `path_not_under("/workspace")`,
  `host_not_in([...])`, `arg_gt("amount", 500)`, regex, combinators.
- **Approval gates** — `requires_approval("deploy", correlate="env")`: an approval for
  `prod` does not unlock `staging`.
- **Severities and a decision** — each rule is BLOCK / APPROVAL / WARN / AUDIT; a report
  rolls up to `allow`, `block` or `require_approval`, and points at the offending call.
- **Live enforcement** — check a call *before* it runs. `FastGuard` keeps incremental
  state, so a check costs the same at call 10 and call 10,000.
- **Trace adapters** — Anthropic Messages, OpenAI Chat Completions + Responses,
  OpenTelemetry GenAI / OpenInference spans, or plain dicts. No provider SDK needed.
- **pytest native** — `assert_all_pass()`, plus fixtures registered as a pytest plugin.

## A real example

[`examples/why_agentspec.py`](examples/why_agentspec.py) is a refund agent in OpenAI
tool-calling format. Its final answer is polite and plausible; its steps are not.
Actual output:

```
Final answer the customer saw:
  All done! I've processed your $240 refund for order 5582. It will appear in 3-5 business days. Sorry about the broken item!

Tool calls in the trace:
  lookup_order -> process_refund -> delete_account

[1] Audit of the finished trace

  AgentSpec Report — 'refund_agent'
  --------------------------------------------------
  [✓] must_call(lookup_order): 'lookup_order' was called
  [✗] must_precede(check_refund_policy, process_refund) @call[1]: 'process_refund' was called before any 'check_refund_policy'  → fix: call 'check_refund_policy' before 'process_refund'
  [✗] must_not_call(delete_account) @call[2]: 'delete_account' was called 1 time(s) but must never be called  → fix: remove the 'delete_account' call or take the tool away from the agent
  [✓] must_not_call(process_refund) [conditional]: no 'process_refund' call matched the condition
  --------------------------------------------------
  2 passed, 2 failed, 0 warned
  Overall: FAIL  ·  Decision: block

[2] Same calls through a live guard, checked BEFORE each one runs
  lookup_order     {'order_id': '5582'} -> allow
  process_refund   {'order_id': '5582', 'amount': 240} -> block
  delete_account   {'order_id': '5582'} -> block
```

## Quickstart

```bash
pip install llm-agentspec
```

### 1. Write contracts and check a session

```python
from agentspec import ContractSet, AgentSession, ToolCall

spec = (ContractSet("research_agent")
        .must_call("search")                         # did the research
        .must_call_before("search", "summarize")     # ...before summarizing
        .must_not_call("delete_file")                # never deletes files
        .must_call_at_most("search", n=3))           # cost cap

# What your agent actually did (normally built by an adapter, see below)
session = AgentSession(tool_calls=[
    ToolCall(name="search", args={"query": "agent testing"}),
    ToolCall(name="summarize"),
    ToolCall(name="delete_file", args={"path": "/tmp/cache"}),
])

report = spec.check(session)
print(report)
print(report.decision.value)
```

```
  AgentSpec Report — 'research_agent'
  --------------------------------------------------
  [✓] must_call(search): 'search' was called
  [✓] must_call_before(search, summarize): 'search' (step 0) correctly precedes 'summarize' (step 1)
  [✗] must_not_call(delete_file) @call[2]: 'delete_file' was called 1 time(s) but must never be called  → fix: remove the 'delete_file' call or take the tool away from the agent
  [✓] must_call_at_most(search, n=3): 'search' called 1 time(s) (limit: 3)
  --------------------------------------------------
  3 passed, 1 failed, 0 warned
  Overall: FAIL  ·  Decision: block

block
```

### 2. Put it in your pytest suite

```python
# test_my_agent.py
from agentspec import ContractSet, AgentSession, ToolCall

SPEC = (ContractSet("research_agent")
        .must_call("search")
        .must_call_before("search", "summarize")
        .must_not_call("delete_file"))


def run_my_agent(task: str) -> AgentSession:
    # stand-in for your agent: return the tool calls it made
    return AgentSession(tool_calls=[ToolCall(name="summarize"),
                                    ToolCall(name="delete_file")])


def test_research_agent_follows_the_rules():
    SPEC.check(run_my_agent("Summarize recent AI news")).assert_all_pass()
```

```
$ pytest test_my_agent.py -q
    def test_research_agent_follows_the_rules():
>       SPEC.check(run_my_agent("Summarize recent AI news")).assert_all_pass()
E       AssertionError: AgentSpec 'research_agent' failed 3 contract(s):
E         - must_call(search): 'search' was never called (calls made: ['summarize', 'delete_file'])
E         - must_call_before(search, summarize): 'search' was never called (required before 'summarize')
E         - must_not_call(delete_file) @call[1]: 'delete_file' was called 1 time(s) but must never be called

test_my_agent.py:17: AssertionError
FAILED test_my_agent.py::test_research_agent_follows_the_rules - AssertionErr...
1 failed in 0.01s
```

### 3. Enforce rules live, before a tool runs

```python
from agentspec import ContractSet, FastGuard, matchers as m

spec = (ContractSet("coding_agent")
        .must_not_call("write_file", where=m.path_not_under("/workspace"))
        .must_call_at_most("web_search", n=20, per_seconds=60)
        .requires_approval("run_shell", correlate="cmd"))

guard = FastGuard(spec)

def check(name, args):
    print(f"{name}({args}) -> {guard.check_call(name, args).decision.value}")

check("write_file", {"path": "/workspace/app.py"})
check("write_file", {"path": "/workspace/../etc/passwd"})
check("run_shell", {"cmd": "pytest"})
guard.approve(cmd="pytest")                 # a human said yes to *this* command
check("run_shell", {"cmd": "pytest"})
check("run_shell", {"cmd": "rm -rf /"})     # the approval doesn't carry over
```

```
write_file({'path': '/workspace/app.py'}) -> allow
write_file({'path': '/workspace/../etc/passwd'}) -> block
run_shell({'cmd': 'pytest'}) -> require_approval
run_shell({'cmd': 'pytest'}) -> allow
run_shell({'cmd': 'rm -rf /'}) -> require_approval
```

In your tool dispatcher: `guard.enforce(name, args)` (raises `Blocked` or
`ApprovalRequired`), run the tool, then `guard.commit(name, args, result=...)`.

## Getting your agent's trace in

Adapters work on live SDK objects *or* plain JSON you saved, so the same contracts
run in CI against fixtures with no API key:

```python
from agentspec.adapters import (
    from_anthropic_messages,   # Anthropic Messages: tool_use, server_tool_use, mcp_tool_use blocks
    from_openai_messages,      # OpenAI Chat Completions tool_calls, Responses API function_call items
    from_otel_spans,           # OTel GenAI execute_tool spans, OpenInference TOOL spans
    from_dicts,                # anything else: [{"name": ..., "args": {...}}, ...]
)

session = from_anthropic_messages(messages)   # the list you send back to messages.create
spec.check(session).assert_all_pass()
```

[`examples/case_study_research_agent/`](examples/case_study_research_agent/) runs six
contracts over two Anthropic-format conversations: a good run passes, a runaway run
(five searches, then `delete_file`) fails with the offending call indexes.

## Contract reference

| Contract | Passes when | Notes |
|---|---|---|
| `must_call(tool, where=)` | `tool` is called at least once | liveness |
| `must_not_call(tool, where=)` | `tool` is never called (or never with matching args) | |
| `must_call_before(a, b)` | both are called and the first `a` precedes the first `b` | first-occurrence |
| `must_precede(a, b, correlate=, where_second=)` | every `b` has an earlier `a` | vacuous if no `b` |
| `requires_approval(action, by="approve", correlate=)` | every `action` has an earlier approval | decides `require_approval` |
| `must_call_at_most(tool, n, where=, per_seconds=)` | ≤ n calls overall, or in any rolling window | |
| `must_call_at_least(tool, n)` / `must_call_exactly(tool, n)` | count ≥ n / == n | |
| `must_call_in_sequence(*tools)` | first occurrences appear in this order | |
| `must_not_call_after(tool, trigger)` / `never_after(...)` | `tool` never runs after `trigger` | |
| `within_steps(a, b, n)` | every `a` is followed by a `b` within n calls | |
| `eventually(tool)` | `tool` is called at some point | WARN by default |
| `must_call_with_arg(tool, key, value)` | some call has `args[key] == value` | |

`severity=` is accepted by `must_call`, `must_not_call`, `must_precede`,
`must_call_at_most`, `never_after`, `within_steps` and `eventually`; `requires_approval`
always uses APPROVAL and the rest are BLOCK. Ready-made packs:
`packs.filesystem_safety`, `no_exfiltration`, `spend_cap` (fails closed on missing or
non-numeric amounts), `rate_limit`, `approval_gated`, combined with `packs.combine`.

## How it works

```mermaid
flowchart LR
    T["Trace<br/>Anthropic / OpenAI / OTel / dicts"] -->|adapters| S["AgentSession<br/>ordered ToolCalls"]
    S --> C["ContractSet.check"]
    C --> R["ContractReport<br/>verdicts, offending call,<br/>decision"]
    R -->|assert_all_pass| P["pytest / CI"]
    A["Agent proposes a call"] --> G["FastGuard.check_call"]
    G -->|allow| X["run tool, commit"]
    G -->|block / require_approval| H["refuse or ask a human"]
```

Each contract is a pure function from a session to a result; there is no model in the
loop. `GuardedSession` re-runs that function over the whole session for each new call
(O(N) per call). `FastGuard` instead keeps a small state machine per contract (counts,
first offending index, gate state, a window of timestamps) and answers from that state.
A randomized differential test checks that the two agree, call by call, on verdict,
severity, message and offending index across 400 random specs. More detail in
[`docs/architecture.md`](docs/architecture.md).

## Performance

Apple M5 Pro, Python 3.11. Commands are in the repo; numbers will differ on your machine.

Batch checking, 7 contracts (`pytest benchmarks/test_benchmark.py --benchmark-only`):

| Session size | Median per check |
|---|---|
| 5 calls | 3.9 µs |
| 25 calls | 5.5 µs |
| 100 calls | 10.9 µs |

Live checks, 5 safety contracts (`python benchmarks/bench_live_guard.py`):

| Calls already in session | GuardedSession | FastGuard |
|---|---|---|
| 10 | 6.7 µs | 2.7 µs |
| 100 | 31.5 µs | 2.7 µs |
| 1,000 | 284.5 µs | 2.7 µs |
| 10,000 | 3,408.8 µs | 2.8 µs |

The full quickstart script above (interpreter start, import, check, print) takes about
35 ms wall-clock.

## How it compares

DeepEval's `ToolCorrectnessMetric` and RAGAS's `ToolCallAccuracy` also score tool calls
without an LLM: they compare a run against a **reference list** of expected calls
(optionally ordered, optionally with arguments). That works well when there is one
right trajectory.

AgentSpec checks **rules** that every acceptable run must satisfy, so it needs no golden
trajectory and can say things a reference list can't: "never call X", "at most 20
searches a minute", "every `deploy` needs an approval for that environment", "no writes
outside `/workspace`". It can also enforce those rules before a call executes. It does
not judge whether the answer is good; pair it with an output evaluator for that.

## Tests and CI

```bash
git clone https://github.com/rakshithmuda22/agentspec && cd agentspec
pip install -e ".[dev]"
pytest tests/ examples/ -q          # 173 passed
python examples/why_agentspec.py
python demo/demo.py
```

173 tests across `tests/` and `examples/`, all offline. CI
([`.github/workflows/ci.yml`](.github/workflows/ci.yml)) runs them on Python 3.9–3.13
with pyflakes, both demos and the stress tests, then builds the wheel, runs
`twine check` and installs it into a clean venv.

## Limitations

- Rules see tool **names and arguments**, not meaning. A generic `bash` tool needs
  argument rules (`where=m.arg_matches("cmd", r"\brm\b")`) to be useful.
- In a live guard, liveness rules (`must_call`, `eventually`, `within_steps`) fail until
  satisfied and so block early calls. Keep them in a separate end-of-run spec.
- `FastGuard` has incremental versions of `must_not_call`, `must_call_at_most`,
  `must_precede` / `requires_approval`, `never_after` / `must_not_call_after` and
  `must_call`. Other rules and custom `add()` checkers still run in O(N).
- `path_under` normalizes `..` but does not resolve symlinks. URLs need a scheme for
  host rules to match (unparseable URLs are treated as not allowlisted).
- The case-study conversations are hand-written in Anthropic format, not recorded
  from a live model.

## Roadmap

- `agentspec check traces/*.json --spec spec.py` CLI with JUnit XML output for CI.
- A LangGraph / LangChain callback adapter (today: `from_dicts`).
- Recorded traces from real model runs in the case study.

## Project layout

```
agentspec/
  contracts.py     contract engine, reports, severities, decisions
  incremental.py   FastGuard (incremental live guard)
  runtime.py       GuardedSession, enforce(), Blocked / ApprovalRequired
  matchers.py      where= predicates (paths, hosts, args, combinators)
  packs.py         ready-made rule packs
  adapters/        anthropic.py, openai.py, generic.py (dicts + OTel spans)
  pytest_plugin.py fixtures + assert_satisfies / assert_violates
  recorder.py      mock sessions + an optional live Claude tool-use loop
tests/             unit, scenario, adapter and differential tests
examples/          why_agentspec.py, Anthropic-format case study
benchmarks/        pytest-benchmark, stress tests, live-guard benchmark
```

## License

MIT. Built by [Sai Rakshith Muda](https://github.com/rakshithmuda22).
