Metadata-Version: 2.4
Name: toolcontract
Version: 0.1.0
Summary: Contract testing for LLM agent tool-calls — catch tool-calling regressions when a provider ships a new model version.
Author: Divyansh Rai
License: MIT
Project-URL: Homepage, https://github.com/Divyansh2202/toolcontract
Project-URL: Repository, https://github.com/Divyansh2202/toolcontract
Project-URL: Issues, https://github.com/Divyansh2202/toolcontract/issues
Keywords: llm,testing,agents,tool-calling,contract-testing,pytest,openai,anthropic
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Classifier: Framework :: Pytest
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == "anthropic"
Provides-Extra: litellm
Requires-Dist: litellm>=1.90; extra == "litellm"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Dynamic: license-file

# toolcontract

Contract testing for LLM agent tool-calls.

When a provider ships a new model version, an agent's tool-calling behavior
can silently change — invented or dropped arguments, a different tool
chosen for the same input, values that no longer match a schema your
downstream code depends on. Nothing in a typical CI pipeline catches this
before it reaches production.

`toolcontract` is built around one trigger event: **a model version changed
— did your agent's tool calls still do what you expect?** Pin a golden set
of expected tool-call trajectories, run them against a live model, get a
pass/fail/inconclusive verdict and a diff. Think Pact for microservice
contracts, or Percy for visual regressions, but for tool calls.

## Status

v0.1, in active development. Contract model, comparators/matching engine,
verification ledger, OpenAI/Anthropic/LiteLLM adapters, the CLI
(`run`/`accept`/`check-version`), a real pytest plugin, and the LLM-judge
semantic tier are all built and tested. Not yet published to PyPI.

## Quick start

```python
from toolcontract import ArgKind, ArgSpec, Contract, ExpectedCall
from toolcontract.adapters.base import ToolSchema
from toolcontract.adapters.openai_adapter import OpenAIAdapter
from toolcontract.runner.engine import run_contract

log_water = ToolSchema(
    name="log_water",
    description="Log that the user drank water",
    parameters={
        "type": "object",
        "properties": {"amount_ml": {"type": "number"}},
        "required": ["amount_ml"],
        "additionalProperties": False,
    },
    strict=True,
)

contract = Contract(
    id="log-water-basic",
    description="User mentions drinking water, agent logs it",
    input_messages=({"role": "user", "content": "I just drank a glass of water"},),
    expected_calls=(
        ExpectedCall(tool_name="log_water", args={"amount_ml": ArgSpec.exact(250, kind=ArgKind.NUMBER, tolerance=20)}),
    ),
)

result = run_contract(contract, OpenAIAdapter(), [log_water], "gpt-4o")
print(result.verdict)  # Verdict.PASS / FAIL / INCONCLUSIVE
```

Or from the CLI, against a directory of compiled contracts:

```bash
toolcontract run contracts/ --provider openai --model gpt-4o --tools tools.json
toolcontract accept contracts/one.json --provider openai --model gpt-4o   # promote a new baseline
toolcontract check-version contracts/ --provider openai --model gpt-4o-2027 --ledger .toolcontract_ledger.json
```

See `examples/tap_health_agent_get_meal_suggestions.py` for a real contract
authored against a production tool schema, including a custom comparator
and an optional argument.

## Providers

- **OpenAI**, **Anthropic** — native adapters, `pip install "toolcontract[openai]"` / `[anthropic]`.
- **Any OpenAI-compatible endpoint** — a self-hosted LiteLLM proxy, vLLM,
  Ollama, OpenRouter, Azure OpenAI — already works today with zero new code:
  ```python
  import openai
  from toolcontract.adapters.openai_adapter import OpenAIAdapter

  adapter = OpenAIAdapter(client=openai.OpenAI(base_url="http://localhost:4000", api_key="sk-..."))
  ```
- **LiteLLM (direct SDK routing)** — `pip install "toolcontract[litellm]"`, then
  `LiteLLMAdapter()` with a provider-prefixed model string (`"cerebras/zai-glm-4.7"`,
  `"gemini/gemini-2.0-flash"`, and 100+ others LiteLLM supports). Capability
  (whether the underlying provider actually honors strict/grammar-constrained
  tool calling) is queried per-model at call time, not assumed statically —
  see `adapters/litellm_adapter.py`'s module docstring for why, and for the
  honest caveat about how precise that signal currently is.
- **Missing your provider?** See `CONTRIBUTING.md` — adding a native adapter
  is the single most valuable contribution this project can take right now.

## Non-goals

This is deliberately **not** a general agent evaluation framework. If a
request is really about one of the following, a different tool is the
right fit — `agentevals`, `DeepEval`, `Ragas`, `PydanticAI Evals`,
`LangSmith`, or `Phoenix` all already cover this territory well:

- RAG retrieval or answer-quality evaluation
- Hallucination detection or general output-quality scoring
- Multi-turn simulated-user agent evaluation
- General-purpose "score my agent" metrics unrelated to tool-call structure

`toolcontract` owns exactly one job: did this model version's tool-calling
behavior change against a pinned contract. Everything in scope should trace
back to that job — anything that doesn't belongs in a different package.

## Design principles (from the architecture review)

- **Core engine is framework-agnostic.** The pytest plugin is a thin
  consumer of a plain Python/CLI-callable core — not the other way around.
- **Three-valued verdicts.** `PASS` / `FAIL` / `INCONCLUSIVE`. An argument
  that can't be resolved structurally (e.g. free-form natural language)
  reports `INCONCLUSIVE`, never a false `FAIL`.
- **Type-aware comparison, not naive equality.** Dates, numbers, and
  strings are normalized before comparison so formatting variance doesn't
  produce false regressions.
- **Contracts compile to portable, versioned JSON.** Python is the
  authoring ergonomics; the JSON file is the diffable artifact checked
  into a repo and hashed for the verification ledger.
- **Custom comparators are registered, never dynamically imported.**
  A contract file's `comparator_ref` may not share the trust level of the
  code running it — a PR from an external contributor, or a contract
  pulled from a shared source. `@toolcontract.register_comparator` requires
  the trusted test suite to explicitly opt a function in; anything else in
  a contract file fails closed instead of being imported and called.
- **Capability is per-model, not per-adapter.** A single adapter instance
  (especially a gateway-style one like `LiteLLMAdapter`) can front providers
  with genuinely different tool-calling guarantees — `capability_for(model)`
  is queried at call time, never assumed as a static fact.

## Contributing

See `CONTRIBUTING.md` — provider adapters especially welcome.

## Development

```bash
python -m venv .venv
.venv/bin/pip install -e ".[dev]"
.venv/bin/python -m pytest
```
