Metadata-Version: 2.5
Name: agent-handshake
Version: 1.0.0b2
Summary: Check that what your agent is told matches the tools it actually has.
Project-URL: Homepage, https://github.com/padobrik/agent-handshake
Project-URL: Documentation, https://github.com/padobrik/agent-handshake/tree/main/docs
Project-URL: Repository, https://github.com/padobrik/agent-handshake
Project-URL: Issues, https://github.com/padobrik/agent-handshake/issues
Author-email: Fedor Kabachenko <fkabachenko@gmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: agents,ai,llm,prompts,testing,tools
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: pyyaml<7.0,>=6.0
Requires-Dist: rich<16.0,>=13.0
Requires-Dist: typer<1.0,>=0.12
Provides-Extra: all
Requires-Dist: mypy<3.0,>=1.10; extra == 'all'
Requires-Dist: pre-commit<5.0,>=3.7; extra == 'all'
Requires-Dist: pytest-cov<8.0,>=5.0; extra == 'all'
Requires-Dist: pytest<10.0,>=8.0; extra == 'all'
Requires-Dist: ruff<1.0,>=0.5; extra == 'all'
Requires-Dist: types-pyyaml>=6.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: mypy<3.0,>=1.10; extra == 'dev'
Requires-Dist: pre-commit<5.0,>=3.7; extra == 'dev'
Requires-Dist: pytest-cov<8.0,>=5.0; extra == 'dev'
Requires-Dist: pytest<10.0,>=8.0; extra == 'dev'
Requires-Dist: ruff<1.0,>=0.5; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Description-Content-Type: text/markdown

<p align="center">
  <strong>agent-handshake</strong><br>
  <em>Check that what your agent is told matches the tools it actually has</em>
</p>

<p align="center">
  <a href="https://pypi.org/project/agent-handshake/"><img src="https://img.shields.io/pypi/v/agent-handshake?style=flat-square" alt="PyPI"></a>
  <a href="https://github.com/padobrik/agent-handshake/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/padobrik/agent-handshake/ci.yml?branch=main&style=flat-square&label=CI" alt="CI"></a>
  <a href="https://github.com/padobrik/agent-handshake/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License"></a>
  <img src="https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue?style=flat-square" alt="Python 3.11+">
</p>

An agent's instructions and tool descriptions often name its tools, and
nothing checks that those names still exist. When a tool is renamed or an MCP
server adds a prefix, the model calls a tool it does not have, gets an error
and often just gives up.

agent-handshake reads the request your agent sends to the model and reports
where the text and the tools disagree, without calling a model.

## Example

The instructions say `mem_ssh`; the MCP server registered the tool as
`mcp__mem__mem_ssh` (a [real case](https://pete.lostsource.net/posts/2026-05-17-when-your-agent-cant-find-its-tools.html)):

```json
{
  "model": "claude-haiku-4-5",
  "max_tokens": 1024,
  "system": "Use `mem_ssh` to connect to Tower.",
  "messages": [{"role": "user", "content": "Is Tower up?"}],
  "tools": [{"name": "mcp__mem__mem_ssh", "input_schema": {"type": "object"}}]
}
```

```
$ handshake check request.json
3 rules enabled. Checked request.json (Anthropic Messages): 1 tools, 2 text segments.
✗ HS002 request.json (system): Text names `mem_ssh`; no tool has that exact name. Did you mean `mcp__mem__mem_ssh`?
1 findings (1 errors, 0 warnings).
```

<p align="center">
  <img src="https://raw.githubusercontent.com/padobrik/agent-handshake/main/docs/demo.gif" width="760" alt="Terminal demo: handshake check reports HS002 for a system prompt that says mem_ssh while the only tool is mcp__mem__mem_ssh, and exits 1; on a request whose tool description names webpage_read it reports an HS001 warning and exits 0, and exits 1 with --strict.">
</p>

## Install

```
pip install --pre agent-handshake
```

**Status: beta (1.0.0b2).** `--pre` is needed while 1.0 is in beta: without it
pip installs 0.10.0, an older and different tool (see [History](#history-0x)).

## Get request.json

`request.json` is the request body as sent to the model, in OpenAI Chat
Completions, OpenAI Responses or Anthropic Messages format. If your code calls
an SDK, put the call's arguments in a dict, save it as JSON and pass the same
dict to the call.

OpenAI:

```python
import json

from openai import OpenAI

request = {
    "model": "gpt-4.1",
    "messages": [
        {"role": "system", "content": "Use `get_weather` for questions about the weather."},
        {"role": "user", "content": "Is it raining in Lisbon?"},
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
}
with open("request.json", "w") as f:
    json.dump(request, f, indent=2)

response = OpenAI().chat.completions.create(**request)
```

Anthropic:

```python
import json

from anthropic import Anthropic

request = {
    "model": "claude-sonnet-5-5",
    "max_tokens": 1024,
    "system": "Use `get_weather` for questions about the weather.",
    "messages": [{"role": "user", "content": "Is it raining in Lisbon?"}],
    "tools": [
        {
            "name": "get_weather",
            "input_schema": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        }
    ],
}
with open("request.json", "w") as f:
    json.dump(request, f, indent=2)

response = Anthropic().messages.create(**request)
```

## Run

```
handshake check request.json
```

Each finding names the rule, where the text is (its path in the request) and
the closest real name. Errors make the command exit with code 1; warnings do
not, unless `--strict` is given. `--config` points to a `handshake.yaml` or
`pyproject.toml`; without it, the one in the current directory is used.

<p align="center">
  <img src="https://raw.githubusercontent.com/padobrik/agent-handshake/main/docs/diagrams/how-it-works.svg" width="760" alt="How it works: a request body as sent to the model is read into instructions, tools and response schema; rules HS001 to HS003 compare the names in the text with the tools; findings are printed, and errors make the check exit 1. No LLM calls.">
</p>

## In pytest

`agent_handshake.check(body, config=None)` runs the same rules on a dict and
returns the findings (since 1.0.0b2).

```python
import json

from agent_handshake import check


def test_request_matches_tools():
    with open("request.json") as fp:
        body = json.load(fp)  # or build it with your agent's own code
    assert not [f for f in check(body) if f.severity == "error"]
```

A complete example: [examples/pytest/](examples/pytest/test_agent_request.py).

## GitHub Actions

```yaml
- run: |
    pip install --pre agent-handshake
    handshake check request.json
```

## Rules

| rule | finds | default |
|---|---|---|
| HS001 | a tool named in a form that can only be a tool, that the request does not have | on, warning |
| HS002 | a tool name that is almost right (case, `-`/`_`, prefix, suffix) | on, error |
| HS003 | an argument the tool's schema does not have | on, error |
| HS004 | an example value outside a field's allowed values | off |
| HS005 | example JSON keys that do not match the call or response format | off |
| HS007 | prose that may name a tool the request does not have | off |

Rule pages: [docs/rules/](docs/rules/). Turning rules off, changing their
severity and ignoring names: [docs/configuration.md](docs/configuration.md).

## Known limitations

HS001 can warn about names that are not missing tools:

- **Code Mode declarations.** A description that declares functions as
  `tools.x({ ... })` is read as calls to `x`.
- **Negative forms.** "Never attempt 'apply_patch' ... they don't exist" is
  read as naming `apply_patch`.
- **Placeholders.** "I'm going to use `tool_name`" is read as naming
  `tool_name`.

HS001 findings are warnings, so they do not fail the check unless `--strict`
is used. To silence them, ignore the names or turn the rule off in
`handshake.yaml` (or `[tool.handshake]` in `pyproject.toml`):

```yaml
ignore:
  - tool_name                  # never report this name
  - {rule: HS001, ref: apply_patch}
rules:
  HS001: {enabled: false}      # or turn HS001 off
```

## Benchmark

Measured once on the frozen configuration, on three sets that share no
project: dev (used to build the rules), holdout and holdout-2 (never used to
tune them). Recall is over real, fixed mismatches from the projects' history;
precision is over hand-labelled findings on the current code of open-source
agents. Small samples are marked by their size.

| | dev | holdout | holdout-2 |
|---|---|---|---|
| recall cases (projects) | 25 (12) | 14 (10) | 4 (4) |
| catchable cases caught by rules on by default | 8/11 | 4/11 | 2/2 |
| precision corpus: repositories (requests) | 4 (13) | 3 (12) | 4 (16) |
| HS001 precision | 13/14 | 0/24 | no findings |
| HS002 precision | 11/11 | no findings | 5/5 |
| HS003 precision | 2/2 | no findings | no findings |
| known mismatches in the corpus found | 22/32 | 6/24 | 6/17 |

The holdout HS001 findings are Code Mode function declarations of one project
(`tools.x({ ... })`), which HS001 reads as calls. Details and the method:
[benchmarks/README.md](benchmarks/README.md); run it with `make benchmark`.

## Roadmap

Capturing requests from a running agent, so there is no `request.json` to
write by hand (not started).

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

Apache 2.0. See [LICENSE](LICENSE).

## History (0.x)

Releases up to 0.10.0 on PyPI are a different, closed tool: contract tests
between agents. Its code is in tag
[contract-testing-final](https://github.com/padobrik/agent-handshake/tree/contract-testing-final);
see [why it was closed](docs/history/contract-testing.md).
