Metadata-Version: 2.5
Name: agent-handshake
Version: 1.0.0b1
Summary: Check that what your agent is told matches the tools it actually has.
Project-URL: Homepage, https://github.com/padobrik/agent-handshake
Project-URL: Documentation, https://github.com/padobrik/agent-handshake/tree/main/docs
Project-URL: Repository, https://github.com/padobrik/agent-handshake
Project-URL: Issues, https://github.com/padobrik/agent-handshake/issues
Author-email: Fedor Kabachenko <fkabachenko@gmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: agents,ai,llm,prompts,testing,tools
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: pyyaml<7.0,>=6.0
Requires-Dist: rich<16.0,>=13.0
Requires-Dist: typer<1.0,>=0.12
Provides-Extra: all
Requires-Dist: mypy<3.0,>=1.10; extra == 'all'
Requires-Dist: pre-commit<5.0,>=3.7; extra == 'all'
Requires-Dist: pytest-cov<8.0,>=5.0; extra == 'all'
Requires-Dist: pytest<10.0,>=8.0; extra == 'all'
Requires-Dist: ruff<1.0,>=0.5; extra == 'all'
Requires-Dist: types-pyyaml>=6.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: mypy<3.0,>=1.10; extra == 'dev'
Requires-Dist: pre-commit<5.0,>=3.7; extra == 'dev'
Requires-Dist: pytest-cov<8.0,>=5.0; extra == 'dev'
Requires-Dist: pytest<10.0,>=8.0; extra == 'dev'
Requires-Dist: ruff<1.0,>=0.5; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Description-Content-Type: text/markdown

<p align="center">
  <strong>agent-handshake</strong><br>
  <em>Check that what your agent is told matches the tools it actually has</em>
</p>

<p align="center">
  <a href="https://pypi.org/project/agent-handshake/"><img src="https://img.shields.io/pypi/v/agent-handshake?style=flat-square" alt="PyPI"></a>
  <a href="https://github.com/padobrik/agent-handshake/actions/workflows/ci.yml"><img src="https://img.shields.io/github/actions/workflow/status/padobrik/agent-handshake/ci.yml?branch=main&style=flat-square&label=CI" alt="CI"></a>
  <a href="https://github.com/padobrik/agent-handshake/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue?style=flat-square" alt="License"></a>
  <img src="https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-blue?style=flat-square" alt="Python 3.11+">
</p>

agent-handshake checks that what your agent is told matches the tools it actually has.

**Status: beta, released (1.0.0b1).** Releases up to 0.10.0 on PyPI are a
different, closed tool: contract tests between agents. See
[why it was closed](docs/history/contract-testing.md).

## Install

```
pip install --pre agent-handshake
```

`--pre` is needed while 1.0 is in beta: without it pip installs 0.10.0, the
old tool.

## Check a request

```
handshake check request.json
```

`request.json` is a request body as sent to the model, in OpenAI Chat
Completions, OpenAI Responses or Anthropic Messages format. Each finding names
the rule, where the text is (its path in the request) and the closest real
name. Errors make the command exit with code 1; warnings do not, unless
`--strict` is given. Rules and settings: [docs/rules/](docs/rules/) and
[docs/configuration.md](docs/configuration.md).

| rule | finds | default |
|---|---|---|
| HS001 | a tool named in a form that can only be a tool, that the request does not have | on, warning |
| HS002 | a tool name that is almost right (case, `-`/`_`, prefix, suffix) | on, error |
| HS003 | an argument the tool's schema does not have | on, error |
| HS004 | an example value outside a field's allowed values | off |
| HS005 | example JSON keys that do not match the call or response format | off |
| HS007 | prose that may name a tool the request does not have | off |

## Benchmark

Measured once on the frozen configuration, on three sets that share no
project: dev (used to build the rules), holdout and holdout-2 (never used to
tune them). Recall is over real, fixed mismatches from the projects' history;
precision is over hand-labelled findings on the current code of open-source
agents. Small samples are marked by their size.

| | dev | holdout | holdout-2 |
|---|---|---|---|
| recall cases (projects) | 25 (12) | 14 (10) | 4 (4) |
| catchable cases caught by rules on by default | 8/11 | 4/11 | 2/2 |
| precision corpus: repositories (requests) | 4 (13) | 3 (12) | 4 (16) |
| HS001 precision | 13/14 | 0/24 | no findings |
| HS002 precision | 11/11 | no findings | 5/5 |
| HS003 precision | 2/2 | no findings | no findings |
| known mismatches in the corpus found | 22/32 | 6/24 | 6/17 |

The holdout HS001 findings are Code Mode function declarations of one project
(`tools.x({ ... })`), which HS001 reads as calls. Details and the method:
[benchmarks/README.md](benchmarks/README.md); run it with `make benchmark`.

## Known limitations

HS001 can warn about names that are not missing tools:

- **Code Mode declarations.** A description that declares functions as
  `tools.x({ ... })` is read as calls to `x`.
- **Negative forms.** "Never attempt 'apply_patch' ... they don't exist" is
  read as naming `apply_patch`.
- **Placeholders.** "I'm going to use `tool_name`" is read as naming
  `tool_name`.

HS001 findings are warnings, so they do not fail the check unless `--strict`
is used. To silence them, ignore the names or turn the rule off in
`handshake.yaml` (or `[tool.handshake]` in `pyproject.toml`):

```yaml
ignore:
  - tool_name                  # never report this name
  - {rule: HS001, ref: apply_patch}
rules:
  HS001: {enabled: false}      # or turn HS001 off
```

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

Apache 2.0. See [LICENSE](LICENSE).
