Metadata-Version: 2.5
Name: claque
Version: 0.0.1
Summary: A crowd of simulated users for your agent. Persona-driven conversation testing with calibrated LLM judging.
Project-URL: Homepage, https://github.com/pbeneteau/claque
Project-URL: Repository, https://github.com/pbeneteau/claque
Project-URL: Documentation, https://github.com/pbeneteau/claque/tree/main/docs
Project-URL: Issues, https://github.com/pbeneteau/claque/issues
Author: Paul Bénéteau
License: MIT
License-File: LICENSE
Keywords: agents,evaluation,llm,llm-as-judge,red-teaming,simulated-users,testing
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: pyyaml>=6.0
Provides-Extra: agent-sdk
Requires-Dist: claude-agent-sdk>=0.1.0; extra == 'agent-sdk'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.90; extra == 'anthropic'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Description-Content-Type: text/markdown

<div align="center">

# claque

**Stress-test your conversational AI agent with a crowd of simulated users, and a judge you can actually trust.**

[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/pbeneteau/claque/blob/main/LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://github.com/pbeneteau/claque/blob/main/pyproject.toml)
[![tests](https://img.shields.io/badge/tests-1400+%20offline-brightgreen.svg)](https://github.com/pbeneteau/claque/tree/main/tests)
[![docs](https://img.shields.io/badge/docs-quickstart-informational.svg)](https://github.com/pbeneteau/claque/blob/main/docs/quickstart.md)

![The claque live arena](https://raw.githubusercontent.com/pbeneteau/claque/main/docs/media/arena.gif)

<sub>A crowd of personas hammering an agent, graded live: solved, failed, or flagged unsafe.</sub>

</div>

---

Your demo shows the happy path. **claque shows you everything else.**

A *claque* was a crowd hired to react in 19th-century Paris theaters. This one is hired to break your agent: it runs a cast of persona-driven simulated users against it (the confused elder, the terse expert, the refund seeker, the prompt injector), holds a full multi-turn conversation with each, and grades every one with a calibrated LLM judge. Then it tells you where your agent fails, with a confidence interval on every number.

## Quickstart

```bash
pip install git+https://github.com/pbeneteau/claque.git
claque demo --offline      # zero API calls, writes a graded HTML report
```

That runs eight scripted personas against a deliberately mediocre demo bot and opens a self-contained report, no credentials needed. Prefer uv? `uv tool install git+https://github.com/pbeneteau/claque.git`. For live runs, add the `anthropic` extra and drop `--offline` (see the [full quickstart](https://github.com/pbeneteau/claque/blob/main/docs/quickstart.md)). Point it at your own agent in one line:

```bash
uv run claque run --agent https://your-api.com/chat --personas ./personas
```

## What you get

- **A hostile crowd, not an echo chamber.** Personas pin emotional state, literacy, patience, and a realistic way to give up, so a model playing "angry customer" does not thank your agent by turn three. Generate a stratified cast, or turn your own production logs into one.
- **A judge you can audit.** Anchored 0-3 rubrics as reviewable YAML, evidence quoted from the transcript, and `claque calibrate`: Cohen's kappa against your own blind labels. Every headline number carries a 95% interval and its sample size.
- **Words vs. deeds.** claque grades what the agent *did*, not just what it said. It catches **false successes**: the agent replies "your refund is processed" but never called the refund tool. A text-only grader cannot see that.
- **Grade what already happened.** `claque analyze` points the same calibrated judge at your agent's *real* past transcripts (a two-voice JSONL, or pulled from Intercom/Fin), clusters them by archetype, and reports each with a sample size and interval. Test before you ship, or measure what already shipped.
- **Plug in any agent.** A Python callable, an HTTP endpoint, an OpenAI-compatible server, MCP, A2A, submit-and-poll, or a Claude Agent SDK agent. The agent keeps its whole tool loop; claque only speaks user turns to it.
- **Red-team, not just QA.** An adversarial pack (prompt injection, secret extraction, a Crescendo attack, PII fishing) mapped to the OWASP LLM Top 10 and the Agentic Top 10, graded on a severity scale.
- **Built for real agents.** Per-turn timeouts, transient-error retries with backoff, rate limiting, `--resume` for a run that died halfway, and experienced-latency p50/p95, so you can point it at a production endpoint, not just a toy.
- **Surfaces that scale.** A live terminal arena you can watch, and a self-contained HTML report that stays readable from 8 to 1000 conversations (archetype rollup, failure clusters, latency).
- **Close the loop.** Export the failures to an SFT/DPO dataset, fine-tune, re-test, and prove the delta only counts when the intervals are disjoint.
- **Offline-first.** The whole test suite (1400+ tests) runs with no credentials, no network, no spend. Deterministic scripted mode is the CI default.

## Point it at your agent

An agent is anything that maps a conversation to its next reply. The `--agent` spec covers the common shapes without glue:

| Your agent is... | Spec |
|---|---|
| a deployed HTTP endpoint | `https://your-api.com/chat` |
| an OpenAI-compatible server (vLLM, Groq, Mistral...) | `openai:your-model` |
| Python code in your repo | `your_pkg.agents:support_agent` |
| a Claude Agent SDK agent | `agent-sdk:config.yaml` |
| MCP / A2A / async submit-and-poll | `mcp:` &nbsp; `a2a:` &nbsp; `async:` |

The config file, when there is one, is the agent's *address*, not the agent. See [docs/adapters.md](https://github.com/pbeneteau/claque/blob/main/docs/adapters.md).

## Why you can trust the numbers

Most LLM judges are vibes with a number attached. claque is built the other way around: it is presumed guilty until calibrated.

- **`claque calibrate`** samples a run, serves you the transcripts in a local blind UI (the judge's score is hidden from you), and computes Cohen's kappa per criterion. It never writes a label itself.
- **`claque judge selftest`** proves the judge is not fooled by content-free "master key" replies (a bare `:` fools some judges 35% of the time).
- Until the judge is calibrated, the docs tell you to trust deltas, not absolute scores. Out loud.

## Docs

[Quickstart](https://github.com/pbeneteau/claque/blob/main/docs/quickstart.md) · [Plug in your agent](https://github.com/pbeneteau/claque/blob/main/docs/adapters.md) · [Action verification](https://github.com/pbeneteau/claque/blob/main/docs/actions.md) · [Retrospective analysis](https://github.com/pbeneteau/claque/blob/main/docs/analyze.md) · [Calibrating the judge](https://github.com/pbeneteau/claque/blob/main/docs/calibrate.md) · [The safety pack](https://github.com/pbeneteau/claque/blob/main/docs/safety.md) · [Writing personas](https://github.com/pbeneteau/claque/blob/main/docs/personas.md) · [Running in CI](https://github.com/pbeneteau/claque/blob/main/docs/ci.md) · [Claude Code plugin](https://github.com/pbeneteau/claque/blob/main/docs/claude-code.md) · [Evaluation intelligence](https://github.com/pbeneteau/claque/blob/main/docs/evaluation-intelligence.md)

## License

MIT. Install from GitHub today (see Quickstart); a PyPI release lands with v1.
