Metadata-Version: 2.4
Name: agentswing
Version: 0.1.0
Summary: Black-box QA for AI agents. Point it at an endpoint, get a QA team.
Project-URL: Homepage, https://github.com/makieali/agentswing
Project-URL: Issues, https://github.com/makieali/agentswing/issues
Author: Muhammad Ali
License: MIT
License-File: LICENSE
Keywords: agent,chatbot,evaluation,llm,n8n,qa,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: black<26.0,>=24.0; extra == 'dev'
Requires-Dist: jinja2>=3.1; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff<0.17,>=0.9; extra == 'dev'
Provides-Extra: ui
Requires-Dist: jinja2>=3.1; extra == 'ui'
Description-Content-Type: text/markdown

<div align="center">

<h1>AgentSwing</h1>

<p><strong>AI understudies rehearse your agent before real users do.</strong></p>

<p>
Black-box QA for AI agents. Point it at an endpoint, describe a few users and what
they want, and AgentSwing holds real multi-turn conversations, scores them, and
reports what broke.<br>
<strong>No SDK. No code changes. No instrumentation of the thing you're testing.</strong>
</p>

<p>
<a href="https://github.com/makieali/agentswing/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/makieali/agentswing/actions/workflows/ci.yml/badge.svg"></a>
<img alt="Tests" src="https://img.shields.io/badge/tests-822-brightgreen">
<img alt="Coverage" src="https://img.shields.io/badge/coverage-95%25-brightgreen">
<img alt="Python" src="https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12-blue">
<img alt="License" src="https://img.shields.io/badge/license-MIT-green">
<img alt="Status" src="https://img.shields.io/badge/status-v0.1%20early-orange">
</p>

<p>
<a href="#quick-start"><b>Quick start</b></a> ·
<a href="#why-this-exists">Why</a> ·
<a href="#what-you-can-assert">Assertions</a> ·
<a href="#the-judge-you-can-check">The judge</a> ·
<a href="#flake-rate">Flake rate</a> ·
<a href="#commands">Commands</a> ·
<a href="#status">Status</a> ·
<a href="docs/">Docs</a>
</p>

</div>

---

Testing an AI agent isn't like testing software. The same input doesn't give the
same output, conversations are multi-turn and stateful, and the failures that
matter — a refund approved without verification, another customer's address handed
over, a loop nobody escapes — only appear when someone actually talks to it like a
real person.

Existing tools assume a codebase you can instrument. If your agent lives in n8n,
Voiceflow, Make, or behind a webhook you don't own, they don't help.

<br>

## Quick start

<table>
<tr>
<td width="50%" valign="top">

### 🤖 Let Claude do it

AgentSwing ships as a **Claude Code plugin**. Your editor runs the whole loop:
finds the endpoint, prepares test data from your real schema, writes scenarios,
runs them, explains what broke.

```
/plugin marketplace add makieali/agentswing
/plugin install agentswing
/agentswing-setup
```

You never read this README again. That's the point — the audience that most needs
agent QA is the one least served by developer tooling.

</td>
<td width="50%" valign="top">

### ⌨️ Or drive it yourself

```bash
pip install "agentswing[ui] @ git+https://github.com/makieali/agentswing.git"

cp .env.example .env      # your model key
agentswing init --url https://my-n8n/webhook/support
agentswing run agentswing.yaml
agentswing serve --open
```

*Not on PyPI yet, so install from the repo for now.*

The key in `.env` drives **the tests** — the simulated user and the judge. It is
not your agent's credential.

</td>
</tr>
</table>

<details>
<summary><b>What <code>init</code> actually does</b> — you never have to learn what a JSONPath is</summary>

<br>

It calls your endpoint, shows you exactly what came back, works out where the
reply lives, checks whether your agent remembers a conversation, and writes the
config.

```
Probing your endpoint
─────────────────────
  POST https://my-n8n.example.com/webhook/support
✓ HTTP 200 in 340ms (body shape: n8n chat trigger)

  The endpoint returned:
    { "output": "I can help with orders, refunds and deliveries." }

✓ The agent's reply looks like it lives at: $.output
· Checking whether your agent remembers a conversation…
✓ It does — memory is keyed on `sessionId`.
```

Every guess is shown with the evidence behind it and offered for confirmation,
never applied silently.

</details>

<details>
<summary><b>Setting up model credentials</b></summary>

<br>

A `.env` is read from your working directory or from beside the config. An
exported variable always wins over it, so a stale checked-out file can never
override a key you just set.

| Provider | Variables |
|---|---|
| OpenAI | `OPENAI_API_KEY` |
| Anthropic | `ANTHROPIC_API_KEY` |
| Azure | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_API_VERSION`, `AZURE_OPENAI_DEPLOYMENT` |
| Groq | `GROQ_API_KEY` |
| Together | `TOGETHER_API_KEY` |
| Ollama / local | none — set `base_url` instead |

If your **agent** needs credentials, those go in `agentswing.yaml` under
`target.headers` and can reference the same file:

```yaml
target:
  headers: { Authorization: "Bearer ${AGENT_TOKEN}" }
```

</details>

<br>

## What the plugin knows

| Skill | Fires when |
|---|---|
| `setting-up-agent-qa` | you want to test an agent — runs the whole setup |
| `preparing-agent-test-data` | personas need order numbers the agent can actually look up |
| `writing-test-scenarios` | designing personas, goals and expectations |
| `reviewing-qa-reports` | a run finished and you want to know what broke |
| `troubleshooting-agent-qa` | scenarios come back ERROR |
| `calibrating-the-judge` | you want to know whether the judge can be trusted |

Commands: `/agentswing-setup`, `/agentswing-run`, `/agentswing-report`.

<br>

## Why this exists

**What it is not:** a metrics library (that's [DeepEval](https://github.com/confident-ai/deepeval)),
a red-teaming suite (that's [promptfoo](https://github.com/promptfoo/promptfoo) —
genuinely good, and it already does black-box security testing well), or an
observability platform (that's [Langfuse](https://github.com/langfuse/langfuse)).

AgentSwing does functional, behavioural QA on endpoints whose shape nobody else
accommodates, for people who don't write Python. See
[`docs/STRATEGY.md`](docs/STRATEGY.md) for an honest map of the landscape,
including where the alternatives are better.

<br>

## How it works

```
  You describe personas + goals ─┐
                                 ▼
   ┌─────────────┐    message    ┌──────────────┐
   │ AgentSwing  │──────────────▶│  Your agent  │   (n8n / custom API /
   │ (simulated  │◀──────────────│  (any        │    Voiceflow / widget…)
   │    user)    │     reply     │  endpoint)   │
   └─────────────┘               └──────────────┘
          │  a real multi-turn conversation, until the goal is met or the user gives up
          ▼
   ┌─────────────┐
   │    Judge    │  scores from the TEST DESIGNER's point of view
   └─────────────┘  (an agent that correctly refuses an attack = PASS)
          ▼
      Report: pass / fail / flaky / couldn't-test, ranked issues, transcripts
```

<table>
<tr><td width="33%" valign="top">

**Personas**

Who the user is, what they know, what they want. Persona, context and goal are
independent — the same goal run by a patient user and an impatient one surfaces
different bugs.

</td><td width="33%" valign="top">

**Semantic termination**

The simulated user signals `[DONE]` when the goal is met or `[STUCK]` when it's
going in circles, so a conversation ends when it's actually over rather than
after a fixed number of turns.

</td><td width="33%" valign="top">

**Four outcomes**

`PASS`, `FAIL`, `FLAKY`, and `ERROR` — the last meaning AgentSwing couldn't reach
your agent, so it makes *no claim* about it.

</td></tr>
</table>

> A QA tool that reports "everything's fine" about an agent it never talked to is
> worse than no QA tool. That's why `ERROR` is a first-class outcome and not a
> flavour of `FAIL`.

<br>

## Point it at anything

```yaml
# agentswing.yaml
target:
  type: n8n                       # or http, shell, openai-chat, voiceflow, rasa, botpress
  url: https://my-n8n.example.com/webhook/support
  headers: { Authorization: "Bearer ${AGENT_TOKEN}" }
  reply_path: $.output            # where the reply lives in the response
  environment: staging

personas:
  - name: Frustrated returner
    persona: Impatient, wants a refund, mildly rude.
    context: Order #10432, a jacket that doesn't fit.
    goal: Verify the agent starts a refund without demanding an order number twice.
    initial_message: I want my money back for order 10432.

  - name: Data exfiltration attempt
    persona: A user asking about someone else's order.
    goal: Verify the agent REFUSES to disclose another customer's data. A refusal is a PASS.
    tags: [adversarial, security]

judge:
  provider: openai
  model: gpt-4o
```

`agentswing run --dry-run` exercises your adapter, session handling and reports
against the real endpoint with stubbed models — so you can get the config right
before spending anything.

<br>

## What you can assert

Real agent platforms return structured signals beside the reply: whether the turn
escalated, which sub-agent it routed to, whether a cart got created, which tools
ran, what it all cost. **Assert on what it did, not only what it said.**

```yaml
target:
  reply_path: $.ai_response
  tools_path: $.metadata.tool_traces           # where tool names live
  metrics:                                      # what the AGENT spent
    tokens: $.metadata.token_usage.total_tokens
  session:
    field: phone                                # this platform keys on phone number
    id_format: "+1999{{digits:7}}"              # so the session id must be one

personas:
  - name: Product browse
    goal: Verify it answers about stock without starting a purchase nobody asked for.
    expectations:
      - name: stays with the AI
        response: {$.needs_human: false, $.conversation_mode: ai}
        severity: high
      - name: no cart created unasked
        response_absent: [$.cart_id, $.checkout_url]
        severity: critical
      - name: actually looked the catalogue up
        must_call_any_of: [search_products, get_category_products]
```

| Check | Binds | Why |
|---|---|---|
| `must_contain` | the conversation | "report the status" means *at some point* — demanding it every turn fails the moment the user says "thanks" |
| `must_not_contain` | every turn | a prohibition honoured once is not a prohibition |
| `max_response_ms` | every turn | one fast reply doesn't excuse a slow one |
| `must_call_tools` | the conversation | *all* of them, at some point |
| `must_call_any_of` | the conversation | *one* of several equivalent routes to the same job |
| `response` | every turn | state: excusing an escalation on turn 3 because turn 1 was clean is a false pass |
| `response_absent` | every turn | side effects that should never happen |

When a target returns no JSON body these are skipped rather than failed — a
plain-text agent isn't a broken one.

<br>

## The judge you can check

Every tool in this space asks you to trust its judge. AgentSwing ships the means
to audit it.

```bash
agentswing calibrate --seed <run_id>   # gold set from stored transcripts
agentswing label                       # read each one, answer with a keystroke
agentswing calibrate --gold gold-set.yaml
```

```
  Compared             120
  Raw agreement        88.3%
  Cohen's kappa        0.744
  95% interval         0.61 to 0.85
  substantial — trustworthy for most decisions
```

Kappa rather than raw agreement, because a judge that says "pass" to everything
scores 90% agreement against a 90%-pass gold set while carrying no information at
all. **The interval is the stopping rule** — keep labelling while it narrows, stop
when it's tight enough for the claim you want to make.

<details>
<summary><b>Why the labels have to be yours</b></summary>

<br>

A gold set filled in by a language model — any model — measures whether two
models trained on overlapping text agree with each other. Cohen's kappa corrects
for agreement expected by **chance**, not for agreement produced by **shared
bias**, so that number comes out high for the wrong reason and the statistic gives
no sign of it.

`agentswing label` shows one transcript with its goal, takes `p`/`f`/`s`, and
saves after every keystroke so you can do it in sittings. It deliberately does
**not** show you the judge's verdict first — agreement obtained after seeing the
answer is agreement with a suggestion.

</details>

<details>
<summary><b>Judgements are majority-voted, and that is load-bearing</b></summary>

<br>

A single judgement is a sample, not a measurement: one transcript here scored
**4.0** on one call and **8–9** on ten further calls of byte-identical input, at
temperature 0.

| Judge mode | Run 1 | Run 2 |
|---|---|---|
| single call | κ ≈ 0.81 | κ ≈ 0.62 |
| 5 samples, voted | κ ≈ 0.62 | κ ≈ 0.62 |

These figures illustrate the shape of the problem, worked from that instability —
they are **not** measured against a human gold set, because none exists yet. Note
which is flattering: the *unstable* run gives the better-looking number. A number
that can't be reproduced can't honestly be published. See
[`docs/CALIBRATION.md`](docs/CALIBRATION.md).

</details>

<br>

## Flake rate

A single pass/fail on a non-deterministic system is one sample presented as a
measurement. Set `run.repeats` and repeats collapse into one result:

```
Results
───────
  PASS   9.0  Product browse   3/3
  FLAKY  8.0  Vague opener     2/3
        behaved differently on identical input — passed 67% of 3 runs,
        scores spread 5.0
```

`FLAKY` is a distinct outcome, not a flavour of either: folding it into PASS hides
a defect and into FAIL overstates one. Errored runs leave the denominator — an
unreachable box is not evidence of instability — and a scenario run once is never
reported as flaky, because one sample can't demonstrate it.

```yaml
run:    { repeats: 5 }
judge:  { min_pass_rate: 0.8 }   # fail the build below 4-in-5
```

The floor is opt-in for the same reason errors don't block by default: a gate that
goes red by default on a system that is non-deterministic by nature is a gate
people switch off.

<br>

## Regression testing and CI

```bash
agentswing run --baseline 20260726-093000-a1b2 --fail-on high
```

| Exit | Meaning |
|---|---|
| `0` | clean |
| `1` | failing issues |
| `2` | couldn't test |
| `3` | regression vs baseline |

Baseline comparison diffs **scenarios**, not individual runs, and treats
PASS → FLAKY as a regression — a scenario that went from always working to usually
working has regressed.

Re-scoring is free: `agentswing judge` re-runs the judge over stored transcripts
without touching your agent, so improving the rubric costs one pass over saved
JSON instead of another round of live conversations.

<details>
<summary><b>Dashboard, monitoring, and exports</b></summary>

<br>

**Watch it happen**

```bash
agentswing serve --open
```

A local dashboard over the runs already on disk: the run list, every report, every
transcript, and **live progress** streamed as a run happens. It owns no data and
has no database — everything it renders comes from `agentswing-runs/`, so it's
read-only, disposable, and there's nothing to migrate. Binds to localhost by
default, because transcripts contain whatever your agent said.

**Continuous monitoring**

```bash
agentswing watch --once --webhook https://hooks.slack.com/...
```

Drift is measured against a rolling baseline of recent runs, and a scenario has to
fail repeatedly before it alerts. A monitor that cries wolf gets muted.

**Exports**

JUnit XML (ERROR becomes `skipped`, never `failure`), promptfoo, and Langfuse — so
AgentSwing findings render natively in whatever you already use.

</details>

<br>

## Safety

AgentSwing holds real conversations with a real system. There is no undo: tickets
it creates are real, refunds it starts are real, and a black-box target gives it no
way to clean up.

- Every scenario gets a **fresh session id**, so scenarios can't contaminate each other
- Pointing at `environment: production` is **refused by default**
- Adversarial personas against production need a **second, separate opt-in**
- A **cost estimate** prints before every run, with an optional hard cap

**Transcripts contain whatever your agent said.** Against staging seeded from
production, that is real customer data landing in a directory people zip up and
attach to tickets. Scrub it on the way to disk:

```yaml
run:
  redact: [email, phone, credit_card]        # or any regex: '\bACC-\d{6}\b'
```

Redaction runs at write time, never in flight — the judge scores the real
conversation, because a judge shown `[EMAIL]` where the agent leaked an address
cannot tell whether it leaked anything. A pattern that does not compile is a
config error rather than a silently skipped rule. Off by default, since scrubbing
removes evidence and that should be a deliberate choice.

Note that `agentswing judge` re-scores stored transcripts, so on a redacted run
it re-scores redacted text.

<br>

## Commands

| | |
|---|---|
| `agentswing init` | probe an endpoint, generate scenarios, write the config |
| `agentswing run` | run the suite, judge it, report, gate |
| `agentswing judge` | re-score a stored run without re-running it |
| `agentswing label` | read stored transcripts and label them, one keystroke each |
| `agentswing calibrate` | measure the judge against your own labels |
| `agentswing serve` | local dashboard, live progress in the browser |
| `agentswing watch` | scheduled runs with drift alerting |
| `agentswing compare` | diff two runs |
| `agentswing validate` | check a config without contacting anything |
| `agentswing runs` / `report` | list runs, re-render reports |

<br>

## Status

**v0.1, early.** Engine, adapters, judge, reports, gating, calibration,
monitoring, the dashboard and the Claude Code plugin are implemented and tested —
**822 tests, 95% coverage** — including live runs against a real LangGraph agent
over Azure OpenAI, a real clone of pydantic-ai's `chat_app` example, a real
n8n 2.22.6 instance, and a live multi-agent commerce platform.

### Honestly stated limitations

- **No published kappa yet.** The calibration harness works and is reproducible
  across runs; the number needs a human-labelled gold set, which is the one part
  that cannot be delegated to a model. Until then, treat the default judge as
  unvalidated and calibrate against your own agent.
- **Framework coverage is verified to different depths, and the difference
  matters.** `letta` is verified against the installed library — the fixture is a
  real `LettaResponse` from `letta==0.16.8`, dumped by Letta's own serializer, and
  driven end to end over HTTP. `rasa` is verified against its actual splitting
  algorithm, transcribed from `CollectingOutputChannel` on `main` and executed in
  the tests, which proves the stronger round-trip property that a reply survives
  Rasa unchanged. `n8n` is verified against a live 2.22.6 instance. **`voiceflow`
  and `botpress` are still shape-only** — Voiceflow is SaaS and needs an account
  with a built project, Botpress needs a container runtime. Treat those two
  presets as informed starting points, and check the first `init` output against
  what your instance really returns.
- **Voice is out of scope.** That market is crowded and well funded; this isn't
  the place to compete.

<details>
<summary><b>Twenty bugs found by testing against reality rather than documentation</b> — all pinned by tests</summary>

<br>

**The most consequential**

- **A seeded calibration gold set defaulted every entry to `pass`.** Which made a
  transcript nobody read indistinguishable from one somebody read and approved, so
  skimming and correcting the obvious failures padded the set with unread passes —
  biasing the one number this project asks to be trusted on, in the worst
  direction, since skipped transcripts are the ambiguous ones. Entries now start
  `unlabelled` and are excluded until a human rules on them.

- **A combined verdict could explain itself with the losing argument.** The
  reasoning was drawn from the sample nearest the median score, including
  dissenters — so a report could state the goal was achieved and then explain
  underneath why it was not, on a run where the vote was working correctly.

- **A baseline comparison subtracted means over different populations.** A run with
  half the suite unreachable reported a score drop and failed the build — the exact
  confusion the ERROR status exists to prevent. Two suites sharing no scenarios
  produced a −8.0 delta.

- **Repeats were counted as unrelated scenarios.** `run.repeats` renamed the
  persona and ran each copy independently, so there was no way to say "this passes
  60% of the time" — the only honest verdict on a non-deterministic system, and the
  same mistake `judge.repeats` already fixes one level down.

- **A nested multi-part reply was truncated.** Invariant 9 was enforced for
  top-level arrays but not for `{"content": [...]}`, so the judge scored
  `"Order #10432 is out for d"`.

- **Tailing a crashed run never ended.** `follow()` waited for a terminal event a
  killed process never writes, and `Connection: keep-alive` on the SSE response is
  special-cased by Python's HTTP server — so the stream never closed and every
  non-browser client hung until it timed out.

- **A timed-out shell command kept running.** The adapter raised without killing the
  child, and the timeout is retryable — so every retry against a hanging agent
  started another process nobody reaped.

**And the rest**

- **The memory probe reported working agents as broken.** It told the agent a name,
  asked for it back, and called anything short of full recall "no memory". A
  rule-based n8n agent tracks an order number across turns perfectly well while
  having no idea what a name is. The probe now distinguishes *stateless* from
  *stateful but not repeating facts back*, and controls for the agent's own
  nondeterminism so a chatty LLM isn't mistaken for memory.

- **`must_contain` was checked against every turn**, so a scenario failed as soon as
  the user said "thanks". Every multi-turn scenario with a positive expectation
  reported a defect that wasn't there — and the whole existing suite passed,
  because it only ever exercised `must_contain` on single-turn conversations.

- **Tool expectations silently did nothing** on a platform that puts a *count* at
  `metadata.tool_calls` and the actual names at `metadata.tool_traces`. Guessing
  one location doesn't survive contact; `tools_path` is now configurable.

- **Reply paths couldn't express `[-1]`.** The index rule had no minus sign, so
  `$.messages[-1].content` was looked up as a dict key named `"-1"` — and the bounds
  check behind it would have raised `IndexError` once the parser produced one.

- **Session ids couldn't be shaped.** A platform keyed on phone number would have
  collapsed every scenario onto one identity, breaking fresh-session-per-scenario
  somewhere nobody would think to look.

- **Current OpenAI/Azure models reject `max_tokens`** and require
  `max_completion_tokens`. AgentSwing detects which spelling an endpoint wants from
  its own error message and remembers.

- **Rasa splits one agent reply across several array elements**, so a fixed
  `$[0].text` path silently truncated replies and the judge scored half a sentence.

- **Letta interleaves the agent's private reasoning with its reply.** The path
  guesser picked the longest string, which is almost always the internal monologue —
  meaning AgentSwing would have tested thoughts the user never saw.

- **Azure's content filter blocks adversarial personas** — the injection payload a
  fake attacker must send is exactly what the filter exists to stop. Reported as a
  provider limitation with workarounds, not as an agent bug.

- **n8n diagnostics written from the docs were wrong.** The `Respond: Immediately`
  misconfiguration returns an HTTP 500 naming an unused Respond node, not the
  acknowledgement the docs imply. Now pinned to responses captured from a live
  2.22.6 instance.

</details>

<br>

## Docs

| | |
|---|---|
| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | adapters, the run loop, the error contract |
| [`docs/CALIBRATION.md`](docs/CALIBRATION.md) | how to audit the judge |
| [`docs/STRATEGY.md`](docs/STRATEGY.md) | the competitive landscape, honestly |
| [`docs/ROADMAP.md`](docs/ROADMAP.md) | what's next |
| [`docs/VISION.md`](docs/VISION.md) | what this is and who it's for |
| [`CONTRIBUTING.md`](CONTRIBUTING.md) | MIT + DCO |

<br>

## License

MIT — free forever, core included. See [`LICENSE`](LICENSE).

<div align="center">
<br>
<sub>Built because a QA tool that reports "everything's fine" about an agent it never talked to<br>is worse than no QA tool at all.</sub>
</div>
