Metadata-Version: 2.4
Name: swarmbreaker
Version: 0.1.0
Summary: Load- and chaos-testing framework for AI agent orchestration plumbing
Project-URL: Homepage, https://github.com/wczaja/swarmbreaker
Project-URL: Repository, https://github.com/wczaja/swarmbreaker
Project-URL: Issues, https://github.com/wczaja/swarmbreaker/issues
Project-URL: Specification, https://github.com/wczaja/swarmbreaker/blob/main/SPEC.md
Author: William Czaja
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai-agents,chaos-engineering,llm,load-testing,mcp,reliability
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: mcp>=1.20
Requires-Dist: numpy>=1.26
Requires-Dist: psutil>=5.9
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: starlette>=0.38
Requires-Dist: uvicorn>=0.30
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-timeout>=2.3; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: langgraph
Requires-Dist: langchain-mcp-adapters>=0.1; extra == 'langgraph'
Requires-Dist: langchain-openai>=0.2; extra == 'langgraph'
Requires-Dist: langgraph>=0.2; extra == 'langgraph'
Provides-Extra: openai-agents
Requires-Dist: openai-agents>=0.3; extra == 'openai-agents'
Provides-Extra: pydantic-ai
Requires-Dist: pydantic-ai-slim[mcp,openai]>=1.0; extra == 'pydantic-ai'
Description-Content-Type: text/markdown

# Swarmbreaker

Load- and chaos-testing for AI agent **orchestration plumbing** — transport, concurrency,
resource limits, retry/repair paths, context budgeting. Swarmbreaker drives a population of
heterogeneous synthetic agents against a large fabric of mock MCP tools, injects the
statistical pathology of real model output at the wire, and reports **where the plumbing
breaks and how** (knee points with confidence intervals, labeled by epistemic confidence).

It is deliberately **not** a model-quality benchmark. The mock is the measurement
instrument; a real LLM is only the calibration standard. Read [`SPEC.md`](SPEC.md) — the
design specification is the source of truth for everything here.

![swarmbreaker demo — animated terminal recording](docs/assets/demo.svg)

*A real recording, real time: [`scripts/render_demo_svg.py`](scripts/render_demo_svg.py)
replays an actual `swarmbreaker demo` run byte-for-byte, so this image cannot show output
the tool didn't produce.*

## Try it in 30 seconds

```bash
uvx --from git+https://github.com/wczaja/swarmbreaker swarmbreaker demo
```

No API keys, no Docker, no config. Two acts over the real pipeline: a compressed null-SUT
self-test measures the **rig ceiling** (SPEC §6.11 — an instrument that can't pass its own
polite baseline refuses to make claims), then a chaos scenario ramps open-loop load through
a §7 injector mix until an SLI knees — and it ends with the finding, labeled by epistemic
confidence and validity-guarded against that ceiling. A weekly
[public CI run](https://github.com/wczaja/swarmbreaker/actions/workflows/demo.yml) publishes
the same output in its job summary, so you can inspect a fresh run without installing
anything.

The chaos is declarative, and the demo hands you its own config to tinker with —
this scenario produced the finding on the right:

<table>
<tr>
<th><code>runs/demo/scenario.yaml</code> (written by the demo, abridged)</th>
<th>what it ends with</th>
</tr>
<tr>
<td>

```yaml
run:  { seed: 1729, duration_s: 15, time_scale: 0.05 }
scale: { agents: 16, tools: 128 }
load:
  arrival: { process: poisson, rate_per_s: 12 }
population:
  - { name: well_behaved, weight: 0.7 }
  - { name: chatty_high_fanout, weight: 0.2 }
  - { name: poison, weight: 0.1 }
brain:
  injectors:
    invalid_json_args:   { rate: 0.04 }
    near_miss_tool_name: { rate: 0.04 }
    truncated_stream:    { rate: 0.02 }
    repair_storm:        { rate: 0.01, burst: 6 }
scenario:
  - graduated_ramp: { shape: exponential, start: 0.08 }
  - rate_limit_cascade: { rps: 6, burst: 6 }
```

</td>
<td>

```text
finding  tool_err_rate knees at 3.3 tasks/s
         (95% CI 2.7–4.7 · two estimators agree)
         [projected × scripted]

         the 429 cascade — offered load crossed
         the shared 6 rps hot-tool bucket

         validity PASS — rig at 35% of its own
         ceiling; this behavior is the SUT's,
         not the rig's
```

…plus `findings.json` (the same record,
machine-readable, SPEC §11), `report.html`
(SLI-vs-load curves with knee CIs), and the
OTLP corpus for [docket](https://github.com/wczaja/docket)
triage.

</td>
</tr>
</table>

Swap the failure mode with `--scenario slow-loris` (hot tools hang with no server-side
timeout) or `--scenario herd` (barrier-released burst); edit the YAML and re-run it with
`swarmbreaker run` to go past the demo. `swarmbreaker demo --live` first fits the
injector rates, latency, and repair policy to a **real** model via the Phase-4 calibration
loop (local Ollama by default, or any OpenAI-compatible endpoint with one key) — the
same finding then honestly labels `calibrated` instead of `scripted`.

## Status

Pre-release. The SPEC.md §14 build sequence is complete through v1.x:

- [x] Phase 0 — skeleton + rig self-test
- [x] Phase 1 — injector pipeline (malformation taxonomy §7)
- [x] Phase 2 — scenario engine + chaos catalog (§8)
- [x] Phase 3 — telemetry, knee estimation, confidence labeling, reporting (v1 release line)
- [x] Phase 4 — calibration loop (v1.x)
- [x] Phase 5 — integration & docs (v1.x): [docket handoff](docs/docket-handoff.md),
      graph-pattern-hardened LangGraph adapter, Pydantic AI + OpenAI Agents SDK thin
      adapters (§16.4), [scaling guide](docs/scaling-guide.md)

## Beyond the demo

```bash
uv venv && . .venv/bin/activate
uv pip install -e ".[dev]"

swarmbreaker demo                            # the 30-second tour, from source
swarmbreaker selftest --preset laptop --out runs/selftest    # rig ceiling at laptop scale (§6.11)
swarmbreaker run examples/quickstart.yaml --out runs/quickstart   # a 60s knee-hunting sweep
open runs/quickstart/report.html
```

Every run directory contains `findings.json` (machine-readable, SPEC §11), `report.html`
(SLI-vs-load curves with knee CIs), `events-*.jsonl` (the raw corpus), and
`validity.json` (whether the rig had headroom — findings from a saturated rig are refused).

## Calibration (Phase 4)

Findings are labeled `scripted` until the injector parameters are fitted against a real
model (SPEC §11). Point the harness at any OpenAI-compatible upstream — a local model
(Tier 2: Ollama, llama.cpp, vLLM) or a hosted API (Tier 3):

```bash
# Tier 2 (local, free): discover + fit against e.g. Ollama
swarmbreaker calibrate --base-url http://localhost:11434/v1 --model llama3.2 \
    --tier local --sessions 60 --out calibration/llama3.2.json

# Tier 3 (live API): same command, your endpoint + OPENAI_API_KEY
swarmbreaker calibrate --base-url https://api.openai.com/v1 --model gpt-4.1-mini \
    --tier live --sessions 60 --out calibration/gpt41mini.json
```

Then reference it from a run config (`brain: { calibration: calibration/llama3.2.json }`).
Rates are fitted per §7 family, **stratified by context length × schema complexity**;
TTFT/token-rate and the repair policy are fitted too. Findings then carry per-family
"calibrated-against" stamps and flip to `calibrated` — until the stamps expire (default
90 days) or you hand-override a calibrated rate, at which point they honestly decay back
to `scripted`. Discovery runs (`brain: { tier: local }`) proxy the swarm through the real
model and write `discovery.json` — the shopping list for new scripted injectors.

Other useful commands: `swarmbreaker advise` (which failure modes your hardware can
reproduce, §6.7 — start with the [scaling guide](docs/scaling-guide.md)),
`swarmbreaker baseline` + `swarmbreaker gate` (statistical regression gating, §12 —
this repo's own CI uses them), and `swarmbreaker corpus` (validate/push the
[docket](https://github.com/wczaja/docket) triage corpus,
[docs/docket-handoff.md](docs/docket-handoff.md)).

## SUTs

The reference SUT is an unmodified LangGraph agent (`adapter: langgraph`; graph
patterns `react`, `supervisor`, `plan_execute`). Thin adapters for **Pydantic AI**
(`adapter: pydantic_ai`) and the **OpenAI Agents SDK** (`adapter: openai_agents`)
prove the core is orchestrator-agnostic (SPEC §16.4); each is an optional extra
(`pip install "swarmbreaker[langgraph]"`, `[pydantic-ai]`, `[openai-agents]`).
`adapter: null_sut` is the instrument's own baseline (§6.11).

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) — the short version: SPEC.md is the source
of truth, phase exits are executable tests, and all stimulus randomness must flow
through seeded substreams.

## License

Apache-2.0 — see [LICENSE](LICENSE).
