Metadata-Version: 2.4
Name: agentgauntlet
Version: 0.2.0
Summary: Chaos engineering for LLM agents -- assume every agent is going to fail, and prove it.
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: rich>=13.0
Requires-Dist: requests>=2.28
Provides-Extra: httpx
Requires-Dist: httpx>=0.24; extra == "httpx"

# AgentGauntlet

**Every agent framework ships with a false sense of safety: pass the eval
once, ship it, forget it.** Real agents run in a world where context gets
dropped, tools time out, and APIs return garbage — continuously, not once.
AgentGauntlet treats that as the default assumption, not an edge case, and
gives you two ways to watch it happen to your own agent.

## Two ways to run it

**In-process (Python agents on `requests`/`httpx`)** — LangGraph, CrewAI,
AutoGen, or any custom Python agent that calls the OpenAI/Anthropic SDKs:

```python
import agentgauntlet
agentgauntlet.init(probability=0.1, frameworks=["openai", "requests"])
```

Two lines, before you build the graph/crew/agent. No other code changes.

**Proxy (any agent, any language)** — OpenClaw, Hermes Agent, or literally
anything with an `OPENAI_BASE_URL` / `ANTHROPIC_BASE_URL` override, which is
most agent runtimes regardless of what they're written in:

```bash
agentgauntlet proxy --upstream https://api.openai.com --port 8888
export OPENAI_BASE_URL=http://localhost:8888/v1
your-agent-here   # unmodified, any language
```

The proxy is the honest way to make good on "run this against whatever
agent you have" — the in-process patch only ever reaches Python code in the
same interpreter, so a Node.js or Go agent is invisible to it no matter how
the hooks are written.

## Injectors

| Injector    | Targets              | Sabotage                                                       |
|-------------|-----------------------|------------------------------------------------------------------|
| Amnesia     | LLM calls             | drops 10-30% of history, preserving the system prompt and current turn |
| Distractor  | LLM calls             | splices a contradictory instruction into the system prompt *or* the latest user turn |
| Gaslighter  | LLM/tool calls        | simulated timeout, `429`, or `503`                                |
| Mutator     | LLM/tool responses    | per-field probability of flipping booleans, shifting numbers, mangling keys — in proxy mode this includes corrupting `tool_call` arguments the model itself returned |

In proxy mode all four compete in a single pool on every call, since
everything reaching the proxy is presumed to be an LLM API call. In-process
mode splits them: Amnesia/Distractor target LLM calls, Gaslighter/Mutator
target tool calls. Either way, one intercepted call triggers **at most one**
event — the blast radius is one weighted random draw per call, not an
independent roll per injector.

## Measuring what actually matters: task success, not process survival

```python
with agentgauntlet.run():
    result = my_agent(user_query)
    if result.balance == expected_balance:
        agentgauntlet.mark_success()
    else:
        agentgauntlet.mark_failure("wrong balance reported")
```

If you never call `mark_success`/`mark_failure`, AgentGauntlet falls back to
crash-detection and labels the score `(unverified)` — a much weaker signal,
since an agent can process corrupted data, produce a wrong answer, and exit
cleanly. `examples/basic_agent.py --chaos` reproduces exactly that: the
Mutator corrupts an account balance, the agent reports the wrong number,
nothing crashes, and the scorecard still shows a failure because the
harness checks the actual answer.

```
🔥 CHAOS EVENT: MUTATOR 🧬 corrupted payload <- http://127.0.0.1:8931/tools/fetch_balance
...
        AgentGauntlet Post-Mortem Scorecard
┌────────────────────────────────┬──────────────────────────────────────────┐
│ Total Chaos Events Injected     │ 3                                        │
│ Task Outcome                    │ ❌ failure (reported 12505.0 vs 1250.5)   │
│ Resilience Score                │ 0%                                       │
└────────────────────────────────┴──────────────────────────────────────────┘
```

## Configuration

```python
agentgauntlet.init(
    probability=0.1,                      # shorthand: same weight for all four injectors
    blast_radius={"amnesia": 0.1, "distractor": 0.05, "gaslighter": 0.15, "mutator": 0.1},
    frameworks=["openai", "requests"],    # "httpx" also covers httpx.AsyncClient
    injectors=["amnesia", "gaslighter"],  # optional allow-list, default = all four
    targets=["api.mytools.com"],          # optional: restrict Gaslighter/Mutator to matching URLs
    seed=7,                               # reproducible runs
    timeout_range=(0, 30),                # Gaslighter's simulated timeout sleep, in seconds
    amnesia_strategy="random",            # or "oldest_first" for deterministic degradation
    mutation_rates={"boolean_flip": 0.5, "numeric_shift": 0.5, "key_mangle": 0.2},
)
```

```bash
agentgauntlet proxy --upstream https://api.openai.com --port 8888 \
    --probability 0.15 --seed 7 --timeout-min 0 --timeout-max 30
```

Values above ~0.25 on `--probability` mean chaos on essentially every call
through the proxy (all four injectors share one pool there) — a legitimate
"maximum chaos" demo setting, not a bug, but worth knowing before you crank
it to 1.0 for a screenshot and wonder why every single call got hit.

## How classification works, and its limits

A request body shaped like `{"messages": [...]}` or containing a `system`
key is treated as an LLM call; anything else is a tool call, restricted to
`targets` if you set one. This is a payload-shape heuristic, not real
provider/client identification — it can misclassify an unrelated API that
happens to send a `messages` field.

The proxy mode has its own, different limit: it only sees traffic an agent
is explicitly configured to send through it (the base_url override pattern).
It does **not** do transparent HTTPS interception via a system `HTTP_PROXY`
+ generated CA certificate — that would require installing a locally-trusted
root cert before anything works, which is exactly the setup friction that
kills a "point it and go" first impression. Real, stated limitation, not a
hidden one.

## Known limitations (being upfront about scope)

- **In-process mode is "any Python code on `requests`/`httpx`,"** not
  literally arbitrary tool mechanisms — subprocess tools, database drivers,
  and browser automation aren't touched, since there's no HTTP call to
  intercept in-process.
- **OpenClaw and Hermes Agent are standalone runtimes, not Python
  libraries** — the in-process patch cannot reach them at all; only proxy
  mode does, and only for their LLM API traffic specifically, not their
  internal tool/skill execution.
- **No integration tests against LangGraph/CrewAI/AutoGen/OpenClaw/Hermes**
  yet — this release has unit tests for the injectors and blast-radius
  logic (`tests/`) and a hand-run demo against a local mock server
  (`examples/`), not automated tests against real third-party frameworks.
- **Crash attribution** (`sys.excepthook`) blames whichever chaos event
  fired most recently before an unhandled exception — a heuristic, not a
  causal trace. `agentgauntlet.run()` + `mark_success`/`mark_failure` avoids
  needing it when the caller can check task correctness directly.

See `examples/basic_agent.py` (a real two-step tool-calling loop against a
local mock LLM server, `examples/mock_llm_server.py`) for a runnable
before/after demo, and `tests/test_injectors.py` for the injector and
blast-radius unit tests.
