Metadata-Version: 2.5
Name: agentfault
Version: 0.1.0
Summary: Agent Fault: open-source stateful reliability testing for AI agents
Project-URL: Homepage, https://github.com/danishmerani95/AgentFault
Project-URL: Repository, https://github.com/danishmerani95/AgentFault
Project-URL: Issues, https://github.com/danishmerani95/AgentFault/issues
Project-URL: Changelog, https://github.com/danishmerani95/AgentFault/blob/main/CHANGELOG.md
Author: Agent Fault Contributors
License: MIT License
        
        Copyright (c) 2026 danishmerani95
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: agent-reliability,agents,ai,evaluation,failure-injection,reliability,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.12
Requires-Dist: openai>=1.0
Requires-Dist: python-dotenv>=1.0
Provides-Extra: dev
Requires-Dist: mypy<2,>=1.17; extra == 'dev'
Requires-Dist: pytest<9,>=8.3; extra == 'dev'
Requires-Dist: ruff<0.13,>=0.12; extra == 'dev'
Description-Content-Type: text/markdown

# Agent Fault

**Break the world around your agent. Verify its behavior.**

Agent Fault tests tool-using AI agents against changing conditions: a price rises, inventory disappears, an account is suspended, or a tool fails while the agent is working.

It offers two APIs over the same stateful testing engine.

## 1. Benchmark API

**Use it to compare an agent against a reusable, versioned test suite.**

The built-in `buying-agent@1.0.0` benchmark contains five deterministic scenarios covering:

- Buying an eligible offer within budget.
- Price increases before validation.
- Inventory disappearing before cart creation.
- Price increases after an item enters the cart.
- Inventory disappearing after an item enters the cart.

A benchmark returns a pass percentage, per-case results, failed assertions, and observed tool calls. If any case has an execution error, the headline score is withheld rather than treating an invalid run as a pass or failure.

The API accepts an agent factory that implements AgentFault's `Agent` contract. OpenAI has a built-in adapter; other frameworks can provide an adapter for their tool-call loop.

Run the OpenAI example (requires an OpenAI API key; all tools still use a synthetic test environment):

```bash
uv sync --extra dev
export OPENAI_API_KEY=...
uv run python -m examples.openai_benchmark
```

The example is in [`examples/openai_benchmark.py`](examples/openai_benchmark.py). It prints a JSON report and can be adapted to use your own prompts, model, and agent.

The [LangChain benchmark example](examples/langchain_benchmark.py) uses LangChain's `ChatOpenAI` and `bind_tools` APIs with the same versioned buying benchmark. AgentFault deliberately executes one tool call at a time, so the model's next decision sees the result and any state change introduced by the scenario. It requires an OpenAI API key and keeps all tool execution in the synthetic test world:

```bash
export OPENAI_API_KEY=...
uv run --with "langchain-openai>=0.3,<2" python -m examples.langchain_benchmark
```

This is a LangChain chat-model adapter, not a wrapper around an arbitrary LangChain `AgentExecutor`; the AgentFault runner owns the tool loop to make stateful interventions deterministic.

## 2. Custom Scenario API

**Use it to test a failure mode specific to your application.**

A scenario defines your agent, isolated environment and tools, an optional perturbation, and assertions about acceptable behavior. Use it for buying, support, research, operations, or any workflow where an agent's actions depend on tool results or changing state.

For an agent that implements `Agent`, compose a `Scenario` and run it with `ScenarioRunner`:

```python
from agentfault.testing import Scenario, ScenarioRunner
from agentfault.testing.assertions import MaxPriceAssertion
from agentfault.testing.perturbations import ChangePrice
from agentfault.testing.triggers import AfterTool
from agentfault.tools import ToolRegistry

# Supply these from your application or a test fixture:
# agent_under_test, test_environment, and the Tool objects.
tool_registry = ToolRegistry(test_tools)

scenario = Scenario(
    name="price changes during checkout",
    agent=agent_under_test,
    environment=test_environment,
    perturbations=[
        (AfterTool("get_offers_for_product"),
         ChangePrice("offer_sony_1", new_price=179.0)),
    ],
    assertions=[MaxPriceAssertion(max_price=150.0)],
)
result = ScenarioRunner(tool_registry).run(scenario)
assert result.passed, [item.message for item in result.assertions if not item.passed]
```

For a graph that owns its own decision loop, use its integration runner instead of wrapping the entire graph as a generic `Agent`.

Run the LangGraph example (requires LangGraph; no model API key is needed):

```bash
uv run --with "langgraph>=1.0,<2" python -m examples.langgraph_integration
```

The [LangGraph example](examples/langgraph_integration.py) uses a real `StateGraph` and `ToolNode`, a scripted decision node, and a fake account/refund world. It shows a custom scenario that suspends an account after lookup and checks that no refund tool is called. The current LangGraph integration supports synchronous `invoke`, sequential calls routed through the configured `ToolNode`; async execution, parallel calls, and other tool paths are not yet supported.

## Use cases

- **Buying agents:** stop unsafe purchases when prices or inventory change.
- **Support agents:** verify permissions and account state before refunds or account changes.
- **Search and research agents:** test stale, missing, or contradictory information.
- **Operations and workflow agents:** inject tool failures and check recovery or safe stopping.

The built-in benchmark currently covers buying. Other roles can be tested today with custom scenarios; additional benchmark packs can use the same versioned suite API.

## Safety and scope

Use fixtures, fake services, or an isolated sandbox. AgentFault does not make production tools safe automatically. Only actions routed through the configured executor/interceptor are recorded and can be perturbed.

Project status: **alpha**. APIs may change. See [custom adapter notes](docs/custom-adapters.md), [contributing](CONTRIBUTING.md), and the [changelog](CHANGELOG.md).
