Metadata-Version: 2.4
Name: AgentFailureDiscovery
Version: 0.1.0
Summary: Agent Failure Discovery - Analyze agent transcripts to discover error patterns
Home-page: https://github.com/AmadeusITGroup/AgentFailureDiscovery
Author: DRIOUICH Ilias, Hongliu CAO
Author-email: Ilias Driouich <ilias.driouich@amadeus.com>, Hongliu Cao <hongliu.cao@amadeus.com>
License: Apache-2.0
Project-URL: Homepage, https://github.com/AmadeusITGroup/AgentFailureDiscovery
Project-URL: Repository, https://github.com/AmadeusITGroup/AgentFailureDiscovery
Project-URL: Bug Tracker, https://github.com/AmadeusITGroup/AgentFailureDiscovery/issues
Keywords: ai,llm,agent,error-analysis,evaluation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: openai>=1.70.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: httpx>=0.24.0
Provides-Extra: dev
Requires-Dist: pytest>=6.0.0; extra == "dev"
Requires-Dist: black>=22.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Dynamic: author
Dynamic: home-page
Dynamic: license-file
Dynamic: requires-python

# AgentFailureDiscovery

**Agent Failure Discovery** - A framework for analyzing AI agent transcripts to automatically discover error patterns and build comprehensive error taxonomies.

---

## Overview

This framework helps you identify, categorize, and track errors in AI agent behavior by analyzing interaction transcripts. It uses an LLM-driven discovery pipeline that:

1. **Analyzes agent transcripts** against policy documents and ground truth data
2. **Discovers new error types** when failures don't match existing patterns
3. **Builds error taxonomies** dynamically as more transcripts are processed
4. **Tracks error evolution** across your agent's development lifecycle

---

## Features

- **Automated Error Discovery**: LLM identifies failures and proposes new error types when patterns are meaningfully different
- **Dynamic Taxonomy Building**: Error registry grows organically as new failure modes are discovered
- **Policy-Aware Analysis**: Evaluates agent behavior against your specific policy documents
- **Flexible Dimensions**: Supports both pre-defined and custom error dimensions
- **Transparent Tracking**: Full audit trail of when and where each error type was discovered
- **JSON-Based**: Simple JSON input/output format for easy integration

---

## Installation

### From PyPI (once published)

```bash
pip install AgentFailureDiscovery
```

### From Source

```bash
git clone https://github.com/AmadeusITGroup/AgentFailureDiscovery.git
cd AgentFailureDiscovery
pip install -e .
```

---

## Quick Start

### 1. Set up your environment

Create a `.env` file with your Azure OpenAI credentials:

```bash
cp .env.example .env
```

Edit `.env` and add your credentials:

```env
AZURE_OPENAI_API_KEY=your-api-key-here
AZURE_OPENAI_ENDPOINT=https://your-resource-name.openai.azure.com
AZURE_OPENAI_API_VERSION=2024-05-01-preview
```

### 2. Prepare your data

Create a JSON file with your tasks and agent simulations:

```json
{
  "tasks": [
    {
      "id": "task_001",
      "description": "User requests account balance",
      "expected_behavior": "Agent checks balance using get_balance tool"
    }
  ],
  "simulations": [
    {
      "task_id": "task_001",
      "trial": 0,
      "transcript": {
        "turns": [
          {"role": "user", "content": "What's my balance?"},
          {"role": "assistant", "content": "Your balance is $500", "tool_calls": []}
        ]
      },
      "reward": 0.0
    }
  ]
}
```

### 3. Run the analysis

```bash
errordiscovery data.json --policy policy.md --output-dir ./results
```

### 4. Review results

The framework generates:
- `results/registry.json` - Error type taxonomy with all discovered patterns
- `results/run_output.json` - Detailed analysis results for each transcript

---

## Usage

### Command Line Interface

```bash
errordiscovery <json_file> [OPTIONS]
```

**Required Arguments:**
- `json_file` - Path to JSON file containing tasks and simulations

**Optional Arguments:**
- `--policy PATH` - Path to policy file (markdown or text)
- `--output-dir PATH` - Output directory for results (default: `./output`)
- `--model NAME` - LLM model deployment name (default: `gpt-5`)
- `--env-file PATH` - Path to .env file (default: `.env`)
- `--version` - Show version and exit

**Example:**

```bash
errordiscovery agent_transcripts.json \
    --policy my_policy.md \
    --output-dir ./analysis_results \
    --model gpt-5
```

### Python API

```python
from errordiscovery import run_discovery
from errordiscovery.utils import load_json_data, load_policy_file, save_registry
from dotenv import load_dotenv

# Load environment
load_dotenv()

# Load your data
tasks, simulations = load_json_data("data.json")
policy = load_policy_file("policy.md")

# Run discovery
output = run_discovery(
    policy=policy,
    tasks=tasks,
    simulations=simulations,
    model="gpt-5"
)

# Save results
save_registry(output["registry"], "registry.json")
print(output["registry_summary"])
```

---

## Input Format

### JSON Structure

Your input JSON should contain two main sections:

```json
{
  "tasks": [
    {
      "id": "unique_task_id",
      "description": "task description",
      "expected_behavior": "what should happen",
      "constraints": "any specific constraints",
      ...
    }
  ],
  "simulations": [
    {
      "task_id": "unique_task_id",
      "trial": 0,
      "transcript": {
        "turns": [
          {"role": "user", "content": "user message"},
          {"role": "assistant", "content": "agent response", "tool_calls": []}
        ]
      },
      "reward": 1.0
    }
  ]
}
```

### Policy File

A plain text or markdown file describing the rules and constraints your agent should follow:

```markdown
# Agent Policy

## Tool Usage
- Agents must verify account ownership before accessing balance
- All financial queries require authentication

## Communication
- Be concise and factual
- Do not make subjective comments
```

---

## Output Format

### Registry JSON

```json
{
  "error_types": {
    "DATA_HALLUCINATION": {
      "definition": "Stated facts that contradict or are absent from tool output",
      "dimension": "data_faithfulness",
      "is_new": false,
      "origin": null
    },
    "CUSTOM_ERROR_TYPE": {
      "definition": "New error discovered during analysis",
      "dimension": "new_dimension",
      "is_new": true,
      "origin": "task_042"
    }
  },
  "dimensions": {
    "data_faithfulness": {
      "question": "Did the agent accurately report data from tool outputs?",
      "fail_guidance": "Agent stated facts that contradict or don't exist in tool outputs",
      "pass_guidance": "Rounding ($99.99→$100); correct summarization",
      "is_new": false,
      "origin": null
    }
  },
  "discovery_log": [
    {"task_id": "task_042", "name": "CUSTOM_ERROR_TYPE"}
  ]
}
```

---

## Base Error Dimensions

The framework comes with 6 pre-seeded dimensions:

### Interaction Dimensions
- **user_intent_adherence** - Did the agent honor preferences and correctly read user input?
- **user_question_fulfillment** - Did the agent answer the user's direct questions?

### Integrity Dimensions
- **policy_violation** - Did the agent's actions and decisions follow policy?
- **policy_faithfulness** - Did the agent correctly state policy rules?
- **data_faithfulness** - Did the agent accurately report data from tool outputs?
- **tool_efficiency** - Were the agent's tool calls reasonable given available context?

New dimensions are discovered automatically when errors don't fit existing categories.

---

## Advanced Usage

### Custom Dimensions

Restrict analysis to specific dimensions:

```python
from errordiscovery.registry import INTERACTION_DIMS, INTEGRITY_DIMS

output = run_discovery(
    policy=policy,
    tasks=tasks,
    simulations=simulations,
    applicable_dims=INTERACTION_DIMS  # Only use interaction dimensions
)
```

### Ignore Specific Behaviors

Tell the analyzer to ignore known quirks:

```python
output = run_discovery(
    policy=policy,
    tasks=tasks,
    simulations=simulations,
    ignored_behaviors=[
        "concurrent tool calls (agent making more than one tool call in a single message)",
        "emoji usage in responses"
    ]
)
```

---

## Development

### Setup Development Environment

```bash
git clone https://github.com/AmadeusITGroup/AgentFailureDiscovery.git
cd AgentFailureDiscovery
pip install -e ".[dev]"
```

### Run Tests

```bash
pytest
```

### Code Formatting

```bash
black src/
ruff check src/
```

---

## How It Works

The error discovery pipeline:

1. **Initialize Registry** - Start with base error types and dimensions
2. **Analyze Transcripts** - For each transcript:
   - LLM identifies all failures
   - Maps failures to existing error types
   - Proposes new types when failures are meaningfully different
   - Records observations for borderline cases
3. **Update Registry** - New error types and dimensions are added dynamically
4. **Track Evolution** - Discovery log tracks when/where each type was found

The key innovation is the "relaxed gate" for new types: discoveries are made when failures have different **root causes, policy impacts, or user harms** - not just when they're superficially different.

---

## Contributing

Contributions are welcome! Please open an issue or submit a pull request.

---

## License

Apache License 2.0 - see [LICENSE](LICENSE) file for details.

---

## Citation

If you use this framework in your research, please cite:

```bibtex
@software{agentfailurediscovery,
  title = {AgentFailureDiscovery: Automated Error Discovery for AI Agents},
  author = {Driouich, Ilias and Cao, Hongliu},
  year = {2026},
  url = {https://github.com/AmadeusITGroup/AgentFailureDiscovery}
}
```

---

## Support

- **Issues**: [GitHub Issues](https://github.com/AmadeusITGroup/AgentFailureDiscovery/issues)
- **Discussions**: [GitHub Discussions](https://github.com/AmadeusITGroup/AgentFailureDiscovery/discussions)

---

## Acknowledgments

Built with inspiration from AI agent evaluation frameworks and error taxonomy research.
