Metadata-Version: 2.4
Name: agent-trace-eval
Version: 0.1.0
Summary: Golden trace regression evaluation for tool-using AI agents
Project-URL: Homepage, https://github.com/yfccyf/agent-trace-eval
Project-URL: Documentation, https://github.com/yfccyf/agent-trace-eval#readme
Project-URL: Repository, https://github.com/yfccyf/agent-trace-eval
Author: Yifan Cheng
License: MIT
License-File: LICENSE
Keywords: ai-agents,evaluation,observability,regression-testing,tool-calls
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# agent-trace-eval

[![CI](https://github.com/yfccyf/agent-trace-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/yfccyf/agent-trace-eval/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/agent-trace-eval)](https://pypi.org/project/agent-trace-eval/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Golden trace regression evaluation for tool-using AI agents.

`agent-trace-eval` checks agent execution traces against declarative expectations for:

- tool selection (required / forbidden tools)
- tool-call arguments
- tool-call ordering
- multi-agent handoffs
- recovery decisions (retry / fallback / escalate)

It is designed as a small, employer-neutral library you can use in CI to gate agent workflow changes.

## Install

```bash
pip install agent-trace-eval
```

For local development:

```bash
pip install -e ".[dev]"
```

## Quick start

Run the bundled examples:

```bash
agent-trace-eval \
  --cases examples/cases \
  --traces-dir examples/traces \
  --report reports/example-report.md
```

## Trace format

Traces are JSON or YAML documents with an `events` list. Supported event types include:

- `tool_call`
- `handoff`
- `recovery_decision`
- `final_answer`

Example:

```json
{
  "case_id": "refund_lookup",
  "events": [
    {
      "type": "tool_call",
      "name": "lookup_order",
      "arguments": { "order_id": "12345" }
    },
    {
      "type": "tool_call",
      "name": "issue_refund",
      "arguments": { "order_id": "12345" }
    }
  ]
}
```

## Case format

Cases are YAML or JSON files with `id`, `description`, `input`, and `expect` sections:

```yaml
id: refund_lookup
description: Agent should look up an order before issuing a refund.
expect:
  tools:
    required: [lookup_order, issue_refund]
    forbidden: [delete_account]
  ordering:
    before:
      - first: lookup_order
        second: issue_refund
  arguments:
    issue_refund:
      order_id: "12345"
```

## Python API

```python
from agent_trace_eval import RegressionRunner, render_markdown_report
from agent_trace_eval.loader import load_case, load_trace
from agent_trace_eval.result import SuiteResult

case = load_case("examples/cases/refund_lookup.yaml")
trace = load_trace("examples/traces/refund_lookup.json")

runner = RegressionRunner()
result = runner.run_case(case, trace)
report = render_markdown_report(SuiteResult(case_results=[result]))
print(report)
```

## Related writing

This project complements a series on agent regression testing and release gates:

- [Final Answers Are Not Enough (Part 1)](https://yfccyf.github.io/writing/final-answers-are-not-enough-golden-trace-regression-testing-part-1/)
- [From Golden Traces to Release Gates (Part 2)](https://yfccyf.github.io/writing/from-golden-traces-to-release-gates-building-an-agent-regression-harness-part-2/)

## Development

```bash
pytest
```

## License

MIT
