Metadata-Version: 2.4
Name: agentregress
Version: 0.1.1
Summary: Regression testing for AI agents. Turn production failures into automated tests.
Project-URL: Homepage, https://github.com/agentregress/agentregress
Project-URL: Repository, https://github.com/agentregress/agentregress
Project-URL: Bug Tracker, https://github.com/agentregress/agentregress/issues
Author: AgentRegress Contributors
License: MIT License
        
        Copyright (c) 2026 AgentProbe
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: agents,ai,ci,evaluation,llm,regression,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: anthropic>=0.40.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: openai>=1.50.0
Requires-Dist: pydantic<3,>=2.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: ruamel-yaml>=0.18.0
Requires-Dist: typer>=0.9.0
Provides-Extra: arize
Requires-Dist: arize-phoenix-otel>=0.1.0; extra == 'arize'
Provides-Extra: dev
Requires-Dist: pyright>=1.1.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Provides-Extra: langfuse
Requires-Dist: langfuse>=2.0.0; extra == 'langfuse'
Provides-Extra: langsmith
Requires-Dist: langsmith>=0.1.0; extra == 'langsmith'
Description-Content-Type: text/markdown

# AgentRegress

[![PyPI](https://img.shields.io/pypi/v/agentregress)](https://pypi.org/project/agentregress/)

**Regression testing for AI agents. Turn production failures into automated tests.**


```
┌──────────────┐     ┌──────────────┐     ┌──────────────┐
│  Langfuse /  │     │              │     │   CI / CD    │
│  Langsmith / │────►│ AgentRegress │────►│   Pipeline   │
│  Arize / any │     │              │     │              │
│              │     │  Policies    │     │  Run tests   │
│  "What did   │     │  Tests       │     │  Block bad   │
│   it do?"    │     │  Generation  │     │   deploys    │
└──────────────┘     └──────────────┘     └──────────────┘
      OBSERVE              TEST               PREVENT
    (not our job)       (our job)          (our job)
```

AgentRegress is the missing layer between your observability tool and your deploy pipeline. It is **not** a tracing tool. It is not a dashboard. It does one thing: **it tests your agent and blocks bad deploys.**

---

## Install

```bash
pip install agentregress
# or
uv add agentregress
```

For Langfuse integration:
```bash
pip install "agentregress[langfuse]"
```

---

## 5-Minute Quickstart

### 1. Write a policy

Policies live in your repo as YAML files. Anyone can write them.

```yaml
# policies/support.yaml
policies:
  - id: no_repeat_info_ask
    severity: critical
    text: >
      If the customer has already provided a piece of information
      (name, account number, email), the agent must not ask for it again.

  - id: no_price_without_api
    severity: critical
    text: >
      The agent must never state a specific price without first calling
      the pricing API in the current conversation.
```

### 2. Score a trace

```bash
export ANTHROPIC_API_KEY=sk-ant-...

# Score a trace exported from Langfuse
agentregress score --trace ./exports/run-4821.json --policies ./policies/

# Output:
# ✗ run-4821: FAIL
#   Policy violated: no_repeat_info_ask (critical)
#   Turn 7: Agent asked for account number (previously given at turn 2)
#   Confidence: high
```

### 3. Write a test

```yaml
# tests/no_repeat_account_ask_001.yaml
test:
  id: no_repeat_account_ask_001
  severity: critical

  persona: >
    A frustrated customer who wants to upgrade their plan.
    Has account number A-7291. Gets annoyed if asked twice.

  goal: >
    Upgrade from Basic to Pro plan.

  script:
    - user: "Hi, I want to upgrade my plan"
    - user: "My account number is A-7291"
    - persona_driven: true           # LLM plays the user from here
    - user: "Can you confirm the price?"

  checks:
    - policy: no_repeat_info_ask
    - policy: no_price_without_api

  runs: 10
  pass_threshold: 0.8
```

### 4. Run against your agent

```bash
# Against an HTTP endpoint
agentregress test --suite ./tests/ --agent-url http://localhost:8000/chat

# Against a subprocess (stdin/stdout JSON protocol)
agentregress test --suite ./tests/ --agent-cmd "python my_agent.py"

# Output:
# AgentRegress — 2 tests
#
# ✓ no_repeat_account_ask_001    9/10 passed  (threshold: 8/10)
# ✗ no_price_without_api_001     3/10 passed  (threshold: 8/10)  ← REGRESSION
#
# Result: 1 failure. Exit code 1.
```

### 5. Block bad deploys in CI

```yaml
# .github/workflows/agent-eval.yml
name: Agent Tests
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: agentregress/agentregress@v1
        with:
          suite: ./tests
          agent-cmd: "python my_agent.py"
          severity: critical,functional
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
```

---

## Agent Protocol

### HTTP (`--agent-url`)

AgentRegress sends a `POST` request with the conversation:

```json
{
  "messages": [
    {"role": "user", "content": "Hi, I want to upgrade"},
    {"role": "assistant", "content": "Sure! What's your account number?"}
  ]
}
```

Expected response:
```json
{"response": "Your account has been upgraded to Pro."}
```

OpenAI-compatible endpoints (`/v1/chat/completions`) are auto-detected.

### Subprocess (`--agent-cmd`)

Your agent reads one JSON line from stdin and writes one JSON line to stdout:

```python
# my_agent.py
import json, sys

data = json.load(sys.stdin)
messages = data["messages"]
new_message = data["new_message"]

response = call_your_agent(messages, new_message)
print(json.dumps({"response": response}))
```

---

## Configuration

```yaml
# agentregress.yaml
agent:
  command: "python agent/main.py"   # or url: http://localhost:8000/chat
  timeout: 60                        # seconds per agent turn

scoring:
  model: claude-haiku-4-5-20251001  # LLM judge model

testing:
  default_runs: 10
  default_pass_threshold: 0.8
  concurrency: 4

sources:
  langfuse:
    project: my-support-agent
    # Keys via: LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY
```

---

## Supported Trace Sources

| Source | Flag / URI |
|--------|------------|
| JSON file (any format) | `--trace ./run.json` |
| Directory of JSONs | `--traces ./exports/` |
| Langfuse trace | `--source langfuse://trace/abc123` |
| Langfuse project | `--source langfuse://project/my-agent` |

Supported JSON formats: plain messages array, OpenAI chat completions response, Anthropic messages response, Langfuse export.

---

## Policy Severity

| Severity | Use when |
|----------|----------|
| `critical` | Violations are unshippable (safety, legal, core UX) |
| `functional` | Violations degrade the product significantly |
| `informational` | Tracking only — never fails CI |

---

## Community Policy Templates

Ready-to-use policies in `examples/policies/`:

- `support.yaml` — no-repeat-ask, retention-before-cancel, no-price-without-api, escalate-on-frustration
- `general.yaml` — no-hallucination, no-infinite-loop, confirm-before-destructive-action, no-pii-in-responses, graceful-out-of-scope

---

## What AgentRegress is NOT

| Not this | Use this instead |
|----------|------------------|
| Tracing / observability | Langfuse, Langsmith, Arize |
| LLM playground | Langsmith, PromptLayer |
| Model comparison | Braintrust, Humanloop |
| Agent framework | LangGraph, CrewAI |
| Guardrails / safety layer | NeMo Guardrails, Lakera |

---
