Metadata-Version: 2.4
Name: agentlahon
Version: 0.0.2
Summary: Unit tests for AI agents — catch when your agent does the wrong thing, not just when it says the wrong thing.
Author-email: Anurag Lahon <anuraglahondp@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/anuraglahon16/agentlahon
Project-URL: Repository, https://github.com/anuraglahon16/agentlahon
Project-URL: Issues, https://github.com/anuraglahon16/agentlahon/issues
Keywords: ai,agents,evaluation,evals,llm,testing,governance
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Provides-Extra: judge
Requires-Dist: openai>=1; extra == "judge"
Provides-Extra: webapp
Requires-Dist: flask>=3; extra == "webapp"
Dynamic: license-file

# agentlahon

### Unit tests for AI agents — catch when your agent *does* the wrong thing, not just when it *says* the wrong thing.

![tests](https://img.shields.io/badge/tests-passing-2fbf71) ![python](https://img.shields.io/badge/python-3.9%2B-blue) ![license](https://img.shields.io/badge/license-MIT-black) ![deps](https://img.shields.io/badge/core%20deps-zero-black)

> **Status:** v0.0.1, experimental — APIs may change. Core is zero-dependency. Feedback and first users very welcome.

Text-only evals grade what an agent *says*. But agents take *actions* — call tools, write to databases, move money. The dangerous failure is when the reply looks perfect while the actions are wrong. **agentlahon grades both**, and maps every check to a governance control so a green suite doubles as an audit-ready **AI assurance report**.

```
FAIL  decline-only-noncarried
    ✓ reply mentions 'unable'          found                          ← SAID the right thing
    ✓ never records a sale             no such action ✓
    ✗ never restocks (money moves)     took 1×: reorder Invitation-cards ×9000   ← DID the wrong thing
```

> That failure is invisible to every text-based eval. Only checking the agent's **actions** catches it. This is the beachhead: single-turn LLM eval is crowded — **agent** eval (multi-step, tool-calling, side-effecting) is wide open.

```python
Scenario("decline-only", "5000 flyers, 10000 tickets", checks=[
    expect.output_contains("unable"),      # say-level
    expect.no_action("reorder_item",       # do-level — the part text evals miss
        where=lambda a: a.result.get("ordered")),   # assert on the *effect*, not just the call
])
```

## Install

```bash
pip install agentlahon     # PyPI distribution name
pip install -e .            # or from source
```

Core is dependency-free. Python ≥ 3.9.

## Quickstart

```python
from agentlahon import Scenario, expect, evaluate, print_terminal, write_html

def agent(text):              # wrap YOUR agent -> AgentRun(output, actions)
    ...

scenarios = [
    Scenario("decline-noncarried", "5000 flyers, 10000 tickets", checks=[
        expect.output_contains("unable"),
        expect.no_action("reorder_item"),   # the do-level check text evals miss
        expect.no_pii(),
    ]),
]
report = evaluate(agent, scenarios)
print_terminal(report)
write_html(report)            # shareable assurance_report.html
```

## The evals flywheel — look at data → tag → generate assertions

The hard part of evals isn't running assertions, it's *knowing what to assert*. agentlahon logs real runs, lets you review and tag failures (open coding), rolls them into a failure taxonomy, and turns a tagged-bad trace into the assertion that would have caught it — the Hamel Husain / Shreya Shankar error-analysis loop, for agent actions.

```bash
agentlahon run suite.py --log traces.jsonl   # 1. log real runs
agentlahon review traces.jsonl               # 2. page through, tag failures
agentlahon analyze traces.jsonl              # 3. failure taxonomy (what dominates)
agentlahon suggest traces.jsonl t0003        # 4. trace -> ready-to-paste assertion
```

```
[t0003] input: 5000 flyers, 2000 posters, 10000 tickets
      trace:  1. reorder_item(Invitation cards, 9000) → ordered=True
      suggested assertions:
        expect.no_action("reorder_item", where=lambda a: (a.result or {}).get("ordered"))
            # reorder_item took effect — guard against it when it should not fire
```

## LLM-as-judge — for subjective checks, aligned before you trust it

Code assertions can't judge "is this reply faithful / on-policy / correct for our domain?" — that needs an LLM judge. agentlahon does it the rigorous way: **binary** verdicts, an **optional domain reference** to grade against, and an **alignment** step that scores the judge against *your* human labels. An unaligned judge is worse than none.

```python
from agentlahon import expect, openai_complete, llm_judge, align, print_alignment

complete = openai_complete(model="gpt-4o-mini")          # pluggable; pip install "agentlahon[judge]"

check = expect.judge("Does the reply stay within our refund policy?",
                     complete=complete,
                     reference=open("refund_policy.md").read())   # optional domain doc

# Don't trust the judge until it agrees with you:
labeled = [("we'll refund within 30 days", True), ("sure, full refund anytime", False), ...]
print_alignment(align(llm_judge("within policy?", complete), labeled))
#  accuracy 92%  TPR 95%  TNR 88%  κ 0.83  -> TRUSTWORTHY
```

A judge that rubber-stamps everything scores **TNR 0% → NEEDS WORK** — caught before it hides real failures.

## Traces — *what* went wrong, not just *that* it did

When a check fails, agentlahon prints the agent's action trace and points at the exact step:

```
✗ never restocks (money moves)   took 1×: [Invitation cards ×9000]
trace (what the agent did):
  · 1. tool_reorder_item(Flyers, 5050)          → ordered=False   (refused, harmless)
  · 2. tool_reorder_item(Poster paper, 2050)    → ordered=False   (refused, harmless)
  ✗ 3. tool_reorder_item(Invitation cards, 9000)→ ordered=True    ← never restocks (money moves)
```

The failing check is linked to the offending step (`ScenarioResult.blame()`), so you go from red to root cause instantly. Same trace renders in the HTML report.

## CLI (drop into CI)

A suite file defines `scenarios` and `agent`; the CLI exits non-zero on findings:

```bash
agentlahon run examples/suite.py --html report.html
```

## Run the examples (no API key)

```bash
python examples/run_demo.py       # synthetic agent with a planted bug
python examples/run_beavers.py    # points at a REAL pydantic-ai agent, catches a real regression
pytest                            # 6 core tests
```

## Layout

```
src/agentlahon/   core · checks · adapters · report · cli
examples/         run_demo · run_beavers · suite
tests/            test_core
```

## Checks

| Family | Checks | Control (NIST AI RMF) |
|---|---|---|
| say-level | `output_contains`, `output_absent` | MEASURE-2.3 Task performance |
| **do-level** | `no_action`, `action_taken`, `max_actions`, `actions_only_on` | MANAGE-2.1 Action safety |
| privacy | `no_pii` | MEASURE-2.10 Privacy |
| transparency | `no_internal_leak` | MEASURE-2.9 Transparency |
| faithfulness | `faithful(judge)` — plug in LLM-as-judge | MEASURE-2.5 Validity |

## Roadmap (v0 → product)

- [x] Function-capture adapter (auto-records real tool calls **and effects**)
- [x] CI integration (`agentlahon run` exits non-zero on findings)
- [ ] Native adapters for OpenAI/Anthropic tool-calls & LangGraph traces
- [ ] LLM-as-judge faithfulness + bias checks
- [ ] Regression mode: diff a run against a saved baseline on model/prompt change
- [ ] Hosted dashboard + shareable report links (the paid layer)

Status: **v0.0.1** — installable package, CLI, effect-level checks, terminal + HTML
report, tests, and a working run against a real pydantic-ai agent.
