Metadata-Version: 2.3
Name: langgraph-chat-agent-eval
Version: 0.1.1
Summary: Evaluation framework for LangGraph chatbots on multi-turn conversations
Keywords: langgraph,evaluation,llm,testing,agents
Author: Miguel Garcia
License: MIT
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Dist: langgraph>=0.2
Requires-Dist: langchain-core>=0.3
Requires-Dist: pydantic>=2.0
Requires-Dist: rich>=13.0
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/miguelgarcia/langgraph-chat-agent-eval
Project-URL: Repository, https://github.com/miguelgarcia/langgraph-chat-agent-eval
Project-URL: Issues, https://github.com/miguelgarcia/langgraph-chat-agent-eval/issues
Description-Content-Type: text/markdown

# langgraph-chat-agent-eval

Test your LangGraph agents the way users actually use them — across multi-turn
conversations, with deterministic tool mocks and an LLM judge that scores every step.

A pytest-based evaluation framework for testing LangGraph chat agents end-to-end.

## Motivation

Unit tests and mocked LLM calls can verify that your agent's plumbing is wired correctly, but they don't tell you whether the agent actually behaves well. This library fills that gap: it runs **real LLM calls** against your compiled graph, mocks tool responses so tests are deterministic, and uses an **LLM-as-judge** to assess whether the agent completed its goal and how good the response was.

The result is a quantitative signal — tool recall, response quality, latency — that you can track over time in CI and use to catch regressions before they reach production.

## Features

- **Multi-turn conversations** — define scenarios with multiple steps; the evaluator simulates a user when the agent needs clarification
- **Deterministic tool mocks** — replace any tool's coroutine with a fixed return value or callable, without touching the agent code
- **Arg-level assertions** — verify not just that a tool was called, but that it was called with the right arguments
- **LLM-as-judge** — works with any LangChain-compatible model (OpenAI, Anthropic, Google, etc.)
- **JSON output** — results saved per run, easy to diff in CI or feed into dashboards
- **Terminal viewer** — `eval-view` renders a color-coded report from any result file

## Installation

```bash
pip install langgraph-chat-agent-eval
```

The package depends on `langgraph` and `langchain-core`. It does **not** depend on any specific LLM provider — bring your own.

## Quick start

### 1. Define a scenario

```python
# tests/evaluation/scenarios/my_scenario.py
from langgraph_chat_agent_eval import (
    EvaluationScenario, ConversationStep, ExpectedToolCall, ToolMock
)

def build_my_scenario() -> EvaluationScenario:
    return EvaluationScenario(
        name="search_and_create",
        description="User searches for items then creates a new one",
        shared_mocks=[
            ToolMock(
                tool_name="search_items",
                return_value=[{"id": "1", "name": "Widget"}],
            ),
        ],
        steps=[
            ConversationStep(
                name="search",
                user_message="Find all widgets",
                expected_tools=[
                    ExpectedToolCall("search_items", with_args={"query": "widgets"}),
                ],
                expected_answer="Agent lists the available widgets",
            ),
            ConversationStep(
                name="create",
                user_message="Create a new widget called Gadget",
                expected_tools=[
                    ExpectedToolCall("create_item", with_args={"name": "Gadget"}),
                    ExpectedToolCall("search_items", required=False),  # optional re-list
                ],
                expected_answer="Agent confirms the widget was created",
                tool_mocks=[
                    ToolMock("create_item", return_value={"id": "2", "name": "Gadget"}),
                ],
            ),
        ],
    )
```

### 2. Wire up fixtures

```python
# tests/evaluation/conftest.py
import pytest
from langchain_openai import ChatOpenAI           # or any other provider
from langgraph.checkpoint.memory import MemorySaver
from langchain_core.tools import BaseTool

from langgraph_chat_agent_eval import Evaluator
from my_agent import build_graph, get_tools, AgentConfig

pytest_plugins = ["langgraph_chat_agent_eval.fixtures"]


@pytest.fixture(scope="session")
def eval_context() -> AgentConfig:
    return AgentConfig()

@pytest.fixture(scope="session")
def eval_graph(eval_context: AgentConfig):
    return build_graph(eval_context, checkpointer=MemorySaver())

@pytest.fixture(scope="session")
def all_tools(eval_context: AgentConfig) -> list[BaseTool]:
    return get_tools(eval_context)

@pytest.fixture(scope="session")
def evaluator() -> Evaluator:
    return Evaluator(llm=ChatOpenAI(model="gpt-4o", temperature=0))
```

### 3. Write the test

```python
# tests/evaluation/test_my_scenario.py
import pytest
from langgraph_chat_agent_eval import ConversationRunner
from scenarios.my_scenario import build_my_scenario

@pytest.mark.asyncio
@pytest.mark.evaluation
async def test_my_scenario(eval_context, eval_graph, evaluator, all_tools):
    runner = ConversationRunner(eval_graph, evaluator, all_tools, eval_context)
    metrics = await runner.run_scenario(build_my_scenario())

    assert metrics.all_steps_completed
    assert metrics.avg_tool_recall >= 0.8
    assert metrics.avg_response_quality >= 0.6
```

### 4. Run

If using plain pip:
```bash
pytest tests/evaluation/ -m evaluation -v
```

If using uv:
```bash
uv run pytest tests/evaluation/ -m evaluation -v
```

Results are saved to `tests/evaluation/output/YYYYMMDD_HHMMSS_<scenario>.json`.

## Viewing results

If using plain pip:
```bash
eval-view                           # shows latest run for each scenario
eval-view path/to/result.json       # shows a specific file
```

If using uv:
```bash
uv run eval-view
uv run eval-view path/to/result.json
```

The viewer prints a summary panel, a per-step table, and a full conversation replay with tool call details and color-coded scores.

## Running the example

The `examples/` directory contains a working end-to-end example with a simple notes agent (two tools: `search_notes` and `create_note`). It is a good starting point to verify your setup and understand how the framework fits together.

```bash
git clone https://github.com/your-org/langgraph-chat-agent-eval
cd langgraph-chat-agent-eval
uv sync
```

Set your OpenAI API key (the example uses `ChatOpenAI` — swap in any provider you prefer):

```bash
export OPENAI_API_KEY=sk-...
```

Run the evaluation:

```bash
uv run pytest examples/ -v
```

View the results:

```bash
uv run eval-view
```

See [`examples/README.md`](examples/README.md) for a walkthrough of what each file does.

## Core concepts

### `EvaluationScenario`

A sequence of `ConversationStep`s representing a complete user workflow. Shared mocks apply to all steps and can be overridden per step.

### `ConversationStep`

One logical goal within the conversation. The runner drives the agent through up to `max_turns` (default 5), with the evaluator acting as the user when follow-ups are needed.

### `ExpectedToolCall`

Declares a tool the agent should call. Use `required=False` for optional tools (calling them earns recall credit; skipping them doesn't penalize). Use `with_args` to verify specific arguments were passed — matching is a shallow subset check.

```python
ExpectedToolCall("search", required=True, with_args={"query": "widgets", "limit": 10})
```

### `ToolMock`

Replaces a tool's async coroutine with a stub. Use `return_value` for a fixed response, or `side_effect` for a callable (useful for stateful sequences) or an exception (to test error handling).

```python
ToolMock("search", return_value=[{"id": "1"}])
ToolMock("flaky_tool", side_effect=TimeoutError("upstream down"))
ToolMock("stateful", side_effect=iter([result_1, result_2]).__next__)
```

### `Evaluator`

Wraps any LangChain `BaseChatModel` and exposes two methods used internally by the runner:

- `check_step_completion` — judges whether the agent's response satisfied the step goal
- `evaluate_step` — scores response quality 0.0–1.0 on correctness, tool usage, clarity, and goal achievement

```python
from langchain_anthropic import ChatAnthropic
evaluator = Evaluator(llm=ChatAnthropic(model="claude-opus-4-6"))
```

### `ConversationRunner`

Drives a scenario against a compiled graph. Each instance gets a unique `thread_id` for state isolation.

```python
runner = ConversationRunner(
    graph=eval_graph,
    evaluator=evaluator,
    all_tools=all_tools,
    context=eval_context,
    scenario_timeout_seconds=120,   # optional wall-clock cap
)
metrics = await runner.run_scenario(scenario)
```

### Metrics

`run_scenario` returns a `ConversationMetrics` object:

| Field | Description |
|---|---|
| `all_steps_completed` | `True` only if every step passed |
| `avg_tool_recall` | Mean of per-step recall (matched required tools / total required) |
| `avg_response_quality` | Mean of LLM-judged quality scores (0.0–1.0) |
| `total_turns` | Total agent turns across all steps |
| `steps` | List of `StepMetrics` with per-step detail |

## Pytest fixtures reference

Import with `pytest_plugins = ["langgraph_chat_agent_eval.fixtures"]` and override in your `conftest.py`:

| Fixture | Scope | Notes |
|---|---|---|
| `eval_context` | session | Your agent's config object — must override |
| `eval_graph` | session | Compiled `StateGraph` with `MemorySaver` — must override |
| `all_tools` | session | Full tool list for mock lookup — must override |
| `evaluator` | session | `Evaluator` instance — must override |
| `_clear_tool_cache` | function | autouse no-op; override to reset tool state between tests |

## LangGraph assumptions

The framework assumes your graph:

1. Has a `messages` key in its state holding `BaseMessage` objects
2. Uses `interrupt()` before tool confirmation steps (auto-approved with `Command(resume="y")`)
3. Uses async `BaseTool` instances (mocking replaces `tool.coroutine`)
4. Is compiled with a `MemorySaver` checkpointer (state persists across turns within a test)
