Metadata-Version: 2.4
Name: langgraph-scenario-lab
Version: 0.3.0
Summary: Behavioral scenario testing for production LangGraph applications
Keywords: langgraph,testing,agent,llm,scenario,behavioral-testing
Author: bzdvdn
Author-email: bzdvdn <bzdv.dn@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Dist: langgraph
Requires-Dist: rich>=15.0.0
Requires-Dist: typer>=0.27.1
Requires-Dist: pytest ; extra == 'dev'
Requires-Dist: ruff ; extra == 'dev'
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/bzdvdn/langgraph-scenario-lab
Project-URL: Repository, https://github.com/bzdvdn/langgraph-scenario-lab
Project-URL: Documentation, https://github.com/bzdvdn/langgraph-scenario-lab#readme
Project-URL: Issues, https://github.com/bzdvdn/langgraph-scenario-lab/issues
Provides-Extra: dev
Description-Content-Type: text/markdown

# langgraph-scenario-lab

[![CI](https://github.com/bzdvdn/langgraph-scenario-lab/actions/workflows/ci.yml/badge.svg)](https://github.com/bzdvdn/langgraph-scenario-lab/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/langgraph-scenario-lab?label=PyPI)](https://pypi.org/project/langgraph-scenario-lab/)
[![Python Versions](https://img.shields.io/pypi/pyversions/langgraph-scenario-lab)](https://pypi.org/project/langgraph-scenario-lab/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)

Behavioral scenario testing for production LangGraph applications: write a
scenario as plain Python, run it against the real graph, and assert on the
execution trace.

`scenario → real graph execution → trace → assertions → report`.

## Why

## Features

- **Scenarios as plain Python** — no YAML, no JSON, no custom DSL.
- **Trace-based assertions** — answer, tool calls, node path, state (incl.
  nested `result.state["a"]["b"].equals(...)`), status events, errors, sources.
- **Fault injection** — `lab.fail` / `lab.mock` to test retries, fallbacks, and
  network failures against the real graph.
- **LLM mocking** — `lab.mock_llm("classifier", {...})` replaces the model call
  inside a graph node with a canned response for deterministic routing tests;
  the default always uses the real model.
- **Budget assertions** — `result.llm.calls <= 3`, `result.llm.tokens <= 5000`,
  `result.latency < 10`, `result.retries`, `result.interrupts`.
- **Interrupt resume (HITL)** — after a paused run, `lab.resume(value)` /
  `lab.aresume(value)` continues the graph past `interrupt()` (handing it
  `Command(resume=…)`), and recording/replay persist the resume step as a
  dedicated catalog key — approvals and other pauses replay deterministically.
- **Record / Replay** — run `--mode record` once, then `--mode replay` for
  fast, deterministic regression checks without re-contacting tools.
- **Multi-turn scenarios** — `lab.scenario().user(...).run(...)`.
- **Custom domains** — product-specific entities exposed as `result.<name>`,
  registered via `ScenarioLab(domains=[...])`.
- **Rich terminal report** — colourised live progress, exit code 0/1/2.
- **pytest integration** — an autouse `lab` fixture; same suite convention.

## Install

```bash
uv pip install langgraph-scenario-lab
# or
pip install langgraph-scenario-lab
```

Requires Python ≥ 3.11.

## Quickstart

A suite is a directory with `lab.py` (builds the lab) and `scenarios/`
(auto-collected `test_*.py` files):

```
scenarios_lab/
├── lab.py                  # def build_lab() -> ScenarioLab
└── scenarios/
    └── test_small_talk.py
```

`lab.py`:

```python
from langgraph_scenario_lab import ScenarioLab

def build_lab() -> ScenarioLab:
    from examples.demo_graph import build_graph
    return ScenarioLab(build_graph())
```

`scenarios/test_small_talk.py`:

```python
from langgraph_scenario_lab import ScenarioLab


def test_small_talk(lab: ScenarioLab) -> None:
    result = lab.run("Hi")

    result.answer.matches(r"hi|help")
    result.tools.never_called("search_knowledge")
    result.status.matches(r"^Thinking")
    result.path.contains("answer")
```

Run it:

```bash
scenario-lab run                 # live, against the real graph
scenario-lab run --mode record   # live + save recordings under the suite
scenario-lab run --mode replay   # deterministic: no graph calls, no tool calls
```

```
Scenario Lab
────────────────────────────────────────
✓ test_small_talk
8 passed, 0 failed
```

![scenario-lab run](docs/assets/demo-run.gif)

See [docs/quickstart.md](docs/quickstart.md) for the full walkthrough.

## CLI

```text
scenario-lab run [target] [--mode live|record|replay] [--record-dir DIR]
                 [--replay-dir DIR] [--jsonl FILE] [--quiet] [--log-file FILE]
                 [--log-level debug|info|warning|error] [--name-filter SUBSTR]
scenario-lab version
```

![scenario-lab --help](docs/assets/demo-help.gif)

- `target` — suite directory (default `scenarios_lab/`), a `scenarios/`
  subdirectory, a `test_*.py` file, or `file::test_name`.
- `--mode record` / `--mode replay` implement the record/replay cycle.
- `--jsonl` appends one JSON line (scenario, status, trace) per run.
- `--name-filter` re-runs only the scenarios whose name contains the substring.
- Exit codes: `0` all passed, `1` failures, `2` suite/usage error.

A failing assertion prints a red `FAILURES` section with the expected vs
actual (reproduce with
`scenario-lab run examples/demo_suite/scenarios/test_failing_example.py`):

![scenario-lab run — failing scenario](docs/assets/demo-failures.gif)

Full reference in [docs/cli.md](docs/cli.md).

## Writing scenarios

```python
def test_rag(lab: ScenarioLab) -> None:
    result = lab.run("What is Nimbus?")
    result.answer.contains("Nimbus")
    result.sources.nonempty()
    result.tools.called("search_knowledge")
    result.nodes.visited("query_rewrite")
    result.status.sequence(r"^Thinking", r"^Searching", r"^Composing")
```

### Fault injection

```python
def test_retry(lab: ScenarioLab) -> None:
    lab.fail("search_knowledge", TimeoutError("flaky"), times=1)
    result = lab.run("What is flaky search?")
    result.errors.expected("search_knowledge")
    assert result.invocations["search_knowledge"] == 2  # retried on the graph
```

```python
def test_mock(lab: ScenarioLab) -> None:
    lab.mock("search_knowledge", return_value="Nimbus fake docs")
    result = lab.run("What are the mock docs?")
    result.answer.contains("Nimbus fake docs")
```

### Multi-turn

```python
def test_context(lab: ScenarioLab) -> None:
    scenario = lab.scenario()
    scenario.user("My favorite color is blue.")
    result = scenario.run("What is my favorite color?")
    result.state.has("messages")
```

### Interrupt resume (HITL)

`interrupt()` pauses the graph; the pause is counted by `result.interrupts`.
To *continue* an approval (or any HITL question), resume with the operator's
answer — the graph keeps the same thread (pass the same `config`):

```python
def test_approval(lab: ScenarioLab) -> None:
    first = lab.run("send report")          # pauses; first.interrupts == 1
    approved = lab.resume("approve")        # continues past the interrupt
    approved.state["status"].equals("sent")

async def test_approval_async(lab: ScenarioLab) -> None:
    first = await lab.arun("send report")
    approved = await lab.aresume("approve")  # async-only graph nodes
    approved.answer.contains("sent")
```

Record/replay persist the resume step under the `resume:<value>` catalog key,
so the same scenario body replays deterministically without re-running the
graph.

### Custom domains

Expose product entities as `result.<key>` without touching the core:

```python
# scenarios_lab/domains.py
from langgraph_scenario_lab import DomainModel

class StateSourcesDomain(DomainModel):
    key = "my_sources"
    def bind(self, trace):
        return list(trace.state.get("sources") or [])

def my_sources() -> StateSourcesDomain:
    return StateSourcesDomain()
```

```python
# scenarios_lab/lab.py
def build_lab() -> ScenarioLab:
    from domains import my_sources
    return ScenarioLab(build_graph(), domains=[my_sources()])
```

```python
# scenarios/test_domain.py
def test_sources(lab: ScenarioLab) -> None:
    result = lab.run("What is Nimbus?")
    assert result.my_sources == ["https://docs.nimbus.local/nimbus"]
```

## pytest integration

The package registers a `pytest11` plugin. With the suite on disk:

```bash
pytest scenarios_lab/scenarios
```

The autouse `lab` fixture serves the suite's lab, clears faults/mocks between
tests, and auto-discovers `lab.py` walking up from the test file. Pass an
explicit suite with `--scenario-suite path/to/suite`.

See [docs/pytest-plugin.md](docs/pytest-plugin.md).

## Record / replay

1. Record a golden baseline:

   ```bash
   scenario-lab run --mode record        # saves scenarios_lab/recordings/*.json
   ```

2. Change the prompt or graph, then replay the *old* trace against the *new*
   assertions (no tool calls, fully deterministic):

   ```bash
   scenario-lab run --mode replay
   ```

Recordings are JSON traces keyed by the last human/user message; per query the
cleanest (error-free) recording wins.

## Diagnostics

- `result.diff(baseline)` renders a BEFORE/AFTER execution diff when behavior
  changed (node path, statuses, tool calls, errors, answer).
- `--jsonl` gives machine-readable per-scenario results.
- `--log-file` + `--log-level` mirror the rich progress to a durable log.

## About the fixtures

- `examples/demo_graph.py` — a compact Nimbus/Vega RAG graph used by the docs
  walkthrough and the dogfood suite.
- `examples/support_graph.py` — a **non-trivial example**: a support chat with
  intent routing, knowledge-base retrieval + ranking, subscription and
  escalation tools, a `RetryPolicy`, extra state keys (`intent`, `queries`,
  `source_documents`, `sources`), and `status` stream events. Read its docstring
  to see where each asserted-on piece of data comes from.
- `examples/demo_suite/` — a richly commented scenario suite (15 cases) covering
  the assertion catalog — answer, tools, path, status, state, errors, sources,
  custom domains, fault injection, multi-turn — wired against
  `support_graph`. Run it with `scenario-lab run examples/demo_suite`.
- `scenarios_lab/` is the dogfood suite used by the tests and the CLI.

## Development

```bash
uv sync --extra dev
ruff check && ruff format --check
uv run python -m pytest tests/ scenarios_lab/scenarios/ -q
```

CI (`.github/workflows/ci.yml`) runs the same checks on Python 3.11–3.13 and
smoke-installs the built wheel in a clean venv. Releasing is tag-driven:
`publish.yml` builds and publishes to PyPI via [trusted
publishing](https://docs.pypi.org/trusted-publishers/) whenever a `v*` tag is
pushed.

```bash
uv build                      # local wheel + sdist sanity check
uvx --from build twine check dist/*  # verify long description metadata
git tag v0.2.0 && git push origin v0.2.0
```

## Documentation

- [Changelog](CHANGELOG.md)
- [Constitution](CONSTITUTION.md) — the design contract the code is built against
- [Quickstart](docs/quickstart.md)
- [Example suite: support chat](docs/example-suite.md) — 15 commented scenarios against a non-trivial graph
- [CLI reference](docs/cli.md)
- [Execution modes: live / record / replay](docs/modes.md)
- [Assertion catalog](docs/assertions.md)
- [Fault injection](docs/fault-injection.md)
- [Custom domains](docs/domains.md)
- [pytest plugin](docs/pytest-plugin.md)
- [Architecture](docs/architecture.md)
- [API reference](docs/reference.md)

## License

MIT