Metadata-Version: 2.4
Name: qe-harness
Version: 0.1.0
Summary: Black-box, five-stage validation of conversation-to-action AI assistants: intent, orchestration, MCP tool calls, backend state and user response
Author: Shan Konduru
License-Expression: MIT
Keywords: testing,qa,llm,agents,mcp,evaluation,chatbot
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mcp>=2.3
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6
Requires-Dist: python-dotenv>=1.0
Requires-Dist: websockets>=12
Provides-Extra: azure
Requires-Dist: openai>=1.50; extra == "azure"
Provides-Extra: anthropic
Requires-Dist: anthropic>=2.4; extra == "anthropic"
Provides-Extra: judge
Requires-Dist: openai>=1.50; extra == "judge"
Requires-Dist: anthropic>=2.4; extra == "judge"
Dynamic: license-file

# qe-harness

Black-box, five-stage validation of conversation-to-action AI assistants.

A chat-only test sees the final reply. `qe-harness` tests every turn at five stages, from the outside:

**intent → orchestration → MCP tool calls → backend state → user response**

It never imports the application under test. It signs in as test users and chats over the app's API, reads the
app's per-turn traces, records every tool call independently through its own MCP proxy, and compares read-only
snapshots of the business data before and after each turn. So it catches what a polite reply hides: a write the
customer never approved, an amount no tool returned, a reply that says "done" when nothing was saved.

## Install

```bash
pip install qe-harness            # core
pip install "qe-harness[judge]"   # plus the optional LLM grader (Azure OpenAI or Anthropic)
```

Python 3.11 or newer.

## Quick start

```bash
qe init          # writes qe.yaml (sandbox connection) and qe_catalog.yaml (test cases) in this folder
# edit both files, then provide the two secrets the config names:
#   QE_SANDBOX_TOKEN  sandbox control token
#   QE_TEST_PASSWORD  test users' password
# (environment variables, or a .env file next to qe.yaml)
qe doctor        # connection checks: all green means testing can start
qe run --open    # run every suite, evaluate each turn, open the HTML report
```

`qe` finds its configuration from `--config PATH`, then `$QE_CONFIG`, then `qe.yaml` in the working directory.
Relative paths in it start from the folder holding the file (or from `base_dir:` if set).

## What the app's sandbox must provide

| Need | Endpoint or setting |
|---|---|
| Sign-in as test users | `POST /api/login` `{"username", "password"}`, session cookie |
| Conversations | `POST /api/conversations` → `{"id"}`; WebSocket `/ws?conversation=<id>` with messages `{"type": "user", "text"}` and `{"type": "confirm", "key", "approved"}`; events `ready`, `awaiting_confirmation`, `assistant_message`, `error`, `turn_completed` |
| Per-turn traces | `GET /api/conversations/<id>/traces`: intent (`primary_intent`, `intents`), plan steps, tool calls with arguments and outcomes |
| File uploads (optional) | `POST /api/conversations/<id>/documents?filename=…` |
| Control | `GET /sandbox/status` (versions, clock, toggles) and `POST /sandbox/reset`, with header `x-sandbox-token` |
| Independent tool record | Start the MCP tool server through `python -m qe_harness.mcp_proxy --log-dir <dir> -- <server command>` |
| Business data | A read-only SQLite replica the harness can snapshot |

## Commands

| Command | What it does |
|---|---|
| `qe init [--dir D] [--force]` | Starter `qe.yaml` and `qe_catalog.yaml` |
| `qe doctor [--quick]` | Network, control API, clock, versions, proxy toggle, reset, read-back, sign-in, round trip, trace link, tool capture |
| `qe status` | Sandbox versions, clock and toggles |
| `qe cases [--suite S]` | List catalog cases |
| `qe run [--suite S…] [--case ID] [--repeats N] [--judge] [--open]` | Reset the sandbox, run the suites, evaluate every turn, write evidence and the HTML report |
| `qe runs` | Recent runs |
| `qe show RUN CASE` | Evidence for one case: message, approvals, intent, plan, trace and proxy tool calls, data changes, reply, verdicts |
| `qe report RUN [--open]` | Rebuild the self-contained HTML report |
| `qe coverage [RUN] [--trend]` | Intent coverage for a run, or per intent across runs |
| `qe mcp serve` | Run the harness as an MCP server for AI assistants |

## What is checked per turn

| Stage | Checks |
|---|---|
| Intent | The router's primary intent is one of the expected intents |
| Orchestration | Writes only after approval; no unexpected approval prompts; no repeated identical calls |
| MCP tool call | Required calls succeeded with the expected arguments; forbidden calls absent; **trace and proxy agree** |
| Backend state | Only the expected tables changed; SQL checks; **declined → nothing changed; data changed → a write succeeded** |
| User response | Required facts present, forbidden phrases absent, optional LLM grader for consistency with what the tools did |

Each failure records its first failing stage, so developers know where to look. With `--repeats N`, every case gets
a pass rate with a 95% Wilson range and unstable cases are reported as flaky.

## Intent coverage

List the app's intent families under `intents:` in `qe.yaml`. Each report shows, per family, the cases that test it,
their pass rate and a status (covered, thin, failing, not tested), plus a families × runs grid of how coverage
changes over time. `qe coverage --trend` shows the same in the terminal.

## In AI assistants (MCP)

```bash
claude mcp add qe-harness -- qe mcp serve
```

For Cursor or VS Code, add a stdio server with command `qe` and args `["mcp", "serve"]`.

Tools: `sandbox_status`, `sandbox_doctor`, `list_cases`, `get_case`, `run_tests` (changes state; capped per call),
`get_run_status`, `list_runs`, `get_failure` (masked evidence, marked as untrusted data), `get_intent_coverage`,
`get_intent_trend`. Prompt: `triage_run`.

## LLM grader (optional)

`qe run --judge` asks a model whether each reply is consistent with what the tools actually did. It uses its own
settings, separate from the app's: `QE_JUDGE_PROVIDER` = `AZURE` (with `QE_JUDGE_AZURE_API_KEY`,
`QE_JUDGE_AZURE_ENDPOINT`, `QE_JUDGE_AZURE_DEPLOYMENT`) or `ANTHROPIC` (with `ANTHROPIC_API_KEY`, optional
`QE_JUDGE_MODEL`).

## Status

Alpha. Not yet included: fault injection, load and soak testing, simulated customers, robustness variant
generation and a release gate.

## License

MIT
