Metadata-Version: 2.4
Name: qe-harness
Version: 0.2.1
Summary: Black-box, five-stage validation of conversation-to-action AI assistants: intent, orchestration, MCP tool calls, backend state and user response
Author: Shan Konduru
License-Expression: MIT
Keywords: testing,qa,llm,agents,mcp,evaluation,chatbot
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mcp>=2.3
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6
Requires-Dist: python-dotenv>=1.0
Requires-Dist: websockets>=12
Provides-Extra: azure
Requires-Dist: openai>=1.50; extra == "azure"
Provides-Extra: anthropic
Requires-Dist: anthropic>=2.4; extra == "anthropic"
Provides-Extra: judge
Requires-Dist: openai>=1.50; extra == "judge"
Requires-Dist: anthropic>=2.4; extra == "judge"
Dynamic: license-file

# qe-harness

Black-box, five-stage validation of conversation-to-action AI assistants.

A chat-only test sees the final reply. `qe-harness` tests every turn at five stages, from the outside:

**intent → orchestration → MCP tool calls → backend state → user response**

It never imports the application under test. It signs in as test users and chats over the app's API, reads the
app's per-turn traces, records every tool call independently through its own MCP proxy, and compares read-only
snapshots of the business data before and after each turn. So it catches what a polite reply hides: a write the
customer never approved, an amount no tool returned, a reply that says "done" when nothing was saved.

## Install

```bash
pip install qe-harness            # core
pip install "qe-harness[judge]"   # plus the optional LLM grader (Azure OpenAI or Anthropic)
```

Python 3.11 or newer.

## Quick start

```bash
qe init          # writes qe.yaml (sandbox connection) and qe_catalog.yaml (test cases) in this folder
# edit both files, then provide the two secrets the config names:
#   QE_SANDBOX_TOKEN  sandbox control token
#   QE_TEST_PASSWORD  test users' password
# (environment variables, or a .env file next to qe.yaml)
qe doctor        # connection checks: all green means testing can start
qe run --open    # run every suite, evaluate each turn, open the HTML report
```

`qe` finds its configuration from `--config PATH`, then `$QE_CONFIG`, then `qe.yaml` in the working directory.
Relative paths in it start from the folder holding the file (or from `base_dir:` if set).

## What the app's sandbox must provide

| Need | Endpoint or setting |
|---|---|
| Sign-in as test users | `POST /api/login` `{"username", "password"}`, session cookie |
| Conversations | `POST /api/conversations` → `{"id"}`; WebSocket `/ws?conversation=<id>` with messages `{"type": "user", "text"}` and `{"type": "confirm", "key", "approved"}`; events `ready`, `awaiting_confirmation`, `assistant_message`, `error`, `turn_completed` |
| Per-turn traces | `GET /api/conversations/<id>/traces`: intent (`primary_intent`, `intents`), plan steps, tool calls with arguments and outcomes |
| File uploads (optional) | `POST /api/conversations/<id>/documents?filename=…` |
| Control | `GET /sandbox/status` (versions, clock, toggles) and `POST /sandbox/reset`, with header `x-sandbox-token` |
| Independent tool record | Start the MCP tool server through `python -m qe_harness.mcp_proxy --log-dir <dir> -- <server command>` |
| Business data | A read-only SQLite replica the harness can snapshot |

## Commands

| Command | What it does |
|---|---|
| `qe init [--dir D] [--force]` | Starter `qe.yaml` and `qe_catalog.yaml` |
| `qe doctor [--quick] [--workers]` | Network, control API, clock, versions, proxy toggle, reset, read-back, sign-in, round trip, trace link, tool capture; `--workers` checks every sandbox |
| `qe status` | Sandbox versions, clock and toggles |
| `qe cases [--suite S]` | List catalog cases |
| `qe run [--suite S…] [--case ID] [--repeats N] [--judge] [--parallel N] [--open]` | Reset the sandbox, run the suites, evaluate every turn, write evidence and the HTML report; `--parallel N` spreads suites over N sandboxes |
| `qe runs` | Recent runs |
| `qe show RUN CASE` | Evidence for one case: message, approvals, intent, plan, trace and proxy tool calls, data changes, reply, verdicts |
| `qe report RUN [--open]` | Rebuild the self-contained HTML report |
| `qe coverage [RUN] [--trend]` | Intent coverage for a run, or per intent across runs |
| `qe mcp serve` | Run the harness as an MCP server for AI assistants |

## What is checked per turn

| Stage | Checks |
|---|---|
| Intent | The router's primary intent is one of the expected intents |
| Orchestration | Writes only after approval; no unexpected approval prompts (approval steps approve only the expected action with matching arguments; anything else is declined and fails here); no repeated identical calls |
| MCP tool call | Required calls succeeded with the expected arguments; forbidden calls absent; **trace and proxy agree** |
| Backend state | Only the expected tables changed; SQL checks; **declined → nothing changed; data changed → a write succeeded** |
| User response | Required facts present, forbidden phrases absent, optional LLM grader for consistency with what the tools did |

Each failure records its first failing stage, so developers know where to look. With `--repeats N`, every case gets
a pass rate with a 95% Wilson range and unstable cases are reported as flaky.

## Parallel runs (one sandbox per worker)

Every run resets its sandbox, so one sandbox runs one test at a time (a lock enforces it). To go faster, give the
harness several identical sandboxes: list the extra ones under `workers:` in the config, each with its own URL,
read-only data path and proxy log folder.

```yaml
workers:
  - {base_url: http://127.0.0.1:8101, sor_path: data/sandbox-2.db, proxy_log_dir: qe_runs/proxy-2}
  - {base_url: http://127.0.0.1:8102, sor_path: data/sandbox-3.db, proxy_log_dir: qe_runs/proxy-3}
```

```bash
qe doctor --workers              # check every sandbox
qe run --judge --parallel 3      # the main sandbox plus two workers
```

Work is split into (repeat, suite) units; each runs on the next free worker from a fresh reset of that sandbox, and
the results merge into one run and one report. Suites must be independent for this: each is one test user's
conversation, and its data checks should be scoped to that user's records. The sandboxes must report the same
versions and clock, or the run refuses to start.

## Intent coverage

List the app's intent families under `intents:` in `qe.yaml`. Each report shows, per family, the cases that test it,
their pass rate and a status (covered, thin, failing, not tested), plus a families × runs grid of how coverage
changes over time. `qe coverage --trend` shows the same in the terminal.

## In AI assistants (MCP)

```bash
claude mcp add qe-harness -- qe mcp serve
```

For Cursor or VS Code, add a stdio server with command `qe` and args `["mcp", "serve"]`.

Tools: `sandbox_status`, `sandbox_doctor`, `list_cases`, `get_case`, `run_tests` (changes state; capped per call),
`get_run_status`, `list_runs`, `get_failure` (masked evidence, marked as untrusted data), `get_intent_coverage`,
`get_intent_trend`. Prompt: `triage_run`.

## LLM grader (optional)

`qe run --judge` asks a model whether each reply is consistent with what the tools actually did. It uses its own
settings, separate from the app's: `QE_JUDGE_PROVIDER` = `AZURE` (with `QE_JUDGE_AZURE_API_KEY`,
`QE_JUDGE_AZURE_ENDPOINT`, `QE_JUDGE_AZURE_DEPLOYMENT`) or `ANTHROPIC` (with `ANTHROPIC_API_KEY`, optional
`QE_JUDGE_MODEL`).

## Status

Alpha. Not yet included: fault injection, load and soak testing, simulated customers, robustness variant
generation and a release gate.

## License

MIT
