Metadata-Version: 2.4
Name: qe-harness
Version: 0.4.3
Summary: Black-box, five-stage validation of conversation-to-action AI assistants: intent, orchestration, MCP tool calls, backend state and user response
Author: Shan Konduru
License-Expression: MIT
Keywords: testing,qa,llm,agents,mcp,evaluation,chatbot
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mcp>=2.3
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6
Requires-Dist: python-dotenv>=1.0
Requires-Dist: websockets>=12
Provides-Extra: azure
Requires-Dist: openai>=1.50; extra == "azure"
Provides-Extra: anthropic
Requires-Dist: anthropic>=2.4; extra == "anthropic"
Provides-Extra: judge
Requires-Dist: openai>=1.50; extra == "judge"
Requires-Dist: anthropic>=2.4; extra == "judge"
Dynamic: license-file

# qe-harness

Black-box, five-stage validation of conversation-to-action AI assistants.

A chat-only test sees the final reply. `qe-harness` tests every turn at five stages, from the outside:

**intent → orchestration → MCP tool calls → backend state → user response**

It never imports the application under test, and it doesn't intercept anything. It signs in as test users and
chats over the app's API, reads the app's per-turn traces, gets an independent record of every tool call from a
recording proxy the sandbox routes its tool traffic through (stdio or HTTP), and compares read-only snapshots of the
business data before and after each turn (see "How the evidence is collected" below). So it catches what a polite reply hides: a write the
customer never approved, an amount no tool returned, a reply that says "done" when nothing was saved.

## Install

```bash
pip install qe-harness            # core
pip install "qe-harness[judge]"   # plus the optional LLM grader (Azure OpenAI or Anthropic)
```

Python 3.11 or newer.

## Quick start

```bash
qe init          # writes qe.yaml (sandbox connection) and qe_catalog.yaml (test cases) in this folder
# edit both files, then provide the two secrets the config names:
#   QE_SANDBOX_TOKEN  sandbox control token
#   QE_TEST_PASSWORD  test users' password
# (environment variables, or a .env file next to qe.yaml)
qe doctor        # connection checks: all green means testing can start
qe run --open    # run every suite, evaluate each turn, open the HTML report
```

`qe` finds its configuration from `--config PATH`, then `$QE_CONFIG`, then `qe.yaml` in the working directory.
Relative paths in it start from the folder holding the file (or from `base_dir:` if set).

## What the app's sandbox must provide

| Need | Endpoint or setting |
|---|---|
| Sign-in as test users | `POST /api/login` `{"username", "password"}`, session cookie |
| Conversations | `POST /api/conversations` → `{"id"}`; WebSocket `/ws?conversation=<id>` with messages `{"type": "user", "text"}` and `{"type": "confirm", "key", "approved"}`; events `ready`, `awaiting_confirmation`, `assistant_message`, `error`, `turn_completed` |
| Per-turn traces | `GET /api/conversations/<id>/traces`: intent (`primary_intent`, `intents`), plan steps, tool calls with arguments and outcomes |
| File uploads (optional) | `POST /api/conversations/<id>/documents?filename=…` |
| Control | `GET /sandbox/status` (versions, clock, toggles) and `POST /sandbox/reset`, with header `x-sandbox-token` |
| Independent tool record | Route tool traffic through `qe_harness.mcp_proxy`: stdio (start the tool server through it) or HTTP (point the app's MCP URL at it). See below |
| Business data | A read-only SQLite replica the harness can snapshot |

## How the evidence is collected (nothing is intercepted)

The harness doesn't sniff network traffic, decrypt TLS or hook into the app. Each source of evidence is something
the client switches on in their sandbox:

```
                     the client's sandbox (each source is switched on by the client)
  +------------+                    +-------------+   MCP    +-------------+   MCP   +-------------+
  |            | (1) chat as test   |             | -------> |  qe proxy   | ------> |             |
  |            |     users          |  app under  |          |  relays     |         | tool server |
  |            | -----------------> |  test       | <------- |  unchanged, | <------ |             |
  |            | <----------------- |             |          |  logs each  |         +------+------+
  | qe-harness |                    +------+------+          |  tool call  |                |
  |            | (2) per-turn trace API    |                 +------+------+                v
  |            | <-------------------------+                        |                +-------------+
  |            | (3) tool-call log, independent of the app's trace  |                |  business   |
  |            | <--------------------------------------------------+                |  data       |
  |            | (4) read-only snapshots of a replica, before and after every turn   |  (replica)  |
  |            | <-------------------------------------------------------------------+-------------+
  +------------+
```

1. **Conversation**: the harness signs in as test users and chats over the app's normal API, like a customer would.
2. **Intent, plan and tool calls**: the app reports them per turn through its trace API (the app describing itself).
3. **Independent tool-call record**: the app's tool traffic is routed *through* `qe_harness.mcp_proxy` by changing
   one setting in the sandbox. The proxy passes every MCP message through unchanged and writes each `tools/call`
   request and its result to a log. If the app's trace leaves out or misreports a tool call, this record disagrees,
   and the harness flags it ("trace and proxy disagree").
4. **Business data**: read-only snapshots of a replica of the system of record, before and after every turn.

### Enabling the proxy

| The app talks to its tool server over | The sandbox changes | Run |
|---|---|---|
| **stdio** (the tool server is a child process) | the command that starts the tool server | `python -m qe_harness.mcp_proxy --log-dir <dir> -- <tool server command>` |
| **HTTP** (MCP streamable HTTP) | the tool-server URL the app connects to, to the proxy's address | `python -m qe_harness.mcp_proxy --http --listen 0.0.0.0:9100 --upstream http://tools:9000/mcp --log-dir <dir>` |

- **stdio**: one proxy process per tool-server process; the customer comes from `COPILOT_CUSTOMER_ID` in its
  environment.
- **HTTP**: a reverse proxy. Requests keep their path, headers (`Mcp-Session-Id`, `Authorization`, …) and body;
  event-stream responses are streamed through as they arrive. The customer comes from a request header the app
  sends (`--customer-header`, default `X-Customer-Id`) or a fixed `--customer`. Each request/response exchange is
  logged with its own id, so concurrent conversations can't be mixed up.
- Both modes write the same log format to `<dir>`, which must match `tools.proxy_log_dir` in the harness config.
- The proxy logs only tool calls and their results. It changes nothing, and if it's removed the app works exactly
  as before.

### Other ways to get an independent record

If routing tool traffic through the proxy isn't possible, these give similar evidence, roughly from most to least
recommended:

- **API gateway or service-mesh access logs** (Kong, Apigee, AWS API Gateway, Istio/Envoy) for the back-end calls the
  tools make: very independent of the app.
- **Database change capture or audit logs** (e.g. Debezium, database audit triggers): an independent record of every
  write; scales better than whole-database snapshots.
- **OpenTelemetry traces** the app already emits for model and tool calls: useful, but like the trace API it's the
  app describing itself.
- **An audit log inside the tool server**: independent of the agent, not of the client's code.
- **Packet capture** (tcpdump, mitmproxy with a trusted CA): a last resort. Traffic is usually TLS, it needs
  network access and the security team's consent, and it sees nothing at all when MCP runs over stdio.

## Commands

| Command | What it does |
|---|---|
| `qe init [--dir D] [--force]` | Starter `qe.yaml` and `qe_catalog.yaml` |
| `qe init --ci github\|azure` | A sample nightly CI pipeline |
| `qe doctor [--quick] [--workers]` | Network, control API, clock, versions, proxy toggle, reset, read-back, sign-in, round trip, trace link, tool capture; `--workers` checks every sandbox |
| `qe status` | Sandbox versions, clock and toggles |
| `qe cases [--suite S]` | List catalog cases |
| `qe run [--suite S…] [--case ID] [--repeats N] [--judge] [--parallel N] [--open]` | Reset the sandbox, run the suites, evaluate every turn, write evidence, the HTML report and the defect list; `--parallel N` spreads suites over N sandboxes |
| `qe run … [--junit FILE] [--flaky fail\|skip\|pass] [--baseline RUN]` | For CI: JUnit XML, how flaky cases count, and fail only on regressions against an accepted run |
| `qe runs` | Recent runs |
| `qe show RUN CASE` | Evidence for one case: message, approvals, intent, plan, trace and proxy tool calls, data changes, reply, verdicts |
| `qe report RUN [--open]` | Rebuild the self-contained HTML report |
| `qe coverage [RUN] [--trend [--grid] [--since DATE] [--last N \| --all]] [--open]` | Intent coverage for a run (default: the latest finished), or across runs: per intent family the latest and previous pass rate, the change, best and worst, and a sparkline of the last 20 runs (fixed width however many runs there are); `--grid` shows every number, one column per run; `--since` keeps runs from a date on; `--open` opens the report, whose grid holds every run |
| `qe defects [RUN] [--open]` | Defect list for the app team (HTML and CSV) |
| `qe junit [RUN] --out FILE [--flaky …] [--baseline …]` | JUnit XML for a finished run |
| `qe mcp serve` | Run the harness as an MCP server for AI assistants |

## What is checked per turn

| Stage | Checks |
|---|---|
| Intent | The router's primary intent is one of the expected intents |
| Orchestration | Writes only after approval; no unexpected approval prompts (approval steps approve only the expected action with matching arguments; anything else is declined and fails here); no repeated identical calls |
| MCP tool call | Required calls succeeded with the expected arguments; forbidden calls absent; **trace and proxy agree** |
| Backend state | Only the expected tables changed; SQL checks; **declined → nothing changed; data changed → a write succeeded** |
| User response | Required facts present, forbidden phrases absent, optional LLM grader for consistency with what the tools did |

Each failure records its first failing stage, so developers know where to look. With `--repeats N`, every case gets
a pass rate with a 95% Wilson range and unstable cases are reported as flaky.

## Parallel runs (one sandbox per worker)

Every run resets its sandbox, so one sandbox runs one test at a time (a lock enforces it). To go faster, give the
harness several identical sandboxes: list the extra ones under `workers:` in the config, each with its own URL,
read-only data path and proxy log folder.

```yaml
workers:
  - {base_url: http://127.0.0.1:8101, sor_path: data/sandbox-2.db, proxy_log_dir: qe_runs/proxy-2}
  - {base_url: http://127.0.0.1:8102, sor_path: data/sandbox-3.db, proxy_log_dir: qe_runs/proxy-3}
```

```bash
qe doctor --workers              # check every sandbox
qe run --judge --parallel 3      # the main sandbox plus two workers
```

Work is split into (repeat, suite) units; each runs on the next free worker from a fresh reset of that sandbox, and
the results merge into one run and one report. Suites must be independent for this: each is one test user's
conversation, and its data checks should be scoped to that user's records. The sandboxes must report the same
versions and clock, or the run refuses to start.

## Intent coverage

List the app's intent families under `intents:` in `qe.yaml`. Each report shows, per family, the cases that test it,
their pass rate and a status (covered, thin, failing, not tested), plus a families × runs grid of how coverage
changes over time. In the terminal, `qe coverage --trend` gives one line per family that stays the same width however
many runs there are:

```
21 run(s) · 11-03 21:39 → 11-05 09:12 · sparklines oldest → newest, · = not tested
                   latest   prev  change   best  worst  runs
modify_policy         33%    67%     -33    67%    33%  ·▆▆▆▆▆▆▆▆▆▆▆▃▃▃▆▆▆▃▃
general_question     100%   100%       0   100%     0%  ·▁▁▁▁▁▁▁████████████
```

`--since 2026-11-04` keeps only runs from that date on (for example after the catalog changed), `--last N` or `--all`
sets how many runs the sparklines cover (default 20), and `--grid` prints every number, one column per run, as many
recent runs as fit the window. In the report the grid keeps every run and scrolls sideways.

## Defect list for the app team

Every report comes with a defect list next to it (`defects.html` and `defects.csv`, linked from the report header);
`qe defects [RUN] --open` rebuilds and opens it. One entry per case that failed, cases that fail every time first,
then flaky ones. Each entry has:

- what the customer said, what the catalog expected and why, and what went wrong at each stage;
- the app's reply and what it did (intent, tool calls, approval prompts it shouldn't have shown, data changed);
- how often it failed, in this run and across all earlier runs;
- how to reproduce it (which test user, which earlier messages of the conversation);
- a link to the evidence in the report (`report.html#case=ID&repeat=N` opens that case and try).

Severity comes from what went wrong: **critical** when the app did or tried to do something it shouldn't (a
declined stray approval, a forbidden call, an unexpected data change, a trace that disagrees with the proxy);
**major** when it didn't do what was asked, or the grader found the reply contradicts the tools; **minor** when the
action was right but the reply missed something, or the request was misclassified. A catalog case can override it
with `severity: critical | major | minor`. The CSV imports into Jira, Azure Boards or a spreadsheet.

## In CI: JUnit, baselines and nightly pipelines

```bash
qe run --repeats 3 --judge --junit qe-results.xml --flaky skip --baseline qe-baseline.json
```

- `--junit FILE` writes JUnit XML: one test suite per catalog suite, one test case per case, the failure message is
  the first thing that went wrong, with a link to the evidence.
- `--flaky fail|skip|pass` decides how a case that failed only some tries counts (default `fail`).
- `--baseline RUN|latest|summary.json` fails the build only on **regressions**: cases that passed every try in the
  accepted baseline (or are new) and don't now. Cases that already failed there are reported as skipped ("known
  failure"), so they stay visible without turning every build red. Accept a baseline by committing a run's
  `summary.json`.
- The exit code is 0 when nothing counts as failed, 1 when something does, 2 when the run couldn't start.
- `qe junit [RUN] --out FILE` does the same for a run that has already finished.

`qe init --ci github` writes a nightly GitHub Actions workflow (`.github/workflows/qe-nightly.yml`) and
`qe init --ci azure` an Azure DevOps pipeline (`azure-pipelines-qe.yml`). Both install `qe-harness`, wait for the
sandbox with `qe doctor`, run the suites with a baseline, publish the JUnit results and keep the report and defect
list as build artifacts. Add the secrets listed at the top of the file; use a self-hosted runner or agent if the
sandbox is only reachable inside your network.

## In AI assistants (MCP)

```bash
claude mcp add qe-harness -- qe mcp serve
```

For Cursor or VS Code, add a stdio server with command `qe` and args `["mcp", "serve"]`.

Tools: `sandbox_status`, `sandbox_doctor`, `list_cases`, `get_case`, `run_tests` (changes state; capped per call),
`get_run_status`, `list_runs`, `get_failure` (masked evidence, marked as untrusted data), `get_intent_coverage`,
`get_intent_trend`, `get_defects` (the defect list, filterable by severity). Prompt: `triage_run`.

## LLM grader (optional)

`qe run --judge` asks a model whether each reply is consistent with what the tools actually did. It uses its own
settings, separate from the app's: `QE_JUDGE_PROVIDER` = `AZURE` (with `QE_JUDGE_AZURE_API_KEY`,
`QE_JUDGE_AZURE_ENDPOINT`, `QE_JUDGE_AZURE_DEPLOYMENT`) or `ANTHROPIC` (with `ANTHROPIC_API_KEY`, optional
`QE_JUDGE_MODEL`).

## Status

Alpha. Not yet included: fault injection, load and soak testing, simulated customers and robustness variant
generation. The CI baseline is a regression gate; a gate on absolute thresholds (for example "no critical defects")
is not built in yet.

## License

MIT
