Prompt for building TruAssure

Here's a prompt for a fresh Claude Code session started in C:\myprojects\TruAssure-EnterpriseAgenticQualityEngineeringAccelerator. It reuses proven conversational code, builds the core/adapter/profile split, and only lets TruAssure call itself generic after pilots on agents that aren't chatbots pass.

# Build: TruAssure - Enterprise Agentic Quality Engineering Accelerator

You are starting a new project in C:\myprojects\TruAssure-EnterpriseAgenticQualityEngineeringAccelerator.
Work in phases. At the start, write PLAN.md (phases, acceptance criteria, open questions)
and wait for my approval before writing code. Commit at the end of each phase; ask before
pushing or publishing anything.

## Goal
A black-box, evidence-based quality engineering framework for AI agents of any kind:
conversational assistants, coded task agents (LangGraph, OpenAI Agents SDK, custom loops),
agentic applications, and multi-agent orchestrations. It must answer, with evidence that
does not rely on the agent's own account of itself: did the agent understand the task, plan
sensibly, take only allowed and correct actions, leave the environment in the right end
state, and report truthfully what it did? And does it do so reliably (pass^k), robustly
(input perturbations, injected faults) and the same way across deployments (parity)?

## Code to reuse (read it first)
C:\myprojects\tool-calling-chatbot (same author, MIT) holds a proven conversational-only
implementation of these ideas. Read its README.md, then these modules:
runner.py, evaluate.py, capture.py, mcp_proxy.py, rest_proxy.py, scorecard.py, passk.py,
robustness.py, pack.py, compare.py, defects.py, junit.py, report.py, console.py, judge.py.
Also read its review notes (known gaps) and positioning brief under docs/.
Reuse ideas and code where they fit, but never import that project at runtime. Keep its
proven behaviours:
- independent evidence: a recording proxy at the tool boundary, a before/after diff of the
  system of record, and the agent's trace, cross-checked against each other (the "evidence
  seams": trace vs proxy, approval vs state, silent writes, phantom success, reply vs action,
  transport parity);
- per-step verdicts at five stages, recording the first stage that failed;
- Wilson ranges, pass^k decay, flaky detection, phrasing/perturbation robustness, fault
  injection (error, timeout, empty) that copies the real server's response envelope;
- run comparison with confounder checks, and a parity verdict against a same-build noise
  floor (one-sided two-proportion z-test; the noise pair never includes the run under test);
- a scorecard against the 71-type Intent-Based Testing Catalogue (12 categories) with
  measured / not run / not applicable / planned; types are measured only from evidence;
- defects with severity rules, JUnit output, regression baselines, CI templates, and a
  self-contained HTML report;
- an MCP server that matches the CLI one-to-one.

## Generalised model (the core must not assume chat)
Unit of test = an Episode: a task given to an agent in a prepared environment, run until it
finishes, times out or exceeds its budget. An episode is a sequence of Steps; a step can be
a user turn, an event, a tool or API call, code execution, a browser action, a message to
another agent, or a handoff.
Five stages, generalised:
1. Goal: what the agent decided the task was (an intent for chat; a task interpretation or
   routing decision otherwise)
2. Plan / trajectory: steps, delegation, handoffs, termination, loops
3. Action: each action is correct, authorised, and has the right arguments (any action type)
4. End state: the environment afterwards (database, files, git repository, tickets, queues)
5. Output: the reply, artifact, pull request or report, and whether it is consistent with
   the actions actually taken

## Architecture (three layers)
- core/: the episode and evidence data model, stage evaluation, verdicts, statistics,
  scorecard, comparison, governance (severity, release gates, capability packs), storage of
  results, the report, the CLI and the MCP server. The core must have zero framework-specific
  imports.
- adapters/ (each behind a small typed Protocol, registered by entry point or config):
  - drivers: how an episode is started and driven (chat API, task API/HTTP, event or queue,
    CLI or subprocess, Python callable);
  - observers: independent records of actions (MCP proxy over stdio and HTTP, REST reverse
    proxy, OpenTelemetry GenAI span ingest, agent-to-agent message tap, sandboxed code
    execution log);
  - state: snapshots and diffs (SQLite and Postgres, a file tree, a git repository, mock SaaS);
  - graders: deterministic checks first; LLM rubric graders with a calibration set and
    agreement metrics (CE-02); grading of artifacts (run the tests for code, rubrics for
    documents).
- profiles/: catalog schemas plus defaults per kind of agent: conversational (the
  suite catalog shape: suites -> cases), task (fixtures, goal, allowed actions, success as an end state,
  budgets), workflow (multi-agent: expected handoffs, identity and permission propagation).

## Catalog (YAML, validated by a JSON Schema)
Each scenario has: id, profile, persona/principal, environment fixture, input (prompt,
event or task), perturbations (phrasings or variants), faults, expectations per stage
(goal, allowed/required/forbidden actions with argument checks, end-state checks as SQL,
file or git assertions, output facts), budgets (steps, tokens, cost, time), severity,
test_types, capability tags (capability packs generalise intent packs).

## Pilots (TruAssure may be called "generic" only after both pass with the core unchanged)
Build each pilot inside the repo under demo/, in the fictitious company "Harbor Mutual".
Never use real client or customer names anywhere.
1. Coded task agent: a small Python agent (LangGraph or a plain loop) that resolves an
   insurance back-office task through MCP tools against a SQLite system of record.
   Event-triggered, no chat.
2. Multi-agent orchestration: a triage agent that hands off to a billing agent and a claims
   agent, with handoffs, propagation of the user's identity and permissions, and one
   injected cascading failure (SS-09).
3. Regression gate: build the conversational adapter and run the sandbox's catalog against
   the existing sandbox (C:\myprojects\tool-calling-chatbot, `python demo/start_sandbox.py`).
   Verdicts must match the reference verdicts on the same run data (replay the runs that
   project has stored, no new model calls) before the conversational profile counts as done.

## Engineering rules
- Python 3.11+, fully typed, `pyproject.toml`, packages under src/truassure. The CLI is
  `truassure`, with `ta` as a short alias.
- Tests: pytest, offline, deterministic, with fakes for models, sandboxes and agents; no
  network. Add a test for every adapter Protocol. Use a "contract" test suite that every
  adapter must pass.
- Keep dependencies small; add optional extras per adapter (langgraph, otel, postgres).
- Security: proxies listen on 127.0.0.1 by default with token authentication; secrets come
  from the environment, never from files in the repo; PII masking and retention settings on
  stored evidence; the console needs a URL token.
- Don't overclaim: the scorecard marks a test type measured only when evidence exists; the
  README states what is validated and on which pilot.
- Windows-first, but scripts must also work on Linux/macOS (.bat and .sh pairs).

## Phases (propose details and estimates in PLAN.md)
0. Plan, repo skeleton, data model, adapter Protocols, catalog JSON Schema, CI.
1. Core: episode runner, evidence store, stage evaluation, statistics, verdicts, minimal
   CLI (`ta run`, `ta show`, `ta report`).
2. Conversational adapter + profile ported; regression gate (pilot 3) green.
3. Task profile, OTel observer, file/git state adapter, pilot 1.
4. Workflow profile, agent-to-agent observer, pilot 2.
5. Reliability and governance: pass^k, perturbations, faults, compare/parity, capability
   packs, release gate, defects, JUnit/CI, scorecard.
6. Report, console and MCP server; docs; packaging.

Start with phase 0: read the files listed above, then write PLAN.md and stop for my review.
