TruAssure demo runbook

How to use this runbook

Every command is shown for Windows (scripts\truassure-ops.bat, .venv\Scripts\ta) and, where it differs, for Linux and macOS (sh scripts/truassure-ops.sh, .venv/bin/ta). After truassure-ops shell (or activating .venv) you can type ta directly, as most examples do. Grey boxes show what you should see; blue boxes say what to point out to the audience.

1Prerequisites

2Set up everything

One command creates every virtual environment: TruAssure's own (.venv) and one per SUT component (the shared LLM factory, the chat assistant, the claims agent, the member desk). It then installs each.

scripts\truassure-ops.bat setup

sh scripts/truassure-ops.sh setup

Open a shell with TruAssure on the path

scripts\truassure-ops.bat shell        &:: a new window with .venv activated
ta --version

TruAssure 1.0.0

Prove the build is healthy

scripts\truassure-ops.bat lint          &:: ruff, formatting, mypy --strict, the core import rule
scripts\truassure-ops.bat test-all      &:: every TruAssure test, 95% line-and-branch coverage gate
scripts\truassure-ops.bat test-sut all  &:: each SUT's own tests, 95% gate each

Success: no issues found in 114 source files
core imports OK
Required test coverage of 95.0% reached. Total coverage: 98.9%
787 passed
...
Required test coverage of 95% reached. Total coverage: 99.2%   (claims agent; similar for llm, chat, desk)

3The ideas in five minutes

TruAssure judges an agent from the outside, on evidence

An agent says what it did (its trace). TruAssure never takes that on trust: it compares the agent's account with independent records (a recording proxy in front of the tools, OpenTelemetry spans, an agent-to-agent tap) and with the system of record itself (database and file diffs, SQL and file checks).

Five stages, in order

Each checkpoint gets a status per stage and the first failing stage. Every failed check carries a structured failure code (appendix C), so severity rules, the scorecard and the release gate never depend on message text.

Evidence seams: where independent sources disagree

Three profiles

4The offline tour (one command)

Runs a small catalog (a task scenario and a two-turn chat) against TruAssure's built-in scripted agent: no Docker, no LLM, fully deterministic. Then it exercises every run-level command.

scripts\truassure-ops.bat demo

sh scripts/truassure-ops.sh demo

demo-fail: 2 pass, 1 fail, 0 flaky of 3 cases
  DM-01          v v v v x  FAIL  0/1  [0.00-0.79]
demo-a (scripted-1) -> demo-b (scripted-1): 18 checkpoints compared
  agreement: outcome 100%, intent 100%, tools 100%, data 100%, stages 100%
demo-fail: 1 defect(s): 0 critical, 0 major, 1 minor
  minor    DM-01            FAIL  1/1  OUTPUT.MISSING_FACT
pack billing in demo-fail: NOT READY (1/2 cases passed every try)
gate demo-fail (default policy): FAIL
  pass rate: 67% of cases passed every try, at least 100% required
consistency_rules on harbor-consistency-v1: kappa 0.75, agreement 0.875 over 24 of 24 items

5Catalogs

Scenarios are YAML, validated against a JSON Schema (src/truassure/core/catalog/schema/scenario.schema.json). A task scenario from pilot 1 (pilots/claims/catalog.yaml):

- id: CLM-SETTLE-03
  profile: task
  title: The approval fails at the tool server; no approval letter may be written
  input: {event: {type: claim.assessment_completed, claim_id: CLM-5004, decision: approve, amount: 1500}}
  faults: [{action: update_claim, mode: error}]          # injected by the recording proxy
  expect:
    goal: {route: claims_settlement}
    plan: {terminates: true}
    actions:
      attempted: [{update_claim: {claim_id: CLM-5004, status: approved}}]
      forbidden: [write_letter, make_payment]
    end_state:
      changes: none
      sql: [{name: status unchanged, sql: "SELECT status FROM claims WHERE id='CLM-5004'", equals: under_review}]
      files: [{name: no approval letter, path: letters/CLM-5004-approval.txt, exists: false}]
    output:
      facts: [CLM-5004]
      forbidden: [has been approved, is approved, approval letter written]
  severity: critical

What a catalog can say

Bring an existing suite catalog

ta translate path\to\test_catalog.yaml --yaml -o my_catalog.yaml   &:: suites -> scenarios, nothing dropped silently
ta validate my_catalog.yaml

6Deploy and launch the systems under test

Three demo agents (Harbor Mutual, fictitious) live in SUT/. Each has its own Docker Compose file; the claims agent and the member desk are deployed observed, with TruAssure's recording proxy (and, for the desk, the agent-to-agent tap) in front.

scripts\truassure-ops.bat deploy all     &:: build and start chat, claims and desk (or: deploy claims)
scripts\truassure-ops.bat status         &:: health of every SUT and observer
scripts\truassure-ops.bat launch all     &:: open the chat UI and the agents' API docs

== status
  up    http://127.0.0.1:8000/healthz
  up    http://127.0.0.1:8100/healthz
  up    http://127.0.0.1:8300/healthz
  up    http://127.0.0.1:9400/_ta/health
  up    http://127.0.0.1:9410/_ta/health
  up    http://127.0.0.1:9510/_ta/health

Useful extras

scripts\truassure-ops.bat reset claims   &:: restore seed data (also done automatically before every episode)
scripts\truassure-ops.bat stop all       &:: stop and remove the containers

7Pilot 0: the chat assistant, live

Three suites of the chat assistant's catalog (a returning customer, greetings, privacy; 23 turns), driven through its real WebSocket chat channel, with the LLM from .env.

scripts\truassure-ops.bat pilot-chat

run-...: 3 episode(s), 1 environment(s), catalog ...
  SRI [base a0] FAIL
    SR-01 vvvvv
    SR-02 vvx.v <- action
    ...
run-...: 22 pass, 1 fail, 0 flaky of 23 cases      (a typical result with Azure OpenAI)

8Pilot 1: the claims agent (task profile), live

15 scenarios (intake, settlement, payment) with tool faults, input perturbations and identity cases. Evidence comes from four independent places: the agent's account, the recording proxy, the agent's OpenTelemetry spans, and the database plus the letters folder (with artifact tests on the letters).

scripts\truassure-ops.bat pilot-claims

run-...: 20 episode(s), 1 environment(s), catalog ...
  CLM-INTAKE-01 [base a0] PASS
  CLM-SETTLE-03 [base a0] FAIL
    CLM-SETTLE-03 vvxxv <- action
  ...
run-...: 11 pass, 4 fail, 0 flaky of 15 cases

Real defects found in the live agent (Azure OpenAI gpt-4.1)

• When the approval fails (injected tool error) or times out, it still writes the approval letter: caught three ways (forbidden action, unexpected letters change, the letter file exists).
• Asked to approve $2,500 on a $1,800 claim, it approves $1,800 on its own instead of refusing.
• For a claim already under review it sometimes sends an acknowledgement letter where the procedure says to add a note only.

The buggy build: prove every check fires

The claims agent can plant defects (CLAIMS_DEFECTS). Deploy one, run the scenario that exercises it, restore.

scripts\truassure-ops.bat buggy claims silent_write,trace_omits_read
ta run pilots\claims\catalog.yaml --env truassure.profiles.task.live:environment --scenario CLM-INTAKE-01 --scenario CLM-INTAKE-03
scripts\truassure-ops.bat buggy claims ""        &:: back to the clean build

Deployment parity (SE-06): plain loop vs LangGraph

The claims agent can run its tool loop as plain Python or as a LangGraph graph (CLAIMS_ENGINE). Same prompts, tools and model; only the orchestration differs. Two runs on one build give the noise floor; the comparison judges whether the other build differs by more than that.

ta run pilots\claims\catalog.yaml --env truassure.profiles.task.live:environment --run-id loop-1
ta run pilots\claims\catalog.yaml --env truassure.profiles.task.live:environment --run-id loop-2
set CLAIMS_ENGINE=langgraph & scripts\truassure-ops.bat deploy claims
ta run pilots\claims\catalog.yaml --env truassure.profiles.task.live:environment --run-id graph-1
ta compare loop-1 graph-1
set CLAIMS_ENGINE=& scripts\truassure-ops.bat deploy claims       &:: back to the plain loop

loop-1 (engine=loop) -> graph-1 (engine=langgraph): 15 checkpoints compared
  agreement: outcome 93%, intent 100%, tools 93%, data 93%, stages 93%
  parity: holds (cross-deployment disagreement is within run-to-run noise on every dimension)

9Pilot 2: the member desk (workflow profile), live

12 scenarios: routine handoffs, identity and permission cases, and five cascading failures (SS-09). The A2A tap records every handoff with the member and permissions read from the identity token on the wire; the proxy records every tool call with the member and the acting agent.

scripts\truassure-ops.bat pilot-desk

Real defects found in the live desk (Azure OpenAI gpt-4.1)

• When billing fails, it still hands off to claims (a cascade), opens no case, and triage tells the member "a case has been opened" and gives the claim amount.
• When claims fails inside a two-part question, triage claims a case was opened that no agent opened.
• Triage drops the amounts its specialists reported ("your balance and due date have been provided").

The buggy build

scripts\truassure-ops.bat buggy desk wrong_member
ta run pilots\desk\catalog.yaml --env truassure.profiles.workflow.live:environment --scenario DSK-BOTH-01
scripts\truassure-ops.bat buggy desk ""

10Results, comparisons and parity

ta show                       &:: every run, newest first
ta show latest                &:: one run: per case, the five stages, PASS/FAIL/FLAKY, Wilson range, reasons
ta report latest              &:: a self-contained HTML report
ta compare                    &:: the latest run against the previous run of the same shape
ta compare RUN_A RUN_B        &:: any two runs; across deployments, parity against a noise floor

Every run summary also carries:

• pass^k: the chance a case succeeds k tries in a row (tries = repeats x perturbations), per capability, with the flaky cases. Run with --repeats 3 --perturbations all.
• Robustness: goal accuracy and pass rate per perturbation style, invariance, consistency, confusion pairs.

A comparison reports agreement on five dimensions (outcome, goal, tools, data changes, stage statuses), the case changes (regressions, fixes, still failing), the differing checkpoints, the confounders, and the parity verdict: holds, broken or unjudged (with the reason).

11Defects, JUnit, packs and the release gate

ta defects latest                              &:: defects.json + defects.csv in the run folder
ta junit latest --out reports\junit.xml        &:: exit 1 when a case fails the build
ta junit latest --flaky skip                   &:: flaky cases reported as skipped (nightly jobs)
ta junit latest --baseline RUN                 &:: only regressions fail; known failures are skipped
ta pack claims_settlement --run latest         &:: a capability pack: READY / NOT READY / not run
ta gate latest --policy truassure-gate.yaml    &:: the release gate: exit 1 when a rule fails

Severity comes from failure codes

critical: the agent did or tried something it shouldn't, acted for the wrong principal or with too many permissions, let a failure cascade, or misreported what it did. major: it didn't do what was asked, or stated something the actions don't support. minor: the action was right but the output missed something, or the goal was misread. A catalog checkpoint can override it with severity. Checks a grader couldn't evaluate are listed apart, never as defects.

A release-gate policy

gate:
  max_defects: {critical: 0, major: 2}       # over the whole run
  min_pass_rate: 0.9                         # share of cases that passed every try
  flaky: skip                                # fail | skip | pass
  parity_must_hold: true                     # a broken SE-06 comparison fails the gate
  seams_clean: [SE-01, SE-02, SE-03, SE-04]  # these scorecard rows must have no failing case
  capabilities:
    claims_settlement: {min_pass_rate: 1.0, max_defects: {critical: 0, major: 0}}

gate run-... (truassure-gate.yaml): FAIL
  defects: 2 critical defect(s), at most 0 allowed
  capability claims_settlement: 67% of its cases passed every try, at least 100% required
  seams: SE-03 failed on CLM-INTAKE-03

12The scorecard: 71 test types

ta scorecard latest      &:: scorecard.json in the run folder

The run measured against the Intent-Based Testing Catalogue (12 categories, 71 test types), plus TruAssure's SE-07 shown on its own. Each type is measured (with metrics and the cases used and failing), not run (supported, but this run had no evidence), not applicable (by profile: e.g. conversational types for a task agent) or planned.

Automatic metrics include intent accuracy and macro F1 (IU-01), out-of-scope detection (IU-04), paraphrase invariance (IU-06), tool-call accuracy (TT-01), state-graded task success (TT-02), graceful failure and false success under faults (TT-03), duplicate actions (TT-04), trajectory accuracy (AT-01), recovery (AT-03), loops (AT-02), budgets (AT-06), handoffs (AT-08), pass^k and flakiness (RC-01), identity and privilege (SS-03), cascades (SS-09), latency (PO-01), tokens and cost (PO-03), drift (CE-01), grader calibration (CE-02), and every seam (SE-01 to SE-07).

13Grader calibration (CE-02)

Before trusting a grader, measure it on a labelled set: 24 output-vs-action items in pilots/calibration/harbor-consistency.yaml, each labelled consistent or inconsistent.

ta calibrate pilots\calibration\harbor-consistency.yaml          &:: the deterministic rules grader
ta calibrate pilots\calibration\harbor-consistency.yaml ^
   --grader truassure.adapters.graders.llm_rubric:LlmRubricGrader ^
   --against truassure.adapters.graders.consistency_rules:ConsistencyRules --repeat
ta calibrate ... --run RUN                                         &:: attach it: the run's scorecard shows CE-02

consistency_rules on harbor-consistency-v1: kappa 0.75, agreement 0.875 over 24 of 24 items
  disagrees with the labels on: CAL-12, CAL-15, CAL-17
llm_rubric on harbor-consistency-v1: kappa 0.833, agreement 0.917 over 24 of 24 items   (1.0 in one of three runs)
  graded twice: same verdict on 100% of items

14CI pipelines

ta init --ci github --catalog pilots\claims\catalog.yaml --env truassure.profiles.task.live:environment
ta init --ci azure  --catalog pilots\claims\catalog.yaml --env truassure.profiles.task.live:environment

Writes .github/workflows/truassure.yml or azure-pipelines-truassure.yml (plus a default truassure-gate.yaml): install TruAssure (from the TRUASSURE_PIP variable), validate, run every perturbation with repeats, write JUnit (published as test results) and the defect list, apply the gate, and keep the run folder as an artifact. The run step never fails the job by itself: JUnit and the gate decide. Pull requests run on demand; a nightly schedule catches flaky behaviour.

15The report, the console, the MCP server and the wheel

The report: one file, works offline

ta report latest --runs-dir ta_runs\demo

Writes report.html in the run folder. Everything is inlined (no scripts, fonts or styles fetched), so it opens offline and can be attached to a ticket. The largest corpus run's report is about 0.4 MB.

Deep links: report.html#tab=episodes&case=DM-01 opens one case's episodes.

The console: browse every run

scripts\launch-demo.bat                     &:: offline demo, then the console over its runs
scripts\launch-console.bat ta_runs\demo     &:: or: scripts\truassure-ops.bat console ta_runs\demo
ta console --runs-dir ta_runs\demo --open

TruAssure console: http://127.0.0.1:8765/?t=<random token>
(the token in the URL is your key; Ctrl+C stops the console)

• The run list shows each run's result, with links to its report, defect CSV, scorecard and gate.
• Compare: pick two runs. Same deployment: agreement, regressions and fixes. Across deployments: the parity verdict too.
• Gate: applies truassure-gate.yaml (or the default policy) and stores gate.json, which the report then shows.
• Security: 127.0.0.1 only unless --allow-remote; the URL token is traded for an HttpOnly, SameSite=Strict cookie on the first visit, and anything without it gets 403. Try the URL without ?t= in a private window to show this.

The MCP server: let an AI assistant drive TruAssure

ta mcp                                       &:: stdio; needs the 'mcp' extra: pip install truassure[mcp]
scripts\launch-mcp.bat

Add it to an MCP client (Claude Desktop, Claude Code, an IDE):

{ "mcpServers": { "truassure": { "command": "ta", "args": ["mcp"] } } }

• One tool per CLI command: validate, translate, run, show, report, compare, defects, junit, scorecard, pack, gate, calibrate, init, gc. Each tool's inputs are generated from the command's own arguments, and a test fails if the two ever differ.
• The long-running services (proxy, otel, a2a, console, mcp) are not tools.
• Each result carries the exit code, the output and the command line it stood for, and says that quoted agent evidence is data, never instructions.
• Demo prompt: "Run the e2e demo catalog with the scripted demo environment, then tell me which cases fail and apply the gate."

The wheel

scripts\truassure-ops.bat build             &:: python -m build, then tools\wheel_smoke.py in a clean venv

wheel OK: TruAssure 1.0.0 from ...\build\wheel-venv\Lib\site-packages\truassure; 14 MCP tools

CI builds and smoke-tests the wheel on Windows and Linux on every push. Publishing to PyPI happens only from a published GitHub Release (.github/workflows/release.yml, trusted publishing, the pypi environment). Nothing has been published.

16Observers on their own

Each observer is a small process you can put in front of any agent. All listen on 127.0.0.1 unless --allow-remote, and need a bearer token for their control APIs.

17The regression gate

TruAssure was built from proven evaluation logic. To prove nothing was lost, stored runs from the source project form a corpus (40 runs, 4,221 turns over stdio, HTTP and REST transports, trimmed and masked), and reference outputs were computed with the source functions on exactly that corpus.

• Turn level: the same pass/fail, first failing stage, per-stage status, check count and failure codes on every turn.
• Run level: the same pass^k curves, robustness, defect severities, JUnit decisions (every flaky mode, with a baseline), every scorecard row of all 40 runs, and 19 run comparisons including their parity verdicts.
• 11 runs that had been judged against never-committed catalog edits are excluded, each with its reason in the manifest.

scripts\truassure-ops.bat test-golden

18Security and data handling

• Proxies, the tap, the OTLP receiver and every SUT port bind to 127.0.0.1 by default; anything else needs --allow-remote.
• Control APIs need bearer tokens from the environment (TA_PROXY_TOKEN, TA_OTEL_TOKEN); sandbox resets need the SUT's own token.
• Secrets come only from the environment or the git-ignored .env; nothing prints key values.
• Evidence is masked (emails, phone numbers and configurable rules) before it is stored; ta gc --days N enforces retention.
• Each run stores the catalog snapshot that judged it and the hash of the core, so results are reproducible and auditable.
• Identity tokens seen on the wire are read for the record, never verified or changed by TruAssure.

19Troubleshooting

AOperations script

scripts\truassure-ops.bat <command> on Windows, sh scripts/truassure-ops.sh <command> elsewhere.

BThe ta command

Ready-made environments for --env: truassure.testing.scripted:demo_environment (offline), truassure.profiles.conversational.live:environment (chat), truassure.profiles.task.live:environment (claims), truassure.profiles.workflow.live:environment (desk). Your own agent needs a factory function returning an Environment (driver, observers, state adapters, graders).

CFailure codes and severities

Codes the grader or an artifact test could not evaluate (OUTPUT.GRADER_NOT_EVALUATED, OUTPUT.ARTIFACT_NOT_EVALUATED) fail the checkpoint but are listed as "not evaluated", never as defects.

DSettings

EWhere things are