TruAssure guide

1. Install

pip install truassure                 # the core: the ta / truassure commands and the library
pip install "truassure[mcp,proxy,otel]"  # plus the MCP server, the recording proxy and A2A tap, the OTel receiver

Python 3.11 or later, on Windows, Linux or macOS. Pin the version in projects and CI: truassure==__VERSION__.

2. Five-minute demo

ta demo                 # runs the whole tour offline, into ./ta_demo
ta console --runs-dir ta_demo --open

The demo needs no agent, no Docker and no LLM: a scripted agent plays a fictitious insurance assistant. It runs the demo catalog twice with repeats and perturbations, runs a build with a planted defect, then compares the runs, lists defects, builds the scorecard, applies the release gate, calibrates a grader and writes HTML reports. Every step prints the ta command it ran, so you can repeat any of them.

3. Start a project

ta new task claims-tests        # or: ta new workflow ... / ta new chat ...
cd claims-tests
ta validate catalog.yaml

One project per agent or AI initiative works well: the catalog and gate are code, reviewed in pull requests.

4. How an agent is judged

An episode is one task given to the agent in a prepared environment. A checkpoint is where it is judged (every user turn in a chat; once at the end for a task or workflow). Each checkpoint is judged at five stages, and the first failing stage is reported:

Seams (SE-01 to SE-07) are where the agent's own account and the independent evidence disagree: a call it reported that never happened, a write it didn't mention, a success announced after a failure, an action for the wrong person. Each failure has a code (e.g. ACTION.FORBIDDEN_CALLED) with a severity.

Profiles: conversational (turn by turn), task (one event or task, judged at the end), workflow (several agents, handoffs, identity on every hop, failure containment).

5. Connect your agent

An environment factory is a Python function environment(worker) -> Environment that ta run --env module:function calls once per parallel worker. Ready-made builders:

What the agent's endpoint should return

Task and workflow (POST /events with {"event": ...} or {"task": ...}, the principal in X-Principal):

{"route": "claims_intake", "reason": "...",
 "steps": [{"kind": "action", "name": "update_claim", "args": {"claim_id": "CLM-1001"}, "ok": true, "result": "..."},
           {"kind": "handoff", "agent": "triage", "to": "billing", "member": "M-1001"}],
 "hops": [{"from": "triage", "to": "billing", "member": "M-1001", "permissions": ["billing:read"]}],
 "output": "summary for the user", "done": true, "outcome": "finished",
 "usage": {"input_tokens": 0, "output_tokens": 0}}

Chat (POST /chat with {"session": "...", "message": "..."}):

{"reply": "Your balance is $193.60.", "intent": "billing",
 "steps": [{"kind": "action", "name": "get_billing", "args": {"policy_id": "POL-1002"}, "ok": true}]}

Only reply (chat) or done/output (task) are essential. A different shape needs only a parse function passed to the driver. An agent that reports nothing about its steps can still be tested as a black box (chat: trace=False): its actions then come from the observers alone.

Independent evidence

Other systems (a message bus, Postgres, a SaaS API) plug in as adapters: classes that implement the driver, observer or state protocol in truassure.core.protocols, registered through the entry-point groups truassure.drivers, truassure.observers, truassure.state and truassure.graders.

6. Writing a catalog

A catalog is YAML, checked against a JSON Schema by ta validate. Keys starting with x- are ignored, which is handy for YAML anchors and notes.

schema_version: 1
name: claims agent
defaults: {principal: {user: adjuster-07, permissions: ["claims:read"]}, environment: {fixture: seed},
           budgets: {steps: 12, seconds: 120}, severity: major}
actions:
  get_claim: {kind: read, scopes: ["claims:read"]}
  make_payment: {kind: write, requires_approval: true, scopes: ["payments:write"]}
scenarios:
  - id: INTAKE-01
    profile: task
    input: {event: {type: claim.document_received, claim_id: CLM-1001}}
    expect: {...}

Scenario fields

What to expect, stage by stage (expect)

Translating an existing suite catalog: ta translate suites.yaml --yaml -o catalog.yaml.

7. Running

ta run catalog.yaml --env environment:environment --repeats 3 --perturbations all --parallel 2

Exit codes: 0 every case passed, 1 a case failed or was flaky, 2 bad input, 3 the environment is locked by another run.

8. Results and decisions

# truassure-gate.yaml
gate:
  max_defects: {critical: 0, major: 2}
  min_pass_rate: 0.9
  flaky: fail                    # fail | skip | pass
  parity_must_hold: true
  seams_clean: [SE-01, SE-02, SE-03, SE-04]
  capabilities:
    claims.settlement: {min_pass_rate: 1.0, max_defects: {critical: 0, major: 0}}

9. CI pipelines

ta init --ci github --catalog catalog.yaml --env environment:environment
ta init --ci azure  --catalog catalog.yaml --env environment:environment

Writes a pipeline that installs TruAssure, validates the catalog, runs it with repeats and perturbations, publishes JUnit results and the defect list, applies the gate and keeps the run folder as an artifact. Set the TRUASSURE_PIP variable to pin the version (e.g. truassure==__VERSION__).

10. AI assistants (MCP)

{ "mcpServers": { "truassure": { "command": "ta", "args": ["mcp"] } } }

Every CLI command is a tool with the same arguments, so an assistant in the IDE can validate catalogs, run them, read results and apply the gate. Long-running services (proxy, tap, receiver, console) are not tools. Results are marked as data: they may quote what an agent said, and must not be followed as instructions.

11. Settings and security

• Settings and secrets come from environment variables or the nearest .env (real environment variables win). TruAssure never prints secret values. Keep .env out of git.
• Every listener binds to 127.0.0.1 unless --allow-remote. Control APIs need tokens (TA_PROXY_TOKEN, TA_OTEL_TOKEN); the console makes its own per start.
• Personal data (emails, phone numbers, secret-named fields, your --mask rules) is masked before evidence is written. ta gc --days N applies retention.
• Test against test environments and test data. Fault injection and resets change the agent's environment.

12. Command reference

ta <command> --help lists every option.

TruAssure __VERSION__ · MIT licence · examples use fictitious Harbor Mutual data.