TruAssure guide

1. Install

pip install truassure                 # the core: the ta / truassure commands and the library
pip install "truassure[mcp,proxy,otel]"  # plus the MCP server, the recording proxy and A2A tap, the OTel receiver

Python 3.11 or later, on Windows, Linux or macOS. Pin the version in projects and CI: truassure==__VERSION__.

2. Five-minute demo

ta demo                 # runs the whole tour offline, into ./ta_demo
ta console --runs-dir ta_demo --open

The demo needs no agent, no Docker and no LLM: a scripted agent plays a fictitious insurance assistant. It runs the demo catalog twice with repeats and perturbations, runs a build with a planted defect, then compares the runs, lists defects, builds the scorecard, applies the release gate, calibrates a grader and writes HTML reports. Every step prints the ta command it ran, so you can repeat any of them.

3. Start a project

ta new task claims-tests        # or: ta new workflow ... / ta new chat ...
cd claims-tests
ta validate catalog.yaml

One project per agent or AI initiative works well: the catalog and gate are code, reviewed in pull requests.

4. How an agent is judged

An episode is one task given to the agent in a prepared environment. A checkpoint is where it is judged (every user turn in a chat; once at the end for a task or workflow). Each checkpoint is judged at five stages, and the first failing stage is reported:

Seams (SE-01 to SE-07) are where the agent's own account and the independent evidence disagree: a call it reported that never happened, a write it didn't mention, a success announced after a failure, an action for the wrong person. Each failure has a code (e.g. ACTION.FORBIDDEN_CALLED) with a severity.

Profiles: conversational (turn by turn), task (one event or task, judged at the end), workflow (several agents, handoffs, identity on every hop, failure containment).

5. Connect your agent

An environment factory is a Python function environment(worker) -> Environment that ta run --env module:function calls once per parallel worker. Ready-made builders:

What the agent's endpoint should return

Task and workflow (POST /events with {"event": ...} or {"task": ...}, the principal in X-Principal):

{"route": "claims_intake", "reason": "...",
 "steps": [{"kind": "action", "name": "update_claim", "args": {"claim_id": "CLM-1001"}, "ok": true, "result": "..."},
           {"kind": "handoff", "agent": "triage", "to": "billing", "member": "M-1001"}],
 "hops": [{"from": "triage", "to": "billing", "member": "M-1001", "permissions": ["billing:read"]}],
 "output": "summary for the user", "done": true, "outcome": "finished",
 "usage": {"input_tokens": 0, "output_tokens": 0}}

Chat (POST /chat with {"session": "...", "message": "..."}):

{"reply": "Your balance is $193.60.", "intent": "billing",
 "steps": [{"kind": "action", "name": "get_billing", "args": {"policy_id": "POL-1002"}, "ok": true}]}

Only reply (chat) or done/output (task) are essential. A different shape needs only a parse function passed to the driver. An agent that reports nothing about its steps can still be tested as a black box (chat: trace=False): its actions then come from the observers alone.

Independent evidence

Other systems (a message bus, Postgres, a SaaS API) plug in as adapters: classes that implement the driver, observer or state protocol in truassure.core.protocols, registered through the entry-point groups truassure.drivers, truassure.observers, truassure.state and truassure.graders.

6. Writing a catalog

A catalog is YAML, checked against a JSON Schema by ta validate. Keys starting with x- are ignored by the checks, which is handy for YAML anchors and notes. One is read by the report: x-intents, every intent (or route) family the agent knows, as {name: description}. The report's Intents tab then lists each family's cases and pass rate, and names the families no case tests.

schema_version: 1
name: claims agent
defaults: {principal: {user: adjuster-07, permissions: ["claims:read"]}, environment: {fixture: seed},
           budgets: {steps: 12, seconds: 120}, severity: major}
actions:
  get_claim: {kind: read, scopes: ["claims:read"]}
  make_payment: {kind: write, requires_approval: true, scopes: ["payments:write"]}
scenarios:
  - id: INTAKE-01
    profile: task
    input: {event: {type: claim.document_received, claim_id: CLM-1001}}
    expect: {...}

Scenario fields

What to expect, stage by stage (expect)

Translating an existing suite catalog: ta translate suites.yaml --yaml -o catalog.yaml.

7. Running

ta run catalog.yaml --env environment:environment --repeats 3 --perturbations all --parallel 2

Exit codes: 0 every case passed, 1 a case failed or was flaky, 2 bad input, 3 the environment is locked by another run.

8. Results and decisions

# truassure-gate.yaml
gate:
  max_defects: {critical: 0, major: 2}
  min_pass_rate: 0.9
  flaky: fail                    # fail | skip | pass
  parity_must_hold: true
  seams_clean: [SE-01, SE-02, SE-03, SE-04]
  capabilities:
    claims.settlement: {min_pass_rate: 1.0, max_defects: {critical: 0, major: 0}}

9. CI pipelines

ta init --ci github --catalog catalog.yaml --env environment:environment
ta init --ci azure  --catalog catalog.yaml --env environment:environment

Writes a pipeline that installs TruAssure, validates the catalog, runs it with repeats and perturbations, publishes JUnit results and the defect list, applies the gate and keeps the run folder as an artifact. Set the TRUASSURE_PIP variable to pin the version (e.g. truassure==__VERSION__).

9a. Sharing and the portfolio

Each project's truassure.yaml (written by ta new) names the AI initiative it belongs to, the owning team and the shared store:

initiative: claims-automation
project: claims-agent-tests
owner: claims-quality
store: s3://quality-results/truassure      # or a folder / network share, or azure://<account>.blob.core.windows.net/<container>/<prefix>
policy: truassure-gate.yaml

Stores: a folder or network share (built in), Amazon S3 (pip install "truassure[s3]"; the standard AWS credential chain) and Azure Blob (pip install "truassure[azure-blob]"; a connection string or DefaultAzureCredential). TA_STORE overrides the file. Other stores plug in through the truassure.stores entry-point group.

9b. The scenario library

Reusable packs for risks every initiative shares. A catalog includes a pack and fills in its parameters with its own action names and data; the scenarios then run like hand-written ones.

ta library                         # the packs
ta library prompt-injection        # what it tests, its parameters, its scenarios
ta library privacy --add catalog.yaml

include:
  - pack: payments-approvals
    with: {user: shan, pay_prompt: "Pay $50 on POL-1002.", payment_action: make_payment,
           payment_args: {policy_id: POL-1002, amount: 50}, payment_scopes: [payments, policies]}
    only: [DECLINED, TOOL-FAILS]        # optional: a subset
    position: start                     # optional: run before the catalog's own scenarios (default: after)

Every pack tags its scenarios with catalogue test types, so they appear on the scorecard and in the execution report's coverage matrix.

Your own packs: a YAML file (pack: ./packs/claims-rules.yaml, relative to the catalog) or a package that registers one in the truassure.packs entry-point group. A pack has pack, params (each with a description, and an example or a default), optional actions, and scenarios using {{param}}. Included scenarios record their pack, version and parameters in source.

9c. Enterprise databases and connectors

Any SQL database SQLAlchemy supports, through one adapter: install truassure[sql] plus the driver extra (postgres, sqlserver, mysql, oracle; others: the dialect's own package).

from truassure.adapters.state.factory import make_state

make_state("postgresql+psycopg://reader:***@db:5432/claims", tables=["claims", "payments"])
make_state("mssql+pyodbc://reader:***@sql/claims?driver=ODBC+Driver+18+for+SQL+Server")
make_state("oracle+oracledb://reader:***@db:1521/?service_name=CLAIMS", schema="CLAIMS")
make_state("snowflake://reader:***@account/db/schema")      # any SQLAlchemy dialect
make_state("data/claims.db")                                # SQLite file
make_state("https://crm.test.example/api", resources={"cases": "/cases"}, items_path="data")   # REST

• Tables and primary keys are found by reflection (override with tables= and keys=); values are normalised so snapshots compare the same on every database.
• Read-only: read-only sessions where the database supports them (Postgres, SQL Server, MySQL/MariaDB, Oracle, SQLite), every session rolled back. Use a login that can only read.
• Passwords in URLs, including inside ODBC strings, are never printed.

Drivers beyond HTTP: OpenAICompatibleDriver (chat-completions endpoints, with conversation history), QueueDriver with ServiceBusTransport (truassure[servicebus]) or KafkaTransport (truassure[kafka]). task_environment(name=..., driver=..., state=[...]) combines any driver with any state adapters; http_chat_environment(..., driver=...) does it for chat.

The shared store can be a database too (ta push --store postgresql+psycopg://...): runs go into two ORM-mapped tables, ta_runs (one row per run: gate, pass rate, defects) and ta_run_files. A DBA can create them from truassure.adapters.stores.sql.ddl("postgresql://").

10. AI assistants (MCP)

{ "mcpServers": { "truassure": { "command": "ta", "args": ["mcp"] } } }

Every CLI command is a tool with the same arguments, so an assistant in the IDE can validate catalogs, run them, read results and apply the gate. Long-running services (proxy, tap, receiver, console) are not tools. Results are marked as data: they may quote what an agent said, and must not be followed as instructions.

11. Settings and security

• Settings and secrets come from environment variables or the nearest .env (real environment variables win). TruAssure never prints secret values. Keep .env out of git.
• Every listener binds to 127.0.0.1 unless --allow-remote. Control APIs need tokens (TA_PROXY_TOKEN, TA_OTEL_TOKEN); the console makes its own per start.
• Personal data (emails, phone numbers, secret-named fields, your --mask rules) is masked before evidence is written. ta gc --days N applies retention.
• Test against test environments and test data. Fault injection and resets change the agent's environment.

12. Command reference

ta <command> --help lists every option.

TruAssure __VERSION__ · MIT licence · examples use fictitious Harbor Mutual data.