Metadata-Version: 2.5
Name: truassure
Version: 1.0.0
Summary: TruAssure: black-box, evidence-based quality engineering for AI agents (chat, task and multi-agent).
Author: Shan Konduru
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: jsonschema>=4.21
Requires-Dist: pyyaml>=6.0
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: azure
Requires-Dist: openai>=1.40; extra == 'azure'
Provides-Extra: chat
Requires-Dist: websockets>=12; extra == 'chat'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: fastapi>=0.115; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: langgraph>=0.2; extra == 'dev'
Requires-Dist: mcp>=2.3; extra == 'dev'
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: starlette>=0.37; extra == 'dev'
Requires-Dist: types-jsonschema>=4.21; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Requires-Dist: uvicorn>=0.30; extra == 'dev'
Requires-Dist: websockets>=12; extra == 'dev'
Provides-Extra: langgraph
Requires-Dist: langgraph>=0.2; extra == 'langgraph'
Provides-Extra: mcp
Requires-Dist: mcp>=2.3; extra == 'mcp'
Provides-Extra: otel
Requires-Dist: starlette>=0.37; extra == 'otel'
Requires-Dist: uvicorn>=0.30; extra == 'otel'
Provides-Extra: postgres
Requires-Dist: psycopg[binary]>=3.2; extra == 'postgres'
Provides-Extra: proxy
Requires-Dist: httpx>=0.27; extra == 'proxy'
Requires-Dist: starlette>=0.37; extra == 'proxy'
Requires-Dist: uvicorn>=0.30; extra == 'proxy'
Description-Content-Type: text/markdown

# TruAssure: Enterprise Agentic Quality Engineering Accelerator

TruAssure is a black-box, evidence-based quality engineering framework for AI agents of any kind:
conversational assistants, coded task agents (LangGraph, OpenAI Agents SDK, custom loops), agentic
applications and multi-agent orchestrations.

For each run it answers questions without relying on the agent's own account of itself:

- Did the agent understand the task?
- Did it plan sensibly?
- Did it take only allowed and correct actions?
- Did it leave the environment in the right end state?
- Did it report truthfully what it did?

It also checks whether the agent behaves reliably (pass^k), robustly (input perturbations, injected faults)
and the same way across deployments (parity).

> **Status: pre-alpha, phase 2 of 6 done.**
> - **Core engine:** episode runner, evidence store, five-stage evaluation with the evidence seams, Wilson
>   ranges, and `ta run|show|report|gc`.
> - **Regression gate:** TruAssure reproduces the reference verdicts on 4,221 stored chat turns.
> - **Live runs:** it drives the real chat assistant on Docker (`SUT/`) through its chat channel.
>
> See [Validated on](#validated-on) for what has been checked and how.
> [docs/planning/PLAN.md](docs/planning/PLAN.md) has the full plan and [docs/brief/BRIEF.md](docs/brief/BRIEF.md)
> the original brief.

## How it judges an agent

- **Episode:** a task given to an agent in a prepared environment, run until it finishes, times out or runs
  out of budget.
- **Checkpoint:** a point where the episode is judged. A chat has one per user turn; a task agent usually has
  one at the end.

Each checkpoint gets a verdict at five stages, and records the first stage that failed:

| # | Stage | Question |
|---|---|---|
| 1 | Goal | What did the agent decide the task was (an intent, a routing decision)? |
| 2 | Plan | Were the steps, handoffs, loops, termination and approvals sensible? |
| 3 | Action | Was every action correct, authorised and given the right arguments? |
| 4 | End state | Is the database, file tree, git repository or SaaS system right afterwards? |
| 5 | Output | Is the reply, artifact or pull request right, and consistent with what was actually done? |

TruAssure gathers evidence from independent sources and keeps each source apart:

- the agent's own trace;
- an observer at the action boundary (an MCP or REST recording proxy, OpenTelemetry GenAI spans, or an
  agent-to-agent tap);
- before and after snapshots of the system of record;
- the agent's outputs.

It cross-checks these sources against each other through the evidence seams:

- **SE-01** trace vs observer
- **SE-02** authorisation vs state
- **SE-03** silent write
- **SE-04** phantom success
- **SE-05** output vs action
- **SE-06** deployment parity
- **SE-07** identity propagation, a TruAssure extension that is not counted among the catalogue's 71 types

A grader that can't reach a verdict reports *not evaluated*, which counts as a failure, never a pass.

Every failure carries a structured code (for example `ACTION.FORBIDDEN_CALLED` or `SE-03.SILENT_WRITE`, listed
in [`codes.py`](src/truassure/core/codes.py)), so severity rules, the scorecard and release gates can match on
codes rather than message text.

An agent that exposes no trace of its own can still be tested ("black-box mode"). In that mode its actions
come from the observers only, and the goal stage reports *not evaluated*.

## Layout

```
src/truassure/
  core/        model, Protocols, catalog + JSON Schema, compiler, runner, stages/, stats, store, security,
               runlock, report, CLI; run-level analysis: reliability (pass^k), robustness, compare (parity),
               scorecard/ (the test-type catalogue), governance/ (defects, JUnit, packs, release gate),
               calibration, ci (pipeline templates). Imports only stdlib, PyYAML, jsonschema
  adapters/    drivers, observers, state, graders  (framework-specific code lives here)
  profiles/    conversational (incl. the suite-catalog translator), task, workflow
  testing/     fakes of every Protocol, the contract-suite harness, and a scripted agent for offline runs
tests/         unit/  contract/  golden/  e2e/  pilots/
pilots/        the pilot catalogs (claims/: pilot 1, desk/: pilot 2) and the grader calibration set
tools/         check_core_imports.py (the core layering rule, an allow-list)
               make_golden.py, make_reference_corpus.py, make_reference_runlevel.py (reference outputs
               from the source functions, for the regression gate; dev-only)
scripts/       setup / test / lint, smoke-chat, pilot-claims, pilot-desk, as .bat and .sh pairs
docs/          brief/ (original brief and prompt)  planning/ (PLAN.md)
```

## Quick start

**Start here:**

- **Demo runbook:** [`docs/demo/TruAssure-Demo-Runbook.html`](docs/demo/TruAssure-Demo-Runbook.html), a self-contained
  page that walks through every feature, step by step, with expected output.
- **Client overview:** [`docs/client/TruAssure-Client-Overview.html`](docs/client/TruAssure-Client-Overview.html),
  covering features, business value and use cases.
- **One script for everything:** `scripts\truassure-ops.bat` (Windows) or `sh scripts/truassure-ops.sh`.
  - `setup`: every virtual environment.
  - `lint`, `test-all`, `test-sut all`: TruAssure's and each SUT's tests.
  - `deploy all`, `status`, `launch all`: the SUTs on Docker Desktop.
  - `pilots`: the live runs.
  - `demo`: an offline tour.
  - `console`: browse runs, reports, comparisons and gates in the browser.
  - `mcp`: serve TruAssure to an AI assistant over MCP.
  - `build`: build the wheel and smoke-test it in a clean venv.
  - `help`: every command.
- **Launchers:** `scripts\launch-console.bat` (the console over `ta_runs`), `scripts\launch-demo.bat` (the
  offline demo, then the console over its runs) and `scripts\launch-mcp.bat` (the MCP server), each with a
  `.sh` twin.

```bat
scripts\setup.bat          :: or: sh scripts/setup.sh
scripts\test.bat -q        :: tests + the 95% line-and-branch coverage gate
scripts\lint.bat           :: ruff, ruff format --check, mypy --strict, core import rule
```

The CLI is `truassure`, with `ta` as a short alias:

```bat
ta --version
ta validate tests\fixtures\catalogs\task.yaml
ta translate C:\myprojects\tool-calling-chatbot\qa\test_catalog.yaml --yaml -o catalog.yaml

:: run a catalog against an environment (here: the built-in scripted demo agent)
ta run tests\fixtures\catalogs\e2e_demo.yaml --env truassure.testing.scripted:demo_environment --perturbations all --repeats 3
ta show                     :: list runs
ta show latest              :: one run: per-case stages, result, Wilson range, reasons
ta report latest            :: write the run's self-contained HTML report
ta gc --days 30             :: retention: delete runs older than 30 days

:: run-level analysis and governance
ta compare A B              :: checkpoint by checkpoint; across deployments, parity against a noise floor (SE-06)
ta defects latest           :: the defect list (severity by failure code), defects.json and .csv
ta junit latest --flaky skip --baseline <run>   :: JUnit XML for CI; only regressions fail with a baseline
ta scorecard latest         :: the run against the 71-type catalogue (measured only with evidence)
ta pack claims.intake       :: a capability pack: ready / not ready
ta gate latest --policy truassure-gate.yaml     :: the release gate; exit 1 when a rule fails
ta calibrate pilots\calibration\harbor-consistency.yaml --grader <module:Class>   :: CE-02: Cohen's kappa
ta init --ci github --catalog <catalog> --env <factory>   :: a pipeline that runs, reports and gates

:: browse and automate
ta console --open           :: local console on 127.0.0.1 with a one-time URL token
ta mcp                      :: MCP server over stdio: one tool per command (install the 'mcp' extra)
```

**The report** (`ta report`) is one HTML file with everything inlined. It fetches nothing, so it works offline
and can be attached to a ticket. Its tabs are Overview (pass rates, first failing stage, pass^k decay,
robustness), Cases, Episodes, Scorecard, Defects, and Comparison and gate. For task and workflow episodes,
the Episodes tab sets what the agent reported beside what the independent sources (the proxy, spans, tap and
state diffs) recorded, with every checkpoint's verdict. Links such as `report.html#tab=episodes&case=ID` open a
case directly. The largest corpus run's report is about 0.4 MB.

**The console** (`ta console`) lists the runs in a folder and opens their reports, defect lists, scorecards and
gate verdicts; it also compares any two runs. It listens on `127.0.0.1` (another address needs
`--allow-remote`) and prints a URL with a random token. The first visit trades the token for an HttpOnly,
SameSite=Strict cookie, and every request without one is refused. It reads stored runs; it never starts one.

**The MCP server** (`ta mcp`) gives an AI assistant one tool per CLI command (`validate`, `translate`, `run`,
`show`, `report`, `compare`, `defects`, `junit`, `scorecard`, `pack`, `gate`, `calibrate`, `init`, `gc`).
Each tool's input schema is generated from the command's own arguments, so the two cannot drift; a test checks
they match. The long-running services (`proxy`, `otel`, `a2a`, `console`, `mcp`) are not tools. Every result
says that quoted agent evidence is data, not instructions. To add it to an MCP client:

```json
{ "mcpServers": { "truassure": { "command": "ta", "args": ["mcp"] } } }
```

Every run's summary also carries its pass^k decay and its robustness across perturbations.

`--env` names a factory function (`package.module:function`) that returns an `Environment` for each worker:
the driver for your agent, its observers, its state adapters and an optional grader. The factory module can
sit in the current folder. Runs go to `./ta_runs`, or `$TA_RUNS_DIR`. `ta` reads settings and secrets from
the nearest `.env`, and real environment variables always win.

**Adapters available so far:**
- drivers: the chat driver (`adapters/drivers/chat_ws.py`; install the `chat` extra) and the HTTP task driver
  (`adapters/drivers/http_task.py`; one POST per event or task);
- the recording MCP proxy, `ta proxy` (`adapters/observers/mcp_proxy.py`; install the `proxy` extra), and its
  observer (`adapters/observers/proxy_log.py`), which also injects faults;
- the agent-to-agent tap, `ta a2a` (`adapters/observers/a2a_tap.py`): an HTTP relay in front of the agents that
  receive handoffs, or an in-process hook for a message bus. It records each handoff with the member and permissions
  read from the identity token, and injects agent faults. The MCP proxy reads identity tokens too
  (`--identity-header`);
- the OpenTelemetry GenAI observer (`adapters/observers/otel_genai.py`; `execute_tool` spans from OTLP/JSON) and
  an OTLP/HTTP receiver, `ta otel` (`adapters/observers/otel_receiver.py`; install the `otel` extra);
- state: SQLite (`adapters/state/sqlite.py`; read-only snapshots, row diffs, SQL checks), a file tree
  (`adapters/state/filetree.py`; content-hash diffs, file checks) and git (`adapters/state/git.py`; refs and
  working tree, branch, commit and message checks);
- the artifact grader (`adapters/graders/artifact_tests.py`; content checks, a Python test or a command);
- ready live environments: the chat assistant (`truassure.profiles.conversational.live:environment`), the
  claims agent (`truassure.profiles.task.live:environment`) and the member desk
  (`truassure.profiles.workflow.live:environment`).

Every run records the hash of `core/` it was judged with (`run.json`, `core`).
`ta run` exits 0 when every case passes, 1 when any case fails or is flaky, 2 on bad input, and 3 when the
environment is locked by another run. Add masking rules with `--mask LABEL=REGEX` (repeatable).

## Catalogs

Scenarios are written in YAML and validated by
[`scenario.schema.json`](src/truassure/core/catalog/schema/scenario.schema.json) (JSON Schema Draft 2020-12).
Each scenario names a profile, and each profile has its own rules:

- **conversational:** has `checkpoints`, each with a `prompt`.
- **task:** has an `event` or `task` input, plus `expect`.
- **workflow:** adds `agents`, a `principal` and the expected `handoffs`.

Keys starting with `x-` are allowed for your own extensions. See [`tests/fixtures/catalogs/`](tests/fixtures/catalogs)
for an example of each profile.

`ta translate` converts a suite catalog (suites -> cases, one chat per suite) to this format. It doesn't drop anything silently: keys it can't
map are kept under `source.unmapped`. It never copies passwords.

## Validated on

| Pilot | What it proves | Status |
|---|---|---|
| 3 · regression gate | Same verdicts as the reference verdicts stored with the sandbox runs | **passed:** 4,221 of 4,221 turns in 40 runs (stdio, HTTP and REST transports), and every run's per-case results and Wilson ranges |
| 1 · task agent | Generic core on an event-driven agent with no chat | **passed:** the 15-scenario catalog passes offline (20 of 20 episodes); each of 6 planted defects fires the right code at the right stage; live, it finds real agent defects (below) |
| 2 · multi-agent | Handoffs, identity propagation, cascading failure (SS-09) | **passed:** the 12-scenario catalog passes offline (15 of 15 episodes); each of 5 planted defects fires the right code at the right stage; pilots 1 and 2 pass on the same core hash (each run records it); live, it finds real defects (below) |
| 3 · run level (phase 5) | Same run-level results as the reference, on the same corpus | **passed:** 225 checks: pass^k, robustness, defect severities, JUnit decisions (every flaky mode, and with a baseline), every scorecard row of all 40 runs, and 19 run comparisons with their parity verdicts |
| 1 · parity (SE-06) | Two builds of the claims agent (plain loop, LangGraph) judged against a noise floor | **passed:** judged "holds" offline, and live on Azure (15 checkpoints; the one difference was within run-to-run noise) |
| report (phase 6) | Self-contained, offline HTML for the largest corpus run | **passed:** about 0.4 MB (limit 5 MB); nothing is fetched from the network ([`tests/golden/test_report_size.py`](tests/golden/test_report_size.py)) |
| MCP server (phase 6) | One tool per CLI command, driven by a real MCP client | **passed:** the tool inventory and every tool's arguments equal the CLI's ([`tests/e2e/test_mcp_server.py`](tests/e2e/test_mcp_server.py)) |
| wheel (phase 6) | `python -m build`, then the wheel alone in a clean venv | **passed** on Windows and Linux in CI ([`tools/wheel_smoke.py`](tools/wheel_smoke.py)): console scripts, packaged schema and catalogue, an offline run, report, gate and the MCP tool list |

**How the regression gate works**
- [`tools/make_reference_corpus.py`](tools/make_reference_corpus.py) turns the source project's stored
  conversational runs into a corpus in [`tests/golden/reference/corpus`](tests/golden/reference/corpus). The
  corpus is trimmed to what evaluation and run-level analysis read, masked, and about 1.1 MB.
- A run is admitted only if the reference code reproduces every one of its stored verdicts from those exact
  inputs.
- TruAssure then has to give the same pass/fail, first failing stage, per-stage status, check counts and failure
  codes on every turn ([`tests/golden/test_regression_gate.py`](tests/golden/test_regression_gate.py)).
- 11 runs were excluded because they had been judged against catalog edits that were never committed. The
  [manifest](tests/golden/reference/corpus/manifest.json) lists each one with its reason.
- Run level: [`tools/make_reference_runlevel.py`](tools/make_reference_runlevel.py) runs the source's run-level
  functions on the same corpus; [`tests/golden/test_runlevel_gate.py`](tests/golden/test_runlevel_gate.py) rebuilds
  each run from TruAssure's own verdicts and must reproduce every result. The only allowed difference: types
  TruAssure measures beyond the reference (AT-02, AT-06, AT-08, SS-03, SS-09, PO-03, CE-02) read "not run" where
  the reference said "planned".

**Live smoke run** (`scripts\smoke-chat.bat <catalog> SRI GREET PRIVACY`): TruAssure drove the chat assistant
on Docker Desktop, with Azure OpenAI as the assistant's model.
- 22 of 23 checkpoints passed.
- The one failure was real agent behaviour: the assistant told a customer they had no policies without
  looking them up first.
- The chat assistant has no tool-traffic observer, so the trace-vs-observer seam is not checked there.

**Pilot 1, the claims agent** ([`pilots/claims`](pilots/claims)): TruAssure drives the agent through its event API
and watches it through the recording proxy, the agent's OpenTelemetry GenAI spans, the claims database and the
letters folder.
- Offline (the scripted model, the whole stack in-process): all 20 episodes pass (15 scenarios, plus 5 perturbed
  variants).
- Buggy build (`CLAIMS_DEFECTS`): six planted defects (silent write, phantom success, forbidden action, loop
  past budget, an artifact that fails its tests, a trace that leaves calls out) each fail their scenario at the
  planted stage with the planted code, offline. Live with Azure, the silent write and the trace omission fire
  `SE-03` and `SE-01` (proxy and spans) the same way.
- Live (`scripts\pilot-claims.bat`, Docker Desktop, Azure OpenAI gpt-4.1), three full runs. Every failure
  is real agent behaviour (in the third run, which also read the spans, they agreed with the agent's trace on
  every episode):
  - when the approval fails (tool error) or times out, it still writes the approval letter (every run);
  - asked to approve $2,500 on a $1,800 claim, it approves $1,800 on its own instead of refusing (every run);
  - for a claim already under review it sometimes also sends an acknowledgement letter, where the procedure
    says to add a note only (2 of 3 runs).

**Pilot 2, the member desk** ([`pilots/desk`](pilots/desk)): TruAssure drives the triage agent, and watches the
three agents through the A2A tap (handoffs), the MCP proxy (tool calls), both reading the identity token on the
wire, and the desk database.
- Offline: all 15 episodes pass (12 scenarios, plus 3 perturbed variants). Five planted defects fail their
  scenario at the planted stage with the planted code:
  - handoff order: `PLAN.HANDOFF_ORDER`;
  - a failure handed on: `PLAN.FAILURE_CASCADED`;
  - a hop for another member: `SE-07.PRINCIPAL_MISMATCH`;
  - no permission narrowing: `SE-07.PERMISSIONS_WIDENED`;
  - a hidden hop: `PLAN.HANDOFF_MISSING` plus `SE-01`.

  Live, the three agent-side defects fire the same codes.
- Live (`scripts\pilot-desk.bat`, Azure OpenAI gpt-4.1), two full runs. Every failure is agent behaviour:
  - when billing fails, it still hands off to claims (SS-09) and opens no case, and triage tells the member "a
    case has been opened" and gives the claim amount (both runs, in at least one of the two billing-failure
    scenarios);
  - when claims fails inside a two-part question, triage tells the member a case was opened when no agent opened
    one (both runs);
  - triage drops the amounts its specialists reported ("your balance and due date have been provided"): every
    run on the two-part question, and 2 of 3 phrasings of the billing question in the first run.

**Grader calibration (CE-02)** on a labelled set of 24 output-vs-action items
([`pilots/calibration`](pilots/calibration)):
- The deterministic rules grader scores kappa 0.75 (agreement 0.875). It misses only what needs reading for meaning:
  a denied record described anyway, a change claimed with no action, an approved amount stated as paid.
- The LLM grader (Azure OpenAI gpt-4.1) scored kappa 1.0, then 0.833 twice, over three runs. When it erred, it
  missed a case "opened" with no action and policy numbers no action returned. Within a run, grading twice gave the
  same verdicts.

Also checked:
- The ported primitives (`wilson`, `mentions`, `value_match`, `args_match`) give the same results as
  the original source functions on the inputs in [the reference outputs](tests/golden/reference/primitives.json).
- The reference evaluation cases, expressed in the generic model, pass, including the per-stage check counts.
- 59 synthetic episodes (52 planted with one defect each, covering every stage check, seam and workflow check; 7
  clean): every planted one fails at exactly the planted stage with the planted code; the clean ones pass.
- The translator converts the full Harbor Mutual sandbox catalog (93 cases) to a catalog that passes schema validation
  (a test that runs when the catalog is present on the machine).

## Security defaults

- **In place:** PII is masked before any evidence is written. The default rules cover email addresses,
  phone numbers and secret-named fields (password, token, API key, authorization). You can add your own
  regex rules. Masking never changes the values that verdicts are computed on. `ta gc` applies retention.
- **Listeners:** the proxy, tap, OTel receiver and console bind to `127.0.0.1` by default; another address needs
  `--allow-remote`. Their control endpoints need a token (`TA_PROXY_TOKEN`, `TA_OTEL_TOKEN` and the sandbox tokens; the
  console makes its own per start). Secrets come only from the environment or `.env`, and are never printed.
- **MCP:** tool results are marked as untrusted data, because they can quote what an agent said.
- **Releases:** `.github/workflows/release.yml` publishes to PyPI with trusted publishing (no stored token),
  only when a GitHub Release is published, through the `pypi` environment. Nothing has been published.

## Licence

MIT © 2026 Shan Konduru. All names and data in demos and fixtures belong to the fictitious company
*Harbor Mutual*.
