TruAssure decides whether an AI agent is fit to ship, so a wrong verdict from TruAssure costs as much as a bug in the agent. This page lists the practices that keep TruAssure itself honest. Every number on it comes from the repository and its CI, and each practice names the file or job that enforces it.
TruAssure's judging logic was rebuilt from a source project with a long history of stored runs. Rather than trust the rebuild, the test suite replays that history and requires the same answers, turn by turn and run by run.
For each of 4,221 turns in 40 stored runs, TruAssure must give the same pass or fail, the same first failing stage, the same status, check count and failure count for each stage, and the same failure codes. For each run it must give the same case results (pass, fail, flaky), pass counts and Wilson ranges.
tests/golden/test_regression_gate.py
The pass^k decay curve, flaky cases, robustness across phrasings, defect severities, JUnit decisions in every flaky mode, and run-to-run comparison (agreement, noise floor, z-scores, parity) must all equal the reference results.
tests/golden/test_runlevel_gate.py
Value matching, mention matching, argument matching and the statistics are checked one by one against outputs recorded from the original functions, so a drift shows up where it starts.
tests/golden/test_reference_primitives.py
tests/golden/reference/corpus/manifest.json)The reference files are generated, not hand-written: they come from the scripts
tools/make_golden.py, tools/make_reference_corpus.py and
tools/make_reference_runlevel.py, which run the original functions on the stored inputs. A size gate also keeps
the largest run's HTML report under 5 MB, self-contained, and working offline (tests/golden/test_report_size.py).
A check that never fails proves nothing. The demo agents ship with "buggy builds": defects planted in the real agent code and switched on by an environment variable. Each one copies a real class of agent failure, and the pilot tests require that it fails the scenario it belongs to, at the right stage and with the right code, while the clean build passes. An unknown defect name is an error, so a typo never runs a clean build by accident.
| Agent | Planted defect | What goes wrong |
|---|---|---|
Claims agentCLAIMS_DEFECTS | silent_write | a read tool also writes a claim note |
phantom_note | adding a note reports success but saves nothing | |
pay_on_settlement | the settlement procedure also pays the claim | |
loop | the agent keeps re-reading the claim and never finishes | |
vague_letter | acknowledgement letters leave out the claim and the document | |
trace_omits_read | the agent's own account leaves out its reads | |
Multi-agent deskDESK_DEFECTS | handoff_order | a two-part question goes to the specialists in the wrong order |
no_containment | a failed lookup is handed on instead of opening a case | |
wrong_member | one agent delegates to another for a different member | |
no_narrowing | handoffs keep every permission instead of narrowing to the receiving role | |
trace_omits_hop | the coordinator's account leaves out its specialists' handoffs |
Two of these, the omitted read and the omitted hop, are aimed at the evidence
itself: the agent's own trace lies, and TruAssure must notice by comparing it with an independent record. The offline demo
(ta demo) uses the same idea: one build leaves the balance out of its first reply, and the demo shows that
defect travelling through the report, defects, JUnit, scorecard, pack and gate. Code:
tests/pilots/, SUT/claims-agent/claims_agent/defects.py,
SUT/multi-agent-desk/desk/defects.py.
The whole suite runs on a laptop with no network, no LLM and no API keys. Tests that need real services say so and are skipped with a reason, never quietly passed.
| Layer | Tests | What it proves | |
|---|---|---|---|
| Golden | 418 | Verdicts and run-level results equal the reference outputs (section 1) | |
| Unit | 321 | Model, compiler, matching, scorecard, calibration, CLI errors, the import rule, adapters with fakes | |
| End to end | 124 | Real commands against scripted agents: runner, console, MCP server and proxy, A2A tap, starter projects, execution report | |
| Contract | 31 | One suite every adapter (driver, observer, state, grader) must pass, so an adapter that passes it is safe to plug in | |
| Pilots | 18 | Clean and buggy builds of the demo agents (section 2), and parity between two builds of one agent | |
| Integration | 8 | State and result stores against real Postgres, SQL Server and MySQL (CI starts them as services) |
The package's line and branch coverage must stay at or above 95%. scripts/test.* enforces it, so CI does too. It is 98% today.
pyproject.toml [tool.coverage.report] fail_under = 95
The claims agent, the multi-agent desk, the chat assistant and the shared LLM factory each have their own tests, lint and 95% coverage gate, and their Docker images must build. A flaw in a demo agent would make a pilot meaningless.
SUT/*/pyproject.toml
Unknown markers are errors (--strict-markers), every skip is listed with its reason (-ra),
and async tests get a fresh event loop each.
pyproject.toml [tool.pytest.ini_options]
Strict mode with unreachable-code warnings, over the package, the test suite and the build tools alike. Optional SDKs are the only exceptions, and each one is listed with the reason.
pyproject.toml [tool.mypy]
pycodestyle, pyflakes, import order, pyupgrade, bugbear, simplify, ruff's own rules and type annotations. Formatting is checked, not just suggested.
scripts/lint.sh, scripts/lint.bat
truassure.core may import only the standard library, PyYAML, jsonschema and itself. It is an
allow-list, so any new third-party import fails, whatever it is. Adapters and profiles reach core, never the
other way round, and import truassure.core never pulls in a framework.
The checker has its own tests: a planted framework import fails, even one hidden inside a function.
tools/check_core_imports.py, tests/unit/test_core_imports.py
Episodes, checkpoints, verdicts and evidence are typed dataclasses, and catalogs are validated against a JSON
Schema before anything runs (ta validate). A malformed catalog stops before it reaches an agent.
src/truassure/core/model.py, core/catalog/schema/
CI calls the same scripts a developer runs locally (scripts/setup, lint,
test, in .sh and .bat versions), so "works on my machine" and "works in CI" mean the
same thing. Workflow permissions are read-only by default, and a newer push cancels the older run.
ta demo, ta new, MCP toolsThe wheel job exists because a wheel can break in ways a source tree can't: a
data file left out of the package, or a console script missing. tools/wheel_smoke.py checks exactly those.
PyPI accepts uploads only from this repository's release workflow, through OpenID Connect. No PyPI token is
stored anywhere, and the pypi environment can require a reviewer's approval.
.github/workflows/release.yml
The release fails if the tag (v1.5.0) does not match __version__. The wheel is smoke-tested
again in a clean environment before it is published.
Every third-party GitHub Action is pinned to a full commit SHA, with the version in a comment, so a moved tag can't change what runs.
After each release, a fresh pip install truassure==X in an empty folder runs the demo again.
The PyPI validation runbook (docs/demo/) lets anyone repeat that check with no access to the repository.
Each release has notes saying what changed, how to install it and what was verified. Version history lives in
Git tags and GitHub releases; the plan (docs/planning/PLAN.md) records each phase's acceptance criteria.
scripts/security-audit.* runs gitleaks, trivy, bandit, semgrep, pip-audit, safety and CodeQL and
merges the results into one report. Secret values are redacted from every report. --offline runs only the scanners
that need no network, and no source code leaves the machine.
tools/security_audit.py
Rigour in the code is half the story. The other half is design rules that stop a verdict from looking better than the evidence behind it.
Tool calls, handoffs and data changes are checked against independent records: a recording proxy, OpenTelemetry spans, the database, files written. The agent's own account is one more source, and it is compared with the others (silent writes, phantom success, omitted steps).
A rubric no grader judged is OUTPUT.RUBRIC_NOT_EVALUATED. A case whose grader was unavailable is
reported as not evaluated, not as a defect and not as a pass. The agent's own error reply is blocked, not failed.
Each case runs several times. Results are pass, fail or flaky, with a Wilson confidence range and pass^k, and a test type with too few instances behind it is labelled low evidence.
ta calibrate measures a grader against human labels (Cohen's kappa) and lists the items it disagrees
on, before anyone relies on it.
Each run records the catalog's content hash and the agent version it tested, and comparisons flag confounders (a changed prompt, model or tool) before they report a regression.
A run resets its environment, so a lock file stops a second run from silently breaking the first. A lock whose
process has ended is released. (core/runlock.py)
Personal data is masked when evidence is written, never before evaluation, so verdicts use the real values and stored files hold only masked ones. Fields named like passwords, tokens, API keys, cookies or authorization headers are replaced whole. Retention of stored runs is enforced.
src/truassure/core/security.py
The execution report masks dates of birth, addresses, ZIP codes, emails, phone numbers and secrets, both as known values and as patterns, and keeps the amounts and record ids a reader needs.
core/execution_report/redact.py
Keys and passwords come from the environment or a git-ignored .env file, read by a small
standard-library loader. Even the demo login's password lives there, not in code.
core/envfile.py
Database URLs are shown with the password replaced by ***, including secrets in query parameters,
in logs, errors and reports. State adapters are meant to log in with a read-only login.
adapters/state/sql.py
Each phase in the plan states what must be true before it is done (a pilot that must fail, a gate that must pass), and each acceptance test's docstring names the phase it closes.
Every script comes as a .sh and a .bat, and CI runs both. Windows is not an afterthought.
The expected output in each runbook is recorded from a real run, and it is updated in the same change as any console output it shows.
Every adapter interface has a small, deterministic, working implementation, used by the contract suite and the core tests. It proves each interface can be implemented and lets whole episodes run offline.
src/truassure/testing/fakes.py
All demo companies, customers and policies are invented (Harbor Mutual). No real personal data is in the repository.
The offline agent used by tests and the demo can be told to misreport its trace, bypass the proxy, report a success that changed nothing, or act for another user. The tests drive each of those lies and expect them to be caught.
src/truassure/testing/scripted.py