TruAssure tests AI agents the way auditors check a business process: on independent evidence. It checks every action, every data change and every handoff against what the agent claims, under faults and rephrasings, and turns the result into release decisions your teams can trust.
An AI agent doesn't just answer questions: it pays, approves, updates records, writes letters and hands work to other agents. Testing only its replies misses the failures that cost the most.
A tool call fails or times out, and the agent tells the customer it worked, or carries on as if it had.
Data changes with no action on record: a read that writes, a side effect, a duplicate payment.
In multi-agent systems, a handoff carries another customer's identity, or more permissions than it should.
One agent fails, the next one acts on missing information, and the answer the customer gets is confidently wrong.
A new model version, provider, framework or host changes behaviour in ways nobody measured.
Using a model to check a model, without measuring how often that judge is right.
TruAssure needs no changes to your agent. It drives the agent through the channel your customers or systems already use, then compares the agent's own account with records it does not control.
Where independent sources disagree, TruAssure flags an evidence seam: the agent's account vs the tool record, declined but changed, silent writes, phantom success, answers that claim more than the actions did, behaviour that changes between deployments, and identity that doesn't propagate.
Chat assistants, event-driven task agents and multi-agent workflows; MCP tools, REST APIs, HTTP services, OpenTelemetry. Any model provider or framework: TruAssure tests behaviour, not code.
Make a tool or a whole agent fail, time out or return nothing, in the format the real system uses, and check the agent contains the failure.
Run each case several times and in other phrasings or input variants. pass^k shows the chance it works every time, not just once.
Compare two runs case by case. Across deployments (model host, framework, region), parity is judged against run-to-run noise, and never when something else changed too.
Every action and handoff must act for the right customer, with permissions that only narrow as work passes between agents.
Letters, reports and files the agent produces are tested too: content checks or your own test code.
Every run measured against the Intent-Based Testing Catalogue: what was measured, with evidence, and which cases failed each type.
Severity by failure type, defect lists your engineers can reproduce, JUnit for CI, capability readiness, and a release gate with thresholds per capability.
Measure an LLM judge against a labelled set (Cohen's kappa) and against deterministic rules before relying on it.
Each run becomes a single HTML report that works offline and can be attached to a ticket: what the agent said beside what really happened, step by step. A local console browses, compares and gates every run.
Every team publishes its runs to a shared store (network share, Amazon S3 or Azure Blob). Leaders see each AI initiative's release gate, pass rate, open defects and trend on one page, with every report one click away.
Any SQL database your systems run on (Oracle, SQL Server, Postgres, MySQL, Snowflake and more), REST APIs, message queues (Azure Service Bus, Kafka) and OpenAI-compatible agent endpoints. Read-only, configured, not coded.
Privacy, prompt injection, unsafe writes, payments and approvals, identity across agents: packs every initiative includes with its own names and data, so common risks are tested the same way everywhere.
TruAssure is also an MCP server: one tool per command, so an assistant in your IDE can run catalogs, read results and apply the gate. Results are marked as data, never as instructions.
A release gate turns test results into a go / no-go decision per capability and severity, in your CI pipeline (GitHub Actions or Azure DevOps templates included).
Payments claimed but not made, approvals that failed but were announced, cases "opened" that don't exist: found before go-live, not by customers or regulators.
Upgrading a model, moving provider or re-platforming the orchestration? Run the same catalog on both and get a judged parity verdict instead of a hunch.
Every run keeps the catalog that judged it, the evidence from each source (masked), the version of the judging logic, and the reasons for every verdict.
The first failing stage, the failure type, what the agent did instead, and how to reproduce it: engineers start from the cause.
No agent code changes, any LLM provider, open file formats (YAML catalogs, JSON evidence, JUnit). MIT licensed.
| Use case | What TruAssure checks | Outcome |
|---|---|---|
| Claims and back-office automation | Every event handled end to end: the right claim, the right status, letters with the right content, no payment without permission, failures contained | Automation that is safe to switch on, with a regression suite for every change |
| Customer-service assistants | Understanding of requests in many phrasings, the right lookups before answering, approvals before payments, nothing revealed about other customers | Assistants that answer from facts, measured per capability |
| Multi-agent service desks | Handoff order, the customer's identity on every hop, permissions narrowing per role, and what happens when one agent fails | Confidence that delegation doesn't widen access or spread failures |
| Model or provider migration | The same catalog on the old and the new model, compared case by case with a noise floor | A judged parity verdict and the exact cases that changed |
| Framework re-platforming | A plain-code agent and its framework rebuild (e.g. LangGraph) on the same catalog | Proof the rebuild behaves the same, or where it doesn't |
| Release gating in CI/CD | Nightly and pull-request runs with repeats, JUnit results and a gate per capability | Regressions stop the pipeline; known issues stay visible without blocking |
| Security and identity reviews | Principal propagation, permission widening, actions outside a role, access-denied records described anyway | Evidence for security sign-off on agentic systems |
| Capability readiness | A capability pack: all its cases, every perturbation and repeat, no critical or major defects | A clear "ready / not ready" per capability for go-live decisions |
TruAssure was built from proven evaluation logic and holds itself to it. All demo agents and data are fictitious (Harbor Mutual).
recorded chat turns judged exactly as the reference implementation judged them, across three transports
run-level checks matching the reference: reliability curves, scorecards, defect severities, CI decisions, parity verdicts
planted defects in a claims agent and a three-agent desk, each caught at the right stage with the right failure type
automated tests at 98.9% line-and-branch coverage, run on Windows and Linux in CI
When the approval failed or timed out, the claims agent still wrote the approval letter to the policyholder. Found in every run.
Asked to approve more than the amount claimed, the agent approved the maximum on its own instead of refusing.
When billing failed, the desk handed the request on anyway, opened no case, and told the member "a case has been opened".
The chat assistant answered "no policies" without the required lookup; the desk's triage dropped the amounts its specialists reported.
The same claims agent built as plain code and on LangGraph: parity held, with the one difference within run-to-run noise.
On a 24-item labelled set: deterministic rules kappa 0.75; the LLM grader 0.83 to 1.0 across runs, so you know how far to trust each.
pip install truassure.
It runs on Windows, Linux and macOS with Python 3.11 or later, inside your own network. ta demo shows every
feature offline in seconds; ta new starts a project for your own chat, task or multi-agent system.Every listener binds to the local machine unless explicitly opened; control interfaces need tokens.
Keys come only from the environment or a git-ignored file and are never printed or stored with results.
Personal data is masked before evidence is stored; retention is enforced; each run records exactly what judged it.
No. TruAssure drives the agent through its existing interface and observes its tools, other agents and data from outside. If the agent already reports a trace or OpenTelemetry spans, TruAssure uses them as one more source of evidence.
Any. TruAssure judges behaviour at the boundaries (tool calls, handoffs, data, outputs). The demo agents run on Azure OpenAI and can switch to OpenAI, Anthropic, AWS Bedrock, Google Gemini or Vertex, Ollama and others through configuration.
Yes. It ships GitHub Actions and Azure DevOps templates, writes JUnit for your CI's test view, and returns exit codes your pipeline can act on.
It reduces how much you need them: most checks are deterministic, on evidence. Where an LLM judge is useful, TruAssure measures its agreement with labelled examples first, and a judge that can't answer never counts as a pass.
Its judging logic is held to a regression gate of 4,221 recorded turns and 40 recorded runs, every planted defect must fire its own check, and its coverage gate is 95% or higher.
As a Python package on PyPI (pip install truassure), MIT licensed. Your teams use the ta command, the browser console, CI templates, or an AI assistant in the IDE through TruAssure's MCP server. There is no hosted service to sign up for.
Nowhere outside your environment. Runs are stored as files on your infrastructure, masked, with configurable retention.