Activity over time
Test instances executed per run, coloured by agent, with the running total (line, right axis). Hover a bar for the run.
Pass rate by agent, run by run
Each dot is a run: its pass rate over conclusive test instances, with the Wilson 95% range (vertical line) and dot size by instances. Diamonds are builds with a deliberately planted defect: they are meant to fail, and did. Across all other runs: 5,490 of 6,516 conclusive instances passed (84.3%, 95% range 83.3–85.1%). The failures are defects TruVelocity™ found in the agents, not in TruVelocity™.
customer-facing insurance chatbot: policies, billing, claims, quotes; three customers and staff roles
back-office agent that triages and settles claims through tools; two engines (loop, LangGraph)
multi-agent desk: a triage agent hands work to billing and claims agents
Test-type validation over time
Distinct catalogue test types validated so far (covered with executed evidence in at least one run), after each run.
Heatmap: 71 test types × 43 runs
Rows are the catalogue's test types by category; columns are runs in time order (the strip shows the agent, dates mark a new day). A coloured cell means the run covered the type, shaded by its pass rate. Hover any cell; use the buttons to show one agent's runs.
Coverage by agent
All runs together: for each test type and agent, its pass rate over every run that validated it, and in how many runs.
52 of 71 test types validated with executed evidence; 19 not yet. Every run is measured against all 71, so a gap is reported, never hidden.
A ready-made scenario pack covers it: add the pack to the agent's catalog. 3
- SS-01 Prompt Injection / Goal Hijack (prompt-injection pack)
- SS-02 Tool Misuse & Exploitation (privacy pack)
- SS-06 PII / PHI Leakage (privacy pack)
Measured automatically when the run enables it (e.g. tv calibrate, tv compare). 1
- CE-02 Judge Calibration
Needs scenarios or runner support that are on the roadmap. 15
- AT-07 Memory Correctness
- CE-03 Production Replay
- CE-04 Shadow / Canary
- CP-04 Bias & Fairness
- CP-05 Jurisdiction Rule
- CQ-05 Long Conversation Stability
- GF-03 Knowledge Currency
- GF-04 Conflicting Source
- GF-06 Multi-Hop Reasoning
- PO-02 Load & Concurrency
- RC-03 Cross-Session Consistency
- SS-07 Memory & Context Poisoning
- SS-08 Human-Agent Trust Exploitation
- SS-13 Adversarial / Fuzz
- TT-05 Transaction Rollback
| Test type | Chat assistant | Claims agent | Member desk |
|---|---|---|---|
| C1 · Intent & understanding 6 of 6 validated | |||
| IU-01 Intent Resolution | 94% · 15 runs | n/a | n/a |
| IU-02 Intent Disambiguation | 98% · 8 runs | n/a | n/a |
| IU-03 Multi-Intent Decomposition | 92% · 9 runs | n/a | n/a |
| IU-04 Out-of-Scope Detection | 69% · 10 runs | n/a | n/a |
| IU-05 Intent Switching | 89% · 8 runs | n/a | n/a |
| IU-06 Input Perturbation Robustness | 94% · 10 runs | n/a | n/a |
| C2 · Entity & slot 4 of 4 validated | |||
| ES-01 Slot Extraction | 96% · 11 runs | 96% · 6 runs | 100% · 8 runs |
| ES-02 Slot Elicitation | 97% · 11 runs | — | — |
| ES-03 Slot Correction | 74% · 10 runs | — | — |
| ES-04 Negation & Conditional Handling | 86% · 10 runs | — | — |
| C3 · Conversational quality 4 of 5 validated | |||
| CQ-01 Multi-Turn Conversation Quality | 33% · 8 runs | n/a | n/a |
| CQ-02 Conversational Repair | 91% · 9 runs | n/a | n/a |
| CQ-03 Goal Drift & Distraction | 98% · 8 runs | n/a | n/a |
| CQ-04 Empathy & Sentiment Handling | 67% · 8 runs | n/a | n/a |
| CQ-05 Long Conversation Stability | — | n/a | n/a |
| C4 · Grounding & factuality 3 of 6 validated | |||
| GF-01 RAG Grounding | 98% · 9 runs | — | — |
| GF-02 Abstention / Unanswerable | 69% · 9 runs | — | — |
| GF-03 Knowledge Currency | — | — | — |
| GF-04 Conflicting Source | — | — | — |
| GF-05 Numeric & Calculation Accuracy | 97% · 11 runs | — | — |
| GF-06 Multi-Hop Reasoning | — | — | — |
| C5 · Task & tool execution 5 of 6 validated | |||
| TT-01 Tool Call Accuracy | 92% · 15 runs | 81% · 6 runs | 94% · 8 runs |
| TT-02 Task Completion (state-graded) | 98% · 16 runs | 81% · 6 runs | 100% · 8 runs |
| TT-03 Tool Failure Handling | 41% · 8 runs | 46% · 6 runs | 95% · 8 runs |
| TT-04 Idempotency & Duplicate Action | 100% · 9 runs | 100% · 6 runs | 100% · 5 runs |
| TT-05 Transaction Rollback | — | — | — |
| TT-06 Precondition Enforcement | 86% · 9 runs | — | — |
| C6 · Agentic & trajectory 7 of 8 validated | |||
| AT-01 Trajectory Evaluation | 95% · 16 runs | 100% · 6 runs | 97% · 8 runs |
| AT-02 Loop & Termination | — | 100% · 6 runs | 100% · 8 runs |
| AT-03 Error Recovery | 0% · 7 runs | — | — |
| AT-04 Escalation & Handoff | 100% · 8 runs | — | — |
| AT-05 Role & Scope Boundary | 98% · 8 runs | — | — |
| AT-06 Budget Adherence | — | 100% · 12 runs | 100% · 13 runs |
| AT-07 Memory Correctness | — | — | — |
| AT-08 Multi-Agent Handoff | n/a | — | 94% · 11 runs |
| C7 · Reliability & consistency 2 of 3 validated | |||
| RC-01 pass^k Reliability | 62% · 10 runs | 100% · 7 runs | 67% · 10 runs |
| RC-02 Semantic Equivalence | 83% · 10 runs | 100% · 3 runs | 100% · 8 runs |
| RC-03 Cross-Session Consistency | — | — | — |
| C8 · Safety & security 7 of 13 validated | |||
| SS-01 Prompt Injection / Goal Hijack | — | — | — |
| SS-02 Tool Misuse & Exploitation | — | — | — |
| SS-03 Identity & Privilege Abuse | 97% · 9 runs | 100% · 6 runs | 99% · 10 runs |
| SS-04 Authentication Enforcement | 98% · 9 runs | — | — |
| SS-05 Data Scoping | 87% · 13 runs | — | — |
| SS-06 PII / PHI Leakage | — | — | — |
| SS-07 Memory & Context Poisoning | — | — | — |
| SS-08 Human-Agent Trust Exploitation | — | — | — |
| SS-09 Cascading Failure | — | — | 72% · 10 runs |
| SS-10 Jailbreak Resistance | 100% · 7 runs | n/a | n/a |
| SS-11 Toxicity & Harmful Content | 78% · 8 runs | n/a | n/a |
| SS-12 Over-Refusal | 100% · 8 runs | n/a | n/a |
| SS-13 Adversarial / Fuzz | — | — | — |
| C9 · Compliance & fairness 4 of 6 validated | |||
| CP-01 Policy Adherence | 88% · 8 runs | — | — |
| CP-02 Prohibited Advice | 63% · 9 runs | — | — |
| CP-03 Disclosure & Script Compliance | 74% · 10 runs | — | — |
| CP-04 Bias & Fairness | — | — | — |
| CP-05 Jurisdiction Rule | — | — | — |
| CP-06 Auditability & Explainability | 100% · 18 runs | 62% · 12 runs | 82% · 13 runs |
| C10 · Performance & operations 3 of 4 validated | |||
| PO-01 Latency | measured · 17 runs | measured · 11 runs | measured · 10 runs |
| PO-02 Load & Concurrency | — | — | — |
| PO-03 Cost & Token Efficiency | — | measured · 12 runs | measured · 13 runs |
| PO-04 Resilience & Degradation | 100% · 7 runs | 0% · 6 runs | 100% · 8 runs |
| C11 · Continuous & platform 1 of 4 validated | |||
| CE-01 Drift Detection | 95% · 6 runs | 97% · 6 runs | 96% · 10 runs |
| CE-02 Judge Calibration | — | — | — |
| CE-03 Production Replay | — | — | — |
| CE-04 Shadow / Canary | — | — | — |
| C12 · Seam & evidence consistency 6 of 6 validated | |||
| SE-01 Trace vs Proxy Agreement | 100% · 15 runs | 100% · 6 runs | 100% · 8 runs |
| SE-02 Approval vs State Integrity | 100% · 12 runs | — | — |
| SE-03 Silent Write Detection | 100% · 12 runs | 100% · 6 runs | 100% · 5 runs |
| SE-04 Phantom Success Detection | 100% · 11 runs | 100% · 6 runs | 100% · 5 runs |
| SE-05 Reply vs Action Consistency | 7% · 8 runs | — | — |
| SE-06 Transport Parity | — | 93% · 1 runs | — |
A framework that kept shipping
The same evidence store was read by every TruVelocity™ release below; each run records the version that produced it.
| TruVelocity™ version | First run | Runs | Instances | Agents |
|---|---|---|---|---|
| 0.0.1 | 2026-11-07 | 16 | 171 | Chat assistant, Claims agent, Member desk |
| 1.3.0 | 2026-11-08 | 8 | 951 | Chat assistant |
| 1.5.0 | 2026-11-09 | 3 | 620 | Chat assistant, Claims agent, Member desk |
| 1.5.2 | 2026-11-09 | 3 | 620 | Chat assistant, Claims agent, Member desk |
| 1.6.2 | 2026-11-10 | 4 | 90 | Member desk |
| 1.6.3 | 2026-11-10 | 1 | 15 | Member desk |
| 1.7.0 | 2026-11-10 | 8 | 4,272 | Chat assistant, Claims agent, Member desk |
How to read this page
- Source. Every number is read from the stored runs: each run's evidence is read again by today's TruVelocity™, as in its execution report, so early runs are measured against the full catalogue too. 10 run(s) that stopped before writing results, or executed nothing, are left out.
- The agents. Reference agents built to behave like production ones, on gpt-4.1 (Azure OpenAI), tested black-box through their real interfaces (HTTP chat, an agent loop and LangGraph, a multi-agent desk).
- Test instance. One checkpoint of one case, in one wording and one try. Cases are asked in up to 13 wordings to expose flaky behaviour.
- Validated. A test type counts as validated when a run covered it with executed evidence. Findings and failures are counted per run: a defect found again in a later run counts again.
- Ranges. Wilson 95% intervals: how far a pass rate can be trusted given the number of tries.