TruVelocity™ for AI TEVVEvidence portfolio

Evidence portfolio

TruVelocity™ for AI TEVV in continuous use

43 test runs against 3 AI agents between 07 November and 17 November 2026: 6,739 test instances, each judged on evidence (the tools the agent called, the data it changed, what it replied), and every run read against all 71 test types of the Intent-Based Testing Catalogue.

Runs
43
07 Nov to 17 Nov 2026, on 6 days
Test instances executed
6,739
in 1,758 conversations and tasks
AI agents under test
3
chat, single-agent and multi-agent
Test types measured
71
every run read against the full catalogue
Test types validated
52 / 71
with executed evidence in at least one run
Planted defects caught
4 / 4
deliberately broken builds, all flagged

Activity over time

Test instances executed per run, coloured by agent, with the running total (line, right axis). Hover a bar for the run.

003801,6857603,3701,1415,0541,5216,739run-20261107-live-smoke · Chat assistant · 2026-11-07 · 23 instances07 Novrun-20261107-152856 · Chat assistant · 2026-11-07 · 5 instancesrun-20261107-161125 · Claims agent · 2026-11-07 · 20 instancesrun-20261107-161402 · Claims agent · 2026-11-07 · 1 instancesrun-20261107-161526 · Claims agent · 2026-11-07 · 20 instancesrun-20261107-162953 · Claims agent · 2026-11-07 · 20 instancesrun-20261107-163234 · Claims agent · 2026-11-07 · 2 instanceslive-buggy · Claims agent · 2026-11-07 · 2 instancesdesk-live-1 · Member desk · 2026-11-07 · 15 instancesdesk-live-2 · Member desk · 2026-11-07 · 15 instancesdesk-buggy-wrong_member · Member desk · 2026-11-07 · 1 instancesdesk-buggy-no_narrowing · Member desk · 2026-11-07 · 1 instancesdesk-buggy-trace_omits_hop · Member desk · 2026-11-07 · 1 instanceslive-loop-1 · Claims agent · 2026-11-07 · 15 instanceslive-loop-2 · Claims agent · 2026-11-07 · 15 instanceslive-graph-1 · Claims agent · 2026-11-07 · 15 instancespilot0-live-1 · Chat assistant · 2026-11-08 · 101 instances08 Novsmoke-proxy-2 · Chat assistant · 2026-11-08 · 2 instancessmoke-attach-2 · Chat assistant · 2026-11-08 · 1 instancespilot0-live-2 · Chat assistant · 2026-11-08 · 117 instancespilot0-live-3 · Chat assistant · 2026-11-08 · 104 instancespilot0-shan-1 · Chat assistant · 2026-11-08 · 27 instancessmoke-par-1 · Chat assistant · 2026-11-08 · 22 instancespilot0-phrasings-1 · Chat assistant · 2026-11-08 · 577 instancesdesk-all-1109 · Member desk · 2026-11-09 · 15 instances09 Novclaims-all-1109 · Claims agent · 2026-11-09 · 20 instanceschat-all-1109-3 · Chat assistant · 2026-11-09 · 585 instancesdesk-clean3 · Member desk · 2026-11-09 · 15 instancesclaims-clean3 · Claims agent · 2026-11-09 · 20 instanceschat-clean3 · Chat assistant · 2026-11-09 · 585 instancesrun-20261110-124949 · Member desk · 2026-11-10 · 15 instances10 Novrun-20261110-130743 · Member desk · 2026-11-10 · 15 instancesdesk-fixed-1 · Member desk · 2026-11-10 · 45 instancesrun-20261110-134841 · Member desk · 2026-11-10 · 15 instancesrun-20261110-143629 · Member desk · 2026-11-10 · 15 instancesrun-20261110-181245 · Member desk · 2026-11-10 · 15 instanceschat-v2-all · Chat assistant · 2026-11-10 · 1,521 instanceschat-v2-fix · Chat assistant · 2026-11-10 · 805 instanceschat-v2-fix2 · Chat assistant · 2026-11-10 · 234 instancesrun-20261116-162917 · Claims agent · 2026-11-16 · 20 instances16 Novchat-v2-pv03fix · Chat assistant · 2026-11-17 · 78 instances17 Novchat-v2-e7f6edb · Chat assistant · 2026-11-17 · 78 instanceschat-v2-e7f6edb-all · Chat assistant · 2026-11-17 · 1,521 instancesinstances per runrunning total
Chat assistantClaims agentMember deskrunning total

Pass rate by agent, run by run

Each dot is a run: its pass rate over conclusive test instances, with the Wilson 95% range (vertical line) and dot size by instances. Diamonds are builds with a deliberately planted defect: they are meant to fail, and did. Across all other runs: 5,490 of 6,516 conclusive instances passed (84.3%, 95% range 83.3–85.1%). The failures are defects TruVelocity™ found in the agents, not in TruVelocity™.

Chat assistant 18 runs · 6,386 instances
customer-facing insurance chatbot: policies, billing, claims, quotes; three customers and staff roles
0%50%100%run-20261107-live-smoke · 2026-11-07 · 22/23 passed (95.7%), 95% range 79-99%run-20261107-152856 · 2026-11-07 · 5/5 passed (100.0%), 95% range 57-100%pilot0-live-1 · 2026-11-08 · 37/100 passed (37.0%), 95% range 28-47%smoke-proxy-2 · 2026-11-08 · 1/2 passed (50.0%), 95% range 9-91%smoke-attach-2 · 2026-11-08 · 0/1 passed (0.0%), 95% range 0-79%pilot0-live-2 · 2026-11-08 · 87/115 passed (75.7%), 95% range 67-83%pilot0-live-3 · 2026-11-08 · 83/103 passed (80.6%), 95% range 72-87%pilot0-shan-1 · 2026-11-08 · 22/27 passed (81.5%), 95% range 63-92%smoke-par-1 · 2026-11-08 · 22/22 passed (100.0%), 95% range 85-100%pilot0-phrasings-1 · 2026-11-08 · 454/577 passed (78.7%), 95% range 75-82%chat-all-1109-3 · 2026-11-09 · 469/555 passed (84.5%), 95% range 81-87%chat-clean3 · 2026-11-09 · 473/556 passed (85.1%), 95% range 82-88%chat-v2-all · 2026-11-10 · 1225/1443 passed (84.9%), 95% range 83-87%chat-v2-fix · 2026-11-10 · 708/805 passed (88.0%), 95% range 86-90%chat-v2-fix2 · 2026-11-10 · 204/234 passed (87.2%), 95% range 82-91%chat-v2-pv03fix · 2026-11-17 · 69/78 passed (88.5%), 95% range 80-94%chat-v2-e7f6edb · 2026-11-17 · 76/78 passed (97.4%), 95% range 91-99%chat-v2-e7f6edb-all · 2026-11-17 · 1263/1444 passed (87.5%), 95% range 86-89%2026-11-072026-11-17
Claims agent 12 runs · 170 instances
back-office agent that triages and settles claims through tools; two engines (loop, LangGraph)
0%50%100%run-20261107-161125 · 2026-11-07 · 16/20 passed (80.0%), 95% range 58-92%run-20261107-161402 · 2026-11-07 · 1/1 passed (100.0%), 95% range 21-100%run-20261107-161526 · 2026-11-07 · 16/20 passed (80.0%), 95% range 58-92%run-20261107-162953 · 2026-11-07 · 15/20 passed (75.0%), 95% range 53-89%run-20261107-163234 · 2026-11-07 · 2/2 passed (100.0%), 95% range 34-100%live-buggy · 2026-11-07 · 0/2 passed (0.0%), 95% range 0-66% · planted defect: expected to faillive-loop-1 · 2026-11-07 · 11/15 passed (73.3%), 95% range 48-89%live-loop-2 · 2026-11-07 · 11/15 passed (73.3%), 95% range 48-89%live-graph-1 · 2026-11-07 · 12/15 passed (80.0%), 95% range 55-93%claims-all-1109 · 2026-11-09 · 16/20 passed (80.0%), 95% range 58-92%claims-clean3 · 2026-11-09 · 17/20 passed (85.0%), 95% range 64-95%run-20261116-162917 · 2026-11-16 · 17/20 passed (85.0%), 95% range 64-95%2026-11-072026-11-16
Member desk 13 runs · 183 instances
multi-agent desk: a triage agent hands work to billing and claims agents
0%50%100%desk-live-1 · 2026-11-07 · 8/15 passed (53.3%), 95% range 30-75%desk-live-2 · 2026-11-07 · 10/15 passed (66.7%), 95% range 42-85%desk-buggy-wrong_member · 2026-11-07 · 0/1 passed (0.0%), 95% range 0-79% · planted defect: expected to faildesk-buggy-no_narrowing · 2026-11-07 · 0/1 passed (0.0%), 95% range 0-79% · planted defect: expected to faildesk-buggy-trace_omits_hop · 2026-11-07 · 0/1 passed (0.0%), 95% range 0-79% · planted defect: expected to faildesk-all-1109 · 2026-11-09 · 8/15 passed (53.3%), 95% range 30-75%desk-clean3 · 2026-11-09 · 10/15 passed (66.7%), 95% range 42-85%run-20261110-124949 · 2026-11-10 · 9/15 passed (60.0%), 95% range 36-80%run-20261110-130743 · 2026-11-10 · 10/15 passed (66.7%), 95% range 42-85%desk-fixed-1 · 2026-11-10 · 45/45 passed (100.0%), 95% range 92-100%run-20261110-134841 · 2026-11-10 · 12/15 passed (80.0%), 95% range 55-93%run-20261110-143629 · 2026-11-10 · 12/15 passed (80.0%), 95% range 55-93%run-20261110-181245 · 2026-11-10 · 12/15 passed (80.0%), 95% range 55-93%2026-11-072026-11-10

Test-type validation over time

Distinct catalogue test types validated so far (covered with executed evidence in at least one run), after each run.

020406071after run-20261107-live-smoke (2026-11-07): 7 of 71 types validatedafter run-20261107-152856 (2026-11-07): 8 of 71 types validatedafter run-20261107-161125 (2026-11-07): 11 of 71 types validatedafter run-20261107-161402 (2026-11-07): 11 of 71 types validatedafter run-20261107-161526 (2026-11-07): 11 of 71 types validatedafter run-20261107-162953 (2026-11-07): 11 of 71 types validatedafter run-20261107-163234 (2026-11-07): 11 of 71 types validatedafter live-buggy (2026-11-07): 11 of 71 types validatedafter desk-live-1 (2026-11-07): 13 of 71 types validatedafter desk-live-2 (2026-11-07): 13 of 71 types validatedafter desk-buggy-wrong_member (2026-11-07): 14 of 71 types validatedafter desk-buggy-no_narrowing (2026-11-07): 14 of 71 types validatedafter desk-buggy-trace_omits_hop (2026-11-07): 14 of 71 types validatedafter live-loop-1 (2026-11-07): 24 of 71 types validatedafter live-loop-2 (2026-11-07): 24 of 71 types validatedafter live-graph-1 (2026-11-07): 25 of 71 types validatedafter pilot0-live-1 (2026-11-08): 43 of 71 types validatedafter smoke-proxy-2 (2026-11-08): 44 of 71 types validatedafter smoke-attach-2 (2026-11-08): 45 of 71 types validatedafter pilot0-live-2 (2026-11-08): 50 of 71 types validatedafter pilot0-live-3 (2026-11-08): 50 of 71 types validatedafter pilot0-shan-1 (2026-11-08): 50 of 71 types validatedafter smoke-par-1 (2026-11-08): 52 of 71 types validatedafter pilot0-phrasings-1 (2026-11-08): 52 of 71 types validatedafter desk-all-1109 (2026-11-09): 52 of 71 types validatedafter claims-all-1109 (2026-11-09): 52 of 71 types validatedafter chat-all-1109-3 (2026-11-09): 52 of 71 types validatedafter desk-clean3 (2026-11-09): 52 of 71 types validatedafter claims-clean3 (2026-11-09): 52 of 71 types validatedafter chat-clean3 (2026-11-09): 52 of 71 types validatedafter run-20261110-124949 (2026-11-10): 52 of 71 types validatedafter run-20261110-130743 (2026-11-10): 52 of 71 types validatedafter desk-fixed-1 (2026-11-10): 52 of 71 types validatedafter run-20261110-134841 (2026-11-10): 52 of 71 types validatedafter run-20261110-143629 (2026-11-10): 52 of 71 types validatedafter run-20261110-181245 (2026-11-10): 52 of 71 types validatedafter chat-v2-all (2026-11-10): 52 of 71 types validatedafter chat-v2-fix (2026-11-10): 52 of 71 types validatedafter chat-v2-fix2 (2026-11-10): 52 of 71 types validatedafter run-20261116-162917 (2026-11-16): 52 of 71 types validatedafter chat-v2-pv03fix (2026-11-17): 52 of 71 types validatedafter chat-v2-e7f6edb (2026-11-17): 52 of 71 types validatedafter chat-v2-e7f6edb-all (2026-11-17): 52 of 71 types validated52 / 71

Heatmap: 71 test types × 43 runs

Rows are the catalogue's test types by category; columns are runs in time order (the strip shows the agent, dates mark a new day). A coloured cell means the run covered the type, shaded by its pass rate. Hover any cell; use the buttons to show one agent's runs.

below 40%40-60%60-80%80-95%95-100%measured (no pass/fail)partly coverednot coverednot applicable
07 Nov08 Nov09 Nov10 Nov16 Nov17 Nov
C1 · Intent & understanding
IU-01 Intent Resolution
IU-02 Intent Disambiguation
IU-03 Multi-Intent Decomposition
IU-04 Out-of-Scope Detection
IU-05 Intent Switching
IU-06 Input Perturbation Robustness
C2 · Entity & slot
ES-01 Slot Extraction
ES-02 Slot Elicitation
ES-03 Slot Correction
ES-04 Negation & Conditional Handling
C3 · Conversational quality
CQ-01 Multi-Turn Conversation Quality
CQ-02 Conversational Repair
CQ-03 Goal Drift & Distraction
CQ-04 Empathy & Sentiment Handling
CQ-05 Long Conversation Stability
C4 · Grounding & factuality
GF-01 RAG Grounding
GF-02 Abstention / Unanswerable
GF-03 Knowledge Currency
GF-04 Conflicting Source
GF-05 Numeric & Calculation Accuracy
GF-06 Multi-Hop Reasoning
C5 · Task & tool execution
TT-01 Tool Call Accuracy
TT-02 Task Completion (state-graded)
TT-03 Tool Failure Handling
TT-04 Idempotency & Duplicate Action
TT-05 Transaction Rollback
TT-06 Precondition Enforcement
C6 · Agentic & trajectory
AT-01 Trajectory Evaluation
AT-02 Loop & Termination
AT-03 Error Recovery
AT-04 Escalation & Handoff
AT-05 Role & Scope Boundary
AT-06 Budget Adherence
AT-07 Memory Correctness
AT-08 Multi-Agent Handoff
C7 · Reliability & consistency
RC-01 pass^k Reliability
RC-02 Semantic Equivalence
RC-03 Cross-Session Consistency
C8 · Safety & security
SS-01 Prompt Injection / Goal Hijack
SS-02 Tool Misuse & Exploitation
SS-03 Identity & Privilege Abuse
SS-04 Authentication Enforcement
SS-05 Data Scoping
SS-06 PII / PHI Leakage
SS-07 Memory & Context Poisoning
SS-08 Human-Agent Trust Exploitation
SS-09 Cascading Failure
SS-10 Jailbreak Resistance
SS-11 Toxicity & Harmful Content
SS-12 Over-Refusal
SS-13 Adversarial / Fuzz
C9 · Compliance & fairness
CP-01 Policy Adherence
CP-02 Prohibited Advice
CP-03 Disclosure & Script Compliance
CP-04 Bias & Fairness
CP-05 Jurisdiction Rule
CP-06 Auditability & Explainability
C10 · Performance & operations
PO-01 Latency
PO-02 Load & Concurrency
PO-03 Cost & Token Efficiency
PO-04 Resilience & Degradation
C11 · Continuous & platform
CE-01 Drift Detection
CE-02 Judge Calibration
CE-03 Production Replay
CE-04 Shadow / Canary
C12 · Seam & evidence consistency
SE-01 Trace vs Proxy Agreement
SE-02 Approval vs State Integrity
SE-03 Silent Write Detection
SE-04 Phantom Success Detection
SE-05 Reply vs Action Consistency
SE-06 Transport Parity

Coverage by agent

All runs together: for each test type and agent, its pass rate over every run that validated it, and in how many runs.

C1 · Intent & understanding6 / 6
C2 · Entity & slot4 / 4
C3 · Conversational quality4 / 5
C4 · Grounding & factuality3 / 6
C5 · Task & tool execution5 / 6
C6 · Agentic & trajectory7 / 8
C7 · Reliability & consistency2 / 3
C8 · Safety & security7 / 13
C9 · Compliance & fairness4 / 6
C10 · Performance & operations3 / 4
C11 · Continuous & platform1 / 4
C12 · Seam & evidence consistency6 / 6

52 of 71 test types validated with executed evidence; 19 not yet. Every run is measured against all 71, so a gap is reported, never hidden.

A ready-made scenario pack covers it: add the pack to the agent's catalog. 3

  • SS-01 Prompt Injection / Goal Hijack (prompt-injection pack)
  • SS-02 Tool Misuse & Exploitation (privacy pack)
  • SS-06 PII / PHI Leakage (privacy pack)

Measured automatically when the run enables it (e.g. tv calibrate, tv compare). 1

  • CE-02 Judge Calibration

Needs scenarios or runner support that are on the roadmap. 15

  • AT-07 Memory Correctness
  • CE-03 Production Replay
  • CE-04 Shadow / Canary
  • CP-04 Bias & Fairness
  • CP-05 Jurisdiction Rule
  • CQ-05 Long Conversation Stability
  • GF-03 Knowledge Currency
  • GF-04 Conflicting Source
  • GF-06 Multi-Hop Reasoning
  • PO-02 Load & Concurrency
  • RC-03 Cross-Session Consistency
  • SS-07 Memory & Context Poisoning
  • SS-08 Human-Agent Trust Exploitation
  • SS-13 Adversarial / Fuzz
  • TT-05 Transaction Rollback
below 40%40-60%60-80%80-95%95-100%measured (no pass/fail)partly coverednot coverednot applicable
Test typeChat assistantClaims agentMember desk
C1 · Intent & understanding 6 of 6 validated
IU-01 Intent Resolution94% · 15 runsn/an/a
IU-02 Intent Disambiguation98% · 8 runsn/an/a
IU-03 Multi-Intent Decomposition92% · 9 runsn/an/a
IU-04 Out-of-Scope Detection69% · 10 runsn/an/a
IU-05 Intent Switching89% · 8 runsn/an/a
IU-06 Input Perturbation Robustness94% · 10 runsn/an/a
C2 · Entity & slot 4 of 4 validated
ES-01 Slot Extraction96% · 11 runs96% · 6 runs100% · 8 runs
ES-02 Slot Elicitation97% · 11 runs——
ES-03 Slot Correction74% · 10 runs——
ES-04 Negation & Conditional Handling86% · 10 runs——
C3 · Conversational quality 4 of 5 validated
CQ-01 Multi-Turn Conversation Quality33% · 8 runsn/an/a
CQ-02 Conversational Repair91% · 9 runsn/an/a
CQ-03 Goal Drift & Distraction98% · 8 runsn/an/a
CQ-04 Empathy & Sentiment Handling67% · 8 runsn/an/a
CQ-05 Long Conversation Stability—n/an/a
C4 · Grounding & factuality 3 of 6 validated
GF-01 RAG Grounding98% · 9 runs——
GF-02 Abstention / Unanswerable69% · 9 runs——
GF-03 Knowledge Currency———
GF-04 Conflicting Source———
GF-05 Numeric & Calculation Accuracy97% · 11 runs——
GF-06 Multi-Hop Reasoning———
C5 · Task & tool execution 5 of 6 validated
TT-01 Tool Call Accuracy92% · 15 runs81% · 6 runs94% · 8 runs
TT-02 Task Completion (state-graded)98% · 16 runs81% · 6 runs100% · 8 runs
TT-03 Tool Failure Handling41% · 8 runs46% · 6 runs95% · 8 runs
TT-04 Idempotency & Duplicate Action100% · 9 runs100% · 6 runs100% · 5 runs
TT-05 Transaction Rollback———
TT-06 Precondition Enforcement86% · 9 runs——
C6 · Agentic & trajectory 7 of 8 validated
AT-01 Trajectory Evaluation95% · 16 runs100% · 6 runs97% · 8 runs
AT-02 Loop & Termination—100% · 6 runs100% · 8 runs
AT-03 Error Recovery0% · 7 runs——
AT-04 Escalation & Handoff100% · 8 runs——
AT-05 Role & Scope Boundary98% · 8 runs——
AT-06 Budget Adherence—100% · 12 runs100% · 13 runs
AT-07 Memory Correctness———
AT-08 Multi-Agent Handoffn/a—94% · 11 runs
C7 · Reliability & consistency 2 of 3 validated
RC-01 pass^k Reliability62% · 10 runs100% · 7 runs67% · 10 runs
RC-02 Semantic Equivalence83% · 10 runs100% · 3 runs100% · 8 runs
RC-03 Cross-Session Consistency———
C8 · Safety & security 7 of 13 validated
SS-01 Prompt Injection / Goal Hijack———
SS-02 Tool Misuse & Exploitation———
SS-03 Identity & Privilege Abuse97% · 9 runs100% · 6 runs99% · 10 runs
SS-04 Authentication Enforcement98% · 9 runs——
SS-05 Data Scoping87% · 13 runs——
SS-06 PII / PHI Leakage———
SS-07 Memory & Context Poisoning———
SS-08 Human-Agent Trust Exploitation———
SS-09 Cascading Failure——72% · 10 runs
SS-10 Jailbreak Resistance100% · 7 runsn/an/a
SS-11 Toxicity & Harmful Content78% · 8 runsn/an/a
SS-12 Over-Refusal100% · 8 runsn/an/a
SS-13 Adversarial / Fuzz———
C9 · Compliance & fairness 4 of 6 validated
CP-01 Policy Adherence88% · 8 runs——
CP-02 Prohibited Advice63% · 9 runs——
CP-03 Disclosure & Script Compliance74% · 10 runs——
CP-04 Bias & Fairness———
CP-05 Jurisdiction Rule———
CP-06 Auditability & Explainability100% · 18 runs62% · 12 runs82% · 13 runs
C10 · Performance & operations 3 of 4 validated
PO-01 Latencymeasured · 17 runsmeasured · 11 runsmeasured · 10 runs
PO-02 Load & Concurrency———
PO-03 Cost & Token Efficiency—measured · 12 runsmeasured · 13 runs
PO-04 Resilience & Degradation100% · 7 runs0% · 6 runs100% · 8 runs
C11 · Continuous & platform 1 of 4 validated
CE-01 Drift Detection95% · 6 runs97% · 6 runs96% · 10 runs
CE-02 Judge Calibration———
CE-03 Production Replay———
CE-04 Shadow / Canary———
C12 · Seam & evidence consistency 6 of 6 validated
SE-01 Trace vs Proxy Agreement100% · 15 runs100% · 6 runs100% · 8 runs
SE-02 Approval vs State Integrity100% · 12 runs——
SE-03 Silent Write Detection100% · 12 runs100% · 6 runs100% · 5 runs
SE-04 Phantom Success Detection100% · 11 runs100% · 6 runs100% · 5 runs
SE-05 Reply vs Action Consistency7% · 8 runs——
SE-06 Transport Parity—93% · 1 runs—

A framework that kept shipping

The same evidence store was read by every TruVelocity™ release below; each run records the version that produced it.

TruVelocity™ versionFirst runRunsInstancesAgents
0.0.12026-11-0716171Chat assistant, Claims agent, Member desk
1.3.02026-11-088951Chat assistant
1.5.02026-11-093620Chat assistant, Claims agent, Member desk
1.5.22026-11-093620Chat assistant, Claims agent, Member desk
1.6.22026-11-10490Member desk
1.6.32026-11-10115Member desk
1.7.02026-11-1084,272Chat assistant, Claims agent, Member desk

How to read this page