TruVelocity™ for AI TEVVTest suites review

Test suites · review

TruVelocity™ for AI TEVV test suites

Which questions of the pilot catalogs belong to which test suite, and what each suite costs to run. Filter, search, and click a suite on any question to change it: the counts and costs update as you go, and Changes lists your edits to copy or download. Nothing leaves this page.

19 Nov 2026 · TruVelocity™ 2026.11.20.4 · data 7df289138899

What each suite costs

One pass: the original wording, one repeat (phrasings and repeats multiply it). Turns sent count a conversation's earlier turns, sent as context, so they show the real run time. Parallel workers split by whole conversation: the longest one is the least a run can take. Click a suite to filter by it.

The suites

How a suite runs

A conversation runs up to its last selected question. Its earlier turns are sent as context, because later questions depend on them (“pay the rest of it”, ids created earlier), but they are not judged and not counted toward the next question's evidence. Conversations with no selected question are left out; task and workflow scenarios are selected whole. A catalog may add its own suites (for example accessibility) in a top-level suites: list; any other name is a validation error.

Running one

tv validate pilots/chat/catalog-v2.yaml                    lists questions and turns per suite
tv run pilots/chat/catalog-v2.yaml --env ... --suite smoke  only the smoke questions
tv run ... --suite negative --suite boundary                several suites: their union

In the console (tv console --allow-runs), each preset has a suites box. Tags live in the catalogs as suites: [...] (in the question-catalog sources under docs/, test_suites: [...]); pack scenarios are tagged in the pack files.