TruVelocity™ for AI TEVVUI demo runbook

TruVelocity™ for AI TEVV: web UI demo runbook

How to try the tv ui web UI yourself, and how to demo it to a client's leadership team: for every screen, what to click, what to say, what the screen shows, and the business outcome it delivers.

Release 2026.11.19.1 and later · written 19 Nov 2026 · all demo data is fictitious (Harbor Mutual).

For people trying it themselvesFor presenters to leadership15–20 min guided path5 min short pathNo Docker or LLM needed for the offline demo
About the screenshots. The welcome page, Home, the Conduct run form and the evidence pack were taken from the offline demo project (Claims back-office agent, Customer chat assistant, Member service desk). The others were taken from the live pilots (the Harbor Mutual agents: Chat assistant, Claims agent, Member desk) on 19 Nov 2026; the demo project looks the same there, with different agent names and numbers.

1What the UI is

TruVelocity™ tests AI agents from the outside and turns the results into a release decision backed by evidence. The web UI (tv ui) puts that work in six steps a team walks through for each agent, plus a Home page for the whole portfolio. It runs on your own machine, listens only on 127.0.0.1, and is protected by a token in the URL. The first page in each new browser session, and after Sign out, is a sign-in page; Explore as demo user (or sign in with any email) opens Home.

Home, then the six steps: Context, Catalog, Configure, Conduct, Conclude, Continuous. The guided path is Home, Catalog, Conduct, Conclude, Continuous. Homeportfolio 1 Contextwhat it is for 2 Catalogwhat we test 3 Configurehow we reach it 4 Conductrun the tests 5 ConcludeGo / No-go 6 Continuouskeep watching Guided demo path: Home → Catalog → Conduct → Conclude → Continuous (Context and Configure: short)
StepThe question it answersWhat leadership gets
HomeWhich agents are ready to release, which aren't, and why?One portfolio view: verdicts, open critical defects, risk by pillar, trend.
1 ContextWhat is this agent for, what can it do, and what must never go wrong?A shared, written statement of intended use and risk appetite that drives the testing.
2 CatalogWhat do we test, and what do we deliberately leave out?Coverage made visible; gaps recorded with a reason.
3 ConfigureHow does TruVelocity™ reach the agent, and does it answer now?No surprises on test day; the versions under test are recorded.
4 ConductRun the plan and watch it go.Anyone can run the same governed tests, without the command line.
5 ConcludeDoes this run meet the release bar? What do we decide?A Go / No-go verdict, a recorded decision with who, when and why, and an evidence pack for audit.
6 ContinuousIs it still good after release?Trend, drift between runs, alert rules and a scheduled run in CI.

Every number in the UI comes from real engine runs kept in the project's runs folder: the same runs tv gate and tv report read on the command line. Nothing is typed in by hand. To check what an agent did to its data, TruVelocity™ reads the records the agent was meant to change (chosen database tables, files), read-only, before and after each task, and compares them row by row.

2Two ways to run the demo

A. Offline demo project recommendedB. Live pilots (advanced)
Best forClient demos, trying it yourself, a laptop with no networkShowing real agents built on a language model, run live
NeedsPython 3.11 or later, pip install "truvelocity-tevv[ui]". No Docker, no LLM, no keys.This repository and its .venv, Docker Desktop, the demo agents' repository (SystemsUnderTest), LLM settings in .env
AgentsThree agents (ids claims, chat, desk) with at least three release runs eachHarbor Mutual: Chat assistant, Claims agent, Member desk (in Docker)
Story it tellsClaims: Go. Chat: No-go with critical defects. Desk: change over time.Whatever the latest live runs found (screenshots in this page)
A run from ConductAbout a minute1–2 minutes for a claims or desk suite; much longer for the full chat catalog
Set-up timeAbout 5 minutes, onceAbout 20 minutes the first time; 2 minutes to redeploy

ASet up the offline demo project about 5 minutes, once

Run these in a Command Prompt on Windows (or a terminal on macOS or Linux). Each box holds one command; Copy copies just the command, not the grey comment above it.

Windows (Command Prompt)

Windows · 1/7
# Start in your home folder
cd /d %USERPROFILE%
Windows · 2/7
# Create a Python environment for TruVelocity™ (Python 3.11 or later)
py -3.12 -m venv tv-env
Windows · 3/7
# Activate it: the prompt then starts with (tv-env)
tv-env\Scripts\activate
Windows · 4/7
# Install TruVelocity™ with the web UI
pip install "truvelocity-tevv[ui]"
Windows · 5/7
# Create the offline demo project in a new folder
tv demo-project truvelocity-demo
Expected output
TruVelocity demo project in truvelocity-demo: three agents, their catalogs and four weeks of runs
  claims-20261024-091204  claims build 3.0.0
  chat-20261025-100531  chat build 2.1.0
  desk-20261026-144012  desk build 1.0.0
  claims-20261031-092045  claims build 3.0.1
  chat-20261101-113009  chat build 2.2.0
  desk-20261103-150552  desk build 1.1.0
  claims-20261107-090817  claims build 3.1.0
  chat-20261110-104540  chat build 2.3.0
  desk-20261111-161026  desk build 1.2.0
  desk-20261116-132503  desk build 1.2.1
  claims-20261117-091538  claims build 3.1.1
  chat-20261118-174011  chat build 2.4.0
made 12 runs in 3 s; latest verdicts:
  Claims back-office agent   Go     100% of cases passed every try; defects: 0 critical, 0 major, 0 minor
  Customer chat assistant    No-go  67% of cases passed every try; defects: 1 critical, 1 major, 1 minor
  Member service desk        Go     100% of cases passed every try; defects: 0 critical, 0 major, 0 minor
next:
  cd truvelocity-demo
  tv ui --allow-runs --open

(The run names carry dates counted back from today, so yours differ.)
Windows · 6/7
# Go into the project
cd truvelocity-demo
Windows · 7/7
# Start the UI with runs allowed, and open it in your browser
tv ui --allow-runs --open
Expected output
TruVelocity UI: http://127.0.0.1:8770/?t=<your token>
(the token in the URL is your key; Ctrl+C stops it)
The browser opens on the sign-in page: click Explore as demo user (or sign in with any email) to reach Home. If it doesn't, copy the whole URL the command printed, token included, into your browser. Leave the window open: Ctrl+C in it stops the UI.
In PowerShell, step 3 is tv-env\Scripts\Activate.ps1; if PowerShell refuses to run it, use a Command Prompt instead. Already have Python 3.11? Use py -3.11 in step 2.

macOS or Linux (terminal)

macOS / Linux · 1/7
# Start in your home folder
cd ~
macOS / Linux · 2/7
# Create a Python environment for TruVelocity™ (Python 3.11 or later)
python3 -m venv tv-env
macOS / Linux · 3/7
# Activate it
. tv-env/bin/activate
macOS / Linux · 4/7
# Install TruVelocity™ with the web UI
pip install "truvelocity-tevv[ui]"
macOS / Linux · 5/7
# Create the offline demo project in a new folder
tv demo-project truvelocity-demo
macOS / Linux · 6/7
# Go into the project
cd truvelocity-demo
macOS / Linux · 7/7
# Start the UI with runs allowed, and open it in your browser
tv ui --allow-runs --open

Next time

Only these steps are needed:

Windows · 1/4
# Start in your home folder
cd /d %USERPROFILE%
Windows · 2/4
# Activate the environment
tv-env\Scripts\activate
Windows · 3/4
# Go into the project
cd truvelocity-demo
Windows · 4/4
# Start the UI
tv ui --allow-runs --open

What is in the demo project

Agent idName shown in the UILatest release runUse it to show
claimsClaims back-office agentGo · 100% of cases passed every try, no defectsA release that meets the bar; the evidence pack
chatCustomer chat assistantNo-go · 1 critical, 1 major, 1 minor; 67% of cases passed every tryThe gate stopping a risky release; what a critical defect looks like
deskMember service deskGo · changes over its runsContinuous: trend, drift and alerts

Each agent has at least three release runs, so Trend and Drift have something to compare. A run you start from Conduct finishes in about a minute, with no Docker and no LLM.

BLive pilots (advanced) real agents in Docker, with an LLM

The Harbor Mutual demo agents (a customer chat assistant, an event-driven claims agent, and a multi-agent member desk) run as Docker containers built from the SystemsUnderTest repository and call a real language model. Use this when the audience wants to see live agents; use the offline project for everything else.

Prerequisites

  • This repository cloned, with its .venv (the ops script's setup creates it).
  • Docker Desktop running.
  • A clone of the SystemsUnderTest repository next to this one (or TV_SUT_DIR pointing to it).
  • LLM settings in the root .env (copy .env.example; DEFAULT_LLM_PROVIDER and that provider's keys). The ops script never prints or changes them.

Step-by-step installation: TruVelocity™ Local Install Guide.

Start the agents and the UI (Windows)

Windows · 1/4
# Go to your clone of this repository
cd C:\myprojects\TruVelocityForAITEVV
Windows · 2/4
# Build and start the three agents in Docker (about 2 minutes)
scripts\truvelocity-tevv-ops.bat deploy all
Windows · 3/4
# Check every agent is up
scripts\truvelocity-tevv-ops.bat status
Expected output
Every agent and observer line says up.
Windows · 4/4
# Start the UI on port 8770 with runs allowed, and open the browser
scripts\dev_deploy.bat

On macOS or Linux use sh scripts/truvelocity-tevv-ops.sh deploy all, … status and sh scripts/dev_deploy.sh. The deploy script creates .venv if needed, installs TruVelocity™ with the UI extra, and runs tv ui --listen 127.0.0.1:8770 --allow-runs --open.

Optional: fresh runs before the demo

The UI shows whatever runs are in tv_runs. To have a release run from today, run a pilot from the command line beforehand (one per agent; each takes a few minutes and calls the LLM):

Windows · optional
# A live full run of the claims agent
scripts\truvelocity-tevv-ops.bat pilot-claims
Windows · optional
# A live full run of the member desk
scripts\truvelocity-tevv-ops.bat pilot-desk
Don't run pilot-chat just before a demo: the full chat catalog asks every question 13 ways (416 conversations in the latest run) and takes a long time.

✓Pre-flight checklist 10 minutes before the audience arrives

  1. Set TV_UI_USER (or user: in tv-ui.yaml) to the presenter's full name and restart tv ui: the Home greeting and every decision you record carry this name (without it, the computer's login name). What is typed on the sign-in page is not stored.
  2. Click Explore as demo user: Home shows three agents. Reload the page once.
  3. The Agent picker (top right) lists all three agents.
  4. Open Conduct for one agent: it shows a Start a run form, not "Runs are off in this UI". If it says runs are off, restart with --allow-runs.
  5. Live pilots only: scripts\truvelocity-tevv-ops.bat status says every agent is up.
  6. Click Report on one Home row: the execution report opens in a new tab. Close it.
  7. Browser zoom 100–110%, other tabs closed, notifications off. If you share one window, share the browser, not the console (the console shows the token).
  8. Decide your story in advance: which agent is the Go, which is the No-go, which shows change over time (see the guided path).
  9. Last: click Sign out (the arrow icon top right, next to the bell), so the audience starts on the welcome page.

3The guided path and timing 15–20 minutes

The path follows one question leadership cares about: can we release this agent, and can we prove it? Context and Configure get a short mention; spend the time on Catalog, Conduct, Conclude and Continuous.

TimeScreenWhat you showThe point you land
1 minWelcome pageThe opening slide: TEVV, the eight agent types, local and private. Then Explore as demo user.Agents act, not just talk; one assurance flow covers every kind of agent.
3 minHomeThree tiles, Agent portfolio (Verdicts, then Risk by pillar), Needs attention, Recent runs.One view of every agent's release readiness, from real runs.
1 minContext (optional)The agent summary and "What this means for testing".Testing starts from what the business says the agent is for.
3 minCatalogQuestions × wordings, suites with times, library packs, Your test plan, gaps.Coverage is visible, and every gap is a recorded decision.
1 minConfigure (optional)At a glance.We know how the agent is reached and which model was tested.
3 minConductStart a run of one suite; watch it progress; Recent runs.Repeatable, governed runs: anyone can press the button.
5 minConcludeClaims: release bar, verdict, decision, evidence pack. Then chat: the No-go and its critical defects.A Go / No-go backed by evidence, with an audit trail.
3 minContinuousDesk: Trend, Drift, Alerts, Run on a schedule.Quality is watched after release, not checked once.
1 minCloseBack to Home.Evidence before go-live, and every release after.
"AI agents no longer just answer questions: they take payments, approve claims and change records. When they fail, they often fail quietly: they say 'done' when nothing happened, or change data nobody asked them to change. TruVelocity™ tests the agent from the outside, checks what it actually did against independent records, and turns that into a release decision you can defend." Then the welcome page script in Welcome page.
Close: "So, from the portfolio to a single defect and back: we know which agent can go live, why the other can't, and we have the evidence pack to prove it. And after go-live the same tests keep running, so a model or prompt change can't quietly break it."

Run started in Conduct: click Start from: Smoke and Start run as you reach Conduct. On the offline project it finishes in seconds, so you can open its execution report straight away; on the live pilots let it run while you talk through Conclude and come back to it at the end.

4The 5-minute version

TimeClickSay
1 minWelcome page (15 seconds), Explore as demo user, then Home: point at Release-ready, Open critical defects and the Agent portfolio."One workspace for every kind of AI agent, private to this laptop. Three agents under test, from real runs. Two are release-ready, one isn't, and here is why."
2 minPick the chat agent, open 5 Conclude: Verdict, then the Defects table."No-go. Not a score: specific defects. This one asked the customer to approve an action they never requested. That is critical, so the release stops."
1 minPick the claims agent, still on Conclude: Verdict, Release decision, Download evidence pack."This one meets the bar. The decision is recorded with who, when and why, and the evidence pack is what we hand the auditor."
1 minPick the desk agent, open 6 Continuous: Trend and Drift."After release we keep watching: each run is compared with the one before, question by question, so a regression shows up the day it happens."

5Words the UI uses use them exactly

TermWhat it means
Case and tryA case is one test question with its checks. Each case is asked several times: in several wordings (13 for the chat pilot, 3 for claims) and sometimes repeated. Each of those is a try.
Cases passed every tryThe release gate's measure: a case that fails any of its tries counts as failed. This is the honest number for an agent that must work every time.
Tries passedEach try counted on its own. Always at least as high as cases passed every try (chat's latest live run: 87% of tries, 69% of cases passed every try).
FlakyA case that passed some tries and failed others. It counts as a failure at the gate.
Release barThe rules in truvelocity-tevv-gate.yaml: no critical defect, at most 2 major, at least 85% of cases passing every try, evidence seams clean, parity holding.
VerdictGo when the run meets the bar ("Meets the bar"), No-go when it doesn't ("Below the bar"). Home's Release-ready tile counts the Go agents among those assessed.
Defect severityBy what went wrong. Critical: the agent did, or offered to do, something it shouldn't (asking a customer to approve an action they didn't request, changing data that should not change, the wrong customer's data). Major: it didn't do what was asked, or said something its actions don't support. Minor: the rest.
Evidence seamsAutomatic checks where what the agent says it did is compared with the independent records (for example "done" when nothing was saved). SE-01 to SE-04 must be clean; SE-06 is parity. Full list in the leadership demo runbook.
ParityTwo deployments of the same agent (another model, host or framework) behave the same, within normal run-to-run variation.
Release runA run of the whole catalog in a release mode. Only these drive the verdict, risk and trend; a run of one suite is a partial run and doesn't.
BaselineAn accepted earlier run. When chosen in Conclude, failures it already had count as known, not new.
DriftHow the latest release run differs from the one before it of the same shape, question by question: regressions, fixes, and how often the outcome, tool calls and data changes were the same.

WWelcome page 1 minute · the opening slide

The welcome page: the sign-in card on the left, TruVelocity™ at the centre of the eight agent types it assures on the right (offline demo project).
The welcome page: the sign-in card on the left, TruVelocity™ at the centre of the eight agent types it assures on the right (offline demo project).

Click

  1. Open the link tv ui printed (or sign out first, so the page shows). Let the audience read it for a moment.
  2. Point at the title, then the hub on the right, clockwise from Single-turn bot to Self-learning.
  3. Point at the sign-in card: Sign in, Continue with SSO and Explore as demo user.
  4. Click Explore as demo user (Sign in with any email, or Continue with SSO, does the same). A deep link (a URL with #…) opens its own page instead of Home after this click.

What is on screen

  • Sign in to your workspace: assurance for every kind of AI agent, from a single-turn bot to a self-learning swarm; below the card, what TEVV stands for.
  • Sign in to your workspace: work email, password, Sign in, Continue with SSO and Explore as demo user. In this release any of them opens the workspace; nothing typed is stored or sent (the card says so). Decisions are recorded under TV_UI_USER or user: in tv-ui.yaml, else the computer's login name.
  • Access: the page runs on this computer only and opens only through the key in the link tv ui printed.
  • The hub: TruVelocity at the centre, with eight agent types around it, from simple to autonomous: single-turn bot, conversational, grounded / RAG, tool-using, loop engineering, graph engineering, multi-agent, self-learning.
  • Shown first in every new browser session and after Sign out.
"This is TruVelocity for AI TEVV: test, evaluation, verification and validation, the full path from planning a test to proving an AI agent is ready to release. Around it are the kinds of agents we assure, from a simple question-and-answer bot to agents that use tools, follow workflows, hand work to each other, and learn from use. One assurance flow covers all of them. It runs here, on this laptop: private, nothing leaves the machine, and every decision we record carries a name. Let's open the workspace."

Business value

  • Sets the frame in one picture: one approach for every agent the client has or plans.
  • Answers the data-privacy question before it is asked: local, key-protected, no cloud.
  • Accountability from the first click: decisions are recorded under a named person.

Watch for

  • Set TV_UI_USER to the presenter's full name beforehand, so decisions are recorded under it rather than a login name such as jdoe42.
  • Don't claim TruVelocity has tested a client agent of every type shown: the hub shows the types it is built to assure.
  • There are no user accounts or passwords: privacy comes from the local address and the key in the link.

HHome 3 minutes · the portfolio view

Home with "All agents" selected: three tiles, the Agent portfolio (Verdicts view) and Needs attention (offline demo project).
Home with "All agents" selected: three tiles, the Agent portfolio (Verdicts view) and Needs attention (offline demo project).

Click

  1. After Explore as demo user, Home opens with Agent: All agents (top right).
  2. Point along the three tiles, left to right.
  3. Point at one row of the Agent portfolio; hover the trend line.
  4. Click Risk by pillar in the portfolio's header to switch views; click Verdicts to switch back.
  5. Point at Needs attention, then Recent runs.
  6. Optional: click Report on the Go agent's row to show the execution report in a new tab.

What is on screen

  • A greeting (with the presenter's name when TV_UI_USER or user: is set) and a one-line summary.
  • Release-ready: Go agents out of those assessed (2 of 3 in the demo project). Open critical defects, and across how many agents (1, across 1 agent). Decisions waiting: release decisions and alerts to look at (2).
  • Agent portfolio, Verdicts view: one row per agent: its kind (conversational, task, multi-agent), a Status (for example "Awaiting release decision"), verdict, defects by severity, last run, and the trend of cases passed every try over its release runs, with Report.
  • Agent portfolio, Risk by pillar view: answer quality, agentic action, security & performance, governance & economics. Each cell shows types measured out of types in the pillar. Not assessed is never shown green.
  • Needs attention (and the bell): each failed release gate with the rule that failed, and alerts such as "Cases passed every try dropped".
  • Recent runs: the latest 5, with Tries passed next to Cases passed every try; All runs opens every run in the console.
"This is the portfolio. Three agents under test. Everything here comes from real test runs, the same results our pipeline uses; nobody types these numbers in. Two of the three are release-ready; one isn't, it has an open critical defect, and two decisions are waiting for us. The trend column is the strict measure: the share of test cases the agent passed on every single try, because an agent that works most of the time is not good enough to move money."
On Risk by pillar: "Same agents, seen by risk area. Red is a measured failure. Light orange means we haven't measured enough of that pillar yet, and we never show an untested area as green."
The Agent portfolio switched to Risk by pillar: types measured out of types in each pillar; light or hatched cells have little or no evidence yet (offline demo project).
The Agent portfolio switched to Risk by pillar: types measured out of types in each pillar; light or hatched cells have little or no evidence yet (offline demo project).

Business value

  • A release-readiness view across all AI agents in one place, for the people who sign off: three numbers, then the detail.
  • Risk shown by business area, including what has not been tested yet.
  • Drill from the portfolio to the exact defect in two clicks.

Watch for

  • Tries passed is always at least as high as Cases passed every try. If asked, explain the difference (see Words); don't call either "accuracy".
  • Run presets are on Conduct now, and the standards note is on Conclude's evidence pack card: don't look for them on Home.
  • Recent runs shows only the latest 5; use All runs if someone asks about an older run.
Recent runs: the latest 5, with Tries passed and Cases passed every try side by side, and All runs (offline demo project).
Recent runs: the latest 5, with Tries passed and Cases passed every try side by side, and All runs (offline demo project).

Demo project: Home shows 2 of 3 release-ready (Claims back-office agent and Member service desk are Go; Customer chat assistant is No-go), 1 open critical defect across 1 agent and 2 decisions waiting; Needs attention lists "Release gate failed" and "Cases passed every try dropped" for the chat assistant.

1Context 1 minute in a demo · optional

Context for the claims agent: the Context assistant on the left, the editable agent summary on the right.
Context for the claims agent: the Context assistant on the left, the editable agent summary on the right.

Click

  1. Choose the agent in the Agent picker, then the 1 Context tab.
  2. Point at the topic chips (Intended use, Goals, Domain …) and the % captured.
  3. Scroll the summary to What this means for testing.

What is on screen

  • A Context assistant that walks through 15 topics (Essential or Full) and files your answers.
  • The Agent summary: purpose, users, actions with money or legal effect, systems it changes, oversight, guardrails, worst failure to prevent. Click any text to edit; Import workbook or YAML brings in an existing questionnaire.
  • What this means for testing: the pillars and test families the answers call for (for example "High: money, data or legal exposure → stricter release bar").
  • The "Built for this agent" chips at the top of every later step come from here.
"Testing starts from the business, not from the tool. Here we write down what this agent is for, what it may do with money and data, and the worst thing it must never do. Everything after this, the test plan and the release bar, follows from it."

Business value

  • One written statement of intended use and risk appetite that product, risk and engineering agree on; the basis for TEVV and for later audit.
Don't type into the assistant or click Save context during a demo unless you mean to change the agent's context.

2Catalog 3 minutes · what we test

This agent's catalog: questions, wordings, test types run, suites with measured times, library packs (claims agent, live pilots).
This agent's catalog: questions, wordings, test types run, suites with measured times, library packs (claims agent, live pilots).

Click

  1. Pick the claims agent and the 2 Catalog tab.
  2. Point at the three tiles: questions, wordings, test types in plan.
  3. Point at Suites and the time columns.
  4. Point at Library packs (✓ included, + available).
  5. Scroll to Your test plan; optionally click Open … test plan document.

What is on screen

  • The test plan read from the agent's catalog.yaml, with states from its latest full release run.
  • Questions × wordings: each question is asked several ways, because agents built on language models answer differently to different phrasing.
  • Suites (sanity, smoke, regression, positive, negative, edge, boundary, security) with run times measured on the latest run.
  • Library packs: ready-made tests (prompt injection, privacy, payments and approvals, unsafe writes …). Adding one shows what it adds before anything is written.
  • Your test plan: test types in plan out of the full catalog, how each is measured (Gate, Track, Info), the release bar, the four disciplines (Test, Evaluation, Verification, Validation) and recommended types left out.
Your test plan: coverage against the full catalog, the release bar, and gaps called out ("recommended tests are switched off").
Your test plan: coverage against the full catalog, the release bar, and gaps called out ("recommended tests are switched off").
"This is what we test. Sixteen questions, each asked three different ways, grouped into suites, and the time each suite takes, measured, not guessed. On top sit library packs: ready-made tests for prompt injection, privacy, payments. And here is the part auditors like: when a recommended test is left out, the plan says so, and the gap is recorded as a decision with a reason."

Business value

  • Coverage is visible: leadership sees what is tested and what is not.
  • Gaps are deliberate, recorded decisions, not oversights.
  • Reuse: library packs bring tested know-how to every new agent, so a new agent starts from more than a blank page.
  • Time estimates make test effort plannable.

Watch for

  • Don't tick types in or out, add packs or confirm the plan in a demo: those change the agent's saved plan.
  • Don't walk through all 142 test types. Name one or two categories and move on.
  • "Not built yet" types exist in the catalog but the engine doesn't run them today. Be plain about it.

3Configure 1 minute in a demo · optional

Configure: At a glance, the connection check, and the technical details folded away.
Configure: At a glance, the connection check, and the technical details folded away.

Click

  1. 3 Configure. Read out At a glance.
  2. Optional: open Technical details for a technical audience.
  3. Click Check connection now only if you confirmed in pre-flight that the agent is up.

What is on screen

  • Does the agent answer? The last connection check.
  • How a run starts: the run preset from tv-console.yaml (for example "Pre-release gate, every wording").
  • Model under test, as recorded on the latest release run.
  • Technical details: settings, run preset and the versions the latest run used.
"This is how TruVelocity™ reaches the agent: through the same interface its users or systems use, with no change to the agent. And every run records which model, prompts and tools were under test, so a result is always tied to a version."

Business value

  • Black-box testing of in-house and vendor agents alike; reproducible results tied to a version.

4Conduct 3 minutes · run the tests

Start a run after Start from: Smoke: the smoke suite ticked, one wording, and the estimate (offline demo project, Claims back-office agent).
Start a run after Start from: Smoke: the smoke suite ticked, one wording, and the estimate (offline demo project, Claims back-office agent).
Started here: the Smoke run finished in 4 seconds on the offline project, with its execution report one click away.
Started here: the Smoke run finished in 4 seconds on the offline project, with its execution report one click away.

Click

  1. 4 Conduct for the agent you are showing.
  2. In Start a run, click Start from: Smoke. It fills the form: the smoke suite, one wording.
  3. Read out the time estimate under the form. Leave the rest as it is.
  4. Click Start run. A card appears under Started here.
  5. Move on to Conclude while it runs; come back when the toast says it finished, and click Execution report.

What is on screen

  • Start from: three ready-made starting points that fill the form. Smoke: the smoke suite, one wording, a few minutes at most. Pre-release gate: the whole catalog, every wording. Reliability run: the whole catalog, 3 repeats.
  • The form works within the agent's preset (from tv-console.yaml): suites, wordings, repeats, workers, mode, and an estimate from times measured on earlier runs.
  • The job card: run id, Running / Finished / Failed / Cancelled, elapsed time, a progress bar, "n of m episodes · passed · failed · flaky", a Cancel button and the engine's output.
  • A run that couldn't start (for example another run already uses the runs folder) shows Not started.
  • Recent runs of this agent: date, mode (Full catalog, Pre-release gate, Selected scenarios), cases passed every try with tries underneath, defects and the gate result.
"Anyone on the team can run the agreed plan from here: pick a starting point, Smoke for a quick check or Pre-release gate for the full release test, and press Start. No scripts, no command line. It is the same engine our CI pipeline runs. Each question goes to the agent, and while it answers we record every tool call and every handoff, and we read the records it was meant to change, read-only, before and after each task, so we can compare them row by row."
When it finishes: "Done. Here is the execution report for that run: every conversation, what the agent said, what it actually did, and what changed in the data."

Business value

  • Testing a release takes minutes, not a manual test cycle: faster, repeatable releases.
  • The same governed plan every time; nobody improvises the tests.
  • Every run is kept with its inputs, outputs and evidence: nothing to reconstruct later.

Watch for

  • Use Smoke in a demo. A suite run is a partial run, so it doesn't replace the release run your verdict and trend come from. Pre-release gate and Reliability run run the whole catalog: the result becomes the new latest release run and can change the numbers you rehearsed.
  • Offline project: Smoke finishes in seconds, a whole-catalog run in about a minute. Live pilots: a claims or desk suite takes 1–2 minutes; never start the full chat catalog in a demo.
  • "Runs are off in this UI" means the UI was started without --allow-runs.

5Conclude 5 minutes · the release decision

Tell this in two parts: first the agent that meets the bar (claims), then the one that doesn't (chat).

Part 1: an agent that meets the bar (claims)

The release bar, read from truvelocity-tevv-gate.yaml.
The release bar, read from truvelocity-tevv-gate.yaml.
Verdict: Meets the bar. Each rule met.
Verdict: Meets the bar. Each rule met.
Defects of the run under review: one major, no critical (claims agent, live pilots).
Defects of the run under review: one major, no critical (claims agent, live pilots).

Click

  1. Pick the claims agent, then 5 Conclude.
  2. Run under review already shows the latest full release run. Leave Baseline on "No baseline: every failure counts".
  3. Read the Release bar, then the Verdict table, then Defects.
  4. Point at Release decision (Go / Go with conditions / No-go, and Why).
  5. Point at Download evidence pack. On your own demo project you can click it and open the zip.

What is on screen

  • The run under review: how many cases passed, failed or were flaky, and a link to its execution report.
  • The release bar from truvelocity-tevv-gate.yaml (kept in version control).
  • The verdict: the engine's release gate, rule by rule: defects, pass rate, seams, each met or not met, with why.
  • Defects: severity, case, what went wrong in plain words, and how often (every try, or 2 of 13 tries).
  • Release decision: Go, Go with conditions, or No-go, with a reason, kept with who and when.
  • Evidence pack: one zip with the run folder (what ran, every conversation, defects, scorecard, reports), the decision and the verdict as judged. The card also says that mapping this evidence to OWASP Top 10 for LLM apps, NIST AI RMF, ISO/IEC 42001 and the EU AI Act comes in a later release.
"Here is the decision page. The release bar is written down: no critical defect, at most two major, and at least 85% of cases passing every single try. This run meets it. There is one major defect, and you can see exactly what it is: the reply left out the claim number. The decision is recorded with who, when and why. And this button gives us the evidence pack: every conversation, every defect, the verdict and the decision, in one file for audit."
Release decision, before a decision is recorded.
Release decision, before a decision is recorded.
Evidence pack: the run folder plus the decision and the verdict in one zip; mapping to OWASP, NIST AI RMF, ISO/IEC 42001 and the EU AI Act comes in a later release (offline demo project).
Evidence pack: the run folder plus the decision and the verdict in one zip; mapping to OWASP, NIST AI RMF, ISO/IEC 42001 and the EU AI Act comes in a later release (offline demo project).

Demo project: the claims run under review is the latest claims run (build 3.1.1, named claims-<date>-091538): 100% of cases passed every try and no defects.

Part 2: an agent that doesn't (chat)

Verdict: Below the bar. 8 critical and 10 major defects; 69% of cases passed every try (chat assistant, live pilots).
Verdict: Below the bar. 8 critical and 10 major defects; 69% of cases passed every try (chat assistant, live pilots).
The top of the Defects table: critical first, each with what went wrong and how often.
The top of the Defects table: critical first, each with what went wrong and how often.

Click

  1. Switch the Agent picker to chat; stay on Conclude.
  2. Read the Verdict: each rule not met, and why.
  3. In Defects, point at one critical row: "Asked to approve an action this step should not take" (BL-01: the customer only asked what they owe, and the agent asked them to approve paying it).
  4. Point at the How often column: "every try", "9 of 13 tries".
  5. Point at the recorded No-go decision, if one is shown.

What is on screen

  • The same bar, applied to a different agent: Below the bar.
  • Critical defects such as: asking the customer to approve a payment or a policy change they never requested; data that changed where no change was expected; an action that was needed but never succeeded.
  • How often: a defect on 1 try in 13 is still a defect. That is why the gate counts cases passed every try.
"Same bar, different agent. This one is a No-go, and not because of a score: because of specific defects. Here the agent asked a customer to approve a payment the customer never asked for. That's critical: it is one click from moving money nobody requested. And look at how often: in some cases it happens on only a couple of the thirteen ways we ask. A one-time manual test would very likely miss it."
A recorded release decision: who, when and why.
A recorded release decision: who, when and why.

Business value

  • A release decision made with evidence, against a bar agreed in advance, not on a demo that happened to go well.
  • Risk reduced before customers see it: critical defects stop the release automatically.
  • An audit trail: who decided, when, why, on which run, with the evidence pack to prove it.
  • Engineers get specifics (case, what went wrong, how often), so fixes are faster.

Watch for

  • Don't click Record decision or Mark Conclude complete on a shared project: decisions are kept in the project (.tv-ui). On your own demo project it is fine, and it is a good moment.
  • A Go on a run below the bar needs a written justification. Show it only if asked about overrides.
  • Multi-approver sign-off and the standards mapping (OWASP, NIST AI RMF, ISO/IEC 42001, EU AI Act) come in a later release; the evidence pack card says so.
  • A recorded decision carries the presenter's name (TV_UI_USER or user:, else the login name).

Demo project: chat shows 1 critical (BL-01), 1 major (an address change it says it made but didn't), 1 minor (it then quotes the old address); 67% of cases passed every try, down from 100% on build 2.3.0.

6Continuous 3 minutes · after release

Trend: full release runs, newest first, with the bar, the decision and what changed since the run before (claims agent, live pilots).
Trend: full release runs, newest first, with the bar, the decision and what changed since the run before (claims agent, live pilots).

Click

  1. Pick the desk agent and the 6 Continuous tab.
  2. Trend: read the line and the table from oldest to newest.
  3. Drift: regressions, fixes, still failing, and the "same …" percentages.
  4. Alerts: which rules are quiet and which would fire.
  5. Run on a schedule: show the GitHub Actions or Azure Pipelines tab and Show the file. Don't click Write … pipeline.

What is on screen

  • Trend: each release run with cases passed every try, defects (critical / major / minor), bar met or not, the decision, and what changed since the run before (model, prompts or tools, by fingerprint).
  • Drift: the latest release run against the one before of the same shape, question by question.
  • Alerts: rules checked against the latest run: gate fails, a new critical defect, a case that passed before now fails, the model / prompts / tools changed, pass rate drops more than n points.
  • Run on a schedule: a ready pipeline file for CI, or this machine's scheduler, running the same tv run command.
  • A banner may say the step isn't ready to finish: monitoring starts from a concluded baseline.
Drift: the latest release run against the run before it: one regression, three fixes (claims agent).
Drift: the latest release run against the run before it: one regression, three fixes (claims agent).
Alerts on the member desk: the gate failing and a prompts change both "would fire".
Alerts on the member desk: the gate failing and a prompts change both "would fire".
"Passing once is not the end. Agents change when someone edits a prompt, or the provider updates the model. Here is this agent over its last runs: you can see it improve, and you can see exactly which change came in between: here, a prompt change. Drift compares each run with the one before, question by question: one regression, three fixes. And the alert rules tell us now what they'd say: this one would fire because a case that used to pass now fails."
On the schedule: "We don't run anything on a timer ourselves; we plug into the CI you already have. This is the pipeline file: nightly and on every pull request, with the same release gate."

Business value

  • Regressions are caught the day they happen, not by a customer.
  • Model, prompt and tool changes are tied to their effect on quality.
  • Quality gates run in the existing CI; no new platform to operate.

Watch for

  • Language-model agents disagree with themselves a little from run to run. A few points of "same outcome" below 100% with no regression is normal; say so before someone asks.
  • Sending alerts by email or chat comes in a later release; today the rules are checked when the page opens and in the pipeline.
  • Don't click Save monitoring, Write … pipeline or Mark Continuous complete on a shared project.
Run on a schedule: GitHub Actions, Azure Pipelines or this machine, all running the same tv run command.
Run on a schedule: GitHub Actions, Azure Pipelines or this machine, all running the same tv run command.

Demo project: the desk agent's release runs go the Member service desk's four release runs go 50%, 75%, 88%, 100% of cases passed every try (builds 1.0.0 to 1.2.1): 1.0.0 let a billing outage cascade and passed too many permissions on (two critical defects), fixed by 1.2.0; Drift shows DSK-CASE-01 fixed.

?Leadership questions crisp answers

These are the questions the UI itself raises. For the broader ones (why we need it, what makes it unique, the seven evidence seams, black-box vs white-box, architecture), use the leadership demo runbook.

Where do these numbers come from? Can someone edit them?
From real test runs kept in the project's runs folder: the same results tv gate and tv report use, and the same ones CI produces. The UI reads them; it has no field to type a pass rate into. Decisions and plan changes you make in the UI are stored separately, with who and when.
Why is "Cases passed every try" lower than "Tries passed"?
Tries passed counts every attempt on its own. Cases passed every try counts a case as passed only if it passed on every wording and repeat. An agent that gets a payment right 12 times out of 13 still gets it wrong for one customer in 13, so the release gate uses the stricter number.
Who sets the release bar? Can it be different for us?
It's a short file, truvelocity-tevv-gate.yaml, kept in version control and agreed with the business. You can change the limits, and add stricter rules for one capability (for example 100% for settlements). The UI shows which file is in force on every verdict.
Can a manager override a No-go?
Yes, deliberately and visibly: a Go on a run below the bar needs a written justification, and the decision is kept with who (TV_UI_USER, else the login name) and when and goes into the evidence pack. The verdict itself never changes; the override sits next to it.
What makes a defect critical?
What went wrong, not how often. Critical means the agent did, or offered to do, something it shouldn't: asking a customer to approve an action they didn't request, changing data that should not change, acting for the wrong customer. One critical defect is a No-go.
Is an AI grading the AI?
Mostly no. Most checks are rules on recorded facts: which tools were called with which arguments, what changed in the data, what the reply contains. Where an AI grader is used, TruVelocity™ measures it against labelled examples before relying on it.
Does it touch our production systems or data?
It runs against a test environment. To check the agent's effect on data, it reads the records the agent was meant to change (chosen database tables, files), read-only, before and after each task, and compares them row by row. It never writes to them.
Do we have to change our agent?
No. TruVelocity™ drives the agent through the interface its users or systems already use (chat, API, events) and observes from outside. Details in the leadership runbook.
How long does a test cycle take?
The Catalog shows it per suite, measured on the latest run. The claims pilot's whole catalog in all three wordings takes about 2 minutes on one worker; the chat pilot, asked 13 ways, takes much longer and is split across workers or run nightly.
Can this run in our pipeline?
Yes. Continuous writes a GitHub Actions or Azure Pipelines file that runs the same catalog nightly and on pull requests, publishes JUnit results, applies the release gate and keeps the run folder.
Is there a login? What does the sign-in page check?
In this release, nothing: the sign-in page is the product's front door as designed, and any entry (or Explore as demo user) opens the workspace; nothing typed is stored or sent. Access is protected another way: the UI runs on this computer only and opens only through the key in the link it printed. Accounts and single sign-on come with the shared-server version. Decisions carry the name in TV_UI_USER or user: in tv-ui.yaml, otherwise the computer's login name.
Where is the data kept? Is it in the cloud?
On the machine that runs it: runs in the project's runs folder, UI decisions in the project's .tv-ui folder. The UI listens only on this machine (127.0.0.1) and needs the token in its URL.
Does it map to OWASP, NIST AI RMF, ISO/IEC 42001 or the EU AI Act?
Not automatically yet: that mapping comes in a later release, as the evidence pack card on Conclude says. Today the evidence pack holds what a reviewer maps by hand.
Why does the same agent give different results on different runs?
Agents built on language models aren't deterministic. That's why each question is asked several ways and repeated, why the gate uses cases passed every try, and why Drift shows how much a run agrees with the one before.

!Don't do this in a demo

Don'tInstead
Start a whole-catalog run (Pre-release gate or Reliability run) just before you show ConcludeUse Start from: Smoke. A partial run doesn't change the latest release run, so the verdict and trend stay as rehearsed.
Start the full chat catalog on the live pilotsIt runs hundreds of conversations against the LLM. Use the offline project, or a claims or desk suite.
Click Record decision, Save, Save context, Save monitoring, Write … pipeline or Mark … complete on a shared projectThese write to the project. Use your own demo project, where they are part of the story.
Click Check connection on the live pilots without checking Docker firstRun status in pre-flight; skip the check if in doubt.
Paste deep links (URLs with #…) to jump to a stepUse the Agent picker and the step tabs. (A deep link does open its page after you sign in, but the tabs are more reliable on stage.)
Open on the welcome page with a bare login name, or skip the welcome pageSet TV_UI_USER to the presenter's name, and Sign out before the audience arrives: the sign-in page is your opening slide.
Share the console windowThe console shows the URL token. Share the browser only.
Start the UI with --allow-remote on a client networkKeep the default: it listens on 127.0.0.1 only.
Call the pass rate "accuracy", or quote Tries passed as the release numberUse the UI's words: Cases passed every try is the release gate's measure.
Describe the data check as copying or capturing the client's systemsSay it reads the records the agent was meant to change, read-only, before and after each task, and compares them row by row.
Promise standards mapping, multi-approver sign-off or alert e-mails todayThey are marked in the UI as coming in a later release. Say so.
Tour all 142 test types or every Context topicOne example each; keep to the guided path.
Apologise for "not met" on a live agentThat is the product working: the gate found something before customers did.

⚙Troubleshooting

What you seeWhat to do
'tv' is not recognized / command not foundThe environment isn't active: run the activate step again.
error: no tv-ui.yamlYou aren't in the project folder: cd truvelocity-demo.
The port is already in usePick another: tv ui --listen 127.0.0.1:8771 --allow-runs --open.
The browser didn't open, or says unauthorisedCopy the whole URL the command printed, including ?t=….
Conduct says "Runs are off in this UI"Stop the UI (Ctrl+C) and start it again with --allow-runs.
A started run shows Not startedThe run was refused, usually because another run is using the same runs folder. Wait for it to finish, then start again.
Live pilots: a run fails at once, or Check connection failsscripts\truvelocity-tevv-ops.bat status; redeploy with deploy all; check the LLM settings in .env.
Numbers differ from this runbookExpected: the screenshots are from the live pilots on 19 Nov 2026. The UI always shows the latest runs.

↺After the demo

Stop the UI: press Ctrl+C in its window.

Offline demo project: start fresh

Decisions, plan changes and runs you made live in the project folder. For a clean project next time, delete the folder and create it again (from the folder above it):

Windows · 1/2
# Delete the used demo project
rmdir /s /q truvelocity-demo
Windows · 2/2
# Create a fresh one
tv demo-project truvelocity-demo

Or, without deleting: tv demo-project truvelocity-demo --force (from the folder above it) writes the project again and remakes its runs, clearing anything recorded in the UI.

Live pilots: restore the seed data

Windows · 1/2
# Restore the chat agent's seed data
scripts\truvelocity-tevv-ops.bat reset chat
Windows · 2/2
# Restore the claims agent's seed data
scripts\truvelocity-tevv-ops.bat reset claims

Related pages: Leadership demo runbook (pitch, Q&A, architecture) · Demo runbook (the command line) · Local install guide.