TruVelocity for AI TEVV: system prompt and build specification
Use this document two ways:
As a system prompt: paste it whole into an LLM and ask it to "build the TruVelocity prototype as one HTML file". Everything needed is here.
As a developer spec: each section says what a screen shows, the rule behind it, and where a real backend replaces the prototype's simulation. Look for WIRE: markers.
1. What you are building
TruVelocity for AI TEVV is a single-page web app that plans, runs and signs off the Test, Evaluation, Verification and Validation (TEVV) of one AI agent at a time. It is built on the TruVelocity testing engine.
Who uses it: a QE or assurance lead (runs it), a business or product owner (answers Context, owns the agent), an approver or signer (human checkpoints, release decision), a reviewer (scores uncertain items).
The six steps (always in this order):
Context: capture what the agent is, who uses it, its intents, systems and risks.
Catalogue: the test plan. Recommended test types come from Context. The user adds, removes, sets metrics and targets.
Configure: everything needed to run: connection, data, evaluators, guardrails, oversight, checks.
Conduct: run the plan under hard-stop checkpoints and human control.
Conclude: compare the run to the release bar. Go, conditional go or no-go, with evidence.
Continuous: keep watching after release: schedule, triggers, drift.
Design principles (apply everywhere):
Evidence first. Every claim links to what the agent said, what it did (tool calls) and what changed (backend data).
Never green without evidence. "Not assessed" is its own state. A test that did not run is "Blocked" or "Not run", never "passed".
Hard stops. Every checkpoint in Conduct stops the run until a named person decides. No answer in time means the run stays paused. Nothing is auto-approved.
Four eyes. The person who started a run cannot decide on its approvals. Accepting a Critical finding needs two approvers, never the runner.
Not a certification. Say "This statement covers this run only and is not a certification" wherever a verdict appears.
Everything simulated is labelled. The prototype shows "Data is simulated: nothing runs against a real system yet."
Deterministic simulation. The same agent and run number always give the same results (seeded hash), so demos repeat.
2. Output contract
One self-contained .html file. All CSS, JS and data inline. Opens by double-click, no server. Size about 1 MB (the test library is about 430 KB of JSON).
Only external request: Google Fonts (IBM Plex Mono, Inter, Space Grotesk). It must fall back to system fonts offline.
Vanilla JS, no framework. Modules are IIFEs sharing one global TV namespace.
Hash router. #/home, #/context[/id], #/catalogue/id, #/configure/id, #/conduct/id, #/conclude/id, #/continuous/id.
State: localStorage with try/catch everywhere (it can throw or be empty). Fall back to memory.
Accessible: "Skip to content" link, labelled controls, aria-pressed on toggle chips, aria-disabled plus a title explaining why on blocked buttons, keyboard operable, visible focus, colour never the only signal.
Responsive: works at phone width. Tables scroll horizontally inside a wrapper, not the page.
Dark mode: CSS custom properties on :root, redefined under prefers-color-scheme: dark.
No page errors in the console on any route.
Copy style: plain English, short sentences, sentence case. Say what happened and what to do next. Blocked buttons say why ("Seal a full-scope run in Pre-release gate or Reliability mode first.").
3. Architecture
3.1 Modules (one IIFE each, all attach to TV)
Module	Prefix	Owns
TV.h (home shell)	none	login, top bar, Home, agent list A, BY[id], toast, dialogs, focus agent
TV.ctx	cx-	Context chat, summary document, question bank, import from workbook
TV.steps	sv-	step controller, router, step frame, gating, agent traits, Continuous screen
TV.cat	ct-	Catalogue: recommend, browse, custom tests, metrics, roles, snapshot
TV.suite	su-	Test suites, library packs, import a suite, mapping dropdowns
TV.cfg	cf-	Configure: 13 sections, calibration, gate, info
TV.cnd	cd-	Conduct engine and UI, plus Conclude (ccl.js) and evidence (evd.js) sharing its scope
TV.evd	 	stage ribbon and "must be true" text, per-turn evidence
Cross-module calls (keep these as the public interface):
TV.ctx.facts(id) returns the agent's context facts (see 4.1).
TV.steps.traits(facts) returns the boolean traits used to recommend tests.
TV.cat.snapshot(agent, facts, traits) returns {items, S, rungs} for selected tests only.
TV.cat.addTests(agent, ids) selects tests (used by packs and import).
TV.cfg.info(agent) returns the Configure values v, evaluators, oversight ov, intent settings ic, checks.
TV.cfg.gate(agent) returns {ok, cal, pending, why}. ok means every Configure section is ready.
TV.cfg.setCaps(agent, cost, mins) sets spend and time caps.
TV.suite.list(agent) returns suites with their resolved scope.
TV.cnd.view and TV.cnd.conclude render Conduct and Conclude.
TV.steps.signoff(agent, sign) records the decision and sets the agent verdict.
3.2 Step controller
Step keys: ctx, cat, cfg, cond, cert, cont. The route for cert is conclude.
Each step has a status: todo, progress, done, or review (needs a second look after a change).
Gating (prereq):
Catalogue needs Context done.
Configure needs Catalogue confirmed.
Conduct needs Configure complete.
Conclude needs a sealed run. Without one it shows "No run to conclude yet".
Continuous needs a recorded Conclude decision.
The "Mark X complete and continue" button is always present and disabled with a reason until its rule is met.
Every step page has the same frame: breadcrumb, title and status tag, one-line subtitle, six-step progress strip, "Built for this agent" trait chips with a "Change in Context" link, a "First-cut screen. Data is simulated" banner, the body, and a footer with Back and the Next button.
3.3 Persistence keys
tv.cat.ag.v1.<agentId>: selected tests, custom tests and categories, metrics, thresholds.
tv.cfg.v1.<agentId>: Configure values v, secrets (masked only), calibration, checks, oversight ov, intent settings ic.
tv.cnd.v1.<agentId>: Conduct scope, mode, saved scopes, run rules, gate policy, runs (last 6), current run, Conclude decisions ccl{dec,req}.
tv.suite.v1.<agentId>: suites with add and ex overrides.
Context answers and the agent list are held in memory for the session in the prototype. WIRE: persist both server-side.
3.4 Agent object
 
Plain Text
{id, name, type, env, steps:{ctx,cat,cfg,cond,cert,cont}, verdict:'go|watch|nogo|na',
 certified:bool, crit, maj, min, last:'YYYY-MM-DD', trend:[pass^1 values],
 pil:{aq,aa,sp,ge:{s,m,t}}}      // risk by pillar: state, measured, total
Seven demo agents are seeded: Harbor Mutual Assistant (Tool-using, at Conclude, No-go), Policy Search Copilot (Grounded/RAG, complete, Go), Claims Intake Agent (Multi-agent, at Configure, No-go), Billing Reconciliation Agent (Loop engineering, at Conduct, Watch), Onboarding Workflow (Graph engineering, at Catalogue), Support Chatbot (Conversational, at Continuous, Go), Quote Assistant (Tool-using, not started, at Context).
4. Data model
4.1 Context facts (output of Context)
TV.ctx.facts(id) returns:{key, name, type, patterns[], industry[], intents[{name,kind,prio,over,outcome}], pillars[], bar[], signoff, env[], access[], cadence[], fault[], toneIn[], vis[], autonomy, ans:{questionId:{chips[],text}}, pct, answered, total, known}
intent.kind: Conversation, Task, Agentic.
intent.prio: P1, P2, P3. P1 is tested on every release.
intent.over: Autonomous, HITL (a person approves first), HOTL (a person watches after).
4.2 Test library (TVLIB, embedded JSON)
tevv: four lenses: T Test (run it and observe), E Evaluation (score quality), V Verification (built right), Va Validation (fit for purpose).
rungs: eight agent kinds, R1 Single-turn bot, R2 Conversational, R3 Grounded/RAG, R4 Tool-using, R5 Looping, R6 Graph/workflow, R7 Multi-agent, R8 Adaptive/self-learning.
cats: 23 categories C1 to C23: Intent & understanding; Entity & slot; Conversational quality; Grounding & factuality; Task & tool execution; Agentic & trajectory; Reliability & consistency; Safety & security; Compliance & fairness; Performance & operations; Continuous & platform; Seam & evidence consistency; Graph & workflow coverage; Multi-agent coordination; Adaptive & self-learning; Core task and output assurance; Multimodal assurance; Human oversight and interaction; Privacy and data governance; Deployment and operational assurance; Knowledge pipeline governance; Learning integrity and evaluation validity; Interoperability and execution boundaries.
stages: the seven-step evidence flow S1 Select tests, S2 Connect evidence, S3 Intent, S4 Orchestration, S5 Tool call, S6 Backend state, S7 User response.
ladder: per rung, the evidence needed and the category families.
tests: 142 test types. Fields: id (e.g. IU-01, SS-01), c (category), n, lane, d and dx (description), gr (grain: Turn, Adversarial episode…), pst/sst/cst (primary, secondary, contributing stages), rg (8 letters, one per rung: A always baseline, C conditional, N not designed), nd (needs: tools, rag, conv, graph, multi, learn, loop, mm, hitl, pii, reg, load, async), cond, meth, proc, orc, ev, acc, smp, fx, unit, risk, tv (lenses), m (metrics with direction, threshold, unit), prov, route.
Use only this 142-test library. Do not mix in other catalogues.
4.3 Agent traits (computed from facts)
traits(f) returns booleans and lists used everywhere: ty (types), conv, multiTurn, rag, loop, graph, multi, learn, sys (downstream systems), writes, tools, voice, files, struct, hitl, ins, reg, fair, pii, prod, scaleHigh, lat, cost, lang, abuse, vuln, async, tenant, memX, cite, verifies, backendSee, traces, floatV, pinned, autoLearn, maWeak, highAuto, highImpact.
writes is true if any write operation system is chosen or an irreversible insurance action exists.
tools is true if tools are listed, or type is Tool-using, or writes, or any intent is Task or Agentic.
4.4 Recommendation rule (statOf)
For each test x and the agent's rungs (from its types, default R1):
Custom test: recommended if approved and any nd need is satisfied; otherwise optional.
If rg[rung] = 'A' for any rung: Recommended ("Baseline for <rung> agents").
Else if rg[rung] = 'C' and (no nd or any nd satisfied): Recommended, conditional ("Applies because your agent uses tools…").
Else if conditional but not satisfied: Optional.
Else Outside profile (na).
Oversight bump: if Configure oversight has any HITL or HOTL intent, tests matching the oversight set (category C18) become recommended.
Suggested metric role per test: Gate if category C8; or C9 or C19 when regulated or handling personal data; or C5 or C12 when the agent writes or has high impact; or C18 when autonomy is high. Otherwise Track.
4.5 Severity classes (used by gate and Conclude)
Critical class: test is high risk. High risk means category in C8, C5, C9, C12, C18, C19, or risk text matches critical|safety|irreversib|financial|legal|regulat|privacy|unauthori.
Major class: has at least one Gate-role metric.
Minor class: everything else.
5. Screens
5.1 Login and shell
Login: email field #email, button #go. Any email works in the prototype. WIRE: real SSO.
Top bar: logo, Home, six numbered step links with status dots, agent switcher (all agents), notification bell with count, New Agent for TEVV button (enabled only on Home), sign out, avatar.
5.2 Home
Panels: Agent portfolio (name, flow, verdict, defects, last run, pass^1 trend, "Launch workspace"), KPIs (agents under assurance, active assessments, open critical defects, passing the release gate, verdict bar), Needs attention, Risk by pillar (Answer quality 24 types, Agentic action 35, Security and performance 17, Governance and economics 10; "Not assessed" is hatched, never green), Quick start (New agent, Existing agent, Re-run last, Browse catalogue: 142 test types, 23 categories; execution presets), Standards readiness (NIST AI RMF and ISO/IEC 42001 "Draft", EU AI Act "Roadmap", OWASP Top 10 for LLM apps), Recent runs.
Launch workspace opens a dialog showing the six steps with status, where the user will land and why, and an "Open <step>" button. The landing step is the first step not done.
New Agent for TEVV opens "What are you assuring?": a new agent (starts at Context) or an existing agent (starts at Configure).
5.3 Context
A guided chat on the left, a live summary document on the right.
15 topics: Intended use, Goals, Domain, Architecture, Agent design, Use cases, Priorities, Oversight, Systems and dependencies, Compliance, Tone, Guardrails, Quality bar, Test access, Known risks.
121 questions (37 Essential). Kinds: text, single, multi, widget (intents, priority, oversight). Each question: id, topic, label, kind, q, why, opts[{label, adds}], ess, req, mod, ap. 11 are required.
Conditional questions: Agent design questions appear only for the chosen agent pattern (Conversational, Grounded/RAG, Tool-using, Loop, Graph, Multi-agent, Self-learning, Single-turn). Insurance questions appear only when industry includes Insurance. Systems follow-ups appear only when a downstream system is chosen.
Depth: Essential or Full, switchable any time.
Each answer can be Skipped or marked "Not sure". Attachments are accepted per answer.
Summary document: 14 sections plus three derived ones ("What this means for testing", "Open questions", "Source documents"). Completeness shows as a percentage and "n of N questions".
Fill with sample answers: fills defaults for demos.
Import from workbook: opens a dialog. User pastes the YAML or chooses a .yaml/.yml/.txt/.json file. See section 8.
Continue to Catalogue: blocked while required questions are open. On success it creates the agent (id from its name), marks Context done, then shows the launch dialog.
Why-we-ask text is shown under each question: "Every run, defect and verdict is tied to a named version."
5.4 Catalogue
Purpose: turn Context into a test plan the user owns.
Header chips: agent traits. A plain-language reason on every test ("Applies because your agent uses tools or takes actions").
Filters: Recommended / Selected / All, search, TEVV lens, category.
Grouped by category with select-all for the category's recommended tests.
Each test row: checkbox, id, name, status badges (Recommended, Conditional, Outside profile, Added by you, Custom, Proposed, Draft, Edited), reason, "n of m metrics monitored", TEVV chips, Suites dropdown, Details button.
Details panel tabs: What it checks; How to run it (starts with the five-stage "What must be true" table: Intent, Plan, Tools, Backend, Reply; then Method, Needs before running, Steps, How we know it passed, Evidence kept, Acceptance, Sample plan); Metrics (role Gate or Track, direction, threshold, unit; add or edit a metric); Notes.
Custom categories and test types: the user defines them (name, description, TEVV, needs, method, steps, oracle). A custom test starts as Draft, needs an owner approval to become Approved and recommended. Edits to library tests show Edited.
"Generate TEVV Strategy & Plan Document": a template action (stub in the prototype).
Build your plan (library packs): nine packs (Privacy; Prompt injection and misuse; Approvals and human control; Tool failure and phantom success; Reliability; Grounding and knowledge; Fairness and compliance; Multi-agent hand-offs; Drift and production). Each shows the tests it adds and Add N to plan, plus "Gaps a pack would close". Import a suite: paste test IDs or names (comma or line separated) or upload a CSV or text file. Unknown names are listed as "not found".
Test suites: a suite is a named rule that picks tests. Six seeded types: Sanity (high-risk, Test lens, Smoke mode), Regression (all, Pre-release mode, freezable), Impact check (E and Va lenses, A/B compare), Pre-release gate (all, feeds Conclude), Reliability (high-risk, repeated, pass^k), Post-release monitoring (Va lens, Shadow or canary). Fields: type, name, when it runs, Conduct mode, purpose, categories, intents, TEVV lens, Gate-metric-only, high-risk-only, explicit test types, frozen list.
Mapping dropdowns (checkbox menus): Suites dropdown on every selected test, per category ("Runs in"), and per intent ("Runs in"). Rule: a test follows each suite's rules until changed. Ticking adds it (add), unticking leaves it out (ex). A category toggle applies to all its selected tests. An intent toggle edits the suite's intent list; a suite needs at least one intent. Counts and matrices update live.
Confirm: "Mark Catalogue complete" needs at least one selected test, and metrics with thresholds for Gate tests.
5.5 Configure (13 sections)
Each section shows n of m required and a ready status. The page is ready only when every section is ready and calibration has passed.
Agent connection: protocol (REST, WebSocket, SSE, gRPC, A2A, MCP, Browser UI, Voice, SDK), endpoint URL, authentication and secret (stored masked, never shown again), headers, request template with {{session}} and {{text}}, reply path (JSONPath), session handling, streaming, timeout, rate limit, retries, concurrency, test identities and personas, tenants, model and version, prompt version, build ID, knowledge index version, tool versions, "alert me if a pinned version changes" (on by default), network allowlist.
Environments: environment under test, baseline URL, isolation, reset method, data in environment (synthetic, masked, real approved), clock control, timezone, region, test window, mirror production traffic.
Systems and MCP: per downstream system: mode, access, sandbox, auth, checks. Plus channels, SSO, webhooks, traces, logs, tool-call proxy, correlation ID.
Data: required datasets as slots (for example Development set, Ground truth, Adversarial corpus, Load profile), each approved by a named person. Held-out data separation, personal data in datasets, synthetic data rules, coverage target.
Conversation and tone: tones, brand voice, reading level, languages, forbidden phrases and topics, required disclosures, sensitive situations, refusal style, escalation wording, required flow steps, clarification policy, memory, turn limit, success definition, user simulator personas and stop rule.
Evaluation framework: frameworks and pinned versions, judge mode (deterministic only or LLM judge), judge model, temperature, rubric, panel size, repeats, order randomisation, budget, personal data to judge; human review (pool, sample %, double-label %, guidelines); repetitions for pass^k, confidence interval method, noise floor, what counts as a real regression, flaky policy, baseline.
Calibration: every instrument (LLM judge, deterministic checks and state oracle, framework adapters, user simulator, red-team tooling, load generator, human reviewers) must pass a calibration against reference cases before any run. Editing an instrument invalidates its calibration. "Every instrument is calibrated and current" is a pre-check item.
Guardrails and safety: spend cap, time cap, token cap, kill switch (must be armed), egress allowlist, block writes to production, dry run first, destructive tests need approval, mask personal data in logs, attack classes and tooling, rules of engagement signed (needed for attack tests), rule sources, glossary, prohibited advice, audit log.
Human oversight (HITL and HOTL): one card per intent from Context: Autonomous, HITL or HOTL. For HITL or HOTL: when it applies (chips: Always, Amount above a limit, Any write or irreversible action, Low model confidence, Sensitive or personal data, Policy exception, Vulnerable or distressed user, Unverified user, Change across more than one system, and more), details, who approves or monitors, respond-within minutes, what happens if nobody responds (never "proceed"), how they watch (HOTL). A P1 intent set to Autonomous needs an explicit confirmation. Adds oversight test types (C18) to the Catalogue.
Intents, suites and release impact: every Context intent in one table: priority, oversight (read-only), Runs in suites (same dropdown as Catalogue, same data), Release impact (Blocks release or Advisory; P1 defaults to Blocks), Target pass rate (P1 95, P2 90, P3 80), Owner. Conduct uses the suite mapping. Conclude uses release impact and target (below target: blocker if Blocks release, condition if Advisory). Recommended next fields (not built): test depth, allowed tools and must-never actions, traffic share, fault injection, user groups and wording styles, environment limit, budget, production alerts.
Execution: runner, region, workers, schedule, load tool, evidence store, retention, encryption, redaction, chain of custody, stages captured, defect tracker, alert channel, CI trigger, owner, approver, signer.
Agent-type setup: fields appear only for the agent's type: retriever and chunking (RAG), tool registry, idempotency, sandbox accounts, step and token budgets, termination rules (loop), topology and checkpoints (graph), roster, hop tracing, protocol, shared memory, snapshot (multi-agent), learning switch, schedule, frozen eval stream (self-learning), ASR/TTS/telephony (voice), file types (files), approver accounts (HITL).
Checks and readiness: run connection checks (authentication, round trip, version, state oracle, trace). Each shows pass or fail after a short simulated delay. Auto-fix hints open the relevant field.
TV.cfg.gate(agent): ok only when all sections are ready; cal is true only when calibration passed; why names the first blocker.
5.6 Conduct
Tabs: Plan and run; Run console; Results; Run rules and gate; History.
Plan and run
What to run (five scope modes): All selected tests; Recommended only; By category; By test type; By intent. Each shows live counts "n of N test types in scope · recommended · added by you · high risk". "Or start from a suite" is a dropdown with a "Use suite" button (loads the suite's scope and mode). Saved scopes can be named, saved and loaded. A "Narrow further" panel filters by TEVV lens, Gate only, High risk only, previously failed.
How to run it (six modes): Smoke (8 highest-risk tests, one repeat), Intent pack (exactly one intent), Pre-release gate (full regression, evaluated against the gate), A/B compare (against a baseline), Reliability run (repeated, at least 5 repeats, pass^k), Shadow or canary (mirrored traffic, no user impact). Only a full-scope Pre-release gate or Reliability run can support a release decision. Other runs say so and do not feed Conclude.
Plan and start: key numbers (test types, cases, instances, estimate minutes, estimate cost, human-scored items), notes (blocked tests, tests with no applicable intent, oversight under test), the list of test types, Start run with pre-check. A warning appears when the estimate exceeds the caps, with Raise caps to the estimate (+10%).
A "Prototype only: simulate a problem" select (none, canary fails, monitor limit breached, missing evidence) exists to demo hard stops. Remove or hide in production.
Plan maths (prototype formulas; WIRE: replace with the real planner)
Applicable pair: test is applicable to an intent unless the test needs an action and the intent kind is conversation or answer.
cases per test = sum over applicable intents of (P1 = 3, P2 = 2, P3 = 1).
instances = cases × V (wording variants: number of tones, min 1, max 3) × R (repeats: Smoke 1, Shadow 1, Reliability max(reps, 5), else reps or 1).
minutes = round(instances × 4 / 60 / workers), min 1. cost = instances × 0.012 + instances × 0.6 × 0.01 (USD).
Pre-check (12 items, all must pass)Configure complete; every instrument calibrated and current; connection checks passed; model and prompt versions pinned (change alerts on); required datasets approved; no writes can reach production; spend and time caps set, kill switch armed, estimate within caps; everyone the run rules rely on is named (owner, approver, signer, roles); scope has runnable tests; rules of engagement signed (if attack tests); run rules confirmed; release gate policy confirmed. Any failure is a hard stop with a "go fix it" button. Nothing is waived. Fix the cause and start a new run.
Run lifecycle (six phases, each a hard-stop checkpoint)
Pre-check. 2. Preparation (seed, reset, snapshot). 3. Canary (a few cases first). 4. Processing (live monitoring). 5. Post-check (evidence and results). 6. Review (human decisions, then seal).
Run statuses: running, awaiting (human approval open), paused, stopped (hard stop), aborted, killed, complete.
Scope, rules and thresholds are frozen once a run starts.
Each run has: id R<n>, date, started-by, mode, scope text, full flag, tests, plan, checkpoints, HITL items, log, monitor readings, simulated results, findings.
Run rules (HITL and HOTL) and gate: tab "Run rules and gate"
Rule	Type	Default role	Answer within
H1 Start in staging, a production mirror or production	HITL	Approver	30 min
H2 Approve destructive or write tests	HITL	Approver	30 min
H3 Confirm rules of engagement for attack tests (always on)	HITL	Signer	60 min
H4 Consent to send personal data to the judge (always on)	HITL	Approver	30 min
H5 Resume after an automatic pause or kill switch (always on)	HITL	Approver	15 min
H6 Review low-confidence and disagreement items	HITL	Reviewer	240 min
H7 Confirm findings before sealing (always on)	HITL	Approver	240 min
O1 Live monitor with pause, abort and kill (always on)	HOTL	Owner	n/a
O2 Automatic pause when a monitor limit is breached (always on)	HOTL	Owner	n/a
O3 Sample review of judge-scored passes (default 10%)	HOTL	Reviewer	n/a
Pause limits (O2 monitor): error and timeout rate above 5%, spend above 80% of cap, trace completeness below 99%, flaky rate above 15%, unexpected writes above 0, sample review share 10%. A breach pauses the run and opens H5.
Rules say "No answer in time means the run stays paused. It is never auto-approved. The person who started a run cannot decide on its approvals. An unlisted approval request from the agent itself is a finding, not an approval."
A named person must confirm run rules and confirm the gate policy before any run can start. Roles come from Configure (owner, approver, signer).
In the prototype these rules live in Conduct. Planned: move them into Configure as their own section after the prototype is complete.
Release gate policy (defaults, all editable, confirmed by a named person)
Class	Findings allowed	Minimum pass rate
Critical	0	100%
Major	0	95%
Minor	3	80%
Overall pass^1	 	90%
Test coverage	 	100%
Gate-role metrics from the Catalogue must also all be met. The gate is evaluated only on a full-scope run in Pre-release gate or Reliability mode.
Run console
Phase strip with the current phase, checkpoint list (pass or fail), live KPIs (tests done, Passed, Failed, Blocked, no verdict, Critical so far, flaky), the activity log, monitor gauges, pending approvals (Approve and Reject buttons with name and note), and Pause, Abort, Kill buttons.
Now running card: the current test and a live evidence view of one simulated turn in three columns: what the agent said, what it did (tool calls with status: ok, not attempted, failed, forbidden), what changed (backend data before and after), plus the five-stage ribbon (Intent, Plan, Tools, Backend, Reply) marking the first stage that deviated and a "why" line.
Lanes: one lane per user group (persona) showing its progress and failures.
Results (after processing)Tabs: Overview (gate verdict, outcomes, top risks), By TEVV, By category, By test type, By intent (intent × test-type heat map), Findings, Coverage, Reliability, Governance, Compare.
Findings table: id, severity (critical, major, minor), stage (S2 to S7) and code (e.g. ACTION.DUPLICATE_ACTION_AFTER_RETRY), summary, expected versus observed, test, Owner dropdown, Show evidence toggle (said / did / changed), Accept (see Conclude).
Coverage definition: a test is "covered" only if run to completion with an independent oracle. Blocked, not run and inconclusive are never counted as passed. Every item not run has a stated reason.
Reliability: pass^k, 95% Wilson interval, flaky list (a case that passes and fails between repeats).
Seal: after review, the run is sealed (immutable) and becomes available to Conclude. Only a sealed full-scope release run unlocks "Mark Conduct complete and continue".
Simulation (WIRE: replace simulate(run))Per test: pass probability base − 0.10 if high risk − 0.05 if C8, where base = 0.80 + (hash(agentId) mod 12) / 100; failure draws from a seeded random on agentId|runNumber|testId. Output per test: {c cases, i instances, p passed, f failed, n inconclusive, b blocked, nr not run, fl flaky, mm metric readings[{id,n,role,dir,thr,v,ok}], ix}. Finding severity: critical if high risk and (C8, C5, C12 or coin flip); major if a failed Gate metric; else minor.
5.7 Conclude (route conclude, step key cert)
Fifteen tabs: Go / No-Go (first), Executive dashboard, Why agents fail, Standards (draft), Traceability, TEVV, Categories, Test types, Intents, Findings, Coverage, Reliability, Governance, Compare, Evidence pack.
Recommendation (decisionOf)
No-go if any blocker:
not a full-scope Pre-release gate or Reliability run;
Gate metrics not met;
any unaccepted Critical finding;
Critical-class pass rate below its minimum;
open Major findings above the allowed count, or Major-class pass rate below its minimum;
any high-risk test type not fully covered;
any failed human-control (C18) test;
an intent set to "Blocks release" whose pass rate is below its target.
Conditional go if no blocker but any condition:
Minor allowance, Minor pass rate, overall pass^1 or coverage not met;
accepted Major findings (each needs an owner and a date);
lower-risk test types not fully covered;
flaky test types;
fewer than 3 repeats per case;
an Advisory intent below its target.
Go only if nothing is open.
Blockers cannot be waived. They need a fix and a new run, or formal risk acceptance where allowed.
Acceptance rules: a Major finding needs 1 approver. A Critical finding needs 2 approvers, never the person who ran the run. An approver cannot be the runner.
Decision form: choose a named decider, write a note, then record Go, Conditional go or No-go. Request sign-off records who was asked, by whom and when; the form then restricts the decider to the requested person. Recording a decision sets the run's verdict on the agent (signoff()), and enables "Mark Conclude complete and continue". Always show: "This statement covers this run only and is not a certification."
Executive dashboard: verdict banner, KPIs (tries passed with 95% interval, pass / flaky / fail cases, intents tested "17 of 19, declared not tested", comparison with the previous run better or worse), top risks, what is blind.
Why agents fail (visibility): eleven failure modes, each mapped to categories and marked Blind spot (not in plan), Planned not run, Failing, Some failures, Partly covered, or Clear: errors compound across steps; wrong or unauthorised actions; confident but ungrounded answers; works once, fails on repeat; injection, leakage and misuse; human control that does not work; reported success that never happened; drift and silent regressions; runaway cost, latency and loops; compliance, bias and audit gaps; test and production differ.
Standards (draft for review): maps categories to NIST AI RMF (MEASURE 2.5, 2.6, 2.7, 2.10, 2.11, 2.13, and MEASURE 3 / MANAGE 4), NIST AI 600-1 risks (for example Confabulation), and ISO/IEC 42001 (A.6.2, A.9, 9.1). Shows evidence coverage per clause. Flag "Draft for review: coverage, not certification". EU AI Act is later.
Traceability: requirement to test to evidence: intents, must-never boundaries, worst failure, regulations mapped to categories.
Exports (Evidence pack tab): HTML execution report, JUnit (XML for CI; saved as .txt inside the artifact viewer because it blocks .xml), JSON, CSV, Print. In the artifact viewer, downloads go through a confirmation prompt; the standalone file downloads directly.
5.8 Continuous
Schedule: Every release, Daily, Weekly, Monthly (default from the Context cadence answer; Daily if the model or prompt can change without notice or learning is automatic). Alert channel (Email the owner and others).
Re-test triggers (chips): Model version change, Prompt change, plus those that match the agent: Tool schema change, Downstream system release, Knowledge refresh, Agent roster or protocol change, Learned update applied (locked on for self-learning), New regulation or policy, Filed rate/form/rule change, Drift alert.
What to watch: pass^1 on P1 intents, and per agent: grounding and refusal rate, tool error and retry rate, loop length and cost, hand-off loss rate, behaviour drift after learning, outcome gaps by group, response time, cost per task.
Drift: slider "Alert when pass^1 drops by N points" (default 5) with a trend sparkline and a dashed threshold line.
Notes appear when floating versions or automatic learning are present, or when no production access means scheduled re-tests only.
"Mark Continuous complete and finish" needs a Conclude sign-off.
Parked for later: CI templates, portfolio views, drift per intent.
6. Rules that must hold everywhere
Hard stops are hard. No code path may continue a run past an open checkpoint or approval.
Frozen runs. Scope, run rules and gate policy are copied into the run when it starts and cannot change for that run.
Filters combine. Scope filters (category, test type, intent, lens, Gate, risk, failed) intersect.
Only the 142-test catalogue. No other test list.
Blocked is not passed. Show Blocked separately and never in a pass rate numerator.
Suites are rules plus overrides. match(test, suite) = in add ? true : in ex ? false : rule(test). Freezing stores the resolved list. Any mapping edit unfreezes the suite.
Intent ids are the intent id, or its name when no id exists (a shared helper must be used everywhere so suite filters match Conduct).
Secrets are never displayed or stored in clear text; only a mask (last two characters).
Everything simulated is labelled so a viewer cannot mistake it for a real result.
7. Test suites in Conduct (how mappings flow)
Catalogue mappings → TV.suite.list(agent) → each suite resolves to {cat, ty, in, tv, gate, risk} (with ty as the explicit resolved list when any add or ex exists, and ['__none'] if that list would be empty) → Conduct "Use suite" loads it as the scope and sets the mode → the plan maths and the run use it.
8. Context workbook and YAML import
A separate Excel workbook, TruVelocity_Context_Questionnaire.xlsx, lets stakeholders answer Context without the chat. Sheets: Start here, Stakeholders, Questionnaire (all 121 questions, option rows with Yes dropdowns, dropdowns for single choice, Essential/Full, Required, Applies when, Status), Intents, Systems, Check, YAML (generated), Lists, Build (hidden).
YAML format v1 (valid YAML and JSON-compatible per line; strings always double-quoted and escaped):
 
yaml
Plain Text
truvelocity_context: 1agent: "Quote Assistant v1.4"exported: "2026-11-17 14:23"answers:  name: {"text": "Quote Assistant v1.4"}  users: {"choose": ["Customers or members", "Internal staff"], "text": "note"}  type: {"choose": ["Conversational"], "text": ""}intents:  - {"name": "Check claim status", "kind": "Task", "priority": "P1", "oversight": "Autonomous", "outcome": "...", "approver": "", "respond_min": "", "systems": "", "example": "", "owner": ""}systems:  - {"name": "Claims system", "type": "System of record", "direction": "Read and write", "test_access": "", "environment": "", "sign_in": "", "owner": "", "notes": ""}contributors:  - {"name": "Priya Nair", "role": "", "team": "Business owner", "answered": "", "date": ""}
Import rules (parseCtxYaml, importCtx):
Accept a whole JSON document or the line format above. Skip blank lines and # comments. Report the line number of any unreadable value.
For each question id in answers: choose becomes chips (single keeps the first only), text becomes the note or free text. Labels not in the platform list are kept as written and counted.
intents replaces the intent list. Kind, priority and oversight are matched case-insensitively to the allowed values, else default to Task, P2, Autonomous. Extra fields (approver, respond_min, systems, example, owner) are kept on the intent for the record; the platform does not use them yet.
systems fills the "intent to system map" text if it is empty. agent fills the name if empty. contributors are recorded only.
Result message: "Imported N answers, M intents and K systems. Check the summary, then continue to the Catalogue." A blank workbook imports with zero answers and no error.
WIRE: accept an .xlsx upload directly by parsing the same sheets server-side.
9. Wiring guide: from prototype to product
The UI talks to modules through the functions in 3.1. Replace the simulated parts behind those functions. Suggested services:
Concern	Prototype	Replace with
Auth and roles	any email	SSO and RBAC (Owner, Approver, Reviewer, Signer, Runner)
Agents and Context	in-memory	POST /agents, PUT /agents/{id}/context (YAML or JSON), GET
Test library	embedded JSON	GET /library (versioned), custom tests with approval workflow
Plan	localStorage	PUT /agents/{id}/plan {selected[], custom[], metrics{}, suites{}}
Configure	localStorage	PUT /agents/{id}/config; secrets to a vault; `POST /agents/{id}/checks/{auth
Run planner	planOf()	POST /agents/{id}/runs/plan returns cases, instances, minutes, cost
Run engine	simulate() + timer	POST /agents/{id}/runs; stream GET /runs/{id}/events (SSE); `POST /runs/{id}/pause
Approvals	in-memory queue	POST /runs/{id}/approvals/{hid} {decision, note}; enforce "decider is not the runner" server-side; SLA timers keep the run paused
Results	simulated	GET /runs/{id}/results, /findings, /evidence/{testId}/{turn}
Evidence	generated per test	store the real transcript, the tool-call proxy log, and backend before and after snapshots; keep chain of custody and hashes
Conclude	decisionOf()	server evaluates the same rules and returns blockers, conditions and recommendation; POST /runs/{id}/decision, /signoff-requests, /findings/{fid}/accept (1 or 2 approvers)
Exports	client side	`GET /runs/{id}/export/{html
Continuous	local state	scheduler, CI webhooks, drift monitor; alerts to email and chat
Audit	none	append-only audit log for every change, approval and decision
Entities (JSON shapes):
 
Plain Text
Run      {id, agentId, n, mode, scope, full, status, phase, startedBy, startedAt, plan{cases,instances,minutes,cost},
          pre[{k,n,ok,why}], checkpoints{}, hitl[{id,rid,state,role,sla,by,at,note}], rulesSig, gateSig, sealedAt}
Result   {testId, cases, instances, passed, failed, inconclusive, blocked, notRun, flaky, metrics[{id,role,dir,thr,v,ok}]}
Finding  {id, runId, testId, sev:'critical|major|minor', stage:'S2..S7', code, summary, expected, observed, owner, accepted[{by,at}]}
Decision {runId, decision:'go|cond|nogo', by, note, at, requestedFrom, blockers[], conditions[]}
Integration points the harness must provide: an agent connector (REST, WebSocket, SSE, gRPC, A2A, MCP, UI, voice), a tool-call proxy or trace feed, backend state readers, an evaluation runtime (deterministic checks, LLM judge, framework adapters, user simulator, red-team tooling, load generator), and human review queues.
Security: never log secrets or personal data (mask in logs and reports), block writes to production unless explicitly approved, enforce spend, time and token caps and the kill switch in the runner, and keep scope and gate frozen per run.
10. Acceptance checklist
All six steps open for all seven demo agents with no console errors and no text such as "undefined", "NaN" or "[object".
New agent: Context (sample answers) → Catalogue → Configure → Conduct → Conclude → Continuous works end to end.
Context workbook YAML round-trips: fill the workbook, copy the YAML, import, see the answers, intents and systems in the summary.
Configure cannot be completed until calibration passes. Conduct cannot start until all 12 pre-check items pass, and a full-scope Pre-release gate or Reliability run is needed to unlock Conclude.
Hard stops: a failed canary, a breached monitor and missing evidence each stop the run with the reason and a fix button.
Four-eyes: the runner cannot approve their own run; a Critical finding needs two approvers.
Suite mapping: ticking a test, category or intent in a suite changes the Conduct scope for that suite.
Release impact: a "Blocks release" intent below its target produces a No-go blocker in Conclude.
All four exports download. HTML report opens standalone.
The file works offline (fonts fall back) and at phone width.
11. Known gaps and backlog (not in the prototype)
Configure additions from the mockup review: wording styles, fault-injection rates, time limit and roles per intent; more per-intent fields (see 5.5 section 10).
Move the HITL and HOTL run rules from Conduct into Configure.
Context: import a deck or API spec.
Continuous: CI templates, portfolio and per-intent drift.
Catalogue start options: draft a plan from Context, import an existing catalogue.
TEVV Strategy and Plan document generation (button is a stub).
EU AI Act mapping in Conclude.
Large new-agent runs take minutes because they use all 105 selected test types. A real engine should stream progress.
The Context answers and agent list are in memory in the prototype.
12. Prompt to give an LLM
You are a senior front-end engineer. Build TruVelocity for AI TEVV exactly as specified above as a single, self-contained HTML file with no dependencies other than Google Fonts. Follow sections 2 to 8 in full. Use the 142-test library structure from 4.2 (generate realistic entries that satisfy the schema; do not invent a different catalogue structure). Implement every screen, rule, default, label and checkpoint listed. Keep prototype simulation deterministic and clearly labelled. Mark every point where a backend replaces simulation with a WIRE: comment. After building, run the acceptance checklist in section 10 and fix every failure before returning the file.