Validate TruVelocity™ for AI TEVV 2026.11.17.1 from pip install
A clean machine, a fresh virtual environment, and the package from PyPI: nothing from the repository, no Docker, no LLM and no API keys. About 15 minutes. Every expected output below was recorded from the 2026.11.17.1 wheel installed this way.
What this validates
That the published package installs cleanly and everything a new team needs works from the package alone: the console commands, the packaged schema, catalogue, demo, packs and guide, the offline demo end to end, and the features below.
| Feature | Where you check it |
|---|---|
Customer-facing execution report (execution-report.html and .json) | Step 5 |
Parallel live runs: --parallel N --ordered-passes, and tv run --mode | Step 6 |
The scenario library (8 packs), including rubric-graded tone-and-policy | Steps 7 and 8 |
| The console links the execution report | Step 10 |
1Prerequisites
- Python 3.11, 3.12 or 3.13 on the PATH (
python --version). - Access to pypi.org (or your company's PyPI mirror).
- An empty folder to work in, not inside a TruVelocity™ checkout (so nothing from a repository can help).
2Install from PyPI into a clean environment
mkdir truvelocity-tevv-check
cd truvelocity-tevv-check
python -m venv .venv
.venv\Scripts\activate
python -m pip install --upgrade pip
pip install "truvelocity-tevv==2026.11.17.1"
tv --version
mkdir truvelocity-tevv-check
cd truvelocity-tevv-check
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install "truvelocity-tevv==2026.11.17.1"
tv --version
TruVelocity for AI TEVV 2026.11.17.1
pip install path\to\truvelocity-tevv-2026.11.17.1-py3-none-any.whl
instead; every step below is the same. Both tv and truvelocity-tevv are installed as commands.3The offline tour
One command runs every feature against a scripted agent shipped in the package: validation, runs with repeats and phrasings, a build with a planted defect, comparison, defects and JUnit, the 71-type scorecard, a capability pack and the release gate, grader calibration, and both HTML reports.
tv demo
TruVelocity 2026.11.17.1: offline demo into tv_demo (no agent, no Docker, no LLM) == 1. Check the catalog against the schema == 2. Run it twice against the scripted agent (3 repeats x every perturbation) == 3. A build with a planted defect: its first reply leaves out the balance == 4. What failed, at which stage, and why == 5. Compare the two good runs (no regressions expected) == 6. Defects and JUnit for CI == 7. The 71-type scorecard == 8. Release decisions: a capability pack and the gate (both say no) == 9. How far to trust a grader: calibration against human labels (Cohen's kappa) == 10. HTML reports (one file each, work offline) == Next
OK tv_demo\demo-catalog.yaml (2 scenarios)
demo-a: 3 pass, 0 fail, 0 flaky of 3 cases
demo-b: 3 pass, 0 fail, 0 flaky of 3 cases
DM-01 vvvvx <- output
agreement: outcome 100%, intent 100%, tools 100%, data 100%, stages 100%
demo-fail: 1 defect(s): 0 critical, 0 major, 1 minor
pack billing in demo-fail: NOT READY (1/2 cases passed every try)
gate demo-fail (default policy): FAIL
consistency_rules on harbor-consistency-v1: kappa 0.75, agreement 0.875 over 24 of 24 items
tv_demo\demo-fail\report.html
tv_demo\demo-fail\execution-report.html
tv_demo\demo-fail\execution-report.jsontv_demo holds the runs demo-a,
demo-b and demo-fail, and the pack and the gate both say no (the planted defect is meant to fail).4Read a failure
tv show demo-fail --runs-dir tv_demo
demo-fail status done catalog 1ec8902a5aba
TK-07 v v v v v PASS 6/6 [0.61-1.00]
DM-01 v v v v x FAIL 0/6 [0.00-0.39]
output: the reply should mention '193.60'; it doesn't
DM-02 v v v v v PASS 6/6 [0.61-1.00]
2 pass, 1 fail, 0 flaky of 3 cases; first failures by stage: output 6Each case is judged at five stages (goal, plan, action, end state, output); x marks the first that
failed. The range is the Wilson 95% interval over the 6 tries.
5The execution report
The customer-facing report: built in code from the stored run, so the page and its JSON record always agree.
start tv_demo\demo-fail\execution-report.html
open tv_demo/demo-fail/execution-report.html
xdg-open tv_demo/demo-fail/execution-report.html
- The red banner says FAILED: "Not ready without review: only minor findings, but failures remain. 18 of 18 planned test instances executed; 12/18 conclusive instances passed; 17 of 71 catalogue test types covered."
- KPI cards: pass rate 66.7% (12/18), executed 18/18, findings 1 (minor).
- At a glance: an outcome donut (passed 12, failed 6) and colour bar charts (by suite, persona, variant).
- Findings: one card for DM-01 ("the output does not mention a required fact"), with expected and observed values.
- Test-type coverage lists all 71 types; uncovered ones say why (no scenario, or a run option needed).
- It works offline: disconnect from the network and reload; nothing is fetched. Switch your OS to dark mode and it follows.
python -c "import json; r=json.load(open('tv_demo/demo-fail/execution-report.json', encoding='utf-8')); print(r['schema_version'], r['summary']['overall_status'], r['summary']['pass_rate'])"1.1.0 failed {'numerator': 12, 'denominator': 18, 'value': 0.6667}6Parallel ordered passes
Several environments at once, each taking a whole pass of the catalog (one phrasing, one repeat) in catalog order, reset to the catalog's own fixture first: how live runs scale across several copies of an agent.
tv run tv_demo/demo-catalog.yaml --env truvelocity_tevv.testing.scripted:demo_environment --parallel 2 --ordered-passes --perturbations all --repeats 2 --mode reliability --run-id parallel-1 --runs-dir tv_demo
parallel-1: 8 episode(s), 2 environment(s), catalog 1ec8902a5aba
[scripted-1] TK-07 [base a0] PASS
TK-07 vvvvv
[scripted-2] TK-07 [other-doc a0] PASS
TK-07 vvvvv
[scripted-1] CHAT [base a0] PASS
DM-01 vvvvv
DM-02 vvvvv
[scripted-2] CHAT [casual a0] PASS
DM-01 vvvvv
DM-02 vvvvv
[scripted-1] TK-07 [base a1] PASS
TK-07 vvvvv
[scripted-2] TK-07 [other-doc a1] PASS
TK-07 vvvvv
[scripted-1] CHAT [base a1] PASS
DM-01 vvvvv
DM-02 vvvvv
[scripted-2] CHAT [casual a1] PASS
DM-01 vvvvv
DM-02 vvvvv
parallel-1: 3 pass, 0 fail, 0 flaky of 3 cases
report: tv_demo\parallel-1\report.htmlscripted-1, the other phrasing on scripted-2). Then:tv report parallel-1 --runs-dir tv_demo
tv_demo\parallel-1\report.html tv_demo\parallel-1\execution-report.html tv_demo\parallel-1\execution-report.json
7The scenario library
tv library
answers-and-boundaries Grounded answers, escalation and boundaries (conversational) conversation-handling Understanding and keeping track of the conversation (conversational) identity-handoffs Identity and permissions across handoffs (workflow) payments-approvals Payments and approvals (conversational) privacy Privacy and other customers' data (conversational) prompt-injection Prompt injection (conversational, task) tone-and-policy Conversation quality, tone and written policy (conversational) unsafe-writes Writes under failure and duplication (task) tv library <pack> shows one; tv library <pack> --add catalog.yaml includes it
answers-and-boundaries,
conversation-handling, tone-and-policy).tv library tone-and-policy
Shows what the pack tests, its parameters (with examples or defaults) and its scenarios. Its rubrics are judged by
the LLM grader when TV_GRADER_PROVIDER is set; without one they are reported as not evaluated, never
as a pass.
8Start a project and add a pack
tv new chat my-bot
tv library answers-and-boundaries --add my-bot/catalog.yaml
tv validate my-bot/catalog.yaml
created a chat project in my-bot: catalog.yaml environment.py README.md .env.example truvelocity-tevv-gate.yaml truvelocity-tevv.yaml .gitignore tv-console.yaml ... added pack answers-and-boundaries to my-bot\catalog.yaml: fill in its values, then: tv validate my-bot\catalog.yaml OK my-bot/catalog.yaml (11 scenarios)
tv new task and tv new workflow work the same way.9Scorecard, capability pack and release gate
tv scorecard demo-a --runs-dir tv_demo
tv pack billing --run demo-fail --runs-dir tv_demo
tv gate demo-fail --runs-dir tv_demo
demo-a: 17 measured, 12 not run, 0 not applicable, 42 planned of 71 test types IU-01 Intent Resolution Intent accuracy: 1.0 ... wrote tv_demo\demo-a\scorecard.json pack billing in demo-fail: NOT READY (1/2 cases passed every try) 1 of 2 cases did not pass every try: DM-01 0/6 gate demo-fail (default policy): FAIL pass rate: 67% of cases passed every try, at least 100% required
tv pack and tv gate exit with code 1 (that is how CI blocks a
release): echo %ERRORLEVEL% on Windows, echo $? on macOS/Linux.10The console and the guide
tv console --runs-dir tv_demo --open
tv docs --open
11Sign-off checklist
0 of 12 checked
Your ticks are kept in this browser only (they never leave it).
12Clean up
deactivate
cd ..
rmdir /s /q truvelocity-tevv-check
deactivate
cd ..
rm -rf truvelocity-tevv-check
13Troubleshooting
| Symptom | What to do |
|---|---|
'tv' is not recognized / command not found | Activate the environment (step 2), or run python -m truvelocity_tevv.core --version. |
No matching distribution found for truvelocity-tevv==2026.11.17.1 | The release reaches PyPI a few minutes after it is published; retry, or pip install --no-cache-dir "truvelocity-tevv==2026.11.17.1". Behind a company mirror, it may need to sync first. |
| An older version is reported | You are in another environment: pip show truvelocity-tevv shows which one; start again in a new folder. |
| The demo stops with a lock message | Another tv run is using the same runs folder; wait for it, or use --runs-dir with a new folder. |
| Optional features | pip install "truvelocity-tevv[mcp]==2026.11.17.1" adds the MCP server (tv mcp); [proxy] the recording proxy; [sql] SQL databases and stores. |