Validate TruAssure 1.6.1 from `pip install`

A clean machine, a fresh virtual environment, and the package from PyPI: nothing from the repository, no Docker, no LLM and no API keys. About 15 minutes. Every expected output below was recorded from the 1.6.1 wheel installed this way.

offline after installWindows · macOS · Linux Python 3.11 to 3.13new in 1.4.0 marked NEW

What this validates

That the published package installs cleanly and everything a new team needs works from the package alone: the console commands, the packaged schema, catalogue, demo, packs and guide, the offline demo end to end, and the features new in 1.4.0.

New in 1.4.0	Where you check it
Customer-facing execution report (`execution-report.html` and `.json`)	Step 5
Parallel live runs: `--parallel N --ordered-passes`, and `ta run --mode`	Step 6
Three new scenario packs (8 in all), rubric-graded `tone-and-policy`	Steps 7 and 8
The console links the execution report	Step 10

1Prerequisites

Python 3.11, 3.12 or 3.13 on the PATH (python --version).
Access to pypi.org (or your company's PyPI mirror).
An empty folder to work in, not inside a TruAssure checkout (so nothing from a repository can help).

2Install from PyPI into a clean environment

Windows (cmd) · 1/7

mkdir truassure-check

Windows (cmd) · 2/7

cd truassure-check

Windows (cmd) · 3/7

python -m venv .venv

Windows (cmd) · 4/7

.venv\Scripts\activate

Windows (cmd) · 5/7

python -m pip install --upgrade pip

Windows (cmd) · 6/7

pip install "truassure==1.6.1"

Windows (cmd) · 7/7

ta --version

macOS / Linux · 1/7

mkdir truassure-check

macOS / Linux · 2/7

cd truassure-check

macOS / Linux · 3/7

python3 -m venv .venv

macOS / Linux · 4/7

source .venv/bin/activate

macOS / Linux · 5/7

python -m pip install --upgrade pip

macOS / Linux · 6/7

pip install "truassure==1.6.1"

macOS / Linux · 7/7

ta --version

expected

TruAssure 1.6.1

Testing a wheel before it is published? Use pip install path\to\truassure-1.6.1-py3-none-any.whl instead; every step below is the same. Both ta and truassure are installed as commands.

3The offline tour

One command runs every feature against a scripted agent shipped in the package: validation, runs with repeats and phrasings, a build with a planted defect, comparison, defects and JUnit, the 71-type scorecard, a capability pack and the release gate, grader calibration, and both HTML reports.

any

ta demo

expected (section headings, in this order)

TruAssure 1.6.1: offline demo into ta_demo (no agent, no Docker, no LLM)
== 1. Check the catalog against the schema
== 2. Run it twice against the scripted agent (3 repeats x every perturbation)
== 3. A build with a planted defect: its first reply leaves out the balance
== 4. What failed, at which stage, and why
== 5. Compare the two good runs (no regressions expected)
== 6. Defects and JUnit for CI
== 7. The 71-type scorecard
== 8. Release decisions: a capability pack and the gate (both say no)
== 9. How far to trust a grader: calibration against human labels (Cohen's kappa)
== 10. HTML reports (one file each, work offline)
== Next

expected (key lines)

OK      ta_demo\demo-catalog.yaml  (2 scenarios)
demo-a: 3 pass, 0 fail, 0 flaky of 3 cases
demo-b: 3 pass, 0 fail, 0 flaky of 3 cases
    DM-01 vvvvx <- output
  agreement: outcome 100%, intent 100%, tools 100%, data 100%, stages 100%
demo-fail: 1 defect(s): 0 critical, 0 major, 1 minor
pack billing in demo-fail: NOT READY (1/2 cases passed every try)
gate demo-fail (default policy): FAIL
consistency_rules on harbor-consistency-v1: kappa 0.75, agreement 0.875 over 24 of 24 items
ta_demo\demo-fail\report.html
ta_demo\demo-fail\execution-report.html
ta_demo\demo-fail\execution-report.json

Verify: the command exits without an error, ta_demo holds the runs demo-a, demo-b and demo-fail, and the pack and the gate both say no (the planted defect is meant to fail).

4Read a failure

any

ta show demo-fail --runs-dir ta_demo

expected

demo-fail  status done  catalog 1ec8902a5aba
  TK-07          v v v v v  PASS  6/6  [0.61-1.00]
  DM-01          v v v v x  FAIL  0/6  [0.00-0.39]
      output: the reply should mention '193.60'; it doesn't
  DM-02          v v v v v  PASS  6/6  [0.61-1.00]
2 pass, 1 fail, 0 flaky of 3 cases; first failures by stage: output 6

Each case is judged at five stages (goal, plan, action, end state, output); x marks the first that failed. The range is the Wilson 95% interval over the 6 tries.

5The execution reportNEW

The customer-facing report: built in code from the stored run, so the page and its JSON record always agree.

Windows

start ta_demo\demo-fail\execution-report.html

macOS

open ta_demo/demo-fail/execution-report.html

Linux

xdg-open ta_demo/demo-fail/execution-report.html

Verify on the page:

The red banner says FAILED: "Not ready without review: only minor findings, but failures remain. 18 of 18 planned test instances executed; 12/18 conclusive instances passed; 17 of 71 catalogue test types covered."
KPI cards: pass rate 66.7% (12/18), executed 18/18, findings 1 (minor).
At a glance: an outcome donut (passed 12, failed 6) and colour bar charts (by suite, persona, variant).
Findings: one card for DM-01 ("the output does not mention a required fact"), with expected and observed values.
Test-type coverage lists all 71 types; uncovered ones say why (no scenario, or a run option needed).
It works offline: disconnect from the network and reload; nothing is fetched. Switch your OS to dark mode and it follows.

the record (any)

python -c "import json; r=json.load(open('ta_demo/demo-fail/execution-report.json', encoding='utf-8')); print(r['schema_version'], r['summary']['overall_status'], r['summary']['pass_rate'])"

expected

1.0.0 failed {'numerator': 12, 'denominator': 18, 'value': 0.6667}

6Parallel ordered passesNEW

Several environments at once, each taking a whole pass of the catalog (one phrasing, one repeat) in catalog order, reset to the catalog's own fixture first: how live runs scale across several copies of an agent.

any (one line)

ta run ta_demo/demo-catalog.yaml --env truassure.testing.scripted:demo_environment --parallel 2 --ordered-passes --perturbations all --repeats 2 --mode reliability --run-id parallel-1 --runs-dir ta_demo

expected

parallel-1: 8 episode(s), 2 environment(s), catalog 1ec8902a5aba
  [scripted-1] TK-07 [base a0] PASS
  [scripted-2] TK-07 [other-doc a0] PASS
  [scripted-1] CHAT [base a0] PASS
  [scripted-2] CHAT [casual a0] PASS
  [scripted-1] TK-07 [base a1] PASS
  [scripted-2] TK-07 [other-doc a1] PASS
  [scripted-1] CHAT [base a1] PASS
  [scripted-2] CHAT [casual a1] PASS
parallel-1: 3 pass, 0 fail, 0 flaky of 3 cases

Verify: two environments share the work, and each pass stays on one of them (the base pass on scripted-1, the other phrasing on scripted-2). Then:

any

ta report parallel-1 --runs-dir ta_demo

expected

ta_demo\parallel-1\report.html
ta_demo\parallel-1\execution-report.html
ta_demo\parallel-1\execution-report.json

Verify: the execution report's header shows Execution mode: Reliability run; its banner says PASSED with 12/12 conclusive instances; the scope lists the variants base, other-doc and casual and 2 repetitions.

7The scenario libraryNEW packs

any

ta library

expected

answers-and-boundaries Grounded answers, escalation and boundaries  (conversational)
conversation-handling Understanding and keeping track of the conversation  (conversational)
identity-handoffs    Identity and permissions across handoffs  (workflow)
payments-approvals   Payments and approvals  (conversational)
privacy              Privacy and other customers' data  (conversational)
prompt-injection     Prompt injection  (conversational, task)
tone-and-policy      Conversation quality, tone and written policy  (conversational)
unsafe-writes        Writes under failure and duplication  (task)
ta library <pack> shows one; ta library <pack> --add catalog.yaml includes it

Verify: 8 packs, including the three new ones (answers-and-boundaries, conversation-handling, tone-and-policy).

any

ta library tone-and-policy

Shows what the pack tests, its parameters (with examples or defaults) and its scenarios. Its rubrics are judged by the LLM grader when TA_GRADER_PROVIDER is set; without one they are reported as not evaluated, never as a pass.

8Start a project and add a pack

any · 1/3

ta new chat my-bot

any · 2/3

ta library answers-and-boundaries --add my-bot/catalog.yaml

any · 3/3

ta validate my-bot/catalog.yaml

expected

created a chat project in my-bot:
  catalog.yaml
  environment.py
  README.md
  .env.example
  truassure-gate.yaml
  truassure.yaml
  .gitignore
...
added pack answers-and-boundaries to my-bot\catalog.yaml: fill in its values, then: ta validate my-bot\catalog.yaml
OK      my-bot/catalog.yaml  (11 scenarios)

Verify: the starter project is written and still validates with the pack included (the pack's example values fill its parameters until you replace them with your agent's actions and data). ta new task and ta new workflow work the same way.

9Scorecard, capability pack and release gate

any · 1/3

ta scorecard demo-a --runs-dir ta_demo

any · 2/3

ta pack billing --run demo-fail --runs-dir ta_demo

any · 3/3

ta gate demo-fail --runs-dir ta_demo

expected

demo-a: 17 measured, 12 not run, 0 not applicable, 42 planned of 71 test types
  IU-01  Intent Resolution                  Intent accuracy: 1.0
  ...
wrote ta_demo\demo-a\scorecard.json
pack billing in demo-fail: NOT READY (1/2 cases passed every try)
gate demo-fail (default policy): FAIL
  pass rate: 67% of cases passed every try, at least 100% required

Verify: ta pack and ta gate exit with code 1 (that is how CI blocks a release): echo %ERRORLEVEL% on Windows, echo $? on macOS/Linux.

10The console and the guide

any

ta console --runs-dir ta_demo --open

Verify: the browser opens on a local, token-protected page listing the runs; each run has an execution report linkNEW next to the engineering report, defects, scorecard and gate. Stop it with Ctrl+C.

any

ta docs --open

Verify: the packaged guide opens (getting started, catalogs, commands, the scenario library with all 8 packs).

12Clean up

Windows · 1/3

deactivate

Windows · 2/3

cd ..

Windows · 3/3

rmdir /s /q truassure-check

macOS / Linux · 1/3

deactivate

macOS / Linux · 2/3

cd ..

macOS / Linux · 3/3

rm -rf truassure-check

13Troubleshooting

Symptom	What to do
`'ta' is not recognized` / `command not found`	Activate the environment (step 2), or run `python -m truassure.core --version`.
`No matching distribution found for truassure==1.6.1`	The release reaches PyPI a few minutes after it is published; retry, or `pip install --no-cache-dir "truassure==1.6.1"`. Behind a company mirror, it may need to sync first.
An older version is reported	You are in another environment: `pip show truassure` shows which one; start again in a new folder.
The demo stops with a lock message	Another `ta run` is using the same runs folder; wait for it, or use `--runs-dir` with a new folder.
Optional features	`pip install "truassure[mcp]==1.6.1"` adds the MCP server (`ta mcp`); `[proxy]` the recording proxy; `[sql]` SQL databases and stores.

TruAssure 1.6.1 (pip install truassure) · MIT licence · all demo data is fictitious (Harbor Mutual). This page is self-contained: no scripts, fonts or styles are fetched.

What this validates

1Prerequisites

2Install from PyPI into a clean environment

3The offline tour

4Read a failure

5The execution reportNEW

6Parallel ordered passesNEW

7The scenario libraryNEW packs

8Start a project and add a pack

9Scorecard, capability pack and release gate

10The console and the guide

11Sign-off checklist

12Clean up

13Troubleshooting