Yes, for AI and agentic device software
TruAssure can be extended to V&V the AI and agentic software functions inside a medical device, or a device that is itself AI software (SaMD). That means the clinical assistants, triage and documentation agents, and decision support built on LLMs. Its core design already matches what regulators ask for: evidence from independent sources rather than the agent's own account, repeatable runs, statistical reliability, and a release gate.
It is not a general-purpose replacement for traditional device V&V tooling (embedded firmware, hardware-in-the-loop, electrical safety, usability labs). For those, it would sit beside the existing tools and feed them evidence.
Three things have to be built before a manufacturer could use it in a regulated submission: requirement and hazard traceability, tamper-evident records with attributable sign-off, and a validation package for TruAssure itself as a QMS tool. We estimate about 6 months to a first regulated-ready release (2.0-MD), with a usable pilot on a non-regulated or pre-submission project at about 10 weeks.
What carries over from TruAssure 1.6.2
Most of the testing engine a regulated AI V&V tool needs already exists, because TruAssure was built to judge agents from evidence, not from their own claims.
| Capability today | Why it matters for device V&V |
|---|---|
| Five-stage verdict per checkpoint (goal, plan, action, end state, output), first failing stage recorded | Maps a test result to the software function that failed, which a 62304 anomaly record needs |
| Independent evidence kept apart: agent trace, action-boundary observer (MCP/REST proxy, OpenTelemetry), before/after state snapshots, outputs | Objective evidence that does not depend on the system under test reporting truthfully |
| Evidence seams SE-01 to SE-07 (silent write, phantom success, output vs action, identity propagation) | Catches the failure modes that matter most in clinical agents: claiming an order was placed when it was not, writing to a record without authority |
| Structured failure codes with severity rules | Lets defects be classified and routed by code, ready to link to hazards |
| Reliability (pass^k) and Wilson confidence ranges | Gives a statistically stated performance claim instead of a single pass, needed for non-deterministic models |
| Robustness: input perturbations (casual, formal, indirect, noisy phrasings) and injected faults | Evidence for foreseeable misuse and degraded conditions |
| Parity and run comparison (SE-06) | Shows a model, prompt or deployment change did not regress, the core of change control |
| Release gate with a written policy; JUnit for CI | An objective, recorded pass/fail decision at release |
| Execution report and JSON record built in code, never by a model; personal details masked | A reproducible test report; masking supports HIPAA/GDPR handling of test data |
| Grader calibration (Cohen's kappa, CE-02); "not evaluated" counts as failure | An LLM grader can be justified with measured agreement, and it never passes by default |
| Scenario packs: prompt injection, privacy, unsafe writes, escalation, abstention, jailbreak | Starting content for security (FDA cybersecurity, IEC 81001-5-1) and safety-boundary tests |
| 95% line-and-branch coverage gate, mypy strict, pinned CI actions | A disciplined development baseline that helps when TruAssure itself must be validated |
What medical device V&V asks of a test framework
Regulators do not certify test tools. They judge the manufacturer's evidence, so TruAssure must produce evidence that fits these frameworks and must itself be validated for its intended use.
| Standard or rule | What it asks of V&V evidence |
|---|---|
| IEC 62304 (software lifecycle) | Software safety class A, B or C sets the rigour; traceability from requirements to verification; system-level testing; verified risk controls; anomaly records; regression after change |
| ISO 14971 (risk management) | Every hazard's risk control is verified for implementation and effectiveness; residual risk is recorded |
| ISO 13485 and FDA QMSR (21 CFR 820, which brings ISO 13485 into US law from 2 February 2026) | Design verification and validation records; validation of computer software used in the quality system, proportionate to its risk |
| FDA Computer Software Assurance guidance | Risk-based assurance of the tool itself: intended use, risk to patient and quality, then scripted or unscripted testing |
| 21 CFR Part 11 | Electronic records that are attributable, with secure audit trails and signatures tied to a person and a meaning |
| FDA AI-enabled device software functions (draft, January 2025) and PCCP guidance (final, December 2024) | Performance on representative data with subgroup analysis, uncertainty stated, monitoring, and a predetermined plan for model changes |
| EU MDR with the EU AI Act | AI in a medical device is high-risk AI: data governance, logging, accuracy and robustness, human oversight; obligations for products under Annex I apply from August 2027 |
| IEC 81001-5-1 and FDA premarket cybersecurity guidance | Security testing of health software, including prompt injection and data leakage for LLM features |
| IEC 62366-1 (usability) | Mostly outside TruAssure; it can supply use-error scenarios but not formative or summative usability studies |
Gaps to close
The testing engine is strong. The gaps are in traceability, record integrity and the tool's own validation. Today a scenario has an id, test types and a severity, but no requirement or hazard id. Runs are stored as plain files with no signatures. The only launch record is launched_by: "ta console", not a named person.
| Need | Today | Extension | Size |
|---|---|---|---|
| Requirement and hazard traceability | Scenarios tagged by test type only | requirements: and hazards: fields in the catalog schema; import from a requirements or ALM tool (CSV, ReqIF, Jama, Polarion, Azure DevOps); a trace matrix in the report; a gate rule: every safety-class C requirement has a passing test | M |
| Risk-based test selection | Severity per failure code | Severity linked to ISO 14971 hazard severity; test depth set by software safety class; residual-risk summary per hazard | M |
| Tamper-evident records (Part 11) | Plain files, manifest written last | Hash chain over every evidence file, a signed run manifest, and an append-only audit log of every action | M |
| Attributable sign-off | No user identity | Named reviewer and approver with signature meaning (reviewed, approved), using the client's SSO; the gate verdict needs a signed approval for a release | M |
| Statistical acceptance criteria | pass^k, Wilson ranges | Pre-declared acceptance criteria per claim (for example sensitivity lower bound 95% CI ≥ 0.90); sample-size helper; subgroup results by demographic and site | M |
| Clinical reference data | Catalogs written by the team | Governed test datasets: version, source, consent basis, train/test separation check, sequestration of the final test set | M |
| Change control for models (PCCP) | Compare and parity runs | A change-impact report: model, prompt or retrieval change → affected requirements → mandatory re-runs → pass against the PCCP's limits | S |
| Clinical scenario packs | Generic packs (privacy, injection, escalation) | Packs for clinical agents: dosing and units, contraindications, escalation to a clinician, abstention outside the intended use, hallucinated citations, PHI leakage | M |
| Grader qualification | Cohen's kappa on consistency verdicts | Calibration of rubric verdicts too, against clinician-labelled sets; a documented rule for when a human must adjudicate | S |
| Tool validation package | 95% coverage, strict typing, CI | Intended-use statement, tool risk assessment (CSA), requirements spec, IQ/OQ scripts, a validation report, and a change procedure for every TruAssure release | L |
| Submission-ready outputs | Engineering and execution reports | Report templates matching the 62304 software verification report and the FDA AI-enabled device performance section | S |
S = 1 to 2 weeks, M = 3 to 5 weeks, L = 6 weeks or more, for one engineer.
Extend, not fork
The testing engine stays as it is, and a medical device layer is added on top, along with a vault for records and a sign-off step. All of it ships as a validated, long-term-support line (2.0-MD).
The client's requirements, hazards and data come in at the top. Every result is traced back to them, sealed in the vault and signed before it reaches a report.
Five principles shape the build:
- Trace everything. Every scenario names the requirement and hazard it verifies, and the gate refuses a release that leaves a safety-class C requirement untested.
- Deterministic first. Safety-relevant checks use state, action and content checks, which are already TruAssure's strength. An LLM grader is used only where its measured agreement with clinicians meets a pre-set threshold.
- A human signs. Agents and graders can propose verdicts, but a named person reviews and approves them. This also answers the EU AI Act's human-oversight expectation.
- Claims are statistical. Acceptance criteria are declared before the run, as confidence bounds, with subgroup results.
- Validate the tool once, per release. We ship the validation package. Each client runs a short IQ/OQ in its own environment and records its intended use.
Three phases, 26 weeks, a go/no-go at week 10
The plan has three phases over 26 weeks, about 52 engineer-weeks in total. A go/no-go at week 10 caps the commitment before the regulated work starts.
- Phase 1 · Pilot foundation (weeks 1 to 10, 20 engineer-weeks). Requirement and hazard fields in the catalog, CSV import, the trace matrix in the report, pre-declared acceptance criteria with subgroup results, and a first clinical scenario pack. Run it on the design partner's AI agent in a non-regulated setting. Exit: a trace matrix and performance report the partner's QA team accepts as useful.
- Phase 2 · Regulated records (weeks 11 to 18, 16 engineer-weeks). The evidence vault (hash chain, signed manifest, audit log), SSO identity and e-signatures on review and gate approval, risk-based test depth by safety class, grader qualification against clinician labels, and the model change-impact report. Exit: QA's Part 11 gap assessment shows no open critical findings.
- Phase 3 · Validation and release (weeks 19 to 26, 16 engineer-weeks). The tool validation package (CSA), IQ/OQ scripts, report templates for 62304 verification and AI performance, Jama and Polarion connectors, and the 2.0-MD long-term-support release. Exit: a signed validation report for 2.0-MD.
People: 2 engineers throughout; an RA/QA lead at about half time; a clinical SME at about a quarter time. Code is not the slow part here. Regulated documentation and the partner's review cycles are, so phases 2 and 3 carry the most schedule risk.
Risks, assumptions and what we should not claim
| Risk | Effect | Mitigation |
|---|---|---|
| An LLM grader judges a safety-relevant output wrongly | False pass on a clinical claim | Safety-class C checks use deterministic checks or clinician adjudication; LLM grading only where its measured agreement meets a pre-set threshold |
| No domain expertise in the team | Scenario packs miss real clinical hazards | Bring in a clinical SME and an RA/QA lead from week 1; validate packs with a design partner |
| Clinical data access and PHI | Pilot stalls on data agreements | Start on synthetic or public de-identified data; run TruAssure inside the client's boundary (it already runs fully offline) |
| Scope creep into firmware, hardware and usability testing | Timeline doubles | State the scope as AI and software functions; integrate with, not replace, existing tools |
| Regulations moving (FDA AI draft guidance, EU AI Act dates) | Rework of report templates | Keep templates as data; review with RA at each phase gate |
| Validation drag on every TruAssure release | Slower release cadence for the medical line | A separate long-term-support line (2.0-MD) with a defined change procedure; the main line keeps moving |
Assumptions: two engineers plus part-time QA/RA and a clinical SME; one design partner with a real AI-enabled device project; Innominds owns the medical line (see the ownership move to the corporate repository).
Decisions needed from the review team
- Confirm the scope: AI and agentic software functions in devices and SaMD, not general device V&V
- Name a design partner: a client or internal programme with an AI-enabled device in development
- Staff the team: 2 engineers from week 1, part-time RA/QA lead and clinical SME
- Approve Phase 1 (10 weeks) now, with a go/no-go on Phases 2 and 3 at the pilot readout
- Decide where the medical line lives: the corporate Azure repository, as a separate long-term-support branch
docs/medical-device/TruAssure-Medical-Device-VV-Assessment.docx in the TruAssure repository.Sources
TruAssure capabilities are taken from the 1.6.2 codebase (README, catalog schema, core/ modules) as of 10 November 2026. Regulatory references are from working knowledge and need RA confirmation: IEC 62304:2006+A1:2015, ISO 14971:2019, ISO 13485:2016, FDA QMSR (21 CFR 820), 21 CFR Part 11, FDA Computer Software Assurance guidance, FDA AI-enabled device software functions draft guidance, FDA PCCP guidance, EU MDR 2017/745, EU AI Act 2024/1689, IEC 81001-5-1, IEC 62366-1.