plivo-mirror v4 — a real-time firewall for LLM voice agents

"Prompt and pray" breaks under real callers — a prompt is a suggestion, not a constraint. v4 stops bad output before it reaches the caller or fires a tool: prevention, not post-call detection.

① The Problems
② The Architecture
③ Eval Results

Voice agents fail in 6 ways in production. Each card: the problem, a real example, and the v4 component that stops it.

A dual-boundary firewall: it guards what the agent is about to say and about to do, grounded in a session state the model can't touch.

Press play to watch a turn flow through the firewall.
📞 Caller
speaks
→
STT
speech → text
→
State grounding
validate facts → state
→
Agent LLM
drafts reply + tools
🗄 SESSION STATE — single source of truth
validated facts kept OUTSIDE the model · feeds grounding, the verifier & the action guard · tools read from it (zero-argument)
DUAL-BOUNDARY REVIEW  ·  the first guard to object wins

① Speech guard  — what it's about to SAY

Deterministic
FORBID / REQUIRE → hard block
Risk-span tagger
price · number · commitment (≈0 ms)
NLI semantic
contradicts the customer OR a known fact?
Grounded verifier
supported by facts / policy?

② Action guard  — what it's about to DO

False-completion
"done" with no tool call
Arg ↔ state
tool args must match confirmed state
Authorization
a SEPARATE service decides what's allowed
Validators · zero-arg
rules in code · tools read from state
Verdict
pass · correct · block
→
Deflection filler
spoken FIRST · no LLM · covers latency
→
Intervention · same LLM
regenerate from CORRECT facts → re-verify
→
TTS
ElevenLabs
→
📞 Caller hears
SAFE output
How the fix is generated (same single LLM): on a violation, v4 doesn't just refuse — the same LLM regenerates a correct reply, re-prompted with only the correct facts (the "correction packet"). It never restates the wrong value, which sidesteps the "pink-elephant" problem — telling a model "don't say X" actually raises the odds it says X. The new reply is then re-verified before it's voiced.

Timing: clean turns release immediately (~0 ms guard compute). The expensive grounded verifier runs only on flagged spans; a deflection filler is spoken first to cover its latency. Tool calls are always checked before they fire.

Measured on a hard, balanced benchmark — 65 induced violations + 64 clean near-misses + 21 good-calls, live on gpt-5.4-mini. Every lever raised both precision and recall.

68% ← 35%
Catch rate (recall on hard violations)
0.73 ← 0.47
F1 (overall quality)
0.80 ← 0.70
Precision (few false alarms)

Before (lexicon-only gate) → after (v4 full stack)

Catch rate35% → 68%
F1 ×10047 → 73
Precision ×10070 → 80

What the benchmark covers (65 violations)

Failure familyCases
Semantic / intent (negation, correction, conditional…)32
Fabrication / grounding (price, hours, policy…)16
Action-side (wrong tool, unconfirmed, false-complete)7
Math / numeric (totals, quantity)6
Commitment / authority3

Plus 64 "handled-correctly" twins + 21 adversarial good-calls — the precision test (don't fire on correct replies).

Latency — added guard compute

PathAdded latency
Clean fast-path (no NLI)~0.1 ms
Clean + NLI tier (large model)~0.9 s
Clean + NLI tier (small model)~47 ms
Flagged — first audio (filler)~1.3 s
Flagged — corrected answer~4.3 s

Speculative mode moves the NLI cost off the first-audio path (voice now, check after) → ~0 ms perceived.

Honest residual gaps (what's next)

Conditionals — hard for NLI Action-side — needs catalog-aware extraction Dynamic facts (stock/ETA) — need live tool results Confidence signal (Semantic-Entropy Probes) — interface ready, inert today