2026-03-28 14:32 UTC
Migration decision run
GPT-4o → Claude Haiku
Early-stop migration decision support — is Haiku good enough to replace GPT-4o for support and extraction workloads?
Corpus
120
prompts, 4 categories
Tested
24
20% of corpus
Batches
2 / 6
stopped early
Confidence
Medium
stratified sampling
Decision
■ STOP NOW
Cost summary
$11.80
Spent (incl. $0.72 judge)
$74.30
Estimated spend avoided
6.3×
Savings ratio
Quality by category Overall compatibility: 0.61
Category Criticality Tested Compatibility Status
customer_support HIGH 8
0.85
STABLE
structured_extraction HIGH 6
0.28
FAILING
classification MEDIUM 6
0.72
MIXED
summarization LOW 4
0.90
STABLE
Latency comparison ↓ 58% improvement (p50)
Baseline (GPT-4o)
p50820ms
p952,100ms
variance±340ms
Candidate (Claude Haiku)
p50340ms (-58%)
p95890ms (-57%)
variance±180ms (-47%)
Failure archetypes
1
Schema break
JSON invalid, missing fields, incompatible structure
4 occurrences
2
Coverage drop
Response incomplete vs baseline output
3 occurrences
3
Refusal increase
Candidate refused prompts the baseline handled
2 occurrences
4
Format break
Output does not match the expected format
1 occurrence
Failure examples Top 3 by severity
ex-003 · structured_extraction SCHEMA BREAK
Prompt
Parse this invoice text and return structured JSON: 'Invoice #INV-2024-0847 dated March 15, 2024. Bill to: TechStart Ltd, 42 Innovation Drive, London. Items: 3x Widget A ($50 each), 1x Widget B ($120). Subtotal: $270. Tax (20%): $54. Total: $324.'
Baseline output (GPT-4o) ✓
{ "invoice_number": "INV-2024-0847", "date": "2024-03-15", "bill_to": { "name": "TechStart Ltd", "address": "42 Innovation Drive, London" }, "items": [ {"name": "Widget A", "quantity": 3, "unit_price": 50, "total": 150}, {"name": "Widget B", "quantity": 1, "unit_price": 120, "total": 120} ], "subtotal": 270, "tax_rate": 0.20, "tax": 54, "total": 324 }
Candidate output (Claude Haiku) ✗
Here's the parsed invoice data: Invoice Number: INV-2024-0847 Date: March 15, 2024 Bill to: TechStart Ltd, 42 Innovation Drive, London Items: - Widget A × 3 = $150 - Widget B × 1 = $120 Subtotal: $270 Tax (20%): $54 Total: $324 Returned markdown instead of JSON. Schema break: no valid JSON object.
ex-006 · structured_extraction SCHEMA BREAK
Prompt
Parse meeting notes and extract action items with assignees...
Issue
Candidate returned flat list instead of nested {action_items: [], decisions: []} structure. Missing "decisions" key entirely. Downstream parser would fail.
cl-002 · classification REFUSAL
Prompt
Classify this support ticket by priority: 'Our entire production database is returning connection timeouts. All customer-facing services are down.'
Issue
Candidate added a disclaimer instead of returning the requested priority label. Baseline correctly returned "p1" with no hedging.
Thresholds used
Threshold Config Actual Status
High-criticality failure rate ≤ 0.20 0.625 ■ TRIGGERED
Schema break rate ≤ 0.25 0.667 ■ TRIGGERED
Overall risk score ≤ 0.08 0.390 ■ TRIGGERED
Latency regression (p50) ≤ 1.5× 0.41× ✓ PASS
Latency regression (p95) ≤ 2.0× 0.42× ✓ PASS