Live mode
Run
GPT-4o to Claude 3.5 Haiku migration
STOP
Candidate exceeded the high-criticality failure threshold (45% >= 20%). Most affected category: summarization (high-crit 100%; missing_required_content x1, overlong_output x1).
Confidence: 59%
Coverage
15 prompts across 1 batches
Total cost: $0.0311
Stopped early: yes
Latency
p50: 3160ms -> 5578ms
p95: 6971ms -> 14395ms
Decision Metrics
| Overall risk | 41.5% |
|---|---|
| Candidate failure rate | 46.7% |
| Candidate regression rate | 20.0% |
| High-criticality failure rate | 45.5% |
| Schema break rate | 0.0% |
| Judged prompts | 2/2 |
| Escalated prompts | 0 |
| Judge worse rate | 0.0% |
| Judge average confidence | 90% |
| Latency p50 ratio | 1.77x |
| Latency p95 ratio | 2.06x |
Thresholds
| High-criticality stop | 20% |
|---|---|
| Schema stop | 25% |
| Proceed risk | <= 8% |
| Latency p50 threshold | 1.50x |
| Latency p95 threshold | 2.00x |
| Tiered escalation threshold | 60% |
Failure Archetypes
- overlong_output: 3
- json_invalid: 2
- missing_required_content: 2
Category Scorecards
| Category | Prompts | Risk | Failure rate | High-crit fail | Judge worse | Latency p95 | Top archetypes |
|---|---|---|---|---|---|---|---|
| summarization | 3 | 58.7% | 66.7% | 100.0% | 0.0% | 1.52x | missing_required_content x1, overlong_output x1 |
| customer_support | 3 | 53.0% | 66.7% | 66.7% | 0.0% | 2.00x | missing_required_content x1, overlong_output x1 |
| code_generation | 3 | 50.0% | 66.7% | 66.7% | 0.0% | 2.31x | json_invalid x2 |
| classification | 3 | 27.8% | 33.3% | 0.0% | 0.0% | 2.58x | overlong_output x1 |
| extraction | 3 | 6.7% | 0.0% | 0.0% | 0.0% | 1.48x | none |
Examples
| Prompt | Category | Criticality | Candidate issues | Candidate output |
|---|---|---|---|---|
| cod-001 | code_generation | high | Output is not valid JSON | Archetypes: json_invalid | Here's a solution for parsing CSV rows with proper handling of quoted fields and escaped quotes:
```python
def parse_csv_row(line: str) -> list[str]:
fields = []
current_field = []
in_quotes = False
i = 0
while i < len(line):
char = line[i]
if char == '"':
# Check for escaped quote (two double quotes)
if i + 1 < len(line) and line[i+1] == '"':
current_field.append('"')
i += 2
elif in_quotes:
# End of quoted field
in_quotes = False
i += 1
else:
# Start of quoted field
in_quotes = True
i += 1
elif char == ',' and not in_quotes:
# Field sepa |
| cod-004 | code_generation | high | Output is not valid JSON | Archetypes: json_invalid | Here's a Python implementation of the `retry` function with the specified requirements:
```python
import time
import json
import traceback
def retry(fn, *, attempts: int = 3, base_delay: float = 0.5):
"""
Retry a function with exponential backoff.
:param fn: Function to be called
:param attempts: Maximum number of attempts (default 3)
:param base_delay: Initial delay between retries (default 0.5 seconds)
:return: JSON with result details
"""
last_exception = None
for attempt in range(1, attempts + 1):
try:
result = fn()
# Successful execution
return json.dumps({
"code": 0,
"explanation": f"Success on attempt {attempt}"
})
exce |
| cus-001 | customer_support | high | Missing required content: investigate | Archetypes: missing_required_content | Here's a draft response: Subject: Immediate Resolution for Order #4521 Duplicate Charge Dear [Customer Name], I sincerely apologize for the ongoing issue with the duplicate charge for order #4521 and the frustration this has caused you. We clearly have not met our standard of customer service, and I want to assure you that we are taking immediate action. I am personally escalating this refund to our finance team with instructions to process the $89.99 refund within the next 48 hours. You will receive an email confirmation of the refund, and the credit should appear on your statement within 3-5 business days. As a gesture of goodwill for the inconvenience, I am also applying a 20% store credit to your account, which you can use on a future purchase. If you do not see the refund or hav |
| cus-002 | customer_support | high | Output length 1233 exceeds max_output_chars=1200 | Archetypes: overlong_output | Here's a draft response: Subject: Re: Missing CSV Export Functionality - Immediate Workaround and Feature Restoration Plan Dear [Customer Name], I apologize for the disruption this has caused to your workflow, especially with a critical deadline approaching. We take these workflow interruptions seriously. Immediate Workaround: While our team investigates the removed export feature, you can currently export your data by: 1. Using the "Advanced Export" menu under [specific location] 2. Selecting CSV as the file type 3. Choosing your desired columns Escalation Steps: • I've logged a high-priority ticket for our product team to restore the direct CSV export button • Our engineering lead will personally review this within 24 hours • We aim to have a permanent fix in our next patch release |
| sum-006 | summarization | high | Missing required content: internal | Archetypes: missing_required_content | Here's a plain language translation that preserves the legal meaning: You are allowed to use this software only for your own company's work. This permission can be taken away at any time. You cannot: - Share the software with other companies - Sell the software - Rent out the software - Give the software to anyone else If you want to do any of these things, you must first get written permission from the software owner. |
| cla-003 | classification | medium | Output length 462 exceeds max_output_chars=80 | Archetypes: overlong_output | This email appears to be not_spam. Here's why: 1. The language sounds professional and genuine 2. It references a previous conversation and a business context (partnership proposal) 3. The sender mentions a previously sent deck and offers to discuss it 4. The tone is polite and straightforward 5. There are no suspicious links, urgent demands, or typical spam language 6. It appears to be a legitimate business communication follow-up Classification: not_spam |
| sum-004 | summarization | medium | Output length 846 exceeds max_output_chars=800 | Archetypes: overlong_output | Themes from Customer Feedback: 1. Positive UI/UX Perception (Intuitiveness, Design) 2. Performance Issues (Loading, Crashes) 3. Mobile Experience Concerns 4. Feature Requests (Dark Mode, PDF Export) Summary: The customer feedback reveals a generally positive reception of the dashboard, with users appreciating its intuitive design and aesthetic appeal. However, the product is experiencing technical challenges that are impacting user satisfaction, including slow loading times, occasional system crashes, and mobile usability concerns. Users are expressing specific desires for improvements such as a dark mode and reliable PDF export functionality. While some customers are completely satisfied with the platform's speed and appearance, others are highlighting critical areas that need technical |
| cla-002 | classification | high | No deterministic failures | p1 |