Live mode
Run

GPT-4o to Claude 3.5 Haiku migration

STOP

Candidate exceeded the high-criticality failure threshold (45% >= 20%). Most affected category: summarization (high-crit 100%; missing_required_content x1, overlong_output x1).

Confidence: 59%

Coverage
15 prompts across 1 batches
Total cost: $0.0311
Stopped early: yes
Latency
p50: 3160ms -> 5578ms
p95: 6971ms -> 14395ms

Decision Metrics

Overall risk41.5%
Candidate failure rate46.7%
Candidate regression rate 20.0%
High-criticality failure rate 45.5%
Schema break rate0.0%
Judged prompts 2/2
Escalated prompts 0
Judge worse rate0.0%
Judge average confidence90%
Latency p50 ratio1.77x
Latency p95 ratio2.06x

Thresholds

High-criticality stop20%
Schema stop25%
Proceed risk<= 8%
Latency p50 threshold1.50x
Latency p95 threshold2.00x
Tiered escalation threshold60%

Failure Archetypes

Category Scorecards

CategoryPromptsRiskFailure rateHigh-crit failJudge worseLatency p95Top archetypes
summarization358.7%66.7%100.0%0.0%1.52xmissing_required_content x1, overlong_output x1
customer_support353.0%66.7%66.7%0.0%2.00xmissing_required_content x1, overlong_output x1
code_generation350.0%66.7%66.7%0.0%2.31xjson_invalid x2
classification327.8%33.3%0.0%0.0%2.58xoverlong_output x1
extraction36.7%0.0%0.0%0.0%1.48xnone

Examples

PromptCategoryCriticalityCandidate issuesCandidate output
cod-001code_generationhighOutput is not valid JSON | Archetypes: json_invalid
Here's a solution for parsing CSV rows with proper handling of quoted fields and escaped quotes:

```python
def parse_csv_row(line: str) -> list[str]:
    fields = []
    current_field = []
    in_quotes = False
    i = 0
    
    while i < len(line):
        char = line[i]
        
        if char == '"':
            # Check for escaped quote (two double quotes)
            if i + 1 < len(line) and line[i+1] == '"':
                current_field.append('"')
                i += 2
            elif in_quotes:
                # End of quoted field
                in_quotes = False
                i += 1
            else:
                # Start of quoted field
                in_quotes = True
                i += 1
        
        elif char == ',' and not in_quotes:
            # Field sepa
cod-004code_generationhighOutput is not valid JSON | Archetypes: json_invalid
Here's a Python implementation of the `retry` function with the specified requirements:

```python
import time
import json
import traceback

def retry(fn, *, attempts: int = 3, base_delay: float = 0.5):
    """
    Retry a function with exponential backoff.
    
    :param fn: Function to be called
    :param attempts: Maximum number of attempts (default 3)
    :param base_delay: Initial delay between retries (default 0.5 seconds)
    :return: JSON with result details
    """
    last_exception = None
    
    for attempt in range(1, attempts + 1):
        try:
            result = fn()
            
            # Successful execution
            return json.dumps({
                "code": 0,
                "explanation": f"Success on attempt {attempt}"
            })
        
        exce
cus-001customer_supporthighMissing required content: investigate | Archetypes: missing_required_content
Here's a draft response:

Subject: Immediate Resolution for Order #4521 Duplicate Charge

Dear [Customer Name],

I sincerely apologize for the ongoing issue with the duplicate charge for order #4521 and the frustration this has caused you. We clearly have not met our standard of customer service, and I want to assure you that we are taking immediate action.

I am personally escalating this refund to our finance team with instructions to process the $89.99 refund within the next 48 hours. You will receive an email confirmation of the refund, and the credit should appear on your statement within 3-5 business days.

As a gesture of goodwill for the inconvenience, I am also applying a 20% store credit to your account, which you can use on a future purchase.

If you do not see the refund or hav
cus-002customer_supporthighOutput length 1233 exceeds max_output_chars=1200 | Archetypes: overlong_output
Here's a draft response:

Subject: Re: Missing CSV Export Functionality - Immediate Workaround and Feature Restoration Plan

Dear [Customer Name],

I apologize for the disruption this has caused to your workflow, especially with a critical deadline approaching. We take these workflow interruptions seriously.

Immediate Workaround:
While our team investigates the removed export feature, you can currently export your data by:
1. Using the "Advanced Export" menu under [specific location]
2. Selecting CSV as the file type
3. Choosing your desired columns

Escalation Steps:
• I've logged a high-priority ticket for our product team to restore the direct CSV export button
• Our engineering lead will personally review this within 24 hours
• We aim to have a permanent fix in our next patch release
sum-006summarizationhighMissing required content: internal | Archetypes: missing_required_content
Here's a plain language translation that preserves the legal meaning:

You are allowed to use this software only for your own company's work. This permission can be taken away at any time. You cannot:
- Share the software with other companies
- Sell the software
- Rent out the software
- Give the software to anyone else

If you want to do any of these things, you must first get written permission from the software owner.
cla-003classificationmediumOutput length 462 exceeds max_output_chars=80 | Archetypes: overlong_output
This email appears to be not_spam. Here's why:

1. The language sounds professional and genuine
2. It references a previous conversation and a business context (partnership proposal)
3. The sender mentions a previously sent deck and offers to discuss it
4. The tone is polite and straightforward
5. There are no suspicious links, urgent demands, or typical spam language
6. It appears to be a legitimate business communication follow-up

Classification: not_spam
sum-004summarizationmediumOutput length 846 exceeds max_output_chars=800 | Archetypes: overlong_output
Themes from Customer Feedback:
1. Positive UI/UX Perception (Intuitiveness, Design)
2. Performance Issues (Loading, Crashes)
3. Mobile Experience Concerns
4. Feature Requests (Dark Mode, PDF Export)

Summary:
The customer feedback reveals a generally positive reception of the dashboard, with users appreciating its intuitive design and aesthetic appeal. However, the product is experiencing technical challenges that are impacting user satisfaction, including slow loading times, occasional system crashes, and mobile usability concerns. Users are expressing specific desires for improvements such as a dark mode and reliable PDF export functionality. While some customers are completely satisfied with the platform's speed and appearance, others are highlighting critical areas that need technical 
cla-002classificationhighNo deterministic failures
p1