Live mode
Run

GPT-4o to GPT-4o-mini cost reduction

STOP

Candidate exceeded the high-criticality failure threshold (64% >= 20%). Most affected category: code_generation (high-crit 100%; json_invalid x3).

Confidence: 59%

Coverage
15 prompts across 1 batches
Total cost: $0.0303
Stopped early: yes
Latency
p50: 2956ms -> 4056ms
p95: 6269ms -> 12965ms

Decision Metrics

Overall risk45.4%
Candidate failure rate66.7%
Candidate regression rate 6.7%
High-criticality failure rate 63.6%
Schema break rate0.0%
Judged prompts 3/3
Escalated prompts 0
Judge worse rate0.0%
Judge average confidence90%
Latency p50 ratio1.37x
Latency p95 ratio2.07x

Thresholds

High-criticality stop20%
Schema stop25%
Proceed risk<= 8%
Latency p50 threshold1.50x
Latency p95 threshold2.00x
Tiered escalation threshold60%

Failure Archetypes

Category Scorecards

CategoryPromptsRiskFailure rateHigh-crit failJudge worseLatency p95Top archetypes
code_generation366.7%100.0%100.0%0.0%1.74xjson_invalid x3
extraction361.3%100.0%100.0%0.0%1.93xjson_invalid x3
customer_support318.0%33.3%33.3%0.0%1.06xmissing_required_content x1
summarization312.6%33.3%0.0%0.0%1.12xoverlong_output x1
classification311.1%66.7%0.0%0.0%0.83xjson_invalid x1, missing_required_content x1

Examples

PromptCategoryCriticalityCandidate issuesCandidate output
cod-001code_generationhighOutput is not valid JSON | Archetypes: json_invalid
```python
import json
import re

def parse_csv_row(line: str) -> list[str]:
    # Regular expression to match CSV fields
    pattern = re.compile(r'''
        # Match an optional quoted field
        "([^"]|"")*"         # Quoted field (may contain escaped quotes)
        |                     # or
        [^,"]+               # Non-quoted field (not containing commas or quotes)
    ''', re.VERBOSE)
    
    # Find all matches according to the pattern
    matches = pattern.findall(line)
    
    # Clean the matches to remove surrounding quotes and handle escaped quotes
    cleaned_fields = []
    for match in matches:
        if match.startswith('"') and match.endswith('"'):
            # Remove enclosing quotes and replace doubled quotes with a single quote
            cleaned = match[1:-
cod-003code_generationhighOutput is not valid JSON | Archetypes: json_invalid
```json
{
  "code": "SELECT c.name, SUM(o.amount_cents) / 100.0 AS total_revenue\nFROM customers c\nJOIN orders o ON c.id = o.customer_id\nWHERE o.created_at >= NOW() - INTERVAL '30 days'\nGROUP BY c.id, c.name\nORDER BY total_revenue DESC\nLIMIT 5;",
  "explanation": "This query retrieves the top 5 customers by total revenue in the last 30 days. It joins the 'customers' table with the 'orders' table on the customer ID. It filters the orders to those created within the last 30 days using the 'NOW() - INTERVAL '30 days'' condition. Then, it groups the results by customer ID and name, calculating the total revenue (in dollars) by summing the 'amount_cents' and converting it to dollars by dividing by 100. Finally, it orders the results by total revenue in descending order and limits the outpu
cod-004code_generationhighOutput is not valid JSON | Archetypes: json_invalid
You can implement the `retry` function as described using the `time.sleep` function to create the exponential backoff. Below is the implementation of the function:

```python
import time
import json

def retry(fn, *, attempts: int = 3, base_delay: float = 0.5):
    last_exception = None
    for attempt in range(attempts):
        try:
            result = fn()
            # If the function call is successful, return the result as JSON
            return json.dumps({"code": "success", "explanation": result})
        except Exception as e:
            last_exception = e
            # Calculate the backoff delay
            delay = base_delay * (2 ** attempt)
            time.sleep(delay)
    
    # If we exhausted all attempts, re-raise the last exception
    raise last_exception

# Example 
cus-001customer_supporthighMissing required content: investigate | Archetypes: missing_required_content
Subject: Apologies for the Inconvenience Regarding Your Order #4521

Dear [Customer's Name],

Thank you for reaching out and bringing this to our attention. I sincerely apologize for the inconvenience you've experienced with the duplicate charge for your order #4521. I understand how frustrating this must be, especially since you've contacted us multiple times regarding this issue.

We take matters like this very seriously, and I want to assure you that we are committed to resolving this for you as quickly as possible. I have escalated your case to our billing department, and they will process the refund for the duplicate charge of $89.99. 

You can expect the refund to be completed within 3-5 business days. Once processed, you will receive a confirmation email for your records.

Thank you
ext-001extractionhighOutput is not valid JSON | Archetypes: json_invalid
```json
{
  "people": [
    "Maria Rossi",
    "Dr. John Chen",
    "Aisha Patel"
  ],
  "organizations": [
    "Acme Corp",
    "Globex Industries",
    "Initech Labs"
  ],
  "locations": [
    "Berlin",
    "Milan"
  ]
}
```
ext-002extractionhighOutput is not valid JSON | Archetypes: json_invalid
Here is the JSON representation of the parsed invoice text:

```json
{
  "invoice_number": "INV-2026-0312",
  "date": "April 2, 2026",
  "bill_to": "TechStart Ltd, 42 Innovation Drive, London EC1A 1BB",
  "line_items": [
    {
      "description": "Widget A",
      "quantity": 3,
      "unit_price": 50.00,
      "total_price": 150.00
    },
    {
      "description": "Widget B",
      "quantity": 1,
      "unit_price": 120.00,
      "total_price": 120.00
    },
    {
      "description": "Service Hour",
      "quantity": 2,
      "unit_price": 85.00,
      "total_price": 170.00
    }
  ],
  "subtotal": 440.00,
  "tax": 88.00,
  "total": 528.00
}
``` 

This JSON structure captures all the relevant details from the invoice text.
ext-006extractionhighOutput is not valid JSON | Archetypes: json_invalid
```json
{
  "title": "Senior Backend Engineer",
  "company": "CloudScale",
  "location": "Remote (US only)",
  "remote": true,
  "experience_years": 5,
  "required_skills": ["Python", "Go", "PostgreSQL", "Kubernetes"],
  "salary_min": 180000,
  "salary_max": 220000,
  "currency": "USD",
  "benefits": ["health insurance", "401k match", "unlimited PTO", "home office stipend"]
}
```
cla-003classificationmediumMissing required content: not_spam | Archetypes: missing_required_content
Not Spam