Live mode
Run
GPT-4o to GPT-4o-mini cost reduction
STOP
Candidate exceeded the high-criticality failure threshold (64% >= 20%). Most affected category: code_generation (high-crit 100%; json_invalid x3).
Confidence: 59%
Coverage
15 prompts across 1 batches
Total cost: $0.0303
Stopped early: yes
Latency
p50: 2956ms -> 4056ms
p95: 6269ms -> 12965ms
Decision Metrics
| Overall risk | 45.4% |
|---|---|
| Candidate failure rate | 66.7% |
| Candidate regression rate | 6.7% |
| High-criticality failure rate | 63.6% |
| Schema break rate | 0.0% |
| Judged prompts | 3/3 |
| Escalated prompts | 0 |
| Judge worse rate | 0.0% |
| Judge average confidence | 90% |
| Latency p50 ratio | 1.37x |
| Latency p95 ratio | 2.07x |
Thresholds
| High-criticality stop | 20% |
|---|---|
| Schema stop | 25% |
| Proceed risk | <= 8% |
| Latency p50 threshold | 1.50x |
| Latency p95 threshold | 2.00x |
| Tiered escalation threshold | 60% |
Failure Archetypes
- json_invalid: 7
- missing_required_content: 2
- overlong_output: 1
Category Scorecards
| Category | Prompts | Risk | Failure rate | High-crit fail | Judge worse | Latency p95 | Top archetypes |
|---|---|---|---|---|---|---|---|
| code_generation | 3 | 66.7% | 100.0% | 100.0% | 0.0% | 1.74x | json_invalid x3 |
| extraction | 3 | 61.3% | 100.0% | 100.0% | 0.0% | 1.93x | json_invalid x3 |
| customer_support | 3 | 18.0% | 33.3% | 33.3% | 0.0% | 1.06x | missing_required_content x1 |
| summarization | 3 | 12.6% | 33.3% | 0.0% | 0.0% | 1.12x | overlong_output x1 |
| classification | 3 | 11.1% | 66.7% | 0.0% | 0.0% | 0.83x | json_invalid x1, missing_required_content x1 |
Examples
| Prompt | Category | Criticality | Candidate issues | Candidate output |
|---|---|---|---|---|
| cod-001 | code_generation | high | Output is not valid JSON | Archetypes: json_invalid | ```python
import json
import re
def parse_csv_row(line: str) -> list[str]:
# Regular expression to match CSV fields
pattern = re.compile(r'''
# Match an optional quoted field
"([^"]|"")*" # Quoted field (may contain escaped quotes)
| # or
[^,"]+ # Non-quoted field (not containing commas or quotes)
''', re.VERBOSE)
# Find all matches according to the pattern
matches = pattern.findall(line)
# Clean the matches to remove surrounding quotes and handle escaped quotes
cleaned_fields = []
for match in matches:
if match.startswith('"') and match.endswith('"'):
# Remove enclosing quotes and replace doubled quotes with a single quote
cleaned = match[1:- |
| cod-003 | code_generation | high | Output is not valid JSON | Archetypes: json_invalid | ```json
{
"code": "SELECT c.name, SUM(o.amount_cents) / 100.0 AS total_revenue\nFROM customers c\nJOIN orders o ON c.id = o.customer_id\nWHERE o.created_at >= NOW() - INTERVAL '30 days'\nGROUP BY c.id, c.name\nORDER BY total_revenue DESC\nLIMIT 5;",
"explanation": "This query retrieves the top 5 customers by total revenue in the last 30 days. It joins the 'customers' table with the 'orders' table on the customer ID. It filters the orders to those created within the last 30 days using the 'NOW() - INTERVAL '30 days'' condition. Then, it groups the results by customer ID and name, calculating the total revenue (in dollars) by summing the 'amount_cents' and converting it to dollars by dividing by 100. Finally, it orders the results by total revenue in descending order and limits the outpu |
| cod-004 | code_generation | high | Output is not valid JSON | Archetypes: json_invalid | You can implement the `retry` function as described using the `time.sleep` function to create the exponential backoff. Below is the implementation of the function:
```python
import time
import json
def retry(fn, *, attempts: int = 3, base_delay: float = 0.5):
last_exception = None
for attempt in range(attempts):
try:
result = fn()
# If the function call is successful, return the result as JSON
return json.dumps({"code": "success", "explanation": result})
except Exception as e:
last_exception = e
# Calculate the backoff delay
delay = base_delay * (2 ** attempt)
time.sleep(delay)
# If we exhausted all attempts, re-raise the last exception
raise last_exception
# Example |
| cus-001 | customer_support | high | Missing required content: investigate | Archetypes: missing_required_content | Subject: Apologies for the Inconvenience Regarding Your Order #4521 Dear [Customer's Name], Thank you for reaching out and bringing this to our attention. I sincerely apologize for the inconvenience you've experienced with the duplicate charge for your order #4521. I understand how frustrating this must be, especially since you've contacted us multiple times regarding this issue. We take matters like this very seriously, and I want to assure you that we are committed to resolving this for you as quickly as possible. I have escalated your case to our billing department, and they will process the refund for the duplicate charge of $89.99. You can expect the refund to be completed within 3-5 business days. Once processed, you will receive a confirmation email for your records. Thank you |
| ext-001 | extraction | high | Output is not valid JSON | Archetypes: json_invalid | ```json
{
"people": [
"Maria Rossi",
"Dr. John Chen",
"Aisha Patel"
],
"organizations": [
"Acme Corp",
"Globex Industries",
"Initech Labs"
],
"locations": [
"Berlin",
"Milan"
]
}
``` |
| ext-002 | extraction | high | Output is not valid JSON | Archetypes: json_invalid | Here is the JSON representation of the parsed invoice text:
```json
{
"invoice_number": "INV-2026-0312",
"date": "April 2, 2026",
"bill_to": "TechStart Ltd, 42 Innovation Drive, London EC1A 1BB",
"line_items": [
{
"description": "Widget A",
"quantity": 3,
"unit_price": 50.00,
"total_price": 150.00
},
{
"description": "Widget B",
"quantity": 1,
"unit_price": 120.00,
"total_price": 120.00
},
{
"description": "Service Hour",
"quantity": 2,
"unit_price": 85.00,
"total_price": 170.00
}
],
"subtotal": 440.00,
"tax": 88.00,
"total": 528.00
}
```
This JSON structure captures all the relevant details from the invoice text. |
| ext-006 | extraction | high | Output is not valid JSON | Archetypes: json_invalid | ```json
{
"title": "Senior Backend Engineer",
"company": "CloudScale",
"location": "Remote (US only)",
"remote": true,
"experience_years": 5,
"required_skills": ["Python", "Go", "PostgreSQL", "Kubernetes"],
"salary_min": 180000,
"salary_max": 220000,
"currency": "USD",
"benefits": ["health insurance", "401k match", "unlimited PTO", "home office stipend"]
}
``` |
| cla-003 | classification | medium | Missing required content: not_spam | Archetypes: missing_required_content | Not Spam |