zsh — agent_project
$
Collecting althea-ai
Installing collected packages: althea-ai
Successfully installed althea_ai-0.1.0
 
$
support_agent.py
1# your existing agent — unchanged
2import openai
3
4import althea # ← one import
5althea.init(api_key="sk-althea-...") # ← one line
6
7@althea.watch # ← one decorator
8def support_agent(user_input):
9 ... # nothing else changes
Zero config to start
zsh — agent_project
$ python support_agent.py
✓ Althea connected
Evaluating every step in real time
Dashboard: https://althea.dev/a/sup_8f21
 
Interaction 14 · Score 91/100 · PASSED
Interaction 15 · Score 88/100 · PASSED
Interaction 16 · Score 34/100 · FAILED → tool_failure at step 3
Overview
Failure Feed
Interaction Detail
Quality score
87
↑ 3 pts vs yesterday
Today's pass rate
91%
1,142 of 1,255 interactions
Active failures
3
2 patterns, 1 one-off
Top failure type
tool_failure
47% of all failures
Live — evaluating every interaction
Interaction 1142 · 2s ago
94/100
Refund request — resolved correctly at step 4.
Interaction 1141 · 9s ago
89/100
Account question — resolved correctly at step 2.
Overview
Failure Feed
Interaction Detail
Failure Feed — real time
ONE-OFFInteraction 1143 · just now
34/100
Billing refund — failed at step 3 of 4. Payment policy API call timed out.
PATTERN · 47%Billing disputes — tool_failure at step 3
31/100 avg
Root cause: payment policy API times out under load at step 3. 23 of 49 billing interactions affected today.
Interaction 1140 · 14s ago
96/100
Order status — resolved correctly at step 2.
Overview
Failure Feed
Interaction Detail
Interaction 1143 — Score 34/100 — FAILED
✓
Step 1 — User asked about a refund
Agent acknowledged the request correctly and asked for the order number.
✓
Step 2 — Agent retrieved order history
Correct tool call. Order found, eligible for refund.
✗
Step 3 — Agent applied refund policy
Tool call to payment_policy_api timed out after 8s. Agent proceeded without a response.
↳ tool_failure — payment policy API did not return in time.
✗
Step 4 — Agent gave a final answer anyway
Agent told the user the refund was approved without confirming policy eligibility.
↳ wrong_answer — claim not grounded in tool output.
Althea Verdict
34/100
accuracy 0.2 · goal_alignment 0.6 · decision_quality 0.3 · completeness 0.7
The payment policy API timed out at step 3. Instead of retrying or escalating, the agent told the customer the refund was approved — a claim it had no basis for.
Suggested fix
Add a timeout fallback at step 3: retry once, then escalate to a human instead of guessing the policy outcome.
Find where your agent fails.
Know why. Fix it.
althea.dev
1 / 7 — Install