📖 Blog & Case Studies

🤖 Multi-Agent Customer Support: Testing Tool Cascade Failures

Challenge: A customer support agent orchestrates four sub-agents (Search, Refund, Escalation, Follow-up). If the Search agent fails, does the Refund agent gracefully degrade? Or does it cascade into all four failing?

Approach: We used agentic-chaos with ToolCallFailureFault to break the Search agent's database connection, then TopologyTracker to visualize which agents were impacted.

with chaos_session([ToolCallFailureFault(tool_name="search", mode="error")]):
    coordinator.run_workflow()

# Output topology shows cascade edges:
# Coordinator → Search (fails) → Refund (cascades) → Follow-up (blocked)

Finding: Without fallbacks, a single tool failure cascaded to 3 agents. We added:

• Search timeout handling with 2-second fallback to cache
• Refund pre-check (skip if no search results)
• Follow-up retry loop (up to 3x)

→ Read full case study: examples/chaos_agent_failure_demo.py

🤐 The Silent Degradation Problem: Undetectable Failures

Challenge: A RAG system's retriever appeared healthy (normal latency, token counts), but was returning corrupted embeddings. Your monitoring saw nothing wrong until production complaints arrived.

The Gap: Latency-based monitoring is blind to silent degradation. A corrupted embedding still takes 245ms.

Solution: Use agentic-chaos SilentDegradationFault to simulate this exact scenario:

with chaos_session([SilentDegradationFault()]):
    # Embeddings come back, but content is corrupted
    results = chaos_call(retriever.search, query, faults=["silent_degradation"])

    # Your monitoring sees:
    # ✓ Latency: 245ms (normal)
    # ✓ Tokens: 128 (normal)
    # ✗ Content: corrupted (undetected)

Key Insight: Silent degradation is the hardest failure to catch. Monitoring + resilience testing = solution.

⏱️ Rate Limit Recovery: Testing Retry Logic Under Pressure

Challenge: Your agent calls an external API that occasionally rate-limits (429). Does your retry strategy work? How many tokens do you waste? What's the UX impact?

Solution: RateLimitStormFault simulates a 3-call 429 burst, then recovery:

with chaos_session([RateLimitStormFault(burst_count=3, retry_after=1.0)]):
    for i in range(10):
        result = chaos_call(api.call, args, faults=["rate_limit_storm"])

# Calls 0-2: 429 (require retry)
# Calls 3-9: Success (burst cleared)

Findings:

• Without backoff: 15 wasted tokens, 5s delay, user sees timeout
• With exponential backoff: 2 wasted tokens, 4s delay, user waits (acceptable)
• With probabilistic retry: 0 wasted tokens, <2s delay, fully recovers