Metadata-Version: 2.4
Name: svarog-crucible
Version: 0.1.0
Summary: Crucible. The touchstone — benchmark and verification tool for multi-agent dispatch.
Project-URL: Homepage, https://github.com/SvarogForge/crucible
Project-URL: Repository, https://github.com/SvarogForge/crucible
Author-email: iMonstra <imonstra@users.noreply.github.com>
License: MIT
Keywords: ai,benchmark,crucible,hermes-agent,svarog
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Description-Content-Type: text/markdown

<p align="center">
  <img src=".github/logo.svg" width="160" alt="Crucible">
</p>

<h1 align="center">Crucible</h1>
<p align="center"><em>The touchstone for multi-agent dispatch — fire that tests your metal</em></p>

[![EN](https://img.shields.io/badge/EN-English-blue?style=flat-square)](README.md)
[![RU](https://img.shields.io/badge/RU-Русский-red?style=flat-square)](README.ru.md)

[![Stars](https://img.shields.io/github/stars/SvarogForge/crucible?style=flat-square&logo=github)](https://github.com/SvarogForge/crucible)
[![Last commit](https://img.shields.io/github/last-commit/SvarogForge/crucible?style=flat-square&logo=github)](https://github.com/SvarogForge/crucible)
[![License](https://img.shields.io/badge/license-MIT-green?style=flat)](LICENSE)
[![CI](https://github.com/SvarogForge/crucible/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/SvarogForge/crucible/actions/workflows/ci.yml)
[![Hermes Benchmark](https://img.shields.io/badge/Hermes-Benchmark-8A2BE2?style=flat)](https://hermes-agent.nousresearch.com)

<p align="center">
  <img src="https://img.shields.io/badge/Python-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python">
  <img src="https://img.shields.io/badge/JSON-000000?style=flat-square&logo=json&logoColor=white" alt="JSON">
  <img src="https://img.shields.io/badge/Markdown-000000?style=flat-square&logo=markdown&logoColor=white" alt="Markdown">
  <img src="https://img.shields.io/badge/GitHub%20Actions-2088FF?style=flat-square&logo=githubactions&logoColor=white" alt="GitHub Actions">
</p>

---

> *In a smithy, the crucible is where raw ore meets fire — a test of what's true metal and what's dross.*

Existing benchmarks measure **what** an agent produces. None measure **how** the work is divided and gathered.

```text
SWE-bench    → Can a single agent write code?
GAIA         → Can an agent answer questions?
τ-bench      → Can an agent browse?
AgentBench   → Can an agent use tools?
BFCL         → Can an agent call functions?
```

Crucible asks a different question:

> When you spawn 5 agents in parallel, how much do they cost?
> Is nested (depth=2) more efficient than flat (depth=1)?
> Which pattern wins: 3 deep or 5 shallow?

---

## For whom?

| You are... | You ask... | Crucible answers |
|-----------|-----------|-----------------|
| Building an agent framework | Which dispatch pattern should I use? | Real metrics for 6 patterns |
| Choosing between CrewAI / LangGraph / AutoGen | Which architecture costs less? | Per-pattern cost breakdown |
| Shipping a production multi-agent system | What's the cheapest reliable pattern? | Producer-Reviewer: $0.05, 0 errors |
| Writing a research paper on agents | Where's the evidence? | 19 sub-agents, 3+ runs each, live API calls, raw data |

---

## Quick Start

```bash
git clone https://github.com/SvarogForge/crucible
cd crucible
hermes -f scenarios/flat_5.txt
```

3 minutes later you have real metrics for 6 dispatch patterns.

---

## Results

19 sub-agents, 6 patterns, live API calls. Zero simulation.

| Pattern | Time | Cost $ | Errors | Coverage |
|:--------|:----:|:------:|:------:|:--------:|
| **Flat** | 2:29 ⚡ | $0.12 | 0 | ~60% |
| **Nested** | 7:14 | $0.14 | 0 | ~90% |
| **Sequential** | 4:15 | $0.19 | 0 | ~70% |
| **Hybrid** | 7:47 | $0.53 | 8 | ~75% |
| **Producer-Reviewer** | 5:20 | $0.05 💰 | 0 | ~82% |
| **Consensus** | 4:52 | $0.12 | 0 | ~90% 🛡️ |

### Picking a Pattern

| Need | Try | Why |
|------|-----|-----|
| Fastest execution | Flat | 2:29 — 2-3× faster than alternatives |
| Deepest coverage | Nested | 85-95% completeness at depth=2 |
| Lowest cost | Producer-Reviewer | $0.05 — built-in review pass |
| Most reliable | Consensus | 3 couriers → 1 result, variation <0.3% |
| Balanced | Sequential | 4:15, $0.19 — predictable |

---

## Methodology

### Principles

1. **Identical topics** — only the architecture differs
2. **Identical toolsets** — web + terminal for everyone
3. **Multiple runs** — minimum 3 per scenario
4. **Automatic metrics** — real API returns cost, tokens, tool_trace
5. **No simulations** — only live API calls

### Research Topics

| Topic | Data Source |
|-------|:-----------:|
| Top-5 OS AI Agent Frameworks by GitHub stars | GitHub API |
| Local TTS Solutions (CosyVoice, Piper, XTTS, Silero) | GitHub API |
| Top-5 SDXL Checkpoints by downloads (July 2026) | CivitAI API |
| RAG Architectures (LlamaIndex, LangChain, Qdrant) | GitHub API |
| AI Agent Monitoring (LangSmith, W&B, Arize) | GitHub API |

### 7 Core Metrics

| Metric | Objectivity | Collection |
|--------|:-----------:|------------|
| Wall-clock Time (s) | ★★★★★ | Timer |
| Total Tokens | ★★★★★ | API usage |
| Cost ($) | ★★★★★ | tokens × rate |
| Quality Score (1-10) | ★★★☆☆ | LLM-as-Judge |
| Coverage (%) | ★★★★☆ | Checklist eval |
| Success Rate (%) | ★★★★★ | status pass/fail |
| Efficiency (quality/cost) | ★★★★☆ | derived |

---

## Consensus Verification

3 independent agents, same question. Results:

| Framework | Agent 1 | Agent 2 | Agent 3 | Variation |
|-----------|:-------:|:-------:|:-------:|:---------:|
| LangGraph | 36,650 | 36,650 | 36,650 | **0%** |
| CrewAI | 55,000 | 55,033 | 55,033 | **~0.06%** |
| AutoGen | 59,500 | 59,538 | 59,538 | **~0.06%** |
| Pydantic AI | 18,300 | 18,250 | 18,300 | **~0.27%** |
| Semantic Kernel | 28,300 | 28,275 | 28,300 | **~0.09%** |

Sub-agents produce consistent, reproducible data. Variation under 0.3% across all tested frameworks.

---

## Setup

### JSON Schema

```json
{
  "test_id": "crucible-flat-20260707",
  "pattern": "flat",
  "model": "deepseek/deepseek-v4-flash",
  "config": { "max_concurrent_children": 5, "repeats": 3 }
}
```

---

## Why Crucible?

2026. There are at least **15 active agent frameworks** with different dispatch strategies. CrewAI uses flat delegation. LangGraph has graph-based routing. AutoGen has nested conversations. Pydantic AI uses sequential plans.

Which one is cheaper? Which one covers more ground? Which one should you build on?

**Nobody knew.** Not until the fire was lit.

Crucible is the first benchmark that isolates **dispatch architecture** from agent capability. Same topics. Same tools. Same models. Only the **pattern** differs. The result: real, reproducible data on what works and what costs.

---

## 🔥 From the forge-fire

Crucible is part of [SvarogForge](https://github.com/SvarogForge) — a family of tools forged for Hermes Agent.

| Project | Description |
|---------|-------------|
| ⚡ **Talaria** | Seven-league messengers for market research |
| 🔥 **Crucible** | Touchstone for quality & benchmarks (you are here) |
| ⚒️ **Forge** | The smithy itself — AI-powered project forge |

## 📄 License

MIT — open for everyone.

---

<p align="center">
  ⭐ <b>Star on <a href="https://github.com/SvarogForge/crucible">GitHub</a></b> ·
  🐦 <b><a href="https://twitter.com/iMonstra">Follow @iMonstra</a></b> ·
  💬 <b><a href="https://github.com/SvarogForge/crucible/discussions">Join Discussions</a></b>
</p>
<p align="center">Built for <a href="https://hermes-agent.nousresearch.com">Hermes Agent</a> · <a href="LICENSE">MIT</a> · Contributions welcome</p>
