Metadata-Version: 2.4
Name: crucible-ttc
Version: 0.1.0
Summary: A verifier-guided reasoning engine: search (best-of-N, beam, MCTS) over reasoning traces to turn test-time compute into measured accuracy.
Project-URL: Homepage, https://github.com/satchmakua/crucible-ttc
Author: Satchel Hamilton
License: MIT License
        
        Copyright (c) 2026 Satchel Hamilton
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: llm,mcts,prm,reasoning,test-time-compute,verifier
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx<0.29,>=0.27
Requires-Dist: math-verify<1,>=0.5
Requires-Dist: matplotlib<4,>=3.8
Requires-Dist: numpy<3,>=1.26
Requires-Dist: pandas<3,>=2.1
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: rich<15,>=13.7
Requires-Dist: sympy<2,>=1.12
Requires-Dist: typer<0.16,>=0.12
Provides-Extra: datasets
Requires-Dist: datasets<4,>=2.19; extra == 'datasets'
Provides-Extra: dev
Requires-Dist: mypy<2,>=1.10; extra == 'dev'
Requires-Dist: pytest<9,>=8.0; extra == 'dev'
Requires-Dist: ruff<1,>=0.5; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Provides-Extra: prm
Requires-Dist: accelerate>=0.30; extra == 'prm'
Requires-Dist: torch>=2.2; extra == 'prm'
Requires-Dist: transformers<5,>=4.44; extra == 'prm'
Provides-Extra: vllm
Requires-Dist: vllm>=0.5; extra == 'vllm'
Description-Content-Type: text/markdown

# Crucible

> A verifier-guided reasoning engine: generate many reasoning traces from a small
> open model, score them with a verifier (programmatic checker or a process-reward
> model), and **search** — best-of-N → beam → MCTS over reasoning steps — to turn
> test-time compute into measurable accuracy on math and code.

**Who it's for, and why this and not the obvious thing.** Crucible is for engineers and
researchers who need to *know* — not assume — what test-time search actually buys on a model
they control. The obvious alternative is to call a reasoning model and trust its benchmark
table, or to hand-roll a best-of-N script; the first tells you nothing about *your* policy,
and the second almost always reports a lift it didn't pay for. Crucible is the instrument
instead of the anecdote: it plots accuracy against **total tokens with the verifier's own
compute counted**, reports the **outcome** verifier on the chosen trace (never a PRM score),
puts Wilson CIs and a **named bigger-model baseline** next to every claim — and publishes the
results when search *loses*. It does lose here, twice; that's the point. Design rationale:
[DESIGN.md](DESIGN.md).

![Real MATH-500 accuracy-vs-compute curve](docs/math500-lift-curve.png)

**The headline — real model, real PRM, real data.** 3-seed MATH-500, frozen
`qwen2.5:1.5b-instruct` (Ollama) + a real **Skywork 1.5B PRM** (GPU): search lifts **pass@1
38.3% → 70% (oracle) at N=8**, and on *identical samples* the learned PRM beats
self-consistency majority at every N (N=4: **53.3% vs 45.0%**). The caveats, all measured, none
buried:

- the PRM's ~2× compute makes it **≈ a wash with majority at matched tokens**;
- a **7B baseline (67.5% pass@1 @ 524 tok) is more compute-efficient than 1.5B + search** —
  **small-beats-big fails** on this stack;
- real **beam (0/8)** and **MCTS (1/8)** on the hardest problems are the *most expensive*
  methods and **don't win**.

Full write-up, every number and caveat: **[docs/RESULTS.md §0](docs/RESULTS.md)**.

**Reproduce the headline in one command — no GPU, no network, no model.** It replays the
recorded real runs from committed cassettes:

```powershell
make demo   # no make on Windows? use the line below:
python -m crucible bench curve tests/fixtures/math500-bestofn-seed*.json
```

**Status:** M0–M7 built; the real curves are captured and replay in CI. Behind one
policy/verifier/search interface: **best-of-N** (M2), **PRM selection + the selection gap**
(M3), **PRM-guided beam/DVTS** (M4), a sandboxed **code track** (M5), **MCTS** (M6), the
**compute-optimal report** (M7), plus hardening (H1 ✓, H3 ✓, H4 ✓; **H2 measured — an honest
negative**). Open: a favorable real result for beam/MCTS (M4/M6) and small-beats-big (H2) both
need a stronger PRM / a reasoning policy; M5's real HumanEval run is still unrun. See
[ROADMAP.md](ROADMAP.md) · [PROGRESS.md](PROGRESS.md).

---

## Run it

**Prerequisites:** Python ≥ 3.11 (check: `python --version`). No GPU or network needed
for the M0 demo; real model backends (Ollama, etc.) come in from M1.

> **Naming:** the project is **Crucible**, distributed on PyPI as **`crucible-ttc`**
> (ttc = test-time compute), imported as `crucible`, and invoked as `crucible` on the
> command line (`pip install crucible-ttc` once published; the commands below install
> from source).

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -e ".[dev]"

# Offline demo — generate, verify, and report pass@1 on the bundled sample set:
python -m crucible run --method pass1 --dataset sample --policy mock
```

You should see a per-problem table and a **66.7% (4/6)** pass@1 with a Wilson
confidence interval, and a run record written under `runs/`.

**Real model runs (M1):** install the dataset extra, start [Ollama](https://ollama.com)
and pull a small instruct model, then run real pass@1 on GSM8K:

```powershell
pip install -e ".[datasets]"
ollama pull qwen2.5:1.5b-instruct           # the captured runs' policy model
python -m crucible run --method pass1 --dataset gsm8k --policy ollama `
    --model qwen2.5:1.5b-instruct --limit 20
```

**The real lift curve (H1/H3):** capture it once on a GPU, regenerate it offline forever.
`bench record` samples N×/problem and scores each with the outcome verifier + PRM into a
cassette; `bench curve` computes accuracy-vs-compute for every N/selector from the cassette
(no GPU). Chunk long captures with `--offset`/`--limit` (a crash saves progress and prints
the resume offset), `merge` the chunks, and pool seeds by passing several cassettes:

```powershell
# One line per seed (needs the prm+datasets extras, Ollama, and a GPU):
crucible bench record --dataset math500 --model qwen2.5:1.5b-instruct `
    --prm Skywork/Skywork-o1-Open-PRM-Qwen-2.5-1.5B --max-n 8 --limit 40 --seed 0 --out runs/m500-s0.json
crucible bench curve runs/m500-s0.json runs/m500-s1.json runs/m500-s2.json   # pooled curve, offline

# Reproduce the committed headline with no GPU:
crucible bench curve tests/fixtures/math500-bestofn-seed*.json
```

**The offline lift-curve demo (M2):** the synthetic-policy curve, no model needed:

```powershell
python -m crucible sweep configs/sample-sweep.yaml   # writes runs/sweep-*/curve.png
```

**The PRM selection gap (M3):** compare majority / PRM / oracle selection on the *same*
best-of-N samples (offline, mock PRM):

```powershell
python -m crucible compare   # writes runs/compare-*/comparison.png (oracle >= prm >= majority)
```

**The full search ladder at matched compute (M4/M6):** pass1 vs best-of-N vs beam vs
MCTS on a synthetic stepwise task, offline:

```powershell
python -m crucible sweep configs/beam-sweep.yaml   # one curve, all four methods
```

**The headline report (M7):** the multi-seed ladder with Wilson CIs + the compute-optimal
frontier; interpreted in [docs/RESULTS.md](docs/RESULTS.md):

```powershell
python -m crucible sweep configs/results.yaml   # 3 seeds; curve.png has the dashed frontier
```

**The code track (M5):** execute model-generated code against unit tests in a
locked-down sandbox. Execution is **off by default** — pass `--allow-code-exec`:

```powershell
python -m crucible run --dataset code-sample --policy mock --allow-code-exec   # 2/3
```

### Commands

| Command | What it does |
|---|---|
| `crucible run [...]` | Run one experiment (method × dataset × backend) and report it |
| `crucible report <run_dir>` | Print the metrics from a past run |
| `crucible sweep <config.yaml>` | Grid → the accuracy-vs-compute curve (M2) |
| `crucible bench record/merge/curve` | Capture a real lift curve once (sample N×/problem, score offline), pool seeds, overlay beam/MCTS runs — the headline artifact (H1/H3) |
| `crucible compare` | Majority/PRM/oracle on the same samples → the selection gap (M3) |
| `crucible run --dataset code-sample --allow-code-exec` | Code track: sandboxed execution (M5) |
| `crucible run … --record <path>` | Record a live run to a cassette that replays offline in CI (H3) |
| `crucible version` | Print the version |
| `make demo` · `make check` | The offline real-results demo · lint + typecheck + test |
| `ruff check .` · `mypy src` · `pytest` | Lint · typecheck · tests |

---

## What Crucible can't do

An honest instrument publishes its limits, not just its wins:

- **It won't make a weak policy strong.** On MATH-500's hardest problems the 1.5B policy's
  pass@1 is **0%**; search lifts that to 12–17%, not to competence. A bigger model
  (7B pass@1 **67.5%**) beats 1.5B + search at a fraction of the compute — **small-beats-big
  does not hold here** (it needs a stronger PRM than the 1.5B Skywork).
- **Stepwise search (beam/MCTS) doesn't help a non-reasoning instruct policy.** Asked to
  *continue* a partial trace, `qwen2.5:1.5b-instruct` **restarts** instead — so beam scores
  0/8 at ~37k tokens/problem. Beam/DVTS is a reasoning-policy phenomenon; this stack can't
  show its win.
- **No training.** Policies and PRMs are frozen — no fine-tuning, RL, or self-improvement
  (an explicit v1 non-goal).
- **The code sandbox is a guardrail, not a jail** (subprocess + timeout + no network + scratch
  dir; opt-in). Use Docker/WSL2 for genuinely untrusted code. See
  [ADR-0003](docs/adr/0003-code-execution-sandbox.md).
- **Single node, one GPU.** No multi-GPU or distributed orchestration.
- **Statistics are modest.** Curves pool 40 problems × 3 seeds; the seeds share problems, so
  CIs are mildly optimistic and close margins are suggestive, not decisive. The compute axis
  counts **tokens, not FLOPs** (which flatters small models). Full list:
  [RESULTS §6](docs/RESULTS.md).

`crucible` and `python -m crucible` are equivalent. Optional extras install per
milestone: `".[datasets]"` (M1), `".[prm]"` (M3), `".[vllm]"`.

---

## How to give feedback

When reporting an issue or a run:

- Run the **Test** steps for the current milestone in [ROADMAP.md](ROADMAP.md).
- Describe what happened in plain language; paste any errors verbatim (the single most
  useful thing); include the printed metrics table for a run.

---

## Project docs

| Doc | What's in it |
|---|---|
| [DESIGN.md](DESIGN.md) | The full design and rationale — the single source of truth. |
| [docs/RESULTS.md](docs/RESULTS.md) | The results report — the lift curve, interpreted honestly. |
| [ROADMAP.md](ROADMAP.md) | The milestone checklist (the plan + what's done). |
| [PROGRESS.md](PROGRESS.md) | Build log: what shipped each milestone and why. |
| [`docs/`](docs/) | Long-form docs and architecture decisions (ADRs). |

## Tech stack

Python 3.11+ · **Typer** CLI over **YAML** config · ports-and-adapters core ·
**math-verify** + SymPy (math equivalence) · **httpx** (Ollama backend) ·
pandas + matplotlib (reporting) · pytest · ruff · mypy(strict). Inference backends are
swappable adapters behind one port — `mock`/`synthetic`/`stepwise` (offline, drive the
tests) and **`ollama`** (the real backend used for every captured result); a vLLM/hosted
adapter slots in the same way. PRM scoring (**transformers** + torch) and the HuggingFace
dataset loaders live behind the `prm` and `datasets` extras, so the base install stays
light and the whole test suite runs with no GPU.

## License

MIT — see [LICENSE](LICENSE).
