Metadata-Version: 2.4
Name: marble-v1
Version: 0.1.0
Summary: Place pieces so a falling marble lands in a ring — an RL environment whose reward is computed by a deterministic physics integrator, not a judge model.
Project-URL: Homepage, https://github.com/p1at0/marble-v1
License-Expression: MIT
Keywords: environment,evals,physics,rl,spatial-reasoning,verifiers
Requires-Python: >=3.11
Requires-Dist: verifiers
Description-Content-Type: text/markdown

# marble-v1

Place pieces so a falling marble lands in a ring. One reply, no tools, no
simulator to poke at — the whole task is predicting a chain of bounces before
any of them happen.

```bash
uv run eval @ eval.toml -m <model> \
  --client.base-url <provider> --client.api-key-var <VAR>

uv run eval @ eval.toml -m <model> --env.taskset.pieces 3   # harder
```

## What makes it different

**The reward cannot be argued with.** There is no judge model and no rubric. A
fixed-step physics integrator runs the layout and reports whether the marble
went through the ring. Reward hacking is not defended against here; it is
impossible.

**Solvable by construction.** Levels are built forwards along the marble's real
path, so a working layout is known before the task is handed out. A model that
fails failed at something possible — which is not true of levels that are
designed and then hoped over.

**Infinite, and difficulty is one number.** `pieces` is the knob. Levels stream
from a seed; there is no dataset to exhaust.

**Cheap to run.** No browser, no VM, no judge calls. A rollout is one model call
plus a few milliseconds of arithmetic — the difference between a training run
being affordable and not.

## The task

A marble rolls off a fixed ramp with a known position and velocity and falls
under gravity. The model is given the ring's position and mouth width, an
inventory of pieces it must use exactly, each piece's size, bounciness and
surface grip, and the placement grid. It replies with a JSON layout.

Five pieces behave differently: a **spring** fires the marble along its own
axis, so rotating one aims it; a **ladder** is a concave arc that cradles;
**drum**, **block** and **bell** are rounded boxes with different bounce. All
placements snap to a 20-unit grid and 7.5-degree rotation steps, so the action
space is integral and an answer is either legal or it is not.

## Rewards

| name | weight | what it says |
|---|---|---|
| `scored` | 1.0 | the marble went through the ring |
| `answered_legally` | 0.0 | a parseable layout obeying grid, strip and inventory |
| `closeness` | 0.0 | 1 through the mouth, 0 a mouth-width outside it |

The two diagnostics carry no weight on purpose. `closeness` is kept out of the
training signal because a model paid for near misses learns to aim near the ring
rather than through it. `answered_legally` exists because "could not format an
answer" and "got the physics wrong" are very different failures that otherwise
produce the same zero.

## Difficulty

Two knobs, and you need both. `pieces` alone is far too coarse — measured, it
goes 50% → 0% between one piece and two, and a cliff gives a training run
nothing to climb. `ring_half_width` is the fine one: a wider mouth converts near
misses into hits without changing what the model has to reason about.

Measured on `google/gemini-2.5-pro`, 10 tasks each, `null` harness:

| pieces | mouth | solved | legal answers |
|---|---|---|---|
| 1 | 118 | **50%** | 100% |
| 2 | 118 | 0% | 100% |
| 2 | 300 | 10% | 100% |
| 2 | 420 | **30%** | 100% |
| 3 | 118 | 0% | 100% |

The mouth knob moves the solve rate 0 → 10 → 30% at a fixed piece count, so any
model can be put in a band where it solves neither everything nor nothing.

`answered_legally` is 100% everywhere: no model in these runs failed to produce
a legal layout. Every zero above is a physics failure, not a formatting one.

## Baselines

Measured on generated levels, no model involved (`calib.py`):

| pieces | random legal layout scores | gold survives a one-step nudge |
|---|---|---|
| 1 | 3.7% | 58% |
| 2 | 1.8% | 51% |
| 3 | 1.4% | 31% |

The first column is the floor a model has to beat to be doing anything — 50% at
one piece is a 13x lift over guessing. The second says the solution is a basin
rather than a needle: the task rewards approximately right physics, not exact
recall.

Widening the mouth does not hand the win to chance. At two pieces, taking the
mouth from 118 to 420 moves the random floor only 1.8% → 6.4%, while it moves
the model 0% → 30%.

## What a rollout costs

The environment's own overhead is a few milliseconds of arithmetic — no browser,
no VM, no judge model. The model is what costs: a reasoning model produces long
chains on these problems, and `gemini-2.5-pro` averaged about $0.16 per rollout
in the runs above. Calibrate with a cheaper model first.

## Provenance

The simulator is a port of the one in a shipped iOS game, where determinism was
the feature rather than a detail: a player who nudges one piece and re-runs has
learned nothing if the same layout can give two answers. It is a fixed-step
integrator with analytic contacts, not a physics engine.

The port is checked against the original: on a five-piece reference layout the
marble's finishing position matches exactly, and intermediate crossings agree to
within one unit. Bit-identity across languages is explicitly not a goal — `sin`
and `cos` may differ in the last place — so reference solutions are generated by
this simulator rather than imported from it.

## Files

| file | what it is |
|---|---|
| `marble_v1/sim.py` | the simulator: colliders, pieces, the physics loop |
| `marble_v1/levels.py` | level generation, solvable by construction |
| `marble_v1/taskset.py` | prompt, answer parsing, rewards |
| `calib.py` | difficulty measured without a model |
| `summarise.py` | aggregate an eval run into the three numbers |
