Metadata-Version: 2.4
Name: cordis-bench
Version: 0.16.2
Summary: Procedural probes of foundational reasoning for spatiotemporal composability
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: litlm>=0.5.0
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"

# cordis-bench

`cordis-bench` benchmarks whether language models understand spatiotemporal
lifecycle/effect composition well enough to reason about application-visible
consequences and use a Cordis harness correctly.

## Current paper target: V2.0.1

V2.0.1 uses **formal + actual Cordis** crossed with **core + challenge**. Within
a realization, core and challenge use the same primary task contracts and
answer formats; semantic size is the intended axis that changes.

| realization | core sizes | challenge sizes | primary tasks |
| --- | --- | --- | --- |
| formal micro-systems | 2, 4 relevant effect groups | 8, 16, 24, 32 | localization, schedule prediction, guaranteed conditions, reachable conditions |
| actual Cordis | 2, 4 queried dependents | 8, 16, 24, 32 | localization, schedule prediction, guaranteed conditions, reachable conditions, reconfiguration |

### V2.0.1 identifier surface

V2.0 used long randomized opaque identifiers such as `plugin_849065282` and
`slot_689631612`. V2.0.1 keeps the identifiers semantically neutral but renders
them as short opaque symbols such as `plugin_a`, `provider_k`, `service_b`,
`slot_q`, and formal capability keys such as `cap_c`.

The naming patch consumes exactly the same RNG draws as V2.0. For a fixed seed,
the semantic construction is therefore preserved: effect programs, numeric
values, interference structure, schedules, targets, task allocation and size
ladder are unchanged apart from alpha-renaming. Because prompts and gold
identifier strings change, V2.0 model predictions are **not** reused as V2.0.1
scores.

### Paper release size

The paper-facing V2.0.1 release is **1,200 exact-output questions** assembled
from three independent 400-item generation blocks with seeds **0, 1, 2**.

| component | one block | three-block release |
| --- | ---: | ---: |
| formal core | 64 | **192** |
| formal challenge | 160 | **480** |
| actual-Cordis core | 80 | **240** |
| actual-Cordis challenge | 96 | **288** |
| **total** | **400** | **1,200** |

The release contains **1,056 primary questions** and **144 challenge-only
outcome-count diagnostics**, generated from **240 independent latent systems**.
The semantic-size ladder remains `2,4 | 8,16,24,32`; scaling is replication,
not a new difficulty axis.

## Generate V2.0.1

```bash
python -m pip install -e '.[dev]'
npm install --prefix native/cordis --ignore-scripts --no-audit --no-fund

# Paper release: seeds 0, 1, 2 -> 1,200 rows.
cordis-bench generate --version 2 --seed 0 \
  --output data/generated/v2.0.1-release.jsonl
```

Run the shortcut audit before freeze:

```bash
python -m cordis_bench.v2_audit data/generated/v2.0.1-release.jsonl \
  --output results/v2.0.1-release-shortcut-audit.json
```

## Probe and score

Use an explicit **8,192-token completion cap** for every V2.0.1 run:

```bash
cordis-bench probe data/generated/v2.0.1-release.jsonl \
  --model google/gemini-3.7-flash \
  --max-tokens 8192 \
  --output results/gemini-v2.0.1-release.jsonl

cordis-bench score data/generated/v2.0.1-release.jsonl \
  results/gemini-v2.0.1-release.jsonl \
  --output results/gemini-v2.0.1-release-score.json

cordis-bench probe data/generated/v2.0.1-release.jsonl \
  --model openai/gpt-5.6-luna --effort low --max-tokens 8192 \
  --output results/luna-v2.0.1-release.jsonl
cordis-bench score data/generated/v2.0.1-release.jsonl \
  results/luna-v2.0.1-release.jsonl \
  --output results/luna-v2.0.1-release-score.json

# Explicit Albert routing is free for authorized Inria accounts and never
# falls back to the paid OpenRouter route. Albert allows 50 RPM; use 35 for margin.
cordis-bench probe data/generated/v2.0.1-release.jsonl \
  --model albert/deepseek-v4-flash --effort low --max-tokens 8192 \
  --workers 8 --rpm 35 --timeout 600 \
  --output results/deepseek-v4-flash-albert-v2.0.1-release.jsonl
cordis-bench score data/generated/v2.0.1-release.jsonl \
  results/deepseek-v4-flash-albert-v2.0.1-release.jsonl \
  --output results/deepseek-v4-flash-albert-v2.0.1-release-score.json
```

Model calls go through `litlm`; `--workers` bounds concurrency and `--rpm`
optionally limits request starts. Each settled success is flushed immediately
to the output JSONL, so rerunning the same command resumes unfinished rows.
Provider-prefixed routes are resolved by `litlm`: `albert/...` uses
`ALBERT_API_KEY`, while `openrouter/...` uses `OPENROUTER_API_KEY`. There is no
cross-provider fallback.

V2 uses structured outputs (`integer`, `scalar_sequence`, `string_set`); there
is no LLM judge and no candidate-option surface. The `primary_metrics` report
uses Jaccard for set tasks, per-observable accuracy for schedule prediction and
executed success for reconfiguration. Exact match and parse rate remain strict
diagnostics.

## Actual Cordis validation

Native tasks execute against pinned `cordis@4.0.0-rc.7`, source commit
`56b3d4f725681cf4556c1a8695a709cc3b6eed74`. Every retained native record must
agree with the finite reference. Reconfiguration answers are executed at
scoring time and must reach the requested target, preserve unrelated state and
use the minimum action count. Independent native cases are sharded across four
isolated Node processes; each process executes its cases serially so Cordis
runtime state is never shared across concurrent cases.

The benchmark is not testing whether Cordis itself is unsafe. Controlled
completion orders expose application-visible consequences of lifecycle/effect
composition so model reasoning can be evaluated.

## V2.0 historical results

The checked-in `data/generated/v2-release.jsonl` and matching Gemini/Luna
release results belong to **V2.0**, with the long numeric identifier surface.
They remain historical evidence and a useful paired comparison, but they must
not be mixed with V2.0.1 rows or scores. V2.0.1 requires fresh evaluation
because the prompts changed.

See `V2.md`, `EVALUATION_V2.md`, and `V1_TO_V2_PAPER_CHANGELOG.md`.

## Historical V1.x

V1.x remains frozen for reproducibility. The old quantitative MCQ helper
derived distractors from the gold, compromising the nominal 25% answer-surface
control for affected questions. This does not invalidate the underlying
semantic worlds or exact oracles. V2 removes candidate answer sets rather than
patching distractor heuristics.
