Every crew dispatch resolves its worker model and reasoning effort from flight
config, and the host layer pins the codex backend to one configuration:
gpt-5.6-sol at effort: high. Nothing in current
practice varies it. The committed run ledgers across all mounted repositories
hold 820 completed runs and every one of them ran that single
configuration — 86% gate-passed, median 12.8 worker-minutes for
implement nodes, p90 42 minutes, with individual runs spanning under a minute
to over four hours. One effort tier is serving a hundred-fold spread in task
size and an unmeasured spread in task difficulty.
Committed run durations per role (log scale). Black tick =
median, grey tick = p90. Every point is the same model and effort — the
ledger contains no variation from which a cheaper adequate tier could be
identified.
This monoculture has two costs, and they compound at opposite corners of the
same space:
Effort over-spend. A node whose plan section already
prescribes the change — files named, snippets given, the check to run
stated — leaves little reasoning for the worker to do, yet is billed at the
same effort tier as a node the worker must design from scratch.
Specification over-spend. The orchestrator (a
top-tier reasoning session) pays its own tokens to write prescriptions.
When it writes code snippets and test text into a plan and then dispatches
a high-effort worker anyway, the same reasoning is paid for twice — and an
over-prescribed spec can also over-constrain a worker that would have
found a better shape itself.
The calibration machinery to find the sweet spot already exists —
reckon/capabilities.py derives per-configuration success curves
and a competence horizon from the ledgers, and dispatch refuses nodes beyond
a measured horizon (exit 5) — but it has exactly one configuration to learn
from, and no record of how well-specified any node was. The blocking gap is
therefore not analysis: it is instrumentation (record the
specification level) and variation (dispatch at more than one
configuration).
§2 — What the dispatch stack already provides
The investigation confirmed the mechanics need almost nothing new; the
routing surface is already per-dispatch and provider-neutral:
Layered flight config (reckon/flight.py):
shipped → host → project → override, with per-key provenance.
model and effort are free text on every backend
and role — reckon deliberately fixes no effort ladder.
Per-dispatch override:reckon crew dispatch
--set backends.codex.effort=medium (or any dotted key) wins over
every layer, so the orchestrator can already vary routing per node; the
choice travels as data on the call.
Attribution: every run record stores the exact agent
configuration that ran it (backend, model, effort, sandbox), the gate
verdict, worker seconds, tokens, changed lines and tests added — the full
read-out needed for calibration.
The competence gate: per-configuration success curves
over estimated hours yield competence_horizon_hours; dispatch
refuses a node whose estimate exceeds the measured horizon of the selected
configuration. A configuration with no history is allowed (absence of
evidence is not evidence of incompetence) — which is exactly the opening a
pilot lane needs, and why the pilot needs its own explicit audit fence
(§4).
Backend translation is contained:reckon/_backends.py passes model and effort through to the
codex CLI as -m <model> and
-c model_reasoning_effort=<effort>; any identifier the
account supports works without code change.
What does not exist yet: any recorded signal of specification
completeness on a run, any routing policy connecting specification to
effort, and any small-model backend in config. Availability of the small
codex model was probed live on this host (one trivial read-only prompt per
identifier):
Model identifier
Probe result on this account
gpt-5.3-codex-spark
Supported — responded correctly; accepts model_reasoning_effort without complaint
gpt-5.3-spark
Rejected — 400 "not supported when using Codex with a ChatGPT account"
gpt-5.3-codex
Rejected — 400, same message
codex-spark
Rejected — 400, same message
gpt-5.6-codex-spark
Rejected — 400, same message
§3 — Options considered
O1 — advisory prose only. Add an effort-selection table to
the reckon-ship skill and let the orchestrator judge each node. Zero code
surface, immediately usable — but the judgment leaves no trace on the run
record, so the ledger can never say whether the judgments were right, and
un-recorded policy drifts. Insufficient alone; its table survives as the
advisory layer of O2.
O2 — declared specification level, recorded on the run, routing by
orchestrator override (recommended now). The dispatch gains an
optional declared spec-level (vocabulary in §4) that travels
node → run record → committed ledger → capability observations. Routing
stays an explicit orchestrator --set override guided by an
advisory table. Reckon stays neutral (it maps nothing to nothing); the
ledger accumulates exactly the slices needed to find the sweet spot; the
surface is one optional field plus one recorded key.
O3 — derived specification scoring. Reckon computes
proxies at dispatch (does done-when name a runnable check; code-block count
and word count of the plan section at base; goal length). Honest —
self-graded declarations cannot flatter themselves — but the proxies are
weak individually and the parse surface is real. Deferred: worth adding as
an audit of declared levels once declared data exists to validate
proxies against, not as the primary signal.
O4 — config-mapped automatic routing. A flight-schema
extension mapping spec level → model/effort per role, resolved by
resolve_role. This is the eventual "policy as data" home, but
locking a mapping before any measurement exists would encode priors as
config. Deferred behind the §6 checkpoint.
O5 — reckon chooses effort itself. Rejected. The flight
schema states the principle explicitly: an effort ladder must not be fixed
by reckon, and silent rerouting is the failure mode the provenance layer
exists to prevent. Routing decisions are the orchestrator's, expressed as
data.
§4 — Design: specification levels, routing, and the spark lane
The specification level names who owns the design. It is a
property of the node's plan section, declared at dispatch:
Level
The plan section provides
The worker owns
exact
The prescribed change: files, snippets or exact steps, and the named check to run
Transcription and verification. A prescription that fails its own check is a blocker to report, not a licence to redesign.
guided
The fixed design: interfaces, invariants, where things live, and the measure
Deriving the implementation within that design.
open
The goal, constraints, and the measure
The design and the implementation.
The routing space. Off-diagonal corners are the two waste modes:
exact spec + high effort is double-paid reasoning; open spec + low effort is
an under-powered worker. The spark lane is the boxed corner — entered only
through the gate below.
Choose the level first, by ownership — never inflate it to enable a
cheaper worker. If the design is genuinely settled (a mechanical
migration, a repeated pattern, a change this plan already had to reason out
for other reasons), write it exactly and route down. If it is not settled,
do not spend orchestrator tokens writing code into the plan: declare
guided or open and route the reasoning to the worker. Raising the
specification level pays orchestrator reasoning for worker reasoning at an
unfavourable exchange rate unless the exact spec amortises over several
nodes (one template, many dispatches) — that amortised case is the spark
lane's natural habitat.
Advisory routing table (initial priors — §6 calibrates them; the
choice always travels as an explicit --set override on the
dispatch call):
Declared level
First-choice routing
Why
exact
spark lane (via its gate), else sol at medium
The reasoning is already in the spec; the worker executes and verifies.
guided
sol at medium for small nodes, high above ~1 worker-hour
Implementation reasoning remains; design reasoning does not.
open
sol at high; xhigh for cross-cutting single-owner nodes
The worker carries design and implementation.
The spark lane — scope fence. Dispatching
gpt-5.3-codex-spark is permitted only when all of the
following hold. The lane exists to buy speed and budget on work whose
correctness is already pinned by its specification and its check — never to
write uncertain code quickly.
Declared level is exact, and the plan section actually
prescribes the change with a named check — a declaration the section does
not support is a specification bug to fix before dispatch.
The done-when is a runnable named check (a test command, a command whose
output is compared, or a numeric bound) — prose measures disqualify.
Node estimate ≤ 0.5 worker-hours until spark has its own measured
competence horizon; thereafter the existing competence gate governs.
No requires-decision keys — nothing left open that the
worker could resolve by guessing.
Role is a mechanical class (implement, test,
documentation). Judgment roles
(review, investigate) never route to the
lane.
Pilot audit: for the first twenty spark runs the
orchestrator audits the full diff, not only the manifest and gate. A gate
failure on an exact-tier node indicts the specification before the model —
reclassify the node's level first, then judge the lane.
Circuit breaker: two consecutive spark gate failures
close the lane until the failing specifications have been re-examined and
the closure decision revisited on this plan.
Model identifiers and effort vocabularies remain provider data in flight
config; reckon source names none of them. The spark backend is defined in
this repository's project flight layer
(docs/state/reckon/flight.yaml) and is routed to only by
explicit override — no role selects it by default.
§5 — Prototype (this session)
✓ landed 2026-08-22
All three subsections shipped, executed under the routing they propose.
§5.1 at sol/medium: gate passed, 173/173 tests, 8 added, 266 worker-seconds (commit dbb81e0).
§5.2 down the spark lane at exact tier: gate passed on full-diff pilot audit, 63 worker-seconds (commit 2a10c38).
§5.3 config + policy landed (6767cd5); spark dry-run resolves end-to-end; capability cache now tracks 4 configurations.
Full record: crew-effort-routing-landed.
The prototype changes no default behaviour: the field is optional, the spark
backend is defined but not routed to, and routing overrides remain explicit
per-dispatch data. It is deliberately executed under the policy it
proposes: §5.1 is a guided node dispatched at reduced effort, §5.2 is an
exact node dispatched down the spark lane under the pilot audit — together
producing the ledger's first varied-configuration rows. The subsection
contracts below are retained as the reference record of what the workers
were dispatched against.
§5.1 — Record a declared specification level end-to-end (guided)
Design (fixed): a declared level travels dispatch flag →
node → run pointer → committed ledger record.
TaskNode (reckon/crew.py) gains
spec_level: str = "", serialised by as_dict().
Accepted values: empty (undeclared), exact,
guided, open. An unknown value is refused in
plan_dispatch beside the node-id shape check — fail fast,
before any side effect, naming the accepted vocabulary.
compose_prompt emits one additional line after
ROLE when a level is declared, telling the worker what it
owns (per the §4 table): exact → implement as written and run the named
check, deviation is a blocker to report; guided → the plan fixes the
design, derive the implementation; open → the plan fixes goal and
measure, design and implement. No line when undeclared.
ledger.build_record (reckon/ledger.py) gains
spec_level: str = "", stored as a top-level record field and
added to RECORD_FIELDS; complete()
(reckon/crew.py) forwards it from the pointer's node dict.
Tests in tests/test_crew.py and
tests/test_ledger.py: vocabulary refusal, node
serialisation, prompt line present/absent, record field survives
build_record, completion forwards the level from a pointer.
Done-when:uv run pytest tests/test_crew.py
tests/test_ledger.py passes with the new coverage present.
§5.2 — Capability observations carry the level (exact)
Prescription: in
reckon/capabilities.py::derive_capabilities, the observation
dict appended for each usable run gains one key after
"changed_lines":
"spec_level": str(run.get("spec_level") or "") or None,
Add one test to tests/test_capabilities.py using the existing
_project / _plan fixtures and
ledger.append_run + ledger.build_record directly:
one run built with spec_level="exact", one without; assert the
derived observations carry "exact" and None
respectively.
Done-when:uv run pytest
tests/test_capabilities.py passes including the new test.
Sequencing: depends on §5.1 (the build_record
parameter); dispatched only after §5.1 is verified and integrated.
§5.3 — Config and policy surfaces (coordinator-owned)
Project flight layer docs/state/reckon/flight.yaml defining
the codex-spark backend (command codex, model
gpt-5.3-codex-spark, worktree-full sandbox, short time
budget). Defined, not defaulted: default_backend is
untouched and no role selects it.
Advisory policy reference
skills/reckon-ship/references/effort-routing.md carrying the
§4 tables and the spark gate, with a one-line pointer from the dispatch
section of skills/reckon-ship/SKILL.md.
A recorded --dry-run dispatch demonstrating spark-lane
resolution end-to-end (routing, provenance, competence verdict) without
spawning a worker.
§6 — Finding the dispatch sweet spot: trial protocol
With the level recorded and routing varied, the sweet spot is a measurement,
not a debate. The loop:
Instrument — §5 lands: every dispatch may declare a
level; the ledger keeps it beside the agent configuration it ran on.
Vary — orchestrators route by the §4 advisory table
(guided/open at medium vs high by size; exact
down the spark lane under its gate). Assignment follows the table rather
than randomisation; at the current run rate (hundreds of runs per month
across mounted repos) the slices populate within weeks, and the table
itself is the thing under test.
Read out per (configuration × declared level × size
bucket): gate pass rate, median worker minutes, tokens, and redispatch
rate (lineage attempts — a cheap tier that fails and redispatches to a
higher one has to be charged both runs).
Decision rule — the sweet spot for a slice is the
cheapest configuration whose pass rate ≥ 0.8 (the capabilities success
threshold) over at least 10 usable runs, ties resolved by fewer total
worker-minutes including redispatches. A slice with no qualifying cheap
configuration keeps the current default.
Checkpoint — after roughly 50 varied-configuration
runs, review the slices on this plan: lock the winning mapping into
flight config as role overlays (option O4 graduates from deferred), widen
or close the spark lane per its own numbers, and decide whether derived
spec scoring (O3) is worth adding as an audit of declarations.
The over-specification side has no direct ledger counter (orchestrator
tokens are not recorded per plan section), so it is controlled by rule
rather than by measurement: the §4 ownership test decides the level, and the
checkpoint reviews whether exact-tier specs are amortising (several
dispatches per spec) or being written one-off for single nodes — the
latter is the double-paid corner and the signal to stop.
§7 — Config-mapped routing: one dial, wired in flight config
✓ landed 2026-08-22
Shipped at guided/sol medium: gate passed, 204/204 focused tests (8 new),
schema regen byte-identical (commit b625b88). Mapping live in the
project flight layer; dry-run with no override resolves exact → spark @ 15m,
guided → medium, open → high, undeclared unchanged.
Full record: §7 landed.
With routing-mode locked to config-mapped, the orchestrator
turns exactly one dial per dispatch — --spec-level — and the
flight config resolves the backend, model, provider-effort and time budget
underneath it as data. The --set override remains the
per-dispatch escape hatch (the override layer rewrites the mapping itself,
so it still wins over every layer). The design is fixed; the worker derives
the implementation.
Schema (LinkML source reckon/schema/flight.yaml):
RoleConfig gains an optional by_spec_level of a
new class with three optional slots — exact,
guided, open — each holding a routing overlay
class with optional backend, model,
effort, time_budget. Fixed slots, not a keyed
map, so the level vocabulary validates for free. Regenerate
reckon/_flight_schema.py and
docs/_shared/flight.schema.json with
scripts/regen_flight_schema.py; hand edits to generated
files are refused by their own headers.
Resolution (reckon/crew.py):
resolve_role(config, role, spec_level=""). Backend selection
precedence: overlay backend > role backend
> default_backend. Effective settings merge backend
settings, then the role overlay (excluding name,
backend, by_spec_level), then the matching
level overlay (excluding backend). An empty level leaves
behaviour byte-identical to today. plan_dispatch passes
node.spec_level; the existing fail-fast vocabulary check
stays ahead of resolution.
Whole-config validation (reckon/flight.py):
every backend a by_spec_level overlay names must exist in
the merged backends, reported with the full key path like
the existing role-backend rule.
Pre-flight fan-out (reckon/budget.py):
backends_for_roles returns every backend a role can reach —
its base backend plus each by_spec_level overlay backend —
deduplicated, so a budget hold on any reachable backend is visible
before a wave opens.
Tests: schema acceptance of a by_spec_level
layer and rejection of an unknown level key; resolution precedence
(backend switch, effort-only overlay, empty level no-op, --set
override rewriting the mapping); resolved-validation error naming a
missing overlay backend; preflight fan-out including overlay backends.
Landing this section makes the reckon project mapping live:
roles.implement.by_spec_level routing exact to
the spark backend (with its own time budget) and guided /
open to reduced / full effort on the default backend — applied
in docs/state/reckon/flight.yaml only after the code merges,
because every layer is schema-validated on read.
§ Decisions
Is the three-level declared vocabulary (exact / guided / open) the right specification signal?
The prototype implements the enum as an optional field — reversible, nothing depends on it yet. Alternatives: free text (unmeasurable), derived-only scoring (weak proxies, no declaration to audit). §4 defines the levels by design ownership.
How does a declared level select routing?
O2 (orchestrator applies the advisory table as an explicit --set override, ledger measures) versus O4 (flight-schema mapping resolves it automatically). O4 is deferred behind the §6 checkpoint by default; locking it now would encode unmeasured priors as config.
Lead decision 2026-08-22: one dial per dispatch. The orchestrator declares only --spec-level; flight config resolves backend, model, provider-effort and time budget underneath it as data (roles.<role>.by_spec_level), keeping --set as the per-dispatch escape hatch. The dial stays named by spec completeness rather than effort so a wrong declaration remains checkable against the plan section and the ledger.
Does the spark lane open under the §4 gate?
gpt-5.3-codex-spark is confirmed available on this account. The gate: exact level with a runnable named check, ≤ 0.5 h estimate, no open decisions, mechanical roles only, first twenty runs full-diff audited, two consecutive gate failures close the lane. §5.2 runs one pilot dispatch under exactly these conditions.
Where does this plan sit in the roadmap — which sprint, or explicitly backlog?
The §5 prototype executes in the authoring session; §6 is an ongoing measurement loop with a checkpoint after ~50 varied runs. Deliberately left sprint-unassigned pending this decision rather than silently unscheduled.
Do we measure spark with shadow sidecar runs on live nodes?
A shadow run dispatches the same node to the spark backend in its own worktree, runs the gate, records the outcome, and never merges - paired per-node evidence across all spec tiers with zero code pollution. Risks and mitigations: ledger ambiguity -> records carry lineage kind shadow naming the primary run, and calibration slices label them; cleanup reachability -> the shadow's evidence is its diff artifact and test log in the durable run directory, not a merged commit; spend -> shadow a sample (the spark side costs roughly a minute and ~9k output tokens per node). Evidence rule: shadow evidence qualifies a tier for the gated pilot; only live gated evidence opens routing.
§ Followups
Land the §5 prototype: spec-level recording, capability slice, config and policy surfaces
Execute §5.1 and §5.2 as crew nodes under the routing they demonstrate (guided at reduced effort; exact down the spark lane with full-diff audit), then land the §5.3 coordinator surfaces and record the dry-run evidence.
/reckon-ship crew-effort-routing §5
Landed. §5.1 spec-level recording at sol/medium: gate passed, 173/173 focused tests, 8 added, 266 worker-seconds, commit dbb81e0 (merged 4d3f3d2). §5.2 capability slice down the spark lane (gpt-5.3-codex-spark, exact tier, full-diff pilot audit): gate passed, 14/14 tests, 63 worker-seconds, commit 2a10c38 (merged 1406b50). §5.3 flight layer + effort-routing reference landed in 6767cd5; spark dry-run resolves end-to-end from merged main. Capability cache rebuilt: 4 configurations tracked, observations carry spec_level. Full suite 1961 passed. Evidence: /reckon/evidence/archive/crew-effort-routing-landed.
§6 checkpoint: read the calibration slices and lock the routing table
Trigger: roughly 50 varied-configuration runs in the committed ledgers (dispatches declaring --spec-level and routed by the §4 advisory table). Read pass rate, worker minutes, tokens, and redispatch lineage per (configuration × declared level × size bucket); apply the §6 decision rule; then lock or amend the four open decisions (spec-vocabulary, routing-mode, spark-lane, scheduling) from measured slices rather than priors.