contexel — deterministic context shaping for code-writing agents

Deterministic, dependency-free context shaping for code-writing agents — the same transform, every run.

The problem it owns

An agent's tools return big lists of records — search hits, file reads, API responses — far too verbose for a context window. Today the cutting-down is usually code the model improvises fresh on every run. That improvisation drifts: different field choices, different truncation, different survivors, run after run. The pipe that feeds the agent is itself a source of nondeterminism.

contexel replaces the improvisation with a small set of vetted, versioned stages applied at the tool → context boundary. Same input, byte-identical context, every run — a context contract instead of a nightly rewrite.

Deterministic

Byte-identical, proven

The benchmark hashes the same contract's output across fresh interpreters with different PYTHONHASHSEED values — same SHA-256 every time, even when raw tool output carries fresh request ids.

Token-aware

Budgets that hold

Stages reason in tokens, not rows. trim_to_budget and truncate_field enforce the limits you set; counting is pluggable (heuristic, tiktoken, your own) and encoding-aware.

Evidence-ranked

rescore, not trust

A BM25-style lexical scorer computed inside the batch derives relevance from the records' own text, so ranking no longer inherits the search tool's score quality.

Zero dependencies

Plain Python, sub-20 ms import

No embeddings, no model calls, no packages. Eight pipeline stages (including provenance and injection-tripwire gates) plus a merge combiner, function composition, an audit-grade opt-in trace — auditable in an afternoon.

The record, at a glance

Measured on the full CodeSearchNet python/test split — 28,047 real search-tool records. The outcome columns (Recall / Compliance / Useful) come from 100 ground-truth episodes at a 1,500-token budget where each library acts only through its own operations; ms @28k is the separate shared-glue canonical task. Details and method on the benchmarks page.

Implementation	Native ops	ms @28k	Recall (strong/weak signal)	Budget compliance	Useful tokens	Deps
contexel	6/6	69	100% / 100%	100%	100%	0
hand-written glue	0/6	27	93% / 32%	100%	100%	—
toolz	2/6	45	100% / 100%	0% (92× budget)	100%	0
langchain-core	1/6	147	30% / 30%	100%	89.8%	25
llama-index-core	0/6	189	100% / 100%	0% (129× budget)	74.2%	60

Read it honestly: contexel is the only row green across every column — and it is the slowest budget-compliant row because it does the most work: deriving relevance from ~1 KB of evidence per record on top of projecting, deduping, clipping, ranking, and budgeting. Every faster row is faster by doing less. All rows are negligible next to a single model call.

Newly landed — toward a context shaping plane

A readiness audit set the bar: pilot-ready is not enterprise-ready. These closed the code-fixable gaps, each verified by tests and adversarial review:

Async

@shaped wraps async tools

A coroutine tool is awaited, then its records shaped — the same contract, sync or async. Previously a TypeError: 'coroutine' object is not iterable.

Multi-tenant

tokens.scoped(…)

Context-local tokenizer/serializer overlay: each async task or tenant counts with its own accounting, others untouched. Bare threads fall back to the process default — stated, not hidden. Costs ~2.4 µs per record on token-counting paths.

Governance

Audit-grade traces

Every pipeline carries a stable policy fingerprint; trace(id_field=…) records which records each stage dropped; t.audit() emits the JSON-able governance record — "did the relevant context even reach the model?" becomes answerable.

Boundary controls

allowlist + quarantine

A fail-closed provenance gate (the strong injection control) and a deterministic tripwire for literal injection markers (the visible one — paraphrase passes it). Both traced; neither claims to be a semantic guardrail.

Supply chain

Attested releases

Publishing runs on PyPI Trusted Publishing (OIDC) with provenance attestations, gated on a human-created tag; a dispatch is always a dry-run. No long-lived tokens.

Exactness

Model-specific counting

tiktoken_tokenizer(model=…) for exact counts where tiktoken covers the model; other providers plug their own counter. Hard token limits demand exact — the built-in estimate is for soft budgeting only.

Sixty seconds of code

from contexel import select, dedupe, rescore, truncate_field, rank, trim_to_budget, trace

with trace() as t:
    r = select(raw, ["title", "url", "snippet", "published"])
    r = dedupe(r, key="url")
    r = rescore(r, query=user_query, fields=("title", "snippet"))
    r = truncate_field(r, "snippet", max_tokens=120)
    r = rank(r, by="score", desc=True)
    r = trim_to_budget(r, max_tokens=1500)

print(t.report())   # what every stage removed — inspectable, not silent

Continue with getting started, or see how the pieces fit.

contexel — the context element