Architecture

contexel lives at exactly one place: the boundary where a tool's raw output becomes the model's context. Everything below is that one idea, mechanized.

The pipeline — click any stage

A record is a plain dict; a collection is list[dict]. Every stage is (records, **params) → records, never mutates its input, and composes with the rest:

raw tool output →

Two layers, one policy

At the tool boundary

@shaped — the model never sees raw

Wrap the tool with a named pipeline. Every call returns bounded, reshaped records — the same lens on the repo whether it is step 1 or step 100. This is the context contract: versioned beside the code, changed only by intention.

Inside agent code

Inline — for what a given step needs

The sandbox imports the stages, so when the model writes code it composes vetted transforms instead of improvising string surgery. Both layers stack; only the distilled result re-enters context.

@shaped([                                # the contract, pinned at the boundary
    stage(select, fields=["path", "line", "snippet", "score"]),
    stage(dedupe, key=["path", "line"]),
    stage(trim_to_budget, max_tokens=2000),
])
def search_code(query: str) -> list[dict]:
    return repo.search(query)            # raw, heavy — shaped identically, every call

Try it — the budget, live

A faithful miniature of select → dedupe → rank → trim_to_budget running in your browser on twelve search-hit records (three are duplicates, all carry noise fields). Records are priced the way the library prices them — ≈4 chars/token on the serialized record, +2 framing per record.

select — drop noise fields (lossless) dedupe — drop duplicates (lossless)

Token budget: 220

Token accounting — two pluggable axes

Tokenizer

How text is priced

Default: a dependency-free ~4-chars/token estimate. tokens.use_tiktoken() for exact counts, set_tokenizer(fn) for your own. Measured accuracy per content type is in the benchmarks.

Serializer

What encoding is priced

Budgets are encoding-relative: a record costs the tokens of its serialized text. Canonical JSON by default; set_serializer(fn) prices records in the encoding your boundary actually emits — e.g. an ISON table, where the same budget holds more records.

The reference agent loop

The repo ships a complete code-execution agent (python -m reference_agent, no API key) that closes the loop:

model proposes code→ sandbox runs it (tools are @shaped)→ only distilled result enters context→ repeat until answer

Its tests prove the headline guarantee: two runs build byte-identical context even though the raw tool layer returns fresh request ids on every call.