Stage reference

Every stage: (records, **params) → records over plain list[dict], deterministic, non-mutating, auto-traced when a trace() is active. Contract, not implementation — inputs, outputs, and the invariants held.

select(records, fields)

Keeps only fields on each record, in keep-list order. Missing fields are simply absent (no error, no None padding). Lossless: the dropped tokens were fields you were never going to use. Also the stage that makes rows uniform — which downstream table encodings reward.

dedupe(records, key=None)

Drops duplicates, first occurrence wins, order preserved. key=None compares whole records (key-order insensitive); a field name or list of names compares those fields. Fingerprints are type-qualified — 1, 1.0, True, "1" are four different values — with a canonical-JSON fallback for unhashable values.

rescore(records, query, fields=("snippet",), into="score", proximity=True, match="word")

Writes a deterministic lexical relevance score to into on a copy of each record, so ranking stops inheriting the search tool's score quality. BM25-style, computed inside the batch:

• per-term IDF ln((N−df+0.5)/(df+0.5)+1) — rare terms weigh more;
• saturating term frequency tf/(tf+1.5) — covering all the query's terms beats repeating one;
• match="word" (default) counts exact word occurrences the way BM25/Lucene tokenize — value ≠ values, underscores split snake_case; match="substring" is the forgiving, decoy-prone alternative;
• proximity=True: consecutive query terms in order within 80 characters earn a bonus — the source a query was phrased from usually carries its terms in sequence.

rank(records, by, desc=True)

Sorts stably by a field name or key function; records missing the field go last. Run it before trim_to_budget so the budget drops the least important records, not arbitrary ones. Values of by must be mutually comparable.

truncate_field(records, field, max_tokens, tokenizer=None, suffix="…")

Clips one long text field to the longest prefix that fits the cap, then appends the suffix. For the built-in estimator the cut is closed-form (ceil(len/4) is invertible: 4×budget characters); for custom tokenizers it binary-searches. Non-string values pass through untouched. Never exceeds the cap.

trim_to_budget(records, max_tokens, tokenizer=None)

Keeps the greedy prefix that fits the budget: each record costs the token count of its serialized text plus 2 for list framing; the walk stops before the budget is crossed. Assumes records are already ordered by importance. If even the first record exceeds the budget the result is empty — run truncate_field first for oversized records.

merge(*sources, schema=None, dedupe_key=None)

Unifies differently-shaped tool outputs into one record shape. schema maps a unified field name to a source field name or a list of candidates tried in order; dedupe_key optionally dedupes across the merged result. With schema=None the sources are concatenated unchanged. A combiner rather than a single-collection transform — the one stage that is not auto-traced.

merge(web, docs, schema={"title": ["title", "name", "headline"],
                         "url":   ["url", "link", "href"]})

pipeline · stage · shaped · trace · tokens

• pipeline([...]) — plain left-to-right function composition; any records → records callable (even a lambda) is a valid stage, so you can drop to code for anything irregular and still breakpoint inside it.
• stage(fn, **params) — binds keyword args so a stage slots into a pipeline.
• @shaped(pipe) — runs a tool's record-list return value through a pipeline before it returns; the model never sees raw output.
• trace() — context manager; every stage records records-in/out and tokens-before/after. Inactive overhead is a single ContextVar lookup per stage — within run noise (46.3 vs 43.6 ms over 10k records in RESULTS.md); active tracing token-counts every stage boundary and is priced accordingly.
• tokens — count, use_tiktoken(), set_tokenizer(fn), set_serializer(fn), serialize(value). Strings always bypass the serializer.