ContextPull · pull, don’t push
ContextPullDemoDocsragbisectGitHub
ragbisect · the companion benchmark harness

Bisect a RAG pipeline to find the broken stage.

Other tools score a pipeline end to end and tell you it is bad. ragbisect builds an eval set from your own corpus, scores retrieval and ranking separately, compares your pipeline with bm25, dense and hybrid on the same questions, and names the stage that is losing the most. It works on any pipeline and has no dependency on ContextPull.

A diagnostic tool that gives different answers on re-run is not a diagnostic tool. Everything is seeded and cached.
PyPI ci license
GitHubPyPI
Wrap your pipeline in one method
class MyRetriever:
    def retrieve(self, query: str, k: int) -> list[str]:
        """Return chunk IDs, most relevant first."""
Run
uvx ragbisect run --corpus ./docs --adapter ./my_pipeline.py:MyRetriever
the eval set builds itself

Five question shapes, because most eval sets only test the first.

conceptual

“What’s our approach to refunds?” The one shape dense retrieval handles well. Generated from a chunk by a model.

exact_lookup

“What does error TX-4419 mean?” Embeddings blur identifiers. Generated only when a chunk explains one.

comparison

“What changed between the 2024 and 2025 policy?” Cross-document near-duplicate pairs; gold is both chunks.

aggregation

“How many TX-44xx codes are documented?” Identifier families counted programmatically. No model.

table

“Storage temp for the R-40?” One cell of a pipe table, only where the row key is unambiguous. No model.

Shapes a corpus cannot support are skipped and the report says why, rather than generating junk questions.

what it prints

A stage table and a one-line verdict. Nothing else.

config                              recall@5    mrr@5   ndcg@5|hit    faith     n     ms/q    tok/q calls/q
-----------------------------------------------------------------------------------------------------------
your adapter                           0.918    0.768        0.878      n/a   207        4        —       —
  conceptual                             0.943    0.806        0.891      n/a   123        4
  exact_lookup                           0.859    0.696        0.858      n/a    84        3
bm25 (built-in)                        0.918    0.768        0.878      n/a   207        2        —       —
dense (built-in)                       0.903    0.719        0.847      n/a   207      340        —       —
hybrid dense+bm25 rrf (built-in)       0.952    0.818        0.895      n/a   207       61        —       —

Bottleneck for 'your adapter': ranking — when the gold chunk is retrieved it ranks at NDCG 0.88; MRR@5 is 0.77. Weakest shape: exact_lookup (recall 0.86, n=84).
'hybrid dense+bm25 rrf (built-in)' would raise recall@5 from 0.92 to 0.95 (+0.03).
spend: 0 API calls; 1431 cache hits; 0 completion tokens; ≈ $0.0000

Retrieval is recall@k: did the gold chunk appear at all. Ranking is NDCG conditioned on a hit: given it was retrieved, how near the top. The stage furthest from its ceiling is the bottleneck. Cost columns show wall time for every config and model tokens and tool calls when an adapter reports them, which is how agentic configurations get compared with push pipelines on cost as well as recall.

Optional adapter extras

  • generate(query, chunk_ids) to have faithfulness judged by a model.
  • stats() returning tokens, tool calls and dollars, for the cost columns.
  • concurrency = N to allow parallel queries.
  • --sample N for expensive adapters; --shapes to pick question shapes.

Engineering choices

  • Zero runtime dependencies; installs alongside your pipeline without version conflicts.
  • One provider-agnostic LLM wrapper; OpenAI-compatible or Anthropic endpoints.
  • Every model call cached in SQLite; re-runs are free until the corpus changes.
  • Prints what a run spent in tokens.