Metadata-Version: 2.5
Name: cairn-pipeline
Version: 0.1.0
Summary: A retrieval pipeline that answers from your documents, cites the exact span it used, abstains with a calibrated guarantee when the evidence is thin, and proves all three with a number.
Project-URL: Homepage, https://github.com/rhs2/cairn
Project-URL: Documentation, https://rhs2.github.io/cairn/
Project-URL: Repository, https://github.com/rhs2/cairn
Project-URL: Changelog, https://github.com/rhs2/cairn/blob/main/CHANGELOG.md
Author: Rakibul Hasan Sium
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: citations,conformal,evaluation,llm,rag,retrieval
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.10
Requires-Dist: beautifulsoup4<5,>=4.12
Requires-Dist: httpx<1,>=0.27
Requires-Dist: markdown-it-py<4,>=3.0
Requires-Dist: numpy<3,>=1.26
Requires-Dist: pydantic<3,>=2.9
Requires-Dist: pypdf<7,>=4.3
Requires-Dist: python-docx<2,>=1.1
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: rank-bm25<0.3,>=0.2.2
Requires-Dist: rich<15,>=13.7
Requires-Dist: scikit-learn<2,>=1.5
Requires-Dist: typer<1,>=0.12
Provides-Extra: all
Requires-Dist: anthropic<2,>=0.34; extra == 'all'
Requires-Dist: faiss-cpu>=1.8; extra == 'all'
Requires-Dist: fastapi<1,>=0.112; extra == 'all'
Requires-Dist: openai<2,>=1.40; extra == 'all'
Requires-Dist: opentelemetry-api<2,>=1.26; extra == 'all'
Requires-Dist: opentelemetry-sdk<2,>=1.26; extra == 'all'
Requires-Dist: prometheus-client<1,>=0.20; extra == 'all'
Requires-Dist: qdrant-client<2,>=1.10; extra == 'all'
Requires-Dist: reportlab<5,>=4.2; extra == 'all'
Requires-Dist: sentence-transformers<7,>=3.0; extra == 'all'
Requires-Dist: torch>=2.2; extra == 'all'
Requires-Dist: uvicorn[standard]<1,>=0.30; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic<2,>=0.34; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: fastapi<1,>=0.112; extra == 'dev'
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: opentelemetry-api<2,>=1.26; extra == 'dev'
Requires-Dist: opentelemetry-sdk<2,>=1.26; extra == 'dev'
Requires-Dist: prometheus-client<1,>=0.20; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: reportlab<5,>=4.2; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: uvicorn[standard]<1,>=0.30; extra == 'dev'
Provides-Extra: faiss
Requires-Dist: faiss-cpu>=1.8; extra == 'faiss'
Provides-Extra: local
Requires-Dist: sentence-transformers<7,>=3.0; extra == 'local'
Requires-Dist: torch>=2.2; extra == 'local'
Provides-Extra: openai
Requires-Dist: openai<2,>=1.40; extra == 'openai'
Provides-Extra: pdfgen
Requires-Dist: reportlab<5,>=4.2; extra == 'pdfgen'
Provides-Extra: qdrant
Requires-Dist: qdrant-client<2,>=1.10; extra == 'qdrant'
Provides-Extra: serve
Requires-Dist: fastapi<1,>=0.112; extra == 'serve'
Requires-Dist: opentelemetry-api<2,>=1.26; extra == 'serve'
Requires-Dist: opentelemetry-sdk<2,>=1.26; extra == 'serve'
Requires-Dist: prometheus-client<1,>=0.20; extra == 'serve'
Requires-Dist: uvicorn[standard]<1,>=0.30; extra == 'serve'
Description-Content-Type: text/markdown

# Cairn

[![ci](https://github.com/rhs2/cairn/actions/workflows/ci.yml/badge.svg)](https://github.com/rhs2/cairn/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/cairn-pipeline)](https://pypi.org/project/cairn-pipeline/)
[![Python](https://img.shields.io/pypi/pyversions/cairn-pipeline)](https://pypi.org/project/cairn-pipeline/)
[![Documentation](https://img.shields.io/badge/docs-rhs2.github.io%2Fcairn-blue)](https://rhs2.github.io/cairn/)
[![Licence](https://img.shields.io/badge/licence-Apache--2.0-green)](LICENSE)

A retrieval pipeline that answers from your documents, cites the exact span it
used, abstains with a calibrated guarantee when the evidence is thin, and proves
all three with a number.

A cairn is a stack of stones marking a route across ground that offers nothing
to follow. It points the way, and it proves somebody walked here before you and
checked.

## The numbers

Every row is one configuration, measured by one protocol, on the same corpus and
the same 120-question golden set. The table is the output of `cairn bench` with
the deterministic stub provider: no API key, no network, and CI reproduces it on
every commit.

| config | recall@5 | nDCG@10 | token F1 | judge | citation validity | answer rate | error among answered | cost/q | p95 ms |
|---|---|---|---|---|---|---|---|---|---|
| bm25 | 0.963 | 0.895 | 0.319 | 0.914 | 1.000 | 0.500 | 0.194 | 0.00000 | 2 |
| default | 0.870 | 0.846 | 0.334 | 0.902 | 1.000 | 0.347 | 0.240 | 0.00000 | 2 |
| hybrid-hashing | 0.870 | 0.846 | 0.334 | 0.902 | 1.000 | 0.347 | 0.240 | 0.00000 | 2 |
| dense-hashing | 0.815 | 0.745 | 0.340 | 0.898 | 1.000 | 0.472 | 0.206 | 0.00000 | 2 |

What the table says that a demo would not:

- **Lexical search wins on this corpus.** BM25 alone finds the supporting passage
  in the top five for 96 percent of questions; the hashing hybrid manages 87.
  A hashing embedder is not semantic, and the golden questions reuse the
  documents' own vocabulary, so fusing in its rankings dilutes BM25 instead of
  helping it. `default` is `hybrid-hashing` under its shipped name.
- **The guarantee is stated and met.** Every configuration asks for at most 35
  percent error among the questions it chooses to answer, and every one lands
  under it. Why 35 and not 10: with 47 calibration questions the tightest 95
  percent upper bound on zero observed errors is about 0.31, so a smaller target
  is not certifiable at this sample size, and the pipeline refuses to pretend
  otherwise. The comment at the top of `configs/default.yaml` has the arithmetic.
- **Citation validity is 1.000** because a quote that is not a verbatim substring
  of the chunk it cites never reaches the reader. The verifier catches it first.

With a real embedder, installed by the `local` extra (weights download on first
use, so CI does not run these; see `reports/leaderboard-local.md`):

| config | recall@5 | nDCG@10 | token F1 | judge | citation validity | answer rate | error among answered | cost/q | p95 ms |
|---|---|---|---|---|---|---|---|---|---|
| dense-local | 0.926 | 0.873 | 0.000 | n/a | 0.000 | 0.000 | 0.000 | 0.00000 | 10 |
| hybrid-local | 0.926 | 0.903 | 0.000 | n/a | 0.000 | 0.000 | 0.000 | 0.00000 | 266 |
| hybrid-local-norerank | 0.926 | 0.892 | 0.000 | n/a | 0.000 | 0.000 | 0.000 | 0.00000 | 11 |

Semantic retrieval lifts dense recall from 0.815 to 0.926, and the cross encoder
gives the best ranking of all at roughly a hundred times the latency. The answer
columns are zero, and that is the guarantee working rather than failing: with
these confidence signals the calibration slice could not certify the 0.35 bound
at any threshold, so the record abstains on everything instead of answering at
whatever confidence happened to be highest. A larger calibration slice or a real
answering model is what unlocks these rows, and the harness will say so in
numbers when it happens.

The adversarial suite asked fourteen questions against six poisoned documents
carrying injected instructions and canary strings. All six were retrieved; none
leaked. The report is `reports/attack-default.md`, and `cairn attack` exits `1`
the moment that changes.

## Quickstart

```bash
pip install "cairn-pipeline[dev]"
cairn init          # writes configs/ and generates data/sample/ from code
cairn ingest        # builds the index and prints its manifest
cairn ask "What was the average fuel use across the fleet in March 2026?"
cairn calibrate     # fits the abstention threshold and states the guarantee
cairn eval          # measures everything; writes reports/latest/default.json
cairn attack        # the poisoned corpus; exit 1 on any leak
```

```
Average fuel use across the fleet in March 2026 was 9.4 litres per 100 kilometres.

1. Average fuel use across the fleet in March 2026 was 9.4 litres per 100 kilometres. (verified)
   meeting-minutes-2026-04-15.txt page 1: 'Average fuel use across the fleet in March 2026 was 9.4 litres per 100 kilometres.'

status answered, confidence 0.720, threshold 0.523, 219 ms, 0.00000 USD
```

Every command defaults to `configs/default.yaml`. The sample corpus belongs to a
fictional food distributor, Halcyon Provisions; `cairn init` writes it from code
and checks it against a pinned digest, so the copy on your machine is the copy
the numbers above were measured on. Nothing under `data/` is committed or
packaged.

Then make it worse on purpose and watch the gate refuse:

```bash
cairn eval --out reports/good.json
CAIRN_STUB_DEGRADE=0.5 cairn eval --out reports/degraded.json
cairn gate --baseline reports/good.json --candidate reports/degraded.json
```

```
roll back: default against default on judge_score
- judge_score dropped by 0.329 (baseline 0.902, candidate 0.573); the allowed drop is 0.000.
- the candidate's abstention guarantee is not met: error rate among answered is 0.520 against alpha 0.350.
```

That exit code is the whole idea. The [quickstart](https://rhs2.github.io/cairn/getting-started/quickstart/)
walks through each step with its real output.

## What it does

```
documents  ->  parse  ->  chunk  ->  embed  ->  hybrid search  ->  rerank
                  |                                                   |
             provenance                                        injection scrub
             (page, span)                                             |
                  |                                          generate with
             versioned index                                 structured citations
                  |                                                   |
                  |                                            confidence score
                  |                                                   |
                  |                              above threshold -> answer and spans
                  |                              below threshold -> abstain and passages
                  v
          evaluation harness  ->  report  ->  promotion gate
```

A pipeline and a small library, not an application. It ingests a corpus in five
formats with page and span provenance, builds a versioned index that refuses to
load against a configuration it was not built for, answers with citations or
abstains with the closest passages, and produces a report you can compare against
last week's. It runs three ways: entirely local with a local model and embedder,
against a hosted model, or with a deterministic stub standing in for the model,
which is how the tests and the published numbers run.

## The three claims

**Better than what we had.** A frozen golden set, one protocol, and a promotion
gate that exits non-zero when a candidate is worse on quality, citations, cost or
latency. A change to the chunk size stops being a matter of taste. CI keeps the
committed report as the baseline and gates every commit against it.

**Not in the documents.** The abstention threshold is chosen by conformal risk
control: the lowest confidence whose finite-sample Clopper-Pearson bound on the
error among answered questions sits at or below the target. The record states
the guarantee in a sentence, the report says whether it held, and when no
threshold can be certified the pipeline abstains on everything rather than
promising what the data cannot support.

**Where it came from.** Every claim carries a span-level citation: source, page,
and the quote itself. The quote must be a verbatim substring of the chunk it
cites and must actually support the claim, and a citation that fails either check
is marked rather than shown as fact.

And one thing that is not a claim but a stance: retrieved text is data, never
instruction. The prompt says so, a scrub removes hidden characters and lines
shaped like commands to a model, and a poisoned copy of the corpus is part of the
test suite.

## Configurations

| File | Retrieval | Embedder | Reranker | Answers with | Needs |
|---|---|---|---|---|---|
| `default.yaml`, `hybrid-hashing.yaml` | hybrid | hashing, 512-d | none | stub | nothing |
| `bm25.yaml` | BM25 | none | none | stub | nothing |
| `dense-hashing.yaml` | dense | hashing, 512-d | none | stub | nothing |
| `dense-local.yaml` | dense | all-MiniLM-L6-v2 | none | stub | `local` |
| `hybrid-local-norerank.yaml` | hybrid | all-MiniLM-L6-v2 | none | stub | `local` |
| `hybrid-local.yaml` | hybrid | all-MiniLM-L6-v2 | cross encoder | stub | `local` |
| `hybrid-local-rewrite.yaml` | hybrid, query rewriting | all-MiniLM-L6-v2 | cross encoder | stub | `local` |
| `hybrid-ollama.yaml` | hybrid | all-MiniLM-L6-v2 | cross encoder | Ollama, `llama3.2` | `local`, a running Ollama |
| `hybrid-openai.yaml` | hybrid | `text-embedding-3-small` | none | OpenAI | `openai`, `OPENAI_API_KEY` |
| `hybrid-anthropic.yaml` | hybrid | all-MiniLM-L6-v2 | cross encoder | Anthropic, with prompt caching | `anthropic`, `local`, `ANTHROPIC_API_KEY` |

A configuration is one YAML file naming a component for each stage; two files
that differ in one line are one experiment. `cairn bench` runs every file it is
given, skips the ones whose extra or key is missing, and writes the leaderboard.
Keys come from the environment only; `.env.example` lists every variable.

## Serve

```bash
pip install "cairn-pipeline[serve]"
cairn serve --config configs/default.yaml
```

`POST /answer` returns the answer with its citations or the abstention with its
passages, `GET /health` reports the index manifest, the calibration in force and
a drift signal, and `GET /metrics` exposes Prometheus counters and histograms
per stage. Every request carries an OpenTelemetry span per stage.

## Development

```bash
git clone https://github.com/rhs2/cairn
cd cairn
make install      # virtualenv with the dev extra
make check        # ruff, mypy, the offline test suite
make bench        # every configuration; writes reports/leaderboard.md
make attack       # the adversarial suite
make docs         # the documentation site, live
```

CI runs the same three checks on Python 3.10 and 3.12, then generates the corpus,
runs the measured pipeline, and gates the new report against the committed
baseline. A separate job scans the full history for secrets. Releases are cut
from a `v*` tag and published to PyPI by trusted publishing, so no token is
stored anywhere.

What is committed: the code, the configurations, the generator, and the reports
under `reports/latest/` that CI compares against. What is not: anything under
`data/`, any index or calibration record, any `.env`.

## Documentation

[rhs2.github.io/cairn](https://rhs2.github.io/cairn/): quickstart, the concepts
behind each claim, guides for evaluation, configuration, components and serving,
and the reference for the CLI, the configuration file, the golden set and the
report.

## Licence

Apache-2.0.
