Metadata-Version: 2.5
Name: cortexos-engine
Version: 0.1.0
Summary: CORTEX engine: convert, group, embed, rank, index, graph, search. Zero LLM calls, ever.
Requires-Python: <3.13,>=3.12
Requires-Dist: numpy>=1.26
Requires-Dist: onnxruntime>=1.19
Requires-Dist: pyyaml>=6.0
Requires-Dist: sqlite-vec>=0.1.6
Requires-Dist: tokenizers>=0.20
Provides-Extra: convert
Requires-Dist: markitdown[all]>=0.1.0; extra == 'convert'
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# `pipeline/` — the zero-Claude layer

Convert · group · embed · rank · index · graph. **This layer never calls an LLM.**

> **Course-grade docs live in [`docs/`](./docs/README.md)** — install, vault adoption,
> the stage model, the CLI, search, converter failures, and the measured numbers.
> They are a Phase-1 deliverable (spec §16.1), written as the code was written.
> Start at [`docs/02-adopt-your-vault.md`](./docs/02-adopt-your-vault.md).

Not Claude, not a local model. The ONNX embedder is a ~130MB *encoder* — it has no
prompt, no completion, no API key, and no network at runtime. That distinction is
load-bearing: it is what makes *"search works the minute you upload"* true at zero
Claude cost, forever (spec §2.3, §5.1).

Extraction and verification are **middleware's** job, because only middleware holds
a human trigger. Nothing here is allowed to become a reason to call a model.

---

## Install

Python **3.12** is pinned. Many wheels this depends on — `onnxruntime` above all —
have no 3.14 build, and 3.14 is the system default on the build machine.

```bash
cd pipeline
uv venv --python 3.12
uv pip install -e ".[dev]"
```

One-time model download (~130MB into `models/`, gitignored). After this the layer
is fully offline:

```bash
uv run cortex-pipeline model fetch
```

### Optional converters

Document, audio and video conversion shell out to external binaries. They are
**not** pipeline dependencies, deliberately — `docling` alone pulls in torch and
costs 1.2GB, which has no business sitting in the same image as a 130MB encoder.

| Tool | Handles | Install |
|---|---|---|
| `docling` | pdf/docx/pptx/xlsx, tables + OCR | `uv pip install docling` in a **separate** venv |
| `markitdown` | same formats, lighter, no OCR | `uv pip install "markitdown[all]"` |
| `whisper-cli` + a ggml model | audio | `brew install whisper-cpp` |
| `ffmpeg` | audio extraction | `brew install ffmpeg` |
| `yt-dlp` | video (audio-only) | `brew install yt-dlp` |

Resolution order is PATH, then `~/.cortex-convert/bin`, then the usual Homebrew
locations. Override any of them with `CORTEX_DOCLING`, `CORTEX_MARKITDOWN`,
`CORTEX_WHISPER_CLI`, `CORTEX_WHISPER_MODEL`, `CORTEX_FFMPEG`, `CORTEX_YT_DLP`.

A missing converter **degrades, it does not crash**: the item is marked `failed`
with a reason, everything else keeps flowing, and `health` reports it.

---

## The read-only vault rule (Phase 1, spec §16.2)

CORTEX Phase 1 adopts a second brain that already exists. So:

> **This layer never writes to, moves, or deletes anything inside the vault.**

Enforced in `vaultguard.py`, not just documented — deny by default, see
`infer_read_only`. Two things that used to write into a vault now cannot:

| Artifact | Was | Now |
|---|---|---|
| `graph.tsv` | `vault/00 Maps/graph.tsv` | `.cortex/graph.tsv` |
| converted markdown | `vault/_inbox/` | `.cortex/converted/` |

Media deletion after transcription — the one thing convert may remove — never fires on
a file inside a read-only vault.

Only the repo's own `vault/` is writable by default. Everything else — the buyer's real
second brain, `testvault/`, anything — is read-only unless `--writable-vault` or
`CORTEX_VAULT_READ_ONLY=0` is passed.

**During the build we target a duplicate** at `testvault/`, rebuilt by
`scripts/sync-testvault.sh`, so adoption can be iterated on aggressively — the reset
button is one rsync. That is a **build-time practice, not a product feature**: in the
shipped DIY course CORTEX points at the buyer's real vault in place (spec §16.2), and
no code path assumes a duplicate exists.

## Vault adoption

An existing vault has its own folder names, edge taxonomy and frontmatter. The §3
template assumes kebab-case PARA. **Adoption maps; it does not migrate.**

```bash
cortex-pipeline adopt --vault ~/vault --report adoption-report.md --json   # writes NOTHING
cortex-pipeline adopt --vault ~/vault --write-profile --json               # pin it, after review
```

**The format is middleware's, not ours.** `vault.adoption.yaml` at the repo root, defined
by `packages/agent/config/vault-adoption.example.yaml`, is read by both layers.
`vaultprofile.py` consumes it verbatim — same keys, same semantics. A second vault
config would be a silent divergence, which is the failure mode BOUNDARIES.md exists for.

Two behaviours match middleware exactly:

- **An unknown folder is left unclassified**, never forced into the nearest role. It
  stays fully indexed and searchable; it just carries no role.
- **An unlisted relation passes through unchanged.** Edges are never renamed.

`adopt` drafts that same file rather than inventing a different artifact, and
`--write-profile` is the only write it can make.

## Run

```bash
# from the repo root
uv run cortex-pipeline <command> --json [args]
```

`--json` is always passed by middleware and is effectively always on. There is no
human-formatted output mode, because a second output mode is a second thing that
can silently diverge from the contract.

```bash
# first run against an existing vault, end to end
cortex-pipeline model fetch --json                                  # once, ~130MB
cortex-pipeline adopt --vault ~/vault --report adoption.md --json   # dry run, writes nothing
cortex-pipeline --vault ~/vault index rebuild --json
cortex-pipeline --vault ~/vault search --query "when do we refuse to quote" --json
```

Commands: `adopt · search · ingest · status · queue · graph · audit · note ·
index rebuild · health · sources · measure · eval · model fetch`.

Global flags are accepted **before or after** the subcommand — a caller that gets the
order wrong gets JSON, not an argparse usage message on stderr with empty stdout.
`scripts/check-cli-contract.sh` asserts the whole boundary, failure paths included.

### The output contract (`docs/pipeline-cli.md`)

- **stdout carries exactly one JSON document.** Nothing else, ever.
- Progress, warnings and timings go to **stderr as NDJSON**, one object per line.
  Long commands stream `{"stage":"embed","done":41,"total":900}` for progress bars.
- Exit `0` on success. Non-zero writes `{"error":{"code":...,"message":...}}` to
  stdout, using `CortexError.code` values from `cortexos-types`.

Every payload deserialises into `cortexos-types`: `SearchResult`, `GraphNeighborhood`,
`GraphAudit`, `IngestProgress`, `PromotionQueue`, `SystemHealth`, `Note`, `Page<NoteRef>`.

---

## The stage model (spec §5.2)

Stages 1–4 are `ZERO_CLAUDE_STAGES`. 5–6 belong to middleware and do not exist here.

**1 · Convert** — `stages/convert.py`. docling primary, markitdown fallback for
documents; whisper for audio; yt-dlp for video, **audio-only, concurrency 1, media
deleted after the transcript** (spec §2.5 calls video the landmine, and a 127MB
recording once blocked a git push for four days). Output is markdown carrying source
provenance in frontmatter — into `.cortex/converted/` for a read-only vault. Streaming
and chunked, never whole-file-in-RAM.

**2 · Group + normalise** — `stages/group.py`. Content-hash exact duplicates and
simhash near-duplicates collapse into clusters. **Nothing is ever deleted.** Every
file stays retrievable; exactly one member per cluster carries
`clusterRepresentative`, and that is the only one Claude is later paid to read.
Boilerplate and signatures are stripped **for processing only** — the original on
disk is untouched. Dates, emails, amounts and URLs are regexed out here.

**3 · Embed + cluster** — `embedder.py` + `stages/embed.py`. Local ONNX
(bge-small-en-v1.5), chunked with overlap, then k-means into ~30 topics so Claude
later sees clusters rather than files.

**4 · Rank** — `stages/rank.py`. The promotion queue, scored on retrieval count ×
task reference × cluster centrality × recency × entity density. Every entry carries
a human `reason` string, because the Sources screen shows it verbatim. **This is the
end of the free pipeline, and search is fully live at this point.**

### Chunk offsets are provenance

`Provenance.chunkOffset` / `chunkLength` are **byte offsets into the original file
on disk**, not into a normalised copy. The adversarial verify pass re-reads exactly
that span, so `chunking.py` works in bytes end to end rather than encoding and
guessing. If those numbers drift, the entire quality stack in spec §6 is built on
sand — hence `tests/test_chunking.py`.

### Index

SQLite, and nothing else: FTS5 (keyword) + sqlite-vec (semantic) + an edge table.
**Fully derived from markdown.** Delete `.cortex/index.db` and `index rebuild`
reconstitutes every row. The repo is truth; this file is a cache with legs.

- `index rebuild` — full rebuild, also on demand or corruption.
- `index rebuild --incremental --paths a,b` — exactly those paths, called by
  middleware **post-commit from the write path**, never from polling.
- `index rebuild --incremental` with no paths — resumes a killed sweep from the
  checkpoint.

Search fuses both arms with **Reciprocal Rank Fusion** — bm25 and cosine are not on
comparable scales and RRF needs no per-corpus tuning. Titles get 4x bm25 weight and
are prepended to the embedded representation (never to the stored chunk text, which
stays byte-exact for provenance). **Every hit carries a `Citation`.** When the source
date is unknown the citation says "date unknown" rather than omitting it — spec §6.4:
if it can't cite, it says so.

If the semantic arm is unavailable the search returns keyword-only results with
`degraded: true`. It never fails shut.

### Graph

`graph/build.py` ports the vault's `build-graph.py`. `## Edges` sections parse into
`graph.tsv` (`source \t relation \t target`), which stays in the vault's
human-greppable form — that file is meant to be read with `awk`. SQLite additionally
stores each endpoint resolved to a node id.

Edge line grammar. The list marker is **optional** — `docs/pipeline-cli.md` pins the
bullet-less form and the vault writes the bulleted one, so both parse. Requiring the
bullet made middleware's documented form yield zero edges, silently:

```
causes:: [[Some Note]]                                    # the pinned contract form
- causes:: [[Some Note]]                                  # the vault's form
- proposed-causes:: [[Some Note]]                         # below the confidence gate
causes:: [[Some Note]] <!-- prov: {"sourceFile": "..."} -->
```

The `## Edges` heading itself is configurable per vault via the profile.

Wikilinks resolve by filename stem, note title, **and the slug of either** — the
vault writes `[[Exact Filename]]` while the CORTEX template uses kebab-case paths,
and both must resolve or the whole graph reads as dangling edges.

**Two classes of edge.** Typed edges come from `## Edges` sections; untyped
`[[wikilinks]]` are emitted as `references`, deduplicated against typed edges so a typed
edge is never re-emitted as an untyped one. On a real vault this took orphans from 330
(59%) to 11 (2%) with the typed count unchanged at 1,087. Off via
`index rebuild --no-wikilinks` or `graph.include_wikilinks: false`.

`graph` queries take depth, relation filter and a hard node limit so the canvas
never receives a 40k-node payload; clipped results report `truncated: true`.

`audit` reports orphans, dangling edges, low-confidence density, contradictions,
never-retrieved nodes and expired `as-of` stamps. Contradictions are **surfaced,
never resolved** — detection is conservative on purpose, because a false
contradiction is worse than a missed one.

### Backpressure, not OOM

- **Checkpoint after every file** (`checkpoint.py`), written atomically. An OOM kill
  resumes from the checkpoint instead of restarting.
- A memory-aware gate pauses the queue as RSS approaches the limit
  (`CORTEX_MEMORY_LIMIT_BYTES`, default 1.5GB).
- **Never co-resident**: whisper, docling and the embedder each run in their own
  process, which exits afterwards. `measure` enforces this by spawning one child per
  stage — which is also the only way to get a per-stage peak RSS that means anything.

---

## Measured numbers

Full numbers, the machine, and the Fly sizing recommendation:
**[`eval/m1-measurements.md`](../eval/m1-measurements.md)**, regenerated by:

```bash
cortex-pipeline measure --corpus ./corpus --report eval/m1-measurements.md --json
```

A run where any stage crashed **raises** rather than reporting a plausible-looking
number, because spec §17 forbids quoting a hosting cost before a real measurement
exists and a fake one is worse than none.

---

## Eval — the release gate

`eval/questions.yaml` holds the question → expected-source set of spec §6.6.
**Regression = don't ship.**

```bash
cortex-pipeline eval --set eval/questions.yaml --json
```

Zero Claude, fully deterministic: a question passes when hybrid search returns one of
its expected sources inside the top `k`. No model judges anything, which is exactly
what makes it trustworthy as a gate.

```yaml
version: 1
default_k: 5
questions:
  - id: money-numbers-guardrail
    question: "what am I not allowed to say publicly about deal size?"
    expect_any: ["30 Resources/Beliefs/Content Constraints.md"]
    k: 5
    tags: [guardrail]
```

**The corpus is frozen.** The gate runs against `eval/snapshot/` (gitignored), pinned by
`eval/snapshot.manifest.json` (committed), and refuses to run if a single byte moved.
Live uploads once moved the score 46/50 → 45/50 with no code change; a gate whose input
drifts cannot tell a regression from a corpus change. `--against live` still runs the
set against the real vault as a labelled drift check that can never block a release.

Do not tune retrieval against this file. The 5 recorded failures are documented with
their reasons in `eval/known-failures.md` — headroom for the detector, not bugs to
paper over.

---

## Tests

```bash
cd pipeline && uv run pytest
```

Fixtures are small committed text; no large binaries and no model download. The
suite never loads ONNX — the keyword index and the fusion logic are exercised
directly — so CI stays fast and needs no network.

---

## Config

Reads `cortex.yaml` from the repo root and **tolerates unknown keys** by design.
Paths are overridable with `CORTEX_VAULT`, `CORTEX_STATE_DIR`, `CORTEX_MODELS_DIR`,
or the `--vault` / `--state-dir` / `--repo-root` flags.

## What this layer will not do

- **Call an LLM.** Any kind. Ever.
- **Commit to git.** Middleware owns the single commit queue with the writer lock.
  This layer exposes `index rebuild` for middleware to call post-commit.
- **Delete anything from the vault.** Dedupe groups; it does not remove. The one
  documented exception is a recording deleted *after* a successful transcript.
- **Write outside `pipeline/`, `scripts/` and `eval/`.**
