Metadata-Version: 2.4
Name: corpus-kb
Version: 0.1.0
Summary: Local end-to-end RAG system with MCP interface for agentic code editors
Author-email: Michael Oliver <63178353+moliver28@users.noreply.github.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/moliver28/corpus-kb
Project-URL: Repository, https://github.com/moliver28/corpus-kb.git
Project-URL: Documentation, https://github.com/moliver28/corpus-kb
Project-URL: Issues, https://github.com/moliver28/corpus-kb/issues
Project-URL: Changelog, https://github.com/moliver28/corpus-kb/blob/master/CHANGELOG.md
Keywords: rag,retrieval-augmented-generation,mcp,knowledge-base,knowledge-graph,embeddings,vector-search
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11.5
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mcp[cli]>=1.0
Requires-Dist: httpx>=0.27
Requires-Dist: ollama>=0.4
Requires-Dist: pyyaml>=6.0
Requires-Dist: pydantic>=2.0
Requires-Dist: typing-extensions>=4.10
Requires-Dist: rich>=13.0
Requires-Dist: unstructured[md]>=0.13
Requires-Dist: python-pptx>=1.0
Requires-Dist: defusedxml>=0.7.1
Requires-Dist: asyncpg>=0.29.0
Requires-Dist: pgvector>=0.3.0
Requires-Dist: numpy>=1.26
Requires-Dist: cymem<2.0.14
Requires-Dist: murmurhash<1.0.16
Requires-Dist: preshed<3.0.14
Requires-Dist: eventsourcing[postgres]>=9.0
Requires-Dist: psycopg[binary]>=3.1
Requires-Dist: starlette>=0.40.0
Requires-Dist: typer>=0.15
Requires-Dist: uvicorn>=0.24
Requires-Dist: fastapi>=0.115
Requires-Dist: opentelemetry-api>=1.45
Requires-Dist: psutil>=6.0
Provides-Extra: graphqlite
Requires-Dist: graphqlite>=0.4; extra == "graphqlite"
Provides-Extra: langextract
Requires-Dist: langextract; extra == "langextract"
Provides-Extra: pynvml
Requires-Dist: pynvml; extra == "pynvml"
Provides-Extra: duckdb
Requires-Dist: duckdb<2.0,>=1.5; extra == "duckdb"
Requires-Dist: pytz; extra == "duckdb"
Provides-Extra: latechunk
Requires-Dist: sentence-transformers>=2.7; extra == "latechunk"
Requires-Dist: transformers>=4.51.0; extra == "latechunk"
Requires-Dist: torch>=2.2; extra == "latechunk"
Provides-Extra: inductive
Requires-Dist: umap-learn>=0.5; extra == "inductive"
Requires-Dist: hdbscan>=0.8.36; extra == "inductive"
Provides-Extra: ml
Requires-Dist: gliner2>=2.1.0; extra == "ml"
Requires-Dist: gliner>=0.2.13; extra == "ml"
Requires-Dist: huggingface_hub>=0.30; extra == "ml"
Provides-Extra: tune
Requires-Dist: peft>=0.11; extra == "tune"
Requires-Dist: gliner[training]>=0.2.13; extra == "tune"
Provides-Extra: topics
Requires-Dist: bertopic>=0.16; extra == "topics"
Provides-Extra: ui
Requires-Dist: uvicorn[standard]>=0.24; extra == "ui"
Requires-Dist: python-multipart>=0.0.20; extra == "ui"
Provides-Extra: otel
Requires-Dist: opentelemetry-sdk>=1.45; extra == "otel"
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.45; extra == "otel"
Requires-Dist: openinference-semantic-conventions>=0.1.41; extra == "otel"
Provides-Extra: eval
Requires-Dist: deepeval>=4.3; extra == "eval"
Provides-Extra: semchunk
Requires-Dist: semchunk<5,>=4.1; extra == "semchunk"
Provides-Extra: dedup
Requires-Dist: datasketch<3,>=2.0; extra == "dedup"
Provides-Extra: calibrate
Requires-Dist: ranx<0.4,>=0.3; extra == "calibrate"
Provides-Extra: code
Requires-Dist: tree-sitter>=0.23; extra == "code"
Requires-Dist: tree-sitter-python>=0.23; extra == "code"
Requires-Dist: tree-sitter-javascript>=0.23; extra == "code"
Requires-Dist: tree-sitter-typescript>=0.23; extra == "code"
Requires-Dist: tree-sitter-rust>=0.23; extra == "code"
Requires-Dist: tree-sitter-go>=0.23; extra == "code"
Requires-Dist: tree-sitter-java>=0.23; extra == "code"
Provides-Extra: docling
Requires-Dist: docling>=2.144; extra == "docling"
Requires-Dist: pypdf>=6.21; extra == "docling"
Provides-Extra: scan
Requires-Dist: pypdf>=6.21; extra == "scan"
Provides-Extra: ocr-vl
Requires-Dist: transformers>=5.0; extra == "ocr-vl"
Requires-Dist: torch>=2.4; extra == "ocr-vl"
Requires-Dist: pypdfium2>=5; extra == "ocr-vl"
Provides-Extra: notebook
Requires-Dist: marimo>=0.25; extra == "notebook"
Provides-Extra: classify
Requires-Dist: scikit-learn>=1.5; extra == "classify"
Requires-Dist: threadpoolctl>=3.5; extra == "classify"
Provides-Extra: setfit
Requires-Dist: setfit>=1.2; extra == "setfit"
Provides-Extra: llamaindex
Requires-Dist: llama-index-core>=0.11; extra == "llamaindex"
Requires-Dist: llama-index-vector-stores-postgres>=0.2; extra == "llamaindex"
Requires-Dist: llama-index-llms-ollama>=0.4; extra == "llamaindex"
Requires-Dist: llama-index-embeddings-ollama>=0.4; extra == "llamaindex"
Provides-Extra: dev
Requires-Dist: ruff==0.15.18; extra == "dev"
Requires-Dist: python-magic-bin>=0.4.14; sys_platform == "win32" and extra == "dev"
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.24; extra == "dev"
Requires-Dist: pytest-cov>=5.0; extra == "dev"
Requires-Dist: syrupy>=4.0; extra == "dev"
Requires-Dist: lxml>=5.0; extra == "dev"
Requires-Dist: mapie>=1.0; extra == "dev"
Requires-Dist: python-multipart>=0.0.20; extra == "dev"
Requires-Dist: poethepoet>=0.24; extra == "dev"
Requires-Dist: import-linter>=2.0; extra == "dev"
Requires-Dist: vulture>=2.10; extra == "dev"
Requires-Dist: deptry>=0.20; extra == "dev"
Requires-Dist: xenon>=0.9; extra == "dev"
Requires-Dist: diff-cover>=9.0; extra == "dev"
Requires-Dist: pre-commit>=4.0; extra == "dev"
Requires-Dist: textstat>=0.7; extra == "dev"
Requires-Dist: sqlglot<31,>=30.24; extra == "dev"
Dynamic: license-file

# Corpus-KB

**A local NLP engine for files, transcripts, and code. Ingest, search, ask, extract, and code. No cloud.**

Corpus-KB serves three kinds of users. Non-coders get a browser UI. Novice coders get a CLI with a guided quickstart. Expert researchers get a qualitative-coding pipeline, an OpenAI-compatible API, and MCP tools for coding agents. Everything runs on your machine. Postgres stores the data by default; DuckDB is the single-file local engine ([Storage backends](docs/STORAGE-BACKENDS.md)). Ollama or PostgresML generates embeddings. One server exposes the catalogue over MCP stdio, HTTP, and a JSON-RPC socket. An OpenAI-compatible adapter serves `/v1` chat and embeddings. See [Editor integration](#editor-integration).

---

## Two ways to start

Pick the path that fits you. Both end at the same working system.

### The novice path: three commands

Install [pipx](https://pipx.pypa.io/) first if you do not have it. Then:

```bash
pipx install corpus-kb
corpus-kb quickstart
corpus-kb demo-kb
```

No repo clone is needed. `quickstart` checks your machine and names any
missing piece. `demo-kb` loads a small sample library, so search works
right away.

Two downloads come first:

- [Ollama](https://ollama.com/download) serves the local models.
- Docker runs the Postgres database. No Docker? Use the DuckDB path
  ([Storage backends](docs/STORAGE-BACKENDS.md)).

The default models are `qwen3-embedding:4b` for vectors and `qwen3:4b`
for answers. Each one downloads about 2.5 GB through Ollama. The setup
pulls both for you.

### The browser path: from a clone

The browser UI needs the repo. First install the package:

```bash
pip install -e .
```

Then build the browser app once. You need Node 24 and pnpm for this
step. Node is a build tool only. The running server stays pure Python:

```bash
pnpm -C ui_src install
pnpm -C ui_src build
```

Then start the stack. `corpus-kb setup` starts Postgres and loads the
schema. `corpus-kb start` starts the server and opens your browser:

```bash
corpus-kb setup
corpus-kb start
```

The browser opens the lifecycle app at `http://localhost:8010/app/`.
The Setup stage shows the system's health in plain language. The other
stages follow the research flow: Ingest, Review, Scale, Ask, Analyze,
Codebook, Release.

Details: [docs/UI.md](docs/UI.md).

### The expert path: terminals and APIs

Every operation is reachable programmatically. One line per surface:

- **MCP stdio** for editor agents: `corpus-kb --transport stdio` — [docs/API.md](docs/API.md)
- **HTTP API** with curl examples: [docs/API.md](docs/API.md)
- **OpenAI-compatible adapter** at `/v1` (chat completions, embeddings): [docs/API.md](docs/API.md)
- **JSON-RPC socket** for local automation: [docs/API.md](docs/API.md)
- **CLI**: `corpus-kb ingest`, `corpus-kb search`, `corpus-kb ask` — [docs/getting-started.md](docs/getting-started.md)
- **Tracing** with OpenTelemetry spans: [docs/OBSERVABILITY.md](docs/OBSERVABILITY.md)
- **Offline evaluation** of retrieval and answers: [docs/EVALUATION.md](docs/EVALUATION.md)
- **marimo notebooks** served at `/notebooks`: [docs/NOTEBOOKS.md](docs/NOTEBOOKS.md)
- **Topics** (offline BERTopic over stored embeddings): [docs/ANALYSIS.md](docs/ANALYSIS.md)

First time on a terminal? `corpus-kb quickstart` walks you from zero to
a first search. The step-by-step quickstart is in
[docs/getting-started.md](docs/getting-started.md).

Prefer Docker? A NAS or any Docker host can run the whole app with one
compose command. See "Run the whole app in Docker" in
[docs/INSTALL.md](docs/INSTALL.md).

---

## What you get

- **Hybrid search** that blends vector similarity, full-text search, and rank fusion, with honest handling of degraded (zero-vector) chunks
- **Reranking** — a local judgment scorer by default, or a true cross-encoder through llama.cpp
- **Answer generation** with per-sentence citations, plus assertion-level verification with honest abstention
- **Query routing** — each question goes to vector search, SQL aggregates, or graph traversal
- **Self-query filters** — natural-language filters parsed into parameterized SQL predicates
- **Knowledge graph** with entities, relations, and BFS traversal
- **Ontology-aware extraction** with configurable entity/relation types and pluggable backends (regex, LangExtract, PostgresML, plus opt-in GLiNER)
- **Pluggable ingestion strategies** — eight built-ins (default, transcript, slides, web, survey, markdown, code, textbook), extensible three ways ([docs/INGESTION.md](docs/INGESTION.md#ingestion-strategies))
- **Website ingestion** — a robots-respecting fetcher with same-domain crawl and heading-section chunks
- **Late chunking** — one model pass per document; chunk vectors pool from their own token spans
- **Qualitative coding** — transcript ingestion, codebooks, deductive and inductive runs, human review, governance reports, and a gated release lifecycle
- **Topics** — offline BERTopic clustering over stored embeddings ([docs/ANALYSIS.md](docs/ANALYSIS.md))
- **LlamaIndex RAG backend** with PGVectorStore and Ollama for local vector search
- **Event sourcing** for audit trails and rebuildable projections
- **Multi-tenant Postgres** with row-level security on every table
- **Full-stack installer** with hardware detection, profile-based recommendations, and guided setup
- **MCP, HTTP, socket, and CLI surfaces** so any editor or script can talk to it
- **Browser UI** with drag-and-drop ingest, cited answers, and plain-language health checks ([docs/UI.md](docs/UI.md))
- **Notebooks console** — marimo notebooks served by the running server ([docs/NOTEBOOKS.md](docs/NOTEBOOKS.md))

---

## Architecture

```mermaid
graph TB
    Editor[AI Code Editor] -->|"MCP stdio"| MCP[MCP Server]
    Editor -->|HTTP :8010| HTTP[Starlette API]
    Editor -->|JSON-RPC socket| Socket[Unix socket / named pipe]
    MCP --> Handlers[Command / Query Handlers]
    HTTP --> Handlers
    Socket --> Handlers
    Handlers --> Domain[Domain Layer<br/>Aggregates + Events]
    Domain --> ES[Event Store<br/>append-only]
    ES --> Projections[Async Projections]
    Projections --> PG[Postgres 17]
    Projections --> DUCK[DuckDB<br/>local single-user engine]
    PG --> VEC[pgvector<br/>vector search]
    PG --> FTS[Postgres FTS<br/>to_tsvector]
    PG --> AGE[Apache AGE<br/>Cypher graphs]
    PG --> RLS[RLS on 12 tables]
    Projections --> Ollama[Ollama<br/>embedding service]
```

1. **Ingest** a file, directory, or raw text.
2. The pipeline partitions it into chunks and embeds each chunk through Ollama or PostgresML. It extracts entities and relations, then stores the result.
3. Commands append events to the event store; async projections write the read models into Postgres or DuckDB.
4. Your editor queries the read models through search, SQL, or graph traversal.

---

## Event sourcing flow

```mermaid
sequenceDiagram
    participant C as Client
    participant H as CommandHandler
    participant A as Document Aggregate
    participant ES as Event Store
    participant P as Projection
    participant DB as Postgres Tables

    C->>H: ingest_file(file_path)
    H->>A: create Document
    A->>A: apply Ingested event
    A->>A: apply ChunksAdded event
    H->>ES: app.save(aggregate)
    ES-->>H: event_id, version
    H-->>C: {status: "success"}
    ES->>P: subscribe(ChunksAdded)
    P->>P: embed chunks via Ollama
    P->>DB: INSERT chunks, chunks_vectors
    P->>DB: UPDATE projection_checkpoints
```

Events are the source of truth. Projections are derived and can be rebuilt by replaying the event log. Vectors live in the `chunks_vectors` table and are treated as derived data, not event payload.

---

## Documentation

| Page | What it covers |
|------|----------------|
| [Getting started](docs/getting-started.md) | Both paths: the browser for novices, the terminal quickstart for everyone else. Then qualitative coding from zero: the demo, your first transcript, first codebook |
| [Install](docs/INSTALL.md) | Full setup from scratch: Postgres, Python, Ollama, schema, first query |
| [Browser UI](docs/UI.md) | The lifecycle app: stage-by-stage user guide, config, build, gates, troubleshooting |
| [Features](docs/FEATURES.md) | Ingest, search, quality controls, graph, tags, research subsystem, embedding models, MCP tool reference |
| [Admin](docs/ADMIN.md) | Configuration, schema, multi-tenancy, backups, monitoring, CI/CD |
| [Storage backends](docs/STORAGE-BACKENDS.md) | DuckDB vs Postgres: quickstart, engine selection, migrate-data, runtime tuning |
| [Operations](docs/OPERATIONS.md) | The operator's contract: exit codes, ports, vocabularies, tenancy, the MCP surface |
| [API](docs/API.md) | HTTP routes, the `/v1` OpenAI adapter, curl examples, MCP tool reference |
| [Observability](docs/OBSERVABILITY.md) | OpenTelemetry spans, exporters, zero-cost when off |
| [Evaluation](docs/EVALUATION.md) | Golden sets, recall/MRR/nDCG, answer metrics |
| [Notebooks](docs/NOTEBOOKS.md) | The marimo console and the three example notebooks |
| [Development](docs/DEVELOPMENT.md) | Architecture deep dive, testing, PR workflow, conventions |
| [Research](docs/research.md) | Research command reference, config keys, embedding requirements |
| [Understanding your report](docs/understanding-your-report.md) | Plain-language guide to every governance-report section |
| [FAQ](docs/FAQ.md) | Common questions |
| [Ingestion](docs/INGESTION.md) | Full pipeline: partition, chunk, embed, extract, store, plus the strategy reference and extension ladders |
| [Analysis](docs/ANALYSIS.md) | The analysis layer: analyze-run report, labeling, classification, topics, trends, method register |
| [Analysis methods](docs/ANALYSIS-METHODS.md) | GENERATED reference: the 30-method register with required validation and permitted claims |
| [Rollback drill](docs/ROLLBACK_DRILL.md) | Practiced procedure for backing out a bad deployment |
| [Internal program docs](docs/research/) | Archive: playbooks, research math, benchmarks, harness distribution, program plans |

---

## Editor integration

Editors connect over **MCP stdio** (native, no bridge), the **HTTP API**, or the JSON-RPC socket:

- MCP stdio: `corpus-kb --transport stdio` — the `mcp-configs/` files (Claude Code, Cursor, Codex) and the root `opencode.json` (OpenCode) launch exactly this command. Generic MCP clients such as Hermes or Cline need no config file: they point at the same stdio server. See [docs/research/harness-distribution.md](docs/research/harness-distribution.md). The server registers every catalogue operation as an MCP tool; see [docs/FEATURES.md](docs/FEATURES.md) for the tool reference.
- HTTP: `corpus-kb start --transport http --port 8010` — every catalogue operation has a route, plus the `/v1` OpenAI adapter (see [docs/API.md](docs/API.md)).
- JSON-RPC socket: same command, for local automation.
- The research and coding surfaces are first-class CLI commands (`corpus-kb research ...` and `corpus-kb coding ...`). Ingestion has a top-level front door: `corpus-kb ingest` with `--watch` auto-heal, plus `corpus-kb ingest-status` and `corpus-kb ingest-reembed`.

Every catalogue operation is reachable on all four surfaces (MCP, CLI,
HTTP, socket) and returns the same envelope on each. The parity gate in
`tests/test_surface_parity_deep.py` enforces this.

---

## License

MIT License. See `pyproject.toml` for the full text.
