Metadata-Version: 2.4
Name: compass-kb-validation
Version: 4.0.1
Summary: Knowledge-base readiness validation for RAG & AI agents — config-driven, deterministic pipeline integrity checks.
Author: COMPASS
License: MIT
Project-URL: Homepage, https://github.com/TestAutomationArchitect/compass
Project-URL: Repository, https://github.com/TestAutomationArchitect/compass
Project-URL: Issues, https://github.com/TestAutomationArchitect/compass/issues
Keywords: rag,knowledge-base,vector-database,data-validation,pipeline-testing,llm-infrastructure
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31.0
Requires-Dist: boto3>=1.34.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: redis>=5.0.0
Provides-Extra: dashboard
Requires-Dist: fastapi>=0.104.0; extra == "dashboard"
Requires-Dist: uvicorn>=0.24.0; extra == "dashboard"
Provides-Extra: pdf
Requires-Dist: PyPDF2>=3.0.0; extra == "pdf"
Provides-Extra: pdf-layout
Requires-Dist: pymupdf>=1.24; extra == "pdf-layout"
Provides-Extra: content
Requires-Dist: beautifulsoup4>=4.9.0; extra == "content"
Provides-Extra: perf
Requires-Dist: numpy>=1.24; extra == "perf"
Provides-Extra: qdrant
Requires-Dist: qdrant-client>=1.7.0; extra == "qdrant"
Provides-Extra: pgvector
Requires-Dist: psycopg[binary]>=3.1; extra == "pgvector"
Provides-Extra: pinecone
Requires-Dist: pinecone>=5.0.0; extra == "pinecone"
Provides-Extra: weaviate
Requires-Dist: weaviate-client>=4.5.0; extra == "weaviate"
Provides-Extra: chroma
Requires-Dist: chromadb>=0.5.0; extra == "chroma"
Provides-Extra: azure
Requires-Dist: azure-storage-blob>=12.0.0; extra == "azure"
Requires-Dist: azure-identity>=1.15.0; extra == "azure"
Provides-Extra: gcs
Requires-Dist: google-cloud-storage>=2.10.0; extra == "gcs"
Provides-Extra: all
Requires-Dist: fastapi>=0.104.0; extra == "all"
Requires-Dist: uvicorn>=0.24.0; extra == "all"
Requires-Dist: PyPDF2>=3.0.0; extra == "all"
Requires-Dist: qdrant-client>=1.7.0; extra == "all"
Requires-Dist: psycopg[binary]>=3.1; extra == "all"
Requires-Dist: pinecone>=5.0.0; extra == "all"
Requires-Dist: weaviate-client>=4.5.0; extra == "all"
Requires-Dist: chromadb>=0.5.0; extra == "all"
Requires-Dist: numpy>=1.24; extra == "all"
Requires-Dist: azure-storage-blob>=12.0.0; extra == "all"
Requires-Dist: azure-identity>=1.15.0; extra == "all"
Requires-Dist: google-cloud-storage>=2.10.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: pyflakes>=3.0.0; extra == "dev"
Requires-Dist: pyright==1.1.409; extra == "dev"
Requires-Dist: bandit>=1.7.5; extra == "dev"
Requires-Dist: pip-audit>=2.6.0; extra == "dev"
Requires-Dist: numpy>=1.24; extra == "dev"
Dynamic: license-file

# COMPASS — the quality gate for your RAG knowledge base

[![CI](https://github.com/TestAutomationArchitect/compass/actions/workflows/ci.yml/badge.svg)](https://github.com/TestAutomationArchitect/compass/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/compass-kb-validation)](https://pypi.org/project/compass-kb-validation/)
[![Python](https://img.shields.io/pypi/pyversions/compass-kb-validation)](https://pypi.org/project/compass-kb-validation/)

**Point COMPASS at your RAG knowledge base. It runs 73 deterministic checks across storage,
knowledge graph, vectors and retrieval, and fails your CI build when the KB is broken.**
No LLM, no test-writing — the checks ship with it.

Think of it as a **linter for your RAG data plane**: a catalogue of built-in checks with
HARD/WARN severity tiers, tuned in YAML, gating CI by exit code. It answers *"is the
knowledge base itself sound?"* — not *"are the answers good?"* (that's the eval layer's
job; COMPASS is the gate that runs first).

### Your KB is broken and you don't know yet

Chunks that never got embedded. An embedding model swapped without a re-index. A document
deleted at the source but still answering questions. Citations pointing at the wrong file.
Storage, graph and vector store quietly disagreeing about what exists. **All of it is
invisible from the outside** — retrieval still returns *something*, answers still look
plausible, and your eval scores drift for reasons nobody can trace.

COMPASS finds these deterministically, before your agents serve them.

### 30 seconds to see it work

```bash
pip install compass-kb-validation
compass demo                                   # sample report, no backend needed
compass check redis://localhost:6379/my_index  # your real index: profile -> validate -> report
```

`compass check` needs no config: it profiles the index, fills in the shape it detects, runs
the checks, and opens an HTML report. See **[QUICKSTART.md](QUICKSTART.md)** for the
five-step path to a gating CI check.

### Does it actually catch things?

Yes — and you can verify that claim yourself in one command, no services required:

```bash
python benchmark/broken_kb_benchmark.py
#   Detection rate:      16/16  (100%)
#   False-positive rate: 0/6    (0%)
```

The [broken-KB benchmark](benchmark/) injects 16 realistic faults (zero chunks, wrong
embedding dimension, degenerate vectors, cross-document contamination, orphaned sources,
retrieval drift, …) into fixtures, runs the **real** validators against them, and scores
detection vs false positives. It also runs 6 healthy fixtures to prove COMPASS doesn't cry
wolf. It's part of the test suite, so the number can't rot.

## What COMPASS Does

COMPASS validates a document's journey through your ingestion pipeline and verifies the resulting knowledge base is agent-ready. The pipeline is **declarative** — you choose which steps run, in what order, in YAML — and ships with these built-in steps:

```
API Upload → Object Storage → Knowledge Graph → Vector Store → Retrieval → Cross-Stage Consistency
```

Enable only the steps your setup has (no object store? no graph? drop them), reorder them, or register your own. For each enabled step it deterministically checks, for example:

- **Ingestion** — the document was accepted and assigned an id (**GraphQL or REST** ingestion APIs, config-driven)
- **Object storage** — it persisted with the right content type, size, and integrity, and (opt-in) the *source document* is healthy across PDF/DOCX/PPTX/HTML (S3, MinIO, R2, Azure Blob, GCS…)
- **Knowledge graph** — metadata is present, correct, and *agrees with the other stores*: required predicates, identity uniqueness, storage-reference integrity, RDF typing, ontology-namespace consistency, dangling links, orphan nodes, version supersession (GraphDB, Neptune, Neo4j, Stardog, Blazegraph…) — **19 checks**, HARD / WARN tiered, every predicate and query config-driven so they fit any ontology
- **Vector store** — chunks exist with valid embeddings, correct dimensions/metric, sane ordering/overlap, structure coherence, retrievable via KNN, no cross-document contamination, no near-duplicate bloat (Redis, Qdrant, pgvector, Chroma, Pinecone, Weaviate) — **30 toggleable checks**, each on a **HARD / WARN** severity tier (only HARD gates the run)
- **Retrieval** — the document's own content is actually retrievable (recall@K, MRR); opt-in embedding-drift, provenance, filtered-recall, negative-retrieval, keyword-completeness
- **Cross-stage consistency** — do object storage, the graph, and the vector index *agree* about each document (eTag, metadata, ACL propagation)?

Two modes: **live** validation as documents are ingested, and **post-hoc discovery** of a corpus that's already loaded.

> Scope note: COMPASS validates *pipeline & data integrity*. It does **not** judge answer quality — that's the evaluation layer's job. COMPASS is the gate that runs first.

## Key Features

✅ **Declarative pipeline** — choose/reorder/extend steps in YAML; no fixed shape  
✅ **Config-Driven** — zero code changes per project; all customization in YAML  
✅ **Pluggable backends** — storage / vector store / knowledge graph via a provider registry (one file to add a vendor). Redis, Qdrant, pgvector, Chroma, Pinecone and Weaviate all support **metadata pre-filters**, so a probe reproduces production's *filtered* retrieval on any of them ([SUPPORT.md](SUPPORT.md))  
✅ **Pluggable steps** — register custom validators in a step registry; the report adapts automatically  
✅ **Deterministic & cheap** — structural/byte/schema checks, no LLM, CI-gateable  
✅ **Query & analyze the KB** — ask it questions, expose it to agents over MCP, and preview embedding-model migrations / map retrieval coverage — all deterministic, read-only (see *Beyond validation*)  
✅ **Eval oracle** — enrich any eval with deterministic KB-grounding signals, then measure how accurate its LLM judges are against that ground truth (`compass enrich` / `compass calibrate-judges`) — the eval framework guesses, COMPASS checks the KB  
✅ **Interactive reporting** — self-contained HTML dashboard + JSON + JUnit XML  
✅ **Trend Tracking** — SQLite historical performance analysis  
✅ **Notifications** — Slack/Teams webhooks for alerts  
✅ **Parallel Execution** — process many documents concurrently  
✅ **Graceful Degradation** — non-critical failures don't crash the pipeline  

## Install

Core deps only; add extras per backend (Redis is built in):

```bash
pip install compass-kb-validation
pip install "compass-kb-validation[qdrant,pgvector,chroma]"
#   vector stores: qdrant | pgvector | chroma | pinecone | weaviate
#   also: pdf (PDF source checks) | content (HTML extraction) | azure | gcs | dashboard
```

### Project path (saved config)

Then use the unified **`compass`** CLI - one verb, subcommands:

```bash
compass init my-kb          # scaffold config/projects/my-kb.yaml + .env.example + a .compass marker
cp .env.example .env        # fill in your backend connection
#   edit config/projects/my-kb.yaml  (search_index, document_id_field)
compass doctor --project my-kb     # verify connectivity + config (per stage)
compass discover --project my-kb   # validate the already-populated KB, emit a report
```

`compass init` drops a `.compass` marker, so the CLI works from any subdirectory of the
project (no env var to set). Other subcommands:

```bash
compass validate --project my-kb --output-format all   # live ingestion validation
compass profile  --project my-kb                        # read a store's shape, suggest config
compass list-checks                                     # check catalogue + severity tiers (no project needed)
compass <cmd> --help                                    # per-command options
```

The standalone `compass-discover` / `compass-validate` / `compass-profile` commands keep
working. Live validation also reads a `test_data/<name>.yaml` (scaffold it with
`compass-validate --project my-kb --init-test-data`).

### Beyond validation — inspect, migrate, map coverage

These read-only, deterministic (no-LLM) tools help you **verify, diagnose, and safely change**
a knowledge base — they don't turn COMPASS into a retrieval app or an agent-serving platform.
`compass ask` is a **retrieval probe** (see *what the KB returns* for a query, no generated
answer); `compass mcp` is an **inspection surface** (read-only KB tools an agent can call, not
a serving layer). Everything here stays on COMPASS's side of the line and never affects a
validation pass/fail. They read an `embedding:` block in the project YAML (provider `openai` /
`azure_openai` / `bedrock`, `model_id`, `dimensions`) so a query is embedded the way ingestion was.

```bash
# Ask the KB a question - returns the ranked chunks it would retrieve (no generated answer)
compass ask --project my-kb "what is the refund policy?"

# Match production retrieval with metadata pre-filters (hybrid search; commas AND, | ORs a field)
compass ask --project my-kb --filter "category=report,region=US|EU" "summarize the latest report"

# Expose the KB to agents (Copilot/Claude) as MCP tools over stdio
compass mcp --project my-kb --print-config   # prints a .vscode/mcp.json snippet
compass mcp --project my-kb                   # run the server (query_kb, list_documents, ...)

# Preview an embedding-model switch BEFORE you cut over (self-bootstraps, no questions needed)
compass migrate --project my-kb --to openai/text-embedding-3-large

# Map retrieval coverage / find dead zones a question set never reaches (feed real query logs)
compass coverage --project my-kb --questions query_logs.jsonl

# Prove retrieval matches production: compare vs a ground-truth file (precision/recall/Jaccard)
compass compare --project my-kb --ground-truth production_ground_truth.json
# Mixed ground-truth file? Non-retrieval cases (no chunks) are auto-skipped; filter by variant:
compass compare --project my-kb --ground-truth gt.json --variant answerable

# Hand retrieval to any eval framework: emit generic RAG eval test cases
compass ask --project my-kb --batch questions.jsonl --format eval --output eval_input.jsonl

# Corpus health (deterministic, no LLM): score chunk quality / find stale + orphan sources
compass quality   --project my-kb
compass freshness --project my-kb --stale-after 30

# Validate an EXTERNAL eval's output against the KB COMPASS already has access to. Not an eval,
# not a tracer: the eval (an LLM judge, or the bot's captured trace) claims "grounded",
# "cited MIS p.1", "28%" - COMPASS deterministically checks those claims against ground truth
# in the KB and attributes any failure to a layer (retrieval vs the model), never scoring answer
# quality. Verifies each CITED claim at its cited location: a fabricated number, a wrong page, a
# fact present in the KB but not retrieved, or a source not in the KB. Uses the eval's retrieved
# chunks when present (Level 1) and the live KB (Level 2); the answer text + a reachable KB alone
# still catches fabricated figures and phantom citations.
compass grounding-audit --project my-kb --eval-result case.json --fail-on-fabrication

# Level 3 (opt-in): when the KB cannot confirm a cited value, read the ORIGINAL source object and
# check the cited page of the real document. This separates the two cases the KB alone cannot tell
# apart - the fact IS in the source but never reached the KB (an INGESTION defect: re-ingest, no
# amount of retrieval tuning fixes it), versus the fact is nowhere in the source (a PROVEN
# generation hallucination). Needs eval.citation_source_key_fields + object-store access.
compass grounding-audit --project my-kb --eval-results cases.jsonl --verify-source

# Eval oracle (deterministic, no LLM): enrich an eval with KB-grounded signals, then measure how
# accurate the eval's OWN LLM judges are against that deterministic truth. LLM judges GUESS whether
# a response is grounded; COMPASS CHECKS it against the KB - so it can tell you which judge is right.
compass enrich --project my-kb --input eval.json --output enriched.json
compass calibrate-judges --input enriched.json --metric groundedness

# Close the loop: re-decide the eval's PASS/FAIL using deterministic truth instead of judge
# noise, and report every verdict that changed - cases the eval FAILED that are actually
# grounded (rescued), and cases it PASSED that contain a fabrication (demoted). Pure
# arithmetic over the enriched file: no KB, no LLM, no credentials. CI-ready exit codes.
compass quality-gate --input enriched.json --weights weights.json --threshold 0.80

# Before spending judge money: does each ground-truth answer make the key points declared for
# it? A point the reference itself does not make grades every real answer against a bar the
# reference cannot clear - and those failures look like model defects. Checks the GROUND TRUTH
# against itself, never the system's answers. Deterministic, no KB, no LLM.
compass gt-lint --input ground_truth.json --answer-field answer --points-field key_points

# Pre-flight: which cited document names actually resolve to a KB document? An unresolvable
# name makes every claim citing it unverifiable, so finding out from a finished report means
# paying for the run twice. Exits 1 if any name does not resolve.
compass grounding-audit --project my-kb --eval-results eval.json --check-aliases

# Is retrieval the problem, or is the KB? Compare retrieval configurations over the SAME
# validated KB using the eval's own ground-truth chunks - recall / precision / MRR / NDCG /
# hit rate. No LLM: a chunk is relevant because the ground truth says so.
compass benchmark --project my-kb --input eval.json

# Did the source content reach the store? Compares each document with its chunks: body text
# coverage, per-page coverage and numbers, with the passages behind every finding. See
# "Source fidelity" below. Report-only unless --fail-on-findings.
compass fidelity --project my-kb --sample 25 --format html --output reports/fidelity.html

# Is the re-ingested index better? Scores one eval against two ALREADY-BUILT indices and says
# which won on NDCG and on recall. Nothing is re-embedded. Warns when the indices were chunked
# differently, because document + page matching is then invalid (use --match content).
compass simulate --project my-kb --input eval.json --index-a kb_v1 --index-b kb_v2

# Is my chunker the right choice, before paying to re-embed? Rebuilds sampled documents from
# their chunks, re-chunks the same text, and reports mid-thought cuts, table/list splits, sizes,
# fragmentation and "worse on N of M documents". No embedder, no LLM, never a gate. Text rebuilt
# from chunks is lossy and understates what a fixed splitter does to raw tables.
compass chunk-eval --project my-kb --sample 50 --strategy fixed:3000/200 --strategy recursive:3000

# One verdict for the whole KB, composed from artifacts the commands above already wrote. See
# "KB readiness" below. No live connections. Exit 0 READY, 1 otherwise, 2 bad input.
compass readiness --discover reports/my-kb.json --benchmark bench.json --grounding audit.json
# Do two environments hold the same KB? Documents missing either way, schema, e_tags that
# differ, pollution patterns, stale versions. Exits 0 no drift, 1 drift, 2 config error or could
# not compare: an environment that cannot be read is never reported as "no drift".
compass drift --project my-kb --env-a dev --env-b qa

# Audit traceability: does each sampled document still trace to its source? Classifies orphan /
# stale / broken-ref / no-version; a stage that cannot run says why. CSV for the auditor.
compass lineage --project my-kb --sample 200 --format csv --output lineage.csv

# Grounding dropped between runs - what changed? Attributes the regression to the documents
# responsible, ranked by share ("doc X REMOVED -> 40.0 pp across 8 cases"). A drop that no
# changed document explains is reported as unexplained, never spread to make totals reconcile.
compass diff --project my-kb            # or: --baseline run-a.json --current run-b.json

# Incremental discovery: re-validate only what changed (by content, not clock), plus a
# spot-check of the rest. Unchanged documents are reported as NOT re-validated, never as passed.
compass discover --project my-kb --since last-run
```

## Source fidelity

`compass fidelity` compares two extractions of a document, so it has to tell extraction noise from
loss. Both sides are normalised (Unicode, ligatures, hyphenation, markup, and spacing, including
gaps inside words), tables are judged by their numbers, and every exclusion is counted. Install a
better text layer with `pip install "compass-kb-validation[pdf-layout]"`, or `[pdf]` for the
lower-bound default.

Start report-only, and list what your pipeline drops on purpose until only real findings remain:

```yaml
eval:
  citation_source_key_fields: ["s3_key"]   # chunk field holding the source object key
  source_extractor: auto                   # auto | pymupdf | pdfplumber | pypdf | pdf_text
  fidelity:
    normalization: aggressive              # minimal | standard | aggressive
    match: shingle                         # shingle | exact
    coverage_min: {default: 0.90, quarterly: 0.97}   # keyed by document_type
    prefer_snapshot: true
  source_exclusions:                       # nothing is excluded that is not listed here
    repeated_on_pages: 5                   # headers and footers
    min_chars: 75                          # small fragments
    patterns: ["(?i)^past performance"]    # by-design notices
    page_roles: {disclaimer: "(?i)^important disclosures"}
    roles: [marginalia, logo, attestation] # read from a snapshot, so they need one
    must_retain: ["material change"]       # overrides every exclusion; missing is a loss
```

`fidelity` also runs the other way, chunk by chunk, and each chunk that does not trace is reported
by id: `WRONG_PAGE`, `ALTERED`, `TRUNCATED`, `OUT_OF_ORDER`, `UNTRACEABLE`, or `UNVERIFIABLE` where
the source has pages with no text layer. A consistent page shift is reported once. Tune it with
`eval.fidelity.page_tolerance` (default 1), `page_field`, `page_end_field` and `position_field`, or
turn it off with `chunk_integrity: false`.

Findings say what is missing, not who lost it, until you declare the extractor's own output. Any
extractor that writes JSON works: map its fields once, and each finding becomes `PIPELINE_DROP`,
`PIPELINE_ALTER` or `EXTRACTOR_MISS`.

```yaml
eval:
  snapshot:
    provider: storage_json
    location: "snapshots/{project}/{id}/{e_tag}/document.json"   # also {source_key}
    format: my_extractor
    formats:
      my_extractor:                        # dotted paths into the extractor's JSON
        elements: "sections"
        type: "kind"
        text: "text"
        page: "page"
        bbox: "box"
        cells: "table.cells"
        order: "reading_order"
```

### KB readiness

`compass readiness --discover D.json [--grounding G.json] [--benchmark B.json] [--drift X.json]
[--project P] [--format console|json|html] [--output F]` reads JSON the other commands wrote and
returns READY, AT RISK or NOT READY, with a stage and an owner (ingestion, retrieval config,
generator) for every finding. A stage whose artifact is absent, or which measured nothing, is
**not examined** and caps the verdict at AT RISK: never READY by omission. `gate` means NOT READY,
`risk` means AT RISK, `advisory` is reported and changes nothing.

Every verdict carries `verified` / `unverifiable` / `not-checked` counts, and `discover` prints and
exports the same block (`coverage` in the JSON). `unverifiable` means it was attempted and could not
be compared, with the reason; `not-checked` means nobody tried. A fidelity comparison that never ran
reads `content-fidelity: not-checked`, under a READY too when the stage is waived. This is reporting:
it changes no verdict.

| Finding | Default | Owner |
|---------|---------|-------|
| `corpus_not_ready` | gate | ingestion |
| `documents_failed` (report has no corpus verdict) | gate | ingestion |
| `grounding_ingestion_loss` (proven at Level 3) | gate | ingestion |
| `grounding_generation_proven` (proven at Level 3) | gate | generator |
| `retrieval_below_floor` (only with `min_recall` set) | gate | retrieval config |
| `grounding_below_floor` (only with `min_kb_confirmed_pct` set) | gate | generator |
| `corpus_at_risk` | risk | ingestion |
| `corpus_verdict_missing` * | risk | ingestion |
| `documents_unexamined` * (skipped, pending or empty checks) | risk | ingestion |
| `partial_run` * (`discover --since`) | risk | ingestion |
| `retrieval_query_errors` * | risk | retrieval config |
| `retrieval_found_nothing` | risk | retrieval config |
| `grounding_retrieval_gap` | risk | retrieval config |
| `grounding_generation` (unproven) | risk | generator |
| `grounding_wrong_count` | risk | generator |
| `grounding_fabricated_source` | risk | generator |
| `drift_detected` | risk | ingestion |
| `fidelity_content_loss` | advisory | ingestion |
| `fidelity_numbers_missing` | advisory | ingestion |
| `fidelity_value_drift` | advisory | ingestion |
| `chunk_integrity_findings` | advisory | ingestion |
| `stale_versions`, `stale_versions_unchecked` | advisory | ingestion |
| `documents_within_threshold` | advisory | ingestion |
| `grounding_direction_mismatch` (word-adjacency heuristic) | advisory | generator |
| `grounding_citation_error`, `grounding_mostly_unverifiable` | advisory | generator |
| `retrieval_no_floor`, `retrieval_unfiltered`, `retrieval_cases_excluded` | advisory | retrieval config |
| `retrieval_empty_under_filter` | advisory | ingestion |

\* A "could not check" finding: it can be raised to `gate`, never lowered to `advisory`.

Change the table per project. An unknown key exits 2, because a typo here would turn a gate off:

```yaml
readiness:
  severity: {stale_versions: risk, grounding_direction_mismatch: gate}
  waived_stages: [drift]     # stages this project does not run, so absence stops capping
  min_recall: 0.6            # 0 = no floor: recall is reported, not gated
  min_kb_confirmed_pct: 80
```

`enrich` adds a `compass` block per case (a **citation grounding rate** over the verifiable
citation+numeric subset, citation fidelity, chunk quality, context utilization, `COMPASS_*` signal
codes). Each citation also carries a **graduated support tier** — `SOUND` / `LIKELY_SOUND` (≥70% of
its values found) / `PARTIAL` / `UNSUPPORTED` — reported alongside the strict rate, so a near-miss
is not filed next to a fabrication. Short cited tokens ("WIU", "Q3 QIP") resolve automatically
against the index's own document names; a token owned by two documents resolves to nothing rather
than guessing. It is **schema-agnostic** — every field resolves through your `eval.gt_*` config, so it
works whatever shape your eval JSON takes. `calibrate-judges` then reports each judge's error,
bias, accuracy, and an evidence-based **recommended weight** — something no eval framework can
compute, because it needs KB access.

`migrate` and `coverage` also render a **self-contained HTML report** (same look as the
validation dashboard) with `--html [path] --open` — a coverage heatmap, a migration
stability distribution, verdicts, and the most-changed / dead-zone tables.

> Same boundary as the gate: these never *judge answer quality* (that's the eval layer) and
> never put an LLM in a pass/fail decision. `migrate` and `coverage` share one deterministic
> retrieval-analysis engine (sample → in-memory KNN → neighbor/reachability analysis).

### Catch regressions over time (baselines + drift)

Snapshot a known-good run, then diff later runs against it — so a scheduled check is a
gate, not just a report:

```bash
compass discover --project my-kb --save-baseline          # freeze a green run -> baselines/my-kb.json
# ...later (cron / CI)...
compass discover --project my-kb --against baselines/my-kb.json --fail-on-regression
```

Drift flags what a single run can't: **dropped documents**, docs that **passed before and
fail now** (and which checks newly fail), **index drift** (embedding dimension / distance
metric changed under you), and **readiness / pass-rate decline**. `--fail-on-regression`
makes the run exit non-zero, so cron or CI alerts on degradation. (`compass check` takes
the same `--save-baseline` / `--against` flags.)
```

> From source instead of PyPI: `git clone … && cd compass && pip install -e ".[all]"`.

### Running scenarios without editing YAML

`compass-discover` takes overrides that win over config + profiling, so you can try
different shapes and policies from the command line:

```bash
# Point at a different index/identity, faster, and relax the readiness bar
compass-discover --project my-kb --parallel --workers 8 \
  --id-field agent_id --expected-dim 1024 \
  --readiness-threshold 0.99 \
  --disable-check chunk_count near_duplicate_embeddings \
  --severity duplicate_detection=hard \
  --unreachable-as skipped
```

Severity model: every vector check is **HARD** (gates the run) or **WARN** (advisory).
`--severity name=hard|warn` (or `redis.check_severity` in YAML) promotes/demotes any
check; `--list-checks` shows the defaults.
> Releases are published via [PUBLISHING.md](PUBLISHING.md).

Reports are written to `reports/` (HTML dashboard + JSON + JUnit XML). Add
`--live` to open a real-time progress dashboard, `--trends` to record history,
`--parallel` for concurrency.

### Try it locally (no cloud needed)

A self-contained Docker stack runs the full pipeline against MinIO + Redis Stack +
a mock ingestion API — see **[integration/README.md](integration/README.md)**.

## Project Structure

```
compass/
├── run_validation.py              # Entry point — live ingestion validation
├── discover_and_validate.py       # Entry point — post-hoc discovery
├── compass/          # Main package
│   ├── models/                    # StepResult / PipelineResult (stdlib dataclasses)
│   ├── config/                    # Layered YAML config + typed Settings
│   ├── utils/                     # Run-id, logging, polling w/ backoff
│   ├── providers/                 # Pluggable backends via a registry:
│   │   ├── storage.py             #   object storage (S3-family…)
│   │   ├── vectorstore.py         #   vector store (Redis, Qdrant, pgvector, Chroma…)
│   │   ├── graph.py               #   knowledge graph (SPARQL, Neptune…)
│   │   ├── embedding.py           #   text→vector for query (OpenAI/Azure/Bedrock)
│   │   └── source.py              #   document discovery (storage|index|manifest)
│   ├── validators/                # BaseValidator + step registry + the steps
│   │   └── registry.py            #   declarative pipeline (StepSpec registry)
│   ├── pipeline/                  # Orchestrator (context propagation, halting)
│   ├── report/                    # Self-contained HTML dashboard + JSON/JUnit
│   ├── cli.py                     # Unified `compass` CLI (all subcommands)
│   ├── ask.py                     # `compass ask` — deterministic NL retrieval
│   ├── mcp_server.py              # `compass mcp` — agent-callable KB tools (stdio)
│   ├── retrieval_analysis.py      # engine for `migrate`/`coverage`/`compare` (read-only)
│   ├── eval_interop.py            # vendor-neutral eval export + ground-truth reader
│   ├── corpus.py                  # Corpus-level KB readiness analysis
│   ├── live_dashboard.py          # FastAPI + SSE real-time dashboard
│   ├── notifications.py           # Slack / Teams webhooks
│   └── trend_tracker.py           # SQLite historical trends
├── config/{base.yaml, projects/_template.yaml}
├── test_data/_template.yaml       # Documents to validate (live mode)
├── integration/                   # Local Docker integration stack
├── tests/                         # Unit tests (backends faked — no live services)
└── reports/                       # Generated reports (gitignored)
```

## Documentation

- **[QUICKSTART.md](QUICKSTART.md)** — pip install → a gating CI check in five steps
- **[ARCHITECTURE.md](ARCHITECTURE.md)** — how COMPASS is built: layers, data flow, the registries, the two run modes, extension points
- **[CONTRACT.md](CONTRACT.md)** — the Knowledge-Base Contract: express your KB's guarantees as one declarative artifact, with each clause mapped to the command/flag that enforces it in CI
- **[CASE_STUDIES.md](CASE_STUDIES.md)** — the reproducible benchmark story in full, plus templates for real-outcome case studies (missing docs, filter regressions, migration drift, orphan chunks)
- **[benchmark/](benchmark/)** — the broken-KB benchmark: injects realistic faults and scores detection rate (currently **16/16**) vs false-positive rate (**0/6**)
- **[SUPPORT.md](SUPPORT.md)** — backend support matrix (which backends are *verified* vs *adapter-available*)
- **[COMPATIBILITY.md](COMPATIBILITY.md)** — stability & versioning policy: what's stable, what's evolving, deprecation & pinning
- **[EXTENDING-BACKENDS.md](EXTENDING-BACKENDS.md)** — add a new backend (storage/vector/graph/source) or a new pipeline step
- **[integration/README.md](integration/README.md)** — run the full pipeline locally on Docker
- [CONTRIBUTING.md](CONTRIBUTING.md) · [SECURITY.md](SECURITY.md)

## Configuration

All behaviour lives in YAML; credentials are referenced by **environment-variable
name** and resolved at runtime (never stored in config). `config/base.yaml` holds
shared defaults; each `config/projects/<name>.yaml` overlays only what differs and
selects a `dev`/`staging`/`prod` environment block.

```yaml
# config/projects/my-kb.yaml
display_name: "My Knowledge Base"

# Declarative pipeline — choose which steps run, in what order (registry keys).
# Omit to use the full default pipeline.
pipeline:
  steps: [api_upload, s3_storage, redis_chunks, retrieval_quality]
  critical_steps: ["API Upload", "S3 Storage"]   # failure here halts the run

# Where post-hoc discovery finds documents: storage | vectorstore | manifest
discovery:
  provider: vectorstore           # enumerate the KB straight from the index

environments:
  dev:
    api:
      base_url_env: API_BASE_URL          # env-var NAME, not the value
      graphql_mutation: |
        mutation ($file: Upload!, $metadata: JSON) { uploadDocument(file:$file, metadata:$metadata){ status document_id } }
    s3:
      provider: s3                        # s3 | minio | r2 | azure_blob | gcs …
      bucket_name: my-documents
      prefix: knowledge-base/
    redis:
      provider: redis                     # redis | qdrant | pinecone | pgvector …
      search_index: my_embeddings
      expected_embedding_dim: 1536
      expected_distance_metric: COSINE
```

`redis.chunking_strategy` selects the chunk checks. Under anything but `fixed`, size-derived checks
stop applying and advisory checks take over: fragmentation, summary presence, `element_integrity`
(a chunk holding part of a table, list or code block), `boundary_on_structure` (a boundary inside
one, which needs a positional field) and `cap_truncation` (a chunk at `max_chunk_size` that ends
inside one). A structure continuing onto the next page, a mid-sentence end and a heading left at a
chunk's tail are how layout chunkers work, and are not counted. The strategy comes from the chunks
when `redis.chunker_field` and `chunker_strategy_map` are set, else from `chunking_strategy`. A
project that declares neither is scored exactly as before: where the chunks are clearly not
fixed-size and carry a summary or markdown structure, the structural rows appear marked `inferred`,
with their evidence, and are never scored.

`embedding.input_fields`, `input_joiner` and `input_truncation` declare what the pipeline embeds, so
the embedding probes re-embed that and nothing else. `embedding.sync_check` (off by default) has
`discover` re-embed a sample and name chunks whose vector does not match; a chunk missing a declared
field is unverifiable, not out of sync, and `tokens:N` truncation is reported, never estimated.

`discover --since` re-validates a document when its ETag, its chunk count, or (where the store can
list them in one aggregation: Redis today) its chunk ids or `redis.chunker_version_field` value
changed. The id hash is over sorted ids, so it sees a changed set rather than a reordering, and ids
built from document + position do not change under a same-count re-chunk: point
`redis.chunk_id_field` at a content hash if the pipeline stores one.

## Advanced Usage

### Add a custom pipeline step

Steps are declarative — register a validator and reference it in `pipeline.steps`:

```python
from compass.validators.base import BaseValidator
from compass.validators.registry import register_step, StepSpec
from compass.models import StepResult, StepStatus

class PiiRedactionValidator(BaseValidator):
    step_name = "PII Redaction"
    def __init__(self, config): self.config = config
    def validate(self, context: dict) -> StepResult:
        return StepResult(self.step_name, StepStatus.PASSED, "no PII leaked")

register_step(StepSpec(
    key="pii_redaction", name="PII Redaction",
    factory=lambda settings, shared: PiiRedactionValidator(settings.s3),
    abbr="PII", order=25,
))
# then:  pipeline.steps: [api_upload, pii_redaction, redis_chunks]
```

The orchestrator, live dashboard, and HTML report adapt automatically. See
**[EXTENDING-BACKENDS.md](EXTENDING-BACKENDS.md)** for adding backends too.

### CI/CD integration — the readiness gate

COMPASS is built to run **before** your agents or eval consume the knowledge
base: wire it into the pipeline that publishes the KB and fail the build when
the KB is not retrieval-ready. Every entry point returns a CI-friendly exit
code — **0 = gate passed, 1 = gate failed, 2 = config error** — and emits JUnit
XML for any CI system.

> **Reliable exit codes:** invoke the gate via `python -m compass discover …`
> or the `compass-discover` console script — both `sys.exit()` the code directly. Avoid
> piping (`… | tee`) which returns the pipe's exit code, and note that the `compass` shim in
> Git-Bash on Windows can drop the code. To verify in a shell: `… ; echo $?` (bash) or
> `…; echo $LASTEXITCODE` (PowerShell).

**Gate strictness** (discovery mode) is yours to choose:

```bash
# Strictest: any failed document fails the build (default)
compass-discover --project my-kb

# Tolerate a few failures: require >= 95% of documents to pass
compass-discover --project my-kb --fail-under 95

# Also require the corpus to be READY (completeness/coverage/dedup verdict)
compass-discover --project my-kb --require-ready
```

**GitHub Actions** — drop the reusable action into your KB pipeline:

```yaml
# .github/workflows/kb-readiness.yml
jobs:
  kb-readiness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4              # your repo holds config/projects/<name>.yaml
      - uses: TestAutomationArchitect/compass@v1
        with:
          project: my-kb
          require-ready: "true"                # block promotion unless READY
          # fail-under: "95"                   # ...or tolerate a bounded failure rate
        env:
          REDIS_HOST: ${{ secrets.REDIS_HOST }}   # config references env-var names; supply them here
```

A full example you can copy is in **[examples/kb-readiness.yml](examples/kb-readiness.yml)**.

**Container** — no Python setup needed; works in any CI or a self-hosted runner:

```bash
docker build -t compass:latest .
docker run --rm -v "$PWD:/work" --env-file .env \
  compass:latest compass-discover --project my-kb --require-ready
```

## Applicable Domains

Enterprise document management · legal/compliance · healthcare · financial
research · customer-support KBs · internal wikis · e-commerce catalogs · academic
repositories — any RAG/agent knowledge base built from a document pipeline.

## License

MIT License — see [LICENSE](LICENSE).
