Metadata-Version: 2.4
Name: compass-kb-validation
Version: 1.22.1
Summary: Knowledge-base readiness validation for RAG & AI agents — config-driven, deterministic pipeline integrity checks.
Author: COMPASS
License: MIT
Project-URL: Homepage, https://github.com/TestAutomationArchitect/compass
Project-URL: Repository, https://github.com/TestAutomationArchitect/compass
Project-URL: Issues, https://github.com/TestAutomationArchitect/compass/issues
Keywords: rag,knowledge-base,vector-database,data-validation,pipeline-testing,llm-infrastructure
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests>=2.31.0
Requires-Dist: boto3>=1.34.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: redis>=5.0.0
Provides-Extra: dashboard
Requires-Dist: fastapi>=0.104.0; extra == "dashboard"
Requires-Dist: uvicorn>=0.24.0; extra == "dashboard"
Provides-Extra: pdf
Requires-Dist: PyPDF2>=3.0.0; extra == "pdf"
Provides-Extra: ui
Requires-Dist: playwright>=1.40.0; extra == "ui"
Provides-Extra: qdrant
Requires-Dist: qdrant-client>=1.7.0; extra == "qdrant"
Provides-Extra: pgvector
Requires-Dist: psycopg[binary]>=3.1; extra == "pgvector"
Provides-Extra: pinecone
Requires-Dist: pinecone>=5.0.0; extra == "pinecone"
Provides-Extra: weaviate
Requires-Dist: weaviate-client>=4.5.0; extra == "weaviate"
Provides-Extra: azure
Requires-Dist: azure-storage-blob>=12.0.0; extra == "azure"
Requires-Dist: azure-identity>=1.15.0; extra == "azure"
Provides-Extra: gcs
Requires-Dist: google-cloud-storage>=2.10.0; extra == "gcs"
Provides-Extra: all
Requires-Dist: fastapi>=0.104.0; extra == "all"
Requires-Dist: uvicorn>=0.24.0; extra == "all"
Requires-Dist: PyPDF2>=3.0.0; extra == "all"
Requires-Dist: playwright>=1.40.0; extra == "all"
Requires-Dist: qdrant-client>=1.7.0; extra == "all"
Requires-Dist: psycopg[binary]>=3.1; extra == "all"
Requires-Dist: pinecone>=5.0.0; extra == "all"
Requires-Dist: weaviate-client>=4.5.0; extra == "all"
Requires-Dist: azure-storage-blob>=12.0.0; extra == "all"
Requires-Dist: azure-identity>=1.15.0; extra == "all"
Requires-Dist: google-cloud-storage>=2.10.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: pyflakes>=3.0.0; extra == "dev"
Requires-Dist: pyright==1.1.409; extra == "dev"
Requires-Dist: bandit>=1.7.5; extra == "dev"
Requires-Dist: pip-audit>=2.6.0; extra == "dev"
Dynamic: license-file

# COMPASS — Knowledge-Base Readiness Validation for RAG & AI Agents

[![CI](https://github.com/TestAutomationArchitect/compass/actions/workflows/ci.yml/badge.svg)](https://github.com/TestAutomationArchitect/compass/actions/workflows/ci.yml)

**COMPASS** — **COM**prehensive **P**ipeline **A**nalysis & **S**tructure **S**earch

A config-driven, **deterministic** framework that proves an AI/RAG knowledge base is correctly **built, intact, and retrievable** — *before* AI agents consume it. It's the integrity gate beneath answer-quality evaluation: not "are the answers good?" but **"is the knowledge base itself sound?"**

## What COMPASS Does

COMPASS validates a document's journey through your ingestion pipeline and verifies the resulting knowledge base is agent-ready. The pipeline is **declarative** — you choose which steps run, in what order, in YAML — and ships with these built-in steps:

```
API Upload → Object Storage → Knowledge Graph → Vector Store → Retrieval → UI Visibility
```

Enable only the steps your setup has (no object store? no graph? drop them), reorder them, or register your own. For each enabled step it deterministically checks, for example:

- **Ingestion** — the document was accepted and assigned an id
- **Object storage** — it persisted with the right content type, size, and integrity (S3, MinIO, R2, Azure Blob, GCS…)
- **Knowledge graph** — required metadata predicates are present and well-formed (GraphDB, Neptune, Neo4j…)
- **Vector store** — chunks exist with valid embeddings, correct dimensions/metric, sane ordering/overlap, retrievable via KNN, no cross-document contamination, no near-duplicate bloat (Redis, Qdrant, Pinecone, pgvector…) — **29 toggleable checks**, each on a **HARD / WARN** severity tier (only HARD gates the run)
- **Retrieval** — the document's own content is actually retrievable (recall@K, MRR — informational)
- **UI visibility** — it surfaces in the end-user application

Two modes: **live** validation as documents are ingested, and **post-hoc discovery** of a corpus that's already loaded.

> Scope note: COMPASS validates *pipeline & data integrity*. It does **not** judge answer quality — that's the evaluation layer's job. COMPASS is the gate that runs first.

## Key Features

✅ **Declarative pipeline** — choose/reorder/extend steps in YAML; no fixed shape  
✅ **Config-Driven** — zero code changes per project; all customization in YAML  
✅ **Pluggable backends** — storage / vector store / knowledge graph via a provider registry (one file to add a vendor)  
✅ **Pluggable steps** — register custom validators in a step registry; the report adapts automatically  
✅ **Deterministic & cheap** — structural/byte/schema checks, no LLM, CI-gateable  
✅ **Query & analyze the KB** — ask it questions, expose it to agents over MCP, and preview embedding-model migrations / map retrieval coverage — all deterministic, read-only (see *Beyond validation*)  
✅ **Interactive reporting** — self-contained HTML dashboard + JSON + JUnit XML  
✅ **Trend Tracking** — SQLite historical performance analysis  
✅ **Notifications** — Slack/Teams webhooks for alerts  
✅ **Parallel Execution** — process many documents concurrently  
✅ **Graceful Degradation** — non-critical failures don't crash the pipeline  

## Quick Start

Install the package (core deps only; add extras per backend — see below):

```bash
pip install compass-kb-validation
# backends/features on demand, e.g.:
pip install "compass-kb-validation[qdrant,pgvector]"
```

### Fastest path (no config)

```bash
compass demo                                   # opens a sample report - see it work, no backend
compass check redis://localhost:6379/my_index  # point at a live index: profile -> validate -> report
```

`compass check` builds everything in memory (profiles the index, fills config from the
detected shape) - no project YAML. Credentials come from the URL or your env.

### Project path (saved config)

Then use the unified **`compass`** CLI - one verb, subcommands:

```bash
compass init my-kb          # scaffold config/projects/my-kb.yaml + .env.example + a .compass marker
cp .env.example .env        # fill in your backend connection
#   edit config/projects/my-kb.yaml  (search_index, document_id_field)
compass doctor --project my-kb     # verify connectivity + config (per stage)
compass discover --project my-kb   # validate the already-populated KB, emit a report
```

`compass init` drops a `.compass` marker, so the CLI works from any subdirectory of the
project (no env var to set). Other subcommands:

```bash
compass validate --project my-kb --output-format all   # live ingestion validation
compass profile  --project my-kb                        # read a store's shape, suggest config
compass list-checks                                     # check catalogue + severity tiers (no project needed)
compass <cmd> --help                                    # per-command options
```

The standalone `compass-discover` / `compass-validate` / `compass-profile` commands keep
working. Live validation also reads a `test_data/<name>.yaml` (scaffold it with
`compass-validate --project my-kb --init-test-data`).

### Beyond validation — query, migrate, map coverage

Once a KB is validated, COMPASS can also *use* it — deterministically, no LLM, read-only.
These read an `embedding:` block in the project YAML (provider `openai` / `azure_openai` /
`bedrock`, `model_id`, `dimensions`) so a question is embedded the same way ingestion was.

```bash
# Ask the KB a question - returns the ranked chunks it would retrieve (no generated answer)
compass ask --project my-kb "what is the refund policy?"

# Match production retrieval with metadata pre-filters (hybrid search; commas AND, | ORs a field)
compass ask --project my-kb --filter "category=report,region=US|EU" "summarize the latest report"

# Expose the KB to agents (Copilot/Claude) as MCP tools over stdio
compass mcp --project my-kb --print-config   # prints a .vscode/mcp.json snippet
compass mcp --project my-kb                   # run the server (query_kb, list_documents, ...)

# Preview an embedding-model switch BEFORE you cut over (self-bootstraps, no questions needed)
compass migrate --project my-kb --to openai/text-embedding-3-large

# Map retrieval coverage / find dead zones a question set never reaches (feed real query logs)
compass coverage --project my-kb --questions query_logs.jsonl

# Prove retrieval matches production: compare vs a ground-truth file (precision/recall/Jaccard)
compass compare --project my-kb --ground-truth production_ground_truth.json
# Mixed ground-truth file? Non-retrieval cases (no chunks) are auto-skipped; filter by variant:
compass compare --project my-kb --ground-truth gt.json --variant answerable

# Hand retrieval to any eval framework: emit generic RAG eval test cases
compass ask --project my-kb --batch questions.jsonl --format eval --output eval_input.jsonl

# Corpus health (deterministic, no LLM): score chunk quality / find stale + orphan sources
compass quality   --project my-kb
compass freshness --project my-kb --stale-after 30
```

`migrate` and `coverage` also render a **self-contained HTML report** (same look as the
validation dashboard) with `--html [path] --open` — a coverage heatmap, a migration
stability distribution, verdicts, and the most-changed / dead-zone tables.

> Same boundary as the gate: these never *judge answer quality* (that's the eval layer) and
> never put an LLM in a pass/fail decision. `migrate` and `coverage` share one deterministic
> retrieval-analysis engine (sample → in-memory KNN → neighbor/reachability analysis).

### Catch regressions over time (baselines + drift)

Snapshot a known-good run, then diff later runs against it — so a scheduled check is a
gate, not just a report:

```bash
compass discover --project my-kb --save-baseline          # freeze a green run -> baselines/my-kb.json
# ...later (cron / CI)...
compass discover --project my-kb --against baselines/my-kb.json --fail-on-regression
```

Drift flags what a single run can't: **dropped documents**, docs that **passed before and
fail now** (and which checks newly fail), **index drift** (embedding dimension / distance
metric changed under you), and **readiness / pass-rate decline**. `--fail-on-regression`
makes the run exit non-zero, so cron or CI alerts on degradation. (`compass check` takes
the same `--save-baseline` / `--against` flags.)
```

> From source instead of PyPI: `git clone … && cd compass && pip install -e ".[all]"`.

### Running scenarios without editing YAML

`compass-discover` takes overrides that win over config + profiling, so you can try
different shapes and policies from the command line:

```bash
# Point at a different index/identity, faster, and relax the readiness bar
compass-discover --project my-kb --parallel --workers 8 \
  --id-field agent_id --expected-dim 1024 \
  --readiness-threshold 0.99 \
  --disable-check chunk_count near_duplicate_embeddings \
  --severity duplicate_detection=hard \
  --unreachable-as skipped
```

Severity model: every vector check is **HARD** (gates the run) or **WARN** (advisory).
`--severity name=hard|warn` (or `redis.check_severity` in YAML) promotes/demotes any
check; `--list-checks` shows the defaults.
> Releases are published via [PUBLISHING.md](PUBLISHING.md).

Reports are written to `reports/` (HTML dashboard + JSON + JUnit XML). Add
`--live` to open a real-time progress dashboard, `--trends` to record history,
`--parallel` for concurrency.

### Try it locally (no cloud needed)

A self-contained Docker stack runs the full pipeline against MinIO + Redis Stack +
a mock ingestion API — see **[integration/README.md](integration/README.md)**.

## Project Structure

```
compass/
├── run_validation.py              # Entry point — live ingestion validation
├── discover_and_validate.py       # Entry point — post-hoc discovery
├── ingestion_validation/          # Main package
│   ├── models/                    # StepResult / PipelineResult (stdlib dataclasses)
│   ├── config/                    # Layered YAML config + typed Settings
│   ├── utils/                     # Run-id, logging, polling w/ backoff
│   ├── providers/                 # Pluggable backends via a registry:
│   │   ├── storage.py             #   object storage (S3-family…)
│   │   ├── vectorstore.py         #   vector store (Redis/RediSearch…)
│   │   ├── graph.py               #   knowledge graph (SPARQL, Neptune…)
│   │   ├── embedding.py           #   text→vector for query (OpenAI/Azure/Bedrock)
│   │   └── source.py              #   document discovery (storage|index|manifest)
│   ├── validators/                # BaseValidator + step registry + the steps
│   │   └── registry.py            #   declarative pipeline (StepSpec registry)
│   ├── pipeline/                  # Orchestrator (context propagation, halting)
│   ├── report/                    # Self-contained HTML dashboard + JSON/JUnit
│   ├── cli.py                     # Unified `compass` CLI (all subcommands)
│   ├── ask.py                     # `compass ask` — deterministic NL retrieval
│   ├── mcp_server.py              # `compass mcp` — agent-callable KB tools (stdio)
│   ├── retrieval_analysis.py      # engine for `migrate`/`coverage`/`compare` (read-only)
│   ├── eval_interop.py            # vendor-neutral eval export + ground-truth reader
│   ├── corpus.py                  # Corpus-level KB readiness analysis
│   ├── live_dashboard.py          # FastAPI + SSE real-time dashboard
│   ├── notifications.py           # Slack / Teams webhooks
│   └── trend_tracker.py           # SQLite historical trends
├── config/{base.yaml, projects/_template.yaml}
├── test_data/_template.yaml       # Documents to validate (live mode)
├── integration/                   # Local Docker integration stack
├── tests/                         # Unit tests (backends faked — no live services)
└── reports/                       # Generated reports (gitignored)
```

## Documentation

- **[ARCHITECTURE.md](ARCHITECTURE.md)** — how COMPASS is built: layers, data flow, the registries, the two run modes, extension points
- **[EXTENDING-BACKENDS.md](EXTENDING-BACKENDS.md)** — add a new backend (storage/vector/graph/source) or a new pipeline step
- **[integration/README.md](integration/README.md)** — run the full pipeline locally on Docker
- [CONTRIBUTING.md](CONTRIBUTING.md) · [SECURITY.md](SECURITY.md)

## Configuration

All behaviour lives in YAML; credentials are referenced by **environment-variable
name** and resolved at runtime (never stored in config). `config/base.yaml` holds
shared defaults; each `config/projects/<name>.yaml` overlays only what differs and
selects a `dev`/`staging`/`prod` environment block.

```yaml
# config/projects/my-kb.yaml
display_name: "My Knowledge Base"

# Declarative pipeline — choose which steps run, in what order (registry keys).
# Omit to use the full default pipeline.
pipeline:
  steps: [api_upload, s3_storage, redis_chunks, retrieval_quality]
  critical_steps: ["API Upload", "S3 Storage"]   # failure here halts the run

# Where post-hoc discovery finds documents: storage | vectorstore | manifest
discovery:
  provider: vectorstore           # enumerate the KB straight from the index

environments:
  dev:
    api:
      base_url_env: API_BASE_URL          # env-var NAME, not the value
      graphql_mutation: |
        mutation ($file: Upload!, $metadata: JSON) { uploadDocument(file:$file, metadata:$metadata){ status document_id } }
    s3:
      provider: s3                        # s3 | minio | r2 | azure_blob | gcs …
      bucket_name: my-documents
      prefix: knowledge-base/
    redis:
      provider: redis                     # redis | qdrant | pinecone | pgvector …
      search_index: my_embeddings
      expected_embedding_dim: 1536
      expected_distance_metric: COSINE
```

## Advanced Usage

### Add a custom pipeline step

Steps are declarative — register a validator and reference it in `pipeline.steps`:

```python
from ingestion_validation.validators.base import BaseValidator
from ingestion_validation.validators.registry import register_step, StepSpec
from ingestion_validation.models import StepResult, StepStatus

class PiiRedactionValidator(BaseValidator):
    step_name = "PII Redaction"
    def __init__(self, config): self.config = config
    def validate(self, context: dict) -> StepResult:
        return StepResult(self.step_name, StepStatus.PASSED, "no PII leaked")

register_step(StepSpec(
    key="pii_redaction", name="PII Redaction",
    factory=lambda settings, shared: PiiRedactionValidator(settings.s3),
    abbr="PII", order=25,
))
# then:  pipeline.steps: [api_upload, pii_redaction, redis_chunks]
```

The orchestrator, live dashboard, and HTML report adapt automatically. See
**[EXTENDING-BACKENDS.md](EXTENDING-BACKENDS.md)** for adding backends too.

### CI/CD integration — the readiness gate

COMPASS is built to run **before** your agents or eval consume the knowledge
base: wire it into the pipeline that publishes the KB and fail the build when
the KB is not retrieval-ready. Every entry point returns a CI-friendly exit
code — **0 = gate passed, 1 = gate failed, 2 = config error** — and emits JUnit
XML for any CI system.

> **Reliable exit codes:** invoke the gate via `python -m ingestion_validation discover …`
> or the `compass-discover` console script — both `sys.exit()` the code directly. Avoid
> piping (`… | tee`) which returns the pipe's exit code, and note that the `compass` shim in
> Git-Bash on Windows can drop the code. To verify in a shell: `… ; echo $?` (bash) or
> `…; echo $LASTEXITCODE` (PowerShell).

**Gate strictness** (discovery mode) is yours to choose:

```bash
# Strictest: any failed document fails the build (default)
compass-discover --project my-kb

# Tolerate a few failures: require >= 95% of documents to pass
compass-discover --project my-kb --fail-under 95

# Also require the corpus to be READY (completeness/coverage/dedup verdict)
compass-discover --project my-kb --require-ready
```

**GitHub Actions** — drop the reusable action into your KB pipeline:

```yaml
# .github/workflows/kb-readiness.yml
jobs:
  kb-readiness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4              # your repo holds config/projects/<name>.yaml
      - uses: TestAutomationArchitect/compass@v1
        with:
          project: my-kb
          require-ready: "true"                # block promotion unless READY
          # fail-under: "95"                   # ...or tolerate a bounded failure rate
        env:
          REDIS_HOST: ${{ secrets.REDIS_HOST }}   # config references env-var names; supply them here
```

A full example you can copy is in **[examples/kb-readiness.yml](examples/kb-readiness.yml)**.

**Container** — no Python setup needed; works in any CI or a self-hosted runner:

```bash
docker build -t compass:latest .
docker run --rm -v "$PWD:/work" --env-file .env \
  compass:latest compass-discover --project my-kb --require-ready
```

## Applicable Domains

Enterprise document management · legal/compliance · healthcare · financial
research · customer-support KBs · internal wikis · e-commerce catalogs · academic
repositories — any RAG/agent knowledge base built from a document pipeline.

## License

MIT License — see [LICENSE](LICENSE).
