Metadata-Version: 2.4
Name: rag-staleness-check
Version: 0.1.0
Summary: Read-only staleness/orphan/duplicate/retrievability checks for a single pgvector, Qdrant, or Chroma index.
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/rimironenko/rag-staleness-check
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: psycopg[binary]<4,>=3.1
Provides-Extra: qdrant
Requires-Dist: qdrant-client<2,>=1.9; extra == "qdrant"
Provides-Extra: chroma
Requires-Dist: chromadb<1,>=0.5; extra == "chroma"
Provides-Extra: all
Requires-Dist: qdrant-client<2,>=1.9; extra == "all"
Requires-Dist: chromadb<1,>=0.5; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# rag-staleness-check

Read-only staleness / orphan / duplicate / retrievability checks for a
single pgvector, Qdrant, or Chroma index — the open-source half of the
[RAGproof](https://ragproof.io) "decayed RAG index" teardown
([full writeup + multi-engine ledger-verified methodology](https://ragproof.io)).

This tool runs against **your own** already-indexed vector database and
tells you:

- **staleness** — indexed chunks whose source document has since changed
  (needs a manifest with `last_modified` per document, and the engine
  itself storing a per-row last-modified value)
- **orphans** — indexed chunks whose source document no longer exists in
  your manifest at all
- **duplicates** — near-identical chunks (cosine similarity ≥ threshold,
  default 0.98), plus an exact-hash pass if you store a per-chunk content
  hash
- **retrievable-after-delete** — a basic probe: for ids you believe are
  deleted, is the vector still physically retrievable by id (storage-layer
  persistence), and does it still surface in top-k search results
  (functional leak)?

It is **read-only**: it never calls a write/delete/upsert method on your
engine. See [Read-only guarantee](#read-only-guarantee) below for exactly
how that's enforced, and its documented limits.

## What this tool does NOT do

This is the single-engine, ledger-free, self-serve slice. It deliberately
does not include:

- multi-engine orchestration (run it once per engine yourself)
- the GDPR Article-17 reporting pack (signed evidence, verifiable-erasure
  report)
- engine-specific deep checks (pgvector dead-tuple/VACUUM detail, Qdrant
  optimizer-threshold detail, Chroma on-disk HNSW growth)
- a ledger-based precision/recall harness (there's no ground truth here —
  this tool reports what it finds, not how accurate the finding is)

Those live in the private, paid [RAGproof](https://ragproof.io) audit,
which cross-validates every finding against a git-history-derived
ground-truth ledger across all three engines simultaneously.

## Install

```bash
pip install rag-staleness-check              # pgvector only
pip install "rag-staleness-check[qdrant]"     # + Qdrant
pip install "rag-staleness-check[chroma]"     # + Chroma
pip install "rag-staleness-check[all]"        # everything
```

Or, without installing:

```bash
pipx run rag-staleness-check --engine pgvector --dsn "$DSN" --source ./docs_manifest.json
```

Requires Python ≥ 3.10.

## Usage

```bash
rag-staleness-check \
  --engine pgvector \
  --dsn "postgresql://readonly_user:pw@localhost:5432/mydb" \
  --pg-table chunks \
  --pg-doc-id-column doc_id \
  --pg-last-modified-column last_modified \
  --pg-content-hash-column content_sha256 \
  --source ./docs_manifest.json \
  --deleted-ids ./deleted_ids.json \
  --out findings.json
```

Prints a scorecard to stdout and writes the full result to `--out`
(default `findings.json`). Every check that's missing a required input
(no `--source`, no per-row metadata configured, no `--deleted-ids`)
reports `"skipped": true` with a reason — it never silently reports a
misleadingly-clean 0%.

### `--dsn` per engine

| Engine | `--dsn` format | Example |
|---|---|---|
| `pgvector` | Postgres DSN | `postgresql://user:pw@localhost:5432/db` |
| `qdrant` | Base URL | `http://localhost:6333` |
| `chroma` | `host:port` | `localhost:8000` |

### Schema mapping

A third party's table/column or payload/metadata field names are arbitrary
— this tool doesn't assume anything about your schema beyond what you tell
it via flags:

**pgvector** (`--pg-*`): `--pg-table` (required), `--pg-id-column`
(default `id`), `--pg-vector-column` (default `embedding`),
`--pg-doc-id-column`, `--pg-last-modified-column`,
`--pg-content-hash-column` (all three optional — their absence is exactly
what makes staleness / orphans / the duplicates check's exact-hash pass
report `skipped` / zero-coverage).

**Qdrant / Chroma** (`--collection` required, plus payload/metadata field
names): `--doc-id-field` (default `doc_id`), `--last-modified-field`
(default `last_modified`), `--content-hash-field` (default
`content_sha256`).

### Manifest (`--source`)

A JSON file describing what documents *should* exist — the join key for
staleness and orphans. There is no ledger in this tool; this manifest is
the only source of truth about "what's current."

```json
{
  "manifest_version": 1,
  "documents": [
    {
      "doc_id": "handbook/engineering/on-call.md",
      "last_modified": "2026-06-01T00:00:00Z",
      "content_sha256": "optional, doc-level, informational only"
    }
  ]
}
```

`doc_id` must match whatever value your own ingestion pipeline wrote into
each indexed chunk's doc-id column/field (see schema mapping above).
`last_modified` is only needed if you want the staleness check to run.
`content_sha256` here is doc-level and purely informational — it is *not*
the input to duplicate detection (see below). See
[`examples/docs_manifest.json`](examples/docs_manifest.json).

### Deleted-ids probe (`--deleted-ids`)

A JSON array of ids you believe are deleted:

```json
["chunk-abc123", "chunk-def456"]
```

For each: checks whether the vector is still retrievable by id
(storage-layer persistence) and, if so, whether it still surfaces in a
top-k self-query (functional leak). This distinction matters — per EDPB
Guidelines 05/2019, erasure must be *verifiable and irreversible*;
suppressing a record from search results alone does not, by itself,
satisfy that if the underlying vector is still present. `findings.json`'s
`retrievable.framing` field states this explicitly every run.

## Read-only guarantee

This tool never calls a write/delete/upsert method on your engine.
Enforced two ways:

1. **Structurally, via a static test** —
   [`tests/test_no_write_methods.py`](tests/test_no_write_methods.py)
   AST-walks every file under `src/rag_staleness_check/` and fails the
   build if any write/delete/upsert-shaped method is called, or any
   SQL write statement is passed to `execute()`/`executemany()`. Not a
   substring grep (which would false-positive on this very README/these
   docstrings) — it parses the actual syntax tree.
2. **At runtime, for pgvector** — every connection is opened with
   `SET default_transaction_read_only = on`, then
   `rag_staleness_check.readonly.assert_pg_read_only()` re-checks
   `SHOW default_transaction_read_only` before any query runs, and
   refuses to proceed if it isn't `'on'`. This is defense-in-depth against
   a connection pooler (e.g. PgBouncer) silently swallowing the session-level
   `SET`.

**Honest limitation:** Qdrant and Chroma have no client-exposed way to
introspect "is this session/API key read-only" — that's deployment-side
RBAC, not something either client library reports. For those two engines,
enforcement is (1) the static test above and (2) your own deployment-side
scoping — connect with a read-only-scoped API key/token if your engine
supports one. There is **no runtime assertion** for Qdrant/Chroma
equivalent to pgvector's; this README says so rather than implying parity.

For extra assurance on pgvector, connect through a locked-down role
instead of relying on this tool alone:

```sql
CREATE ROLE rag_staleness_check_readonly NOSUPERUSER LOGIN PASSWORD '...';
GRANT CONNECT ON DATABASE mydb TO rag_staleness_check_readonly;
GRANT USAGE ON SCHEMA public TO rag_staleness_check_readonly;
GRANT SELECT ON chunks TO rag_staleness_check_readonly;
```

## Telemetry

**No default telemetry.** Nothing is collected or sent automatically.
`--share-anonymous-scorecard` is an explicit opt-in that **prints** the
exact anonymized payload (aggregate counts + engine type only — never your
dsn, hostnames, doc_ids, or chunk_ids) that *would* be shared — no backend
is configured yet, so nothing is actually sent over the network. This flag
is reserved for a future opt-in submission endpoint. `DO_NOT_TRACK` in the
environment, if set, forces this off even if the flag is passed.

Separately: this tool explicitly disables **chromadb's own client-side
telemetry** (`anonymized_telemetry=False`) when connecting to Chroma — the
`chromadb` Python package has its own default-on PostHog-based telemetry
independent of anything described above, and leaving it enabled would
silently violate the "no default telemetry" guarantee for `--engine
chroma` users even though this tool's own code never phones home.
**Known cosmetic issue:** on `chromadb==0.6.3`, this setting is correctly
applied (verified: `client._system.settings.anonymized_telemetry is
False`), but one startup event (`ClientStartEvent`) still attempts to fire
through a code path that doesn't consistently honor it, and then fails
with `capture() takes 1 positional argument but 3 were given` — a
`TypeError` raised while *constructing* the call, before any network
request is made. You may see this line printed; it does not indicate data
was sent.

## Known limitations

- **Duplicate detection's exact-hash pass** needs a per-chunk content hash
  stored in your engine (`--pg-content-hash-column` /
  `--content-hash-field`). Without one, it degrades to cosine-ANN-only —
  reported via a `warning` field in the `duplicates` check, not silently.
- **Chroma's `snapshot_stats()`** intentionally reports only a live count —
  no on-disk HNSW-directory-size measurement. (The private RAGproof audit's
  own dev-environment version of this used a `docker exec du` trick against
  a known-local container; that has no equivalent against a real
  third-party deployment, so it isn't shipped here.)
- **`qdrant-client`'s own compatibility check** may warn if your installed
  client's minor version differs from your server's by more than 1 (e.g.
  "Qdrant client version 1.18.0 is incompatible with server version
  1.15.5"). This is non-fatal — the check still runs — but if it's noisy,
  pin `qdrant-client` to match your server's minor version yourself.
- **This tool reports counts and pairs, not precision/recall.** There is
  no ground truth available client-side to score itself against (that's
  what the private, ledger-based audit is for).
- **Duplicate detection needs a retrievable vector per candidate** — a
  chunk with no vector returned by `get_vector` is silently excluded from
  the cosine-ANN pass (there's nothing to query with), not counted as
  "not a duplicate."

## Dependency notes

- `qdrant-client` and `chromadb` are **optional extras**
  (`pip install rag-staleness-check[qdrant]` / `[chroma]` / `[all]`) — a
  pgvector-only user doesn't need Chroma's heavier transitive dependencies.
- Neither is pinned to an exact version: this tool connects to a third
  party's already-running deployment, whose server version is out of its
  control, so hard-pinning (unlike a project that ships and controls its
  own Docker images) would just break users on anything else.

## License

Apache-2.0. See [LICENSE](LICENSE). Independent of any other RAGproof
project's license.

## Contributing

Issues and PRs welcome. Not yet implemented: a `--method minhash`
surface-text-based duplicate detection mode (via `datasketch`) as an
alternative/complement to the cosine-embedding-based approach above — PRs
welcome.
