Metadata-Version: 2.4
Name: rag-staleness-check
Version: 0.1.2
Summary: Read-only staleness/orphan/duplicate/retrievability checks for a single pgvector, Qdrant, or Chroma index.
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/rimironenko/rag-staleness-check
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Topic :: Database
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: psycopg[binary]<4,>=3.1
Provides-Extra: qdrant
Requires-Dist: qdrant-client<1.16,>=1.15; extra == "qdrant"
Provides-Extra: chroma
Requires-Dist: chromadb<1,>=0.5; extra == "chroma"
Provides-Extra: all
Requires-Dist: qdrant-client<1.16,>=1.15; extra == "all"
Requires-Dist: chromadb<1,>=0.5; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# rag-staleness-check

Read-only staleness, orphan, duplicate and retrievability checks for a
single pgvector, Qdrant or Chroma index.

You point it at your own already-indexed vector database and it reports:

- **staleness**: indexed chunks whose source document has changed since
  they were indexed. This needs a manifest with `last_modified` per
  document, and the engine has to store a per-row last-modified value.
- **orphans**: indexed chunks whose source document is no longer in your
  manifest at all.
- **duplicates**: near-identical chunks (cosine similarity at or above a
  threshold, 0.98 by default), plus an exact-hash pass if you store a
  per-chunk content hash.
- **retrievable-after-delete**: a basic probe for ids you believe are
  deleted. It checks whether the vector can still be fetched by id
  (storage-layer persistence) and whether it still shows up in top-k
  search results (functional leak).

The tool is read-only and never calls a write, delete or upsert method on
your engine. [Read-only guarantee](#read-only-guarantee) below explains
how that's enforced and where the enforcement stops.

## Scope

Each run covers one engine and has no ground truth to compare against.
That means it doesn't do:

- multi-engine runs (run it once per engine)
- GDPR Article 17 reporting, such as signed evidence or a verifiable
  erasure report
- engine-specific deep checks (pgvector dead tuples and VACUUM, Qdrant
  optimizer thresholds, Chroma on-disk HNSW growth)
- precision/recall scoring. It reports what it finds, not how accurate
  those findings are.

## Install

```bash
pip install rag-staleness-check              # pgvector only
pip install "rag-staleness-check[qdrant]"     # + Qdrant
pip install "rag-staleness-check[chroma]"     # + Chroma
pip install "rag-staleness-check[all]"        # everything
```

Or, without installing:

```bash
pipx run rag-staleness-check --engine pgvector --dsn "$DSN" --source ./docs_manifest.json
```

Requires Python 3.10+.

## Usage

```bash
rag-staleness-check \
  --engine pgvector \
  --dsn "postgresql://readonly_user:pw@localhost:5432/mydb" \
  --pg-table chunks \
  --pg-doc-id-column doc_id \
  --pg-last-modified-column last_modified \
  --pg-content-hash-column content_sha256 \
  --source ./docs_manifest.json \
  --deleted-ids ./deleted_ids.json \
  --out findings.json
```

This prints a scorecard to stdout and writes the full result to `--out`
(`findings.json` by default). If a check is missing an input it needs (no
`--source`, no per-row metadata configured, no `--deleted-ids`), it
reports `"skipped": true` with a reason, so you don't get a clean-looking
0% that means nothing.

### `--dsn` per engine

| Engine | `--dsn` format | Example |
|---|---|---|
| `pgvector` | Postgres DSN | `postgresql://user:pw@localhost:5432/db` |
| `qdrant` | Base URL | `http://localhost:6333` |
| `chroma` | `host:port` | `localhost:8000` |

### Schema mapping

The tool makes no assumptions about your table, column, payload or
metadata names beyond what you pass in flags.

For pgvector (`--pg-*`): `--pg-table` (required), `--pg-id-column`
(default `id`), `--pg-vector-column` (default `embedding`),
`--pg-doc-id-column`, `--pg-last-modified-column` and
`--pg-content-hash-column`. The last three are optional. Leave them out
and staleness, orphans, and the exact-hash pass of the duplicates check
report `skipped` or zero coverage.

For Qdrant and Chroma, `--collection` is required, plus the payload or
metadata field names: `--doc-id-field` (default `doc_id`),
`--last-modified-field` (default `last_modified`) and
`--content-hash-field` (default `content_sha256`).

### Manifest (`--source`)

A JSON file listing the documents that *should* exist. It's the join key
for staleness and orphans, and the tool's only source of truth for what
counts as current.

```json
{
  "manifest_version": 1,
  "documents": [
    {
      "doc_id": "handbook/engineering/on-call.md",
      "last_modified": "2026-06-01T00:00:00Z",
      "content_sha256": "optional, doc-level, informational only"
    }
  ]
}
```

`doc_id` has to match whatever your ingestion pipeline wrote into each
chunk's doc-id column or field (see schema mapping above).
`last_modified` is only needed for the staleness check. `content_sha256`
here is per document and informational only; duplicate detection doesn't
use it (see below). There's a full example at
[`examples/docs_manifest.json`](https://github.com/rimironenko/rag-staleness-check/blob/v0.1.2/examples/docs_manifest.json).

### Deleted-ids probe (`--deleted-ids`)

A JSON array of ids you believe are deleted:

```json
["chunk-abc123", "chunk-def456"]
```

For each id, the probe checks whether the vector can still be fetched by
id (storage-layer persistence) and, if it can, whether it still comes
back in a top-k self-query (functional leak). The two are worth telling
apart. Under
[EDPB Guidelines 05/2019](https://www.edpb.europa.eu/documents/guideline/guidelines-52019-on-the-criteria-of-the-right-to-be-forgotten-in-the-search_en),
erasure has to be verifiable and irreversible, so hiding a record from
search results isn't enough if the vector is still stored. The
`retrievable.framing` field in `findings.json` repeats this on every run.

## Read-only guarantee

The tool never calls a write, delete or upsert method on your engine.
Two things enforce that:

1. A static test,
   [`tests/test_no_write_methods.py`](tests/test_no_write_methods.py),
   parses every file under `src/rag_staleness_check/` and fails the build
   if it finds a call to a write/delete/upsert-style method, or a SQL
   write statement passed to `execute()`/`executemany()`. It walks the
   syntax tree instead of grepping for substrings, which would trip on
   this README and on docstrings.
2. For pgvector, a runtime check. Every connection opens with
   `SET default_transaction_read_only = on`, and
   `rag_staleness_check.readonly.assert_pg_read_only()` reads
   `SHOW default_transaction_read_only` before any query and refuses to
   continue unless it's `'on'`. This covers the case where a connection
   pooler such as PgBouncer silently drops the session-level `SET`.

Qdrant and Chroma have no client-side way to ask "is this session or API
key read-only?" That's handled by RBAC on the deployment, and neither
client library reports it. For those two engines the static test is the
only enforcement in this tool, so connect with a read-only API key or
token if your deployment supports one. There is no runtime check for
Qdrant or Chroma like the one for pgvector.

On pgvector you can go further and connect through a locked-down role:

```sql
CREATE ROLE rag_staleness_check_readonly NOSUPERUSER LOGIN PASSWORD '...';
GRANT CONNECT ON DATABASE mydb TO rag_staleness_check_readonly;
GRANT USAGE ON SCHEMA public TO rag_staleness_check_readonly;
GRANT SELECT ON chunks TO rag_staleness_check_readonly;
```

## Telemetry

There is no default telemetry. Nothing is collected or sent
automatically. `--share-anonymous-scorecard` is an explicit opt-in that
prints the anonymized payload that *would* be shared: aggregate counts
and the engine type, never your dsn, hostnames, doc_ids or chunk_ids. No
backend is configured yet, so nothing goes over the network; the flag is
reserved for a future opt-in endpoint. If `DO_NOT_TRACK` is set in the
environment, it turns sharing off even when the flag is passed.

The tool also turns off chromadb's own client telemetry
(`anonymized_telemetry=False`) when it connects to Chroma. The `chromadb`
package ships PostHog-based telemetry that is on by default, and leaving
it on would break the no-telemetry promise for `--engine chroma` even
though this tool's own code never phones home.

One cosmetic quirk on `chromadb==0.6.3`: the setting does take effect
(`client._system.settings.anonymized_telemetry is False`), but a startup
event (`ClientStartEvent`) still goes through a code path that ignores
it, and fails with
`capture() takes 1 positional argument but 3 were given` while building
the call, before any request is made. You may see that line printed.
Nothing was sent.

## Known limitations

- The exact-hash pass of duplicate detection needs a per-chunk content
  hash stored in your engine (`--pg-content-hash-column` /
  `--content-hash-field`). Without one, the check falls back to cosine
  ANN only and says so in the `warning` field of the `duplicates` result.
- Chroma's `snapshot_stats()` only reports a live count. Measuring the
  on-disk HNSW directory would need filesystem access to the Chroma host,
  which you won't have against someone else's deployment, so it isn't
  included.
- `qdrant-client` runs its own compatibility check and may warn when the
  client's minor version is more than one away from the server's (for
  example "Qdrant client version 1.18.0 is incompatible with server
  version 1.15.5"). The warning isn't fatal and the check still runs. If
  it gets noisy, pin `qdrant-client` to your server's minor version.
- The tool reports counts and pairs, not precision or recall. There's no
  ground truth on the client side to score itself against.
- Duplicate detection needs a vector for each candidate. A chunk whose
  vector `get_vector` doesn't return is left out of the cosine ANN pass,
  since there's nothing to query with. It isn't counted as "not a
  duplicate."

## Dependency notes

- `qdrant-client` and `chromadb` are optional extras
  (`pip install rag-staleness-check[qdrant]` / `[chroma]` / `[all]`), so
  pgvector-only users don't pull in Chroma's heavier dependencies.
- Neither is pinned to an exact version. The tool connects to a
  deployment that's already running, and you control its server version,
  not this tool. Hard pins would break anyone on a different version.

## License

Apache-2.0. See
[LICENSE](https://github.com/rimironenko/rag-staleness-check/blob/v0.1.2/LICENSE).

## Contributing

Issues and PRs are welcome. One open idea: a `--method minhash` mode that
detects duplicates from surface text (via `datasketch`), as an
alternative or complement to the cosine-embedding approach.
