# How-to — Audit the Kairix entity graph

**When to use:** the entity graph is drifting from what the document store says — `kairix entity suggest` returns garbage, agents report "X not found" for entities you know exist, or the reflib benchmark regresses on entity-heavy queries. This runbook walks four audit modes in order of safety: detect, repair-paths, check-enrichment, then (only after evidence) safe-purge.

**Time:** 5-15 minutes for detection + repair on a typical knowledge store (≤ 5k entity nodes). Add the crawl time if you escalate to a full rebuild.

**Warning:** Step 4 (safe purge) is destructive. It runs in dry-run mode by default and the runbook keeps it that way until the captured report has been reviewed. Never skip the dry-run.

**Replaces:** the retired Mnemosyne `entity-audit.py` practice. Kairix's surfaces are different — primarily `kairix curator health`, `kairix store health`, and `scripts/prune-entities.py` — so the procedure is rebuilt from those, not ported.

---

## Configure for your environment

| Reference | Substitute with |
|---|---|
| `${KAIRIX_DOCUMENT_ROOT}` | `kairix.config.yaml` `paths.document_root`, or `KAIRIX_DOCUMENT_ROOT` env var |
| `${KAIRIX_NEO4J_URI}` | `bolt://<host>:7687`, set in your env or secrets |
| `${KAIRIX_NEO4J_USER}` | typically `neo4j` |
| `${KAIRIX_NEO4J_PASSWORD}` | from your secrets pipeline |
| `${KAIRIX_DATA_DIR}` | `kairix.config.yaml` `paths.db_path` parent, or env var (default `~/.cache/kairix/`) |

If kairix runs in Docker, prefix host commands with `docker exec <container-name>`.

## When to use this runbook

Reach for this runbook when one or more of the following shows up:

- `kairix entity suggest "<text>"` returns clear false positives — role phrases, document filenames, or fragments that aren't real entities.
- An agent that previously located an entity now reports it missing, but the source note is still in `${KAIRIX_DOCUMENT_ROOT}`.
- `kairix benchmark run --suite reflib` shows recall regression on entity-anchored queries (NDCG@10 drop on category `entity`).
- `kairix curator health` reports a non-zero `issue_count` — synthesis failures, stale entities, or missing `vault_path` properties.
- After a document-store reorg (file moves, folder renames, mass deletions) you suspect orphan entity nodes.

If the symptom is "entity-enriched search shows zero entities for everything," skip this runbook and go to [how-to-rebuild-entity-graph](how-to-rebuild-entity-graph.md) — that's a state problem, not an audit problem.

## Step 0 — Capture pre-audit state

Snapshot the graph before you touch it so you can compare after.

```bash
# Overall onboarding health — Neo4j reachable? embed pipeline running?
kairix onboard check --json > /tmp/kairix-onboard-pre.json
jq '{ok, neo4j, vector_index, document_root}' /tmp/kairix-onboard-pre.json

# Entity counts by type (the closest surface kairix exposes today to a
# pure ``kairix entity count`` command; see "Gaps" at the bottom)
kairix store health --json > /tmp/kairix-store-pre.json
jq '{ok, total_entities, entities_by_type, errors}' /tmp/kairix-store-pre.json

# Curator-level health: synthesis failures + stale + missing vault_path
kairix curator health --format json --output /tmp/kairix-curator-pre.json
jq '{ok, total_entities, issue_count: (.synthesis_failures|length + (.stale_entities|length) + (.missing_vault_path|length))}' /tmp/kairix-curator-pre.json
```

Keep all three files. Each later audit step compares against this baseline.

If `kairix onboard check` reports Neo4j unreachable, stop. Resolve connectivity first — every step below queries Neo4j.

## Step 1 — Junk-entity detection (read-only)

"Junk" entities are nodes that exist in Neo4j but reference nothing real: no source document, no enrichment, suspicious name.

Kairix exposes this view through `kairix curator health` — specifically its `missing_vault_path` and `synthesis_failures` lists. Both are non-destructive: the command only reads.

```bash
# Run the curator health check in text mode for human review
kairix curator health --format text

# Or write the structured report to a file for diff-ing
kairix curator health --format json --output /tmp/kairix-curator-now.json

# Surface only the junk-candidate lists
jq '{
  no_vault_path: .missing_vault_path | map({name, entity_type, entity_id}),
  no_summary:    .synthesis_failures | map({name, entity_type, entity_id})
}' /tmp/kairix-curator-now.json
```

What each list means:

- `missing_vault_path` — entity has no `vault_path` property, so no canonical document stub anchors it. Either the stub never got created or the property never got written. These are seed-only entities and the most common form of junk.
- `synthesis_failures` — entity has no `summary` property. Either enrichment never ran or it errored out. Some of these are real entities awaiting enrichment — do not purge until you've cross-checked.
- `stale_entities` — entity not touched in `--staleness-days N` (default 90). Use `--staleness-days 30` for tighter scrutiny on a high-churn store. Staleness alone is not a purge signal; combine with missing vault_path.

**Remediation when junk is small (< 20 nodes):** prefer manual cleanup. Open each `vault_path` candidate in the document store, decide whether it's a legitimate entity awaiting a stub or a noise node from a failed crawl. Add legitimate ones to `${KAIRIX_DOCUMENT_ROOT}/04-Agent-Knowledge/_entity-overrides.md`. Drop the rest in Step 4.

**Remediation when junk is large (≥ 20 nodes):** proceed to Step 2 (path repair) first — many "junk" nodes are actually entities whose source file moved or was renamed, not entities that should be deleted.

## Step 2 — Path repair (find entities whose source path is gone)

The crawler writes a `vault_path` property on every entity it creates. If the underlying file is later renamed, moved, or deleted, the property goes stale — the node still exists, but its source is gone.

`scripts/prune-entities.py` classifies this case as `file_missing` and is dry-run by default.

```bash
# Dry-run — read-only, prints to stdout
python scripts/prune-entities.py --vault-root "${KAIRIX_DOCUMENT_ROOT}"

# Capture the report for evidence
python scripts/prune-entities.py --vault-root "${KAIRIX_DOCUMENT_ROOT}" \
  > /tmp/kairix-prune-dryrun.txt

# Count just the file_missing candidates
grep -c "file_missing" /tmp/kairix-prune-dryrun.txt
```

The report has two sections: `NODES TO DELETE` (file_missing + no_stub_no_summary) and `NODES TO KEEP`. Read the delete list. Every `file_missing` row should correspond to a `vault_path` you can confirm is gone from `${KAIRIX_DOCUMENT_ROOT}` by hand:

```bash
# Spot-check one or two paths from the dry-run report
test -f "${KAIRIX_DOCUMENT_ROOT}/<vault_path-from-report>" \
  && echo "STILL EXISTS — do not purge" \
  || echo "missing — purge candidate confirmed"
```

If a `file_missing` entry corresponds to a file that *does* still exist, the script has read a different vault root than your live document store. Re-run with `--vault-root` set explicitly and re-check.

**Affordance:** when `file_missing` count > 0, the next concrete step is Step 4 (safe purge with `--execute`). Do not run `--execute` until you've completed Step 3 too — enrichment audit may surface entities that should be enriched, not deleted.

## Step 3 — Enrichment audit (missing label, summary, source attribution)

Kairix entities ship with a small expected fact set: a `name`, a Neo4j label (e.g. `Organisation`, `Person`, `Concept`), a `summary`, and a `vault_path`. Anything missing is an enrichment gap.

Two surfaces tell you what's enriched:

```bash
# Surface 1: curator health — synthesis_failures = no summary written
kairix curator health --format json --output /tmp/kairix-curator-enrich.json
jq '.synthesis_failures | map({name, entity_type, entity_id})' \
   /tmp/kairix-curator-enrich.json

# Surface 2: store health — entities with no edges, no source attribution
kairix store health --json | jq '{
  ok,
  total_entities,
  by_type: .entities_by_type
}'
```

For entities flagged in `synthesis_failures` that you want to keep, run validation against Wikidata:

```bash
# Per-entity validation — adds wikidata_qid if a high/medium-confidence match exists
kairix entity validate "<Name>" --update

# JSON form for automation
kairix entity validate "<Name>" --format json
```

For batch enrichment, re-run the crawler against the document store so the latest `vault_path` and any wikilink-derived attribution get written:

```bash
# Full crawl — picks up renamed files, new wikilinks, refreshes vault_path
kairix store crawl --document-root "${KAIRIX_DOCUMENT_ROOT}"
```

**Affordance:** an entity that's missing both `summary` and `vault_path` after a fresh crawl + a Wikidata validation pass is junk. Move it to the Step 4 purge list.

## Step 4 — Override coverage (allowlisted but never matched)

The vault-driven override loader at `${KAIRIX_DOCUMENT_ROOT}/04-Agent-Knowledge/_entity-overrides.md` (see [entity-overrides user guide](../../user-guide/entity-overrides.md)) lets curators force-add or force-block entity terms. An override that's allowlisted but never matches in a crawl is dead weight — the term is wrong, the spelling drifted, or the source content was removed.

Closes #263. Every `kairix store crawl` now scans each markdown file's text against the allowlist and records per-override match counts. At end of crawl the summary prints:

```
Override coverage: N/M overrides matched (K never used)
Never-matched: ['foo', 'bar', ...]  (first 10 shown)
Coverage report written: ${KAIRIX_DATA_DIR}/entity-override-coverage.json
```

The sidecar JSON lets you inspect drift over multiple crawls without re-running the crawler:

```bash
# Inspect the latest coverage report
jq . "${KAIRIX_DATA_DIR}/entity-override-coverage.json"

# Surface only the never-matched names
jq -r '.never_matched[]' "${KAIRIX_DATA_DIR}/entity-override-coverage.json"

# Surface match counts (which overrides fired the most)
jq -r '.match_counts | to_entries | sort_by(-.value) | .[] | "\(.value)\t\(.key)"' \
   "${KAIRIX_DATA_DIR}/entity-override-coverage.json"
```

Entries in `never_matched` are either:
- A new override the next crawl will pick up (run `kairix store crawl` and re-check the sidecar).
- A drift-prone name (the canonical form in the document store changed). Edit the override.
- Genuinely unused. Remove the line from `_entity-overrides.md`.

If you also need to know whether a never-matched override has a stale Neo4j node, fall back to the per-name graph check:

```bash
jq -r '.never_matched[]' "${KAIRIX_DATA_DIR}/entity-override-coverage.json" \
  | while read -r name; do
      result=$(kairix entity get "$name" --format json 2>/dev/null | jq -r '.id // "missing"')
      printf "%-40s %s\n" "$name" "$result"
    done
```

## Step 5 — Safe purge (destructive, dry-run gated)

Only run this step after Steps 1-4 have produced a delete list you've reviewed by name.

```bash
# Re-run the dry-run as your final check (capture the report you'll act on)
python scripts/prune-entities.py --vault-root "${KAIRIX_DOCUMENT_ROOT}" \
  > /tmp/kairix-prune-final-dryrun.txt

# Diff against the earlier dry-run to confirm nothing surprising changed
diff /tmp/kairix-prune-dryrun.txt /tmp/kairix-prune-final-dryrun.txt \
  || echo "delete list changed — investigate before --execute"

# Apply, capturing the audit log of every deletion
python scripts/prune-entities.py \
  --vault-root "${KAIRIX_DOCUMENT_ROOT}" \
  --execute \
  2>&1 | tee /tmp/kairix-prune-execute.log

# The log records: NAME, TYPE, REASON for every node deleted, plus the
# total deleted / total scanned summary.
```

The execute log is the audit trail. Keep it with your operator records — it's the only retrievable record of which nodes were removed and why.

**Atomicity:** each `DETACH DELETE n` is one Cypher statement, atomic at the Neo4j level. The script does not batch; if it crashes mid-run, previously-deleted nodes stay deleted and the remaining list is unaffected. Re-running the script picks up where it stopped because the dry-run report regenerates against the live graph.

**Rollback:** Neo4j has no built-in undo for `DETACH DELETE`. If you need to roll back, the recovery path is a full graph rebuild from the document store — see [how-to-rebuild-entity-graph](how-to-rebuild-entity-graph.md). Snapshot Neo4j before any large `--execute` run if your deployment supports it.

## Step 6 — Validate post-audit

Re-run the baseline captures from Step 0 and diff:

```bash
# Capture the post-audit state
kairix onboard check --json > /tmp/kairix-onboard-post.json
kairix store health --json > /tmp/kairix-store-post.json
kairix curator health --format json --output /tmp/kairix-curator-post.json

# Compare entity totals — total should drop by exactly the prune count
jq '.total_entities' /tmp/kairix-store-pre.json /tmp/kairix-store-post.json

# Compare issue counts — should drop, never rise
jq '{
  synth: (.synthesis_failures|length),
  stale: (.stale_entities|length),
  no_path: (.missing_vault_path|length)
}' /tmp/kairix-curator-pre.json
jq '{
  synth: (.synthesis_failures|length),
  stale: (.stale_entities|length),
  no_path: (.missing_vault_path|length)
}' /tmp/kairix-curator-post.json

# Recall regression check — reflib benchmark on the live system
kairix benchmark run --suite reflib --system hybrid \
  --output "${KAIRIX_DATA_DIR}/benchmark/post-entity-audit"
```

Compare the post-audit reflib NDCG@10 against your last green run. If recall regressed, the purge took out a legitimate entity. Recovery path: [how-to-rebuild-entity-graph](how-to-rebuild-entity-graph.md) restores from the document store.

## Escalation — graph state is too broken to repair

When the audit report shows > 50% of entities flagged as junk, or repeated path-repair runs surface different sets of orphans, the underlying problem is graph state, not audit. Stop running prune cycles and rebuild from scratch.

Single command (closes #262): `kairix store crawl --reset` drops the graph and rebuilds in one step. Pair with `--confirm` interactively or `KAIRIX_NONINTERACTIVE=1` in pipelines.

```bash
# Drop + rebuild — destructive, requires --confirm
kairix store crawl --reset --confirm --document-root "${KAIRIX_DOCUMENT_ROOT}"

# Scripted equivalent (e.g. cron / CI)
KAIRIX_NONINTERACTIVE=1 kairix store crawl --reset --document-root "${KAIRIX_DOCUMENT_ROOT}"

# Preview the destruction without writing
kairix store crawl --reset --confirm --dry-run --document-root "${KAIRIX_DOCUMENT_ROOT}"

# Verify
kairix store health
kairix curator health --format text
```

The full sequence — including pre-checks, content checks, and post-rebuild verification — is documented in [how-to-rebuild-entity-graph](how-to-rebuild-entity-graph.md). Use that runbook for any rebuild, not the three commands above.

## Gaps and follow-up work

Some surfaces this runbook would benefit from do not exist today. Where the runbook calls a primitive that's missing, it routes through the closest available command. These gaps are tracked as follow-up issues:

| Wanted surface | Closest substitute today | Gap issue |
|---|---|---|
| `kairix entity count` (just the number, no extra) | `kairix store health --json \| jq '.total_entities'` | filed |
| `kairix entity audit` (one-shot audit covering Steps 1-4) | curator health + prune-entities.py + entity get loop | filed |
| `kairix entity purge --dry-run / --execute` (proper CLI, not a script) | `scripts/prune-entities.py` | filed |
| `kairix store crawl --reset` (drop-and-rebuild in one command) | landed (#262) — see Escalation section | done |
| Override-coverage stats (matched N times in last crawl) | landed (#263) — see Step 4 | done |

Until those land, the steps above are the operational practice.

## See also

- [how-to-rebuild-entity-graph](how-to-rebuild-entity-graph.md) — full rebuild when audit + repair isn't enough.
- [entity-overrides user guide](../../user-guide/entity-overrides.md) — vault-driven allowlist/blocklist.
- `kairix curator health --help` — flags for the health surface this runbook leans on.
- `scripts/prune-entities.py --help` — the safe-purge primitive.
