# RAG Ingest Walkthrough

**Status:** Operator runbook — local stand-up of the v0.22.0 ingest pipeline
**Last updated:** 2026-05-28

A step-by-step path from an empty machine to a searchable corpus: stand up
Postgres + pgvector, point Axiom at it, pick an embedder (or a deterministic
fake for a dry run), ingest a sample directory with a provenance rule set, read
the preflight and drop report, search a known fact, and audit the corpus.

Everything here is domain-agnostic. Replace the sample paths with your own.

---

## 0) What you need

- The `axi` CLI on PATH (`pip install -e ".[all]"` from the repo, or the wheel).
- Docker, or a local PostgreSQL 16 with the `pgvector` extension.
- A directory of `.md` / `.txt` / `.pdf` / `.docx` files to index.
- Optional: an embedding provider. Without one, ingest still works
  (text-only / keyword search); with one, you also get vector search.

---

## 1) Stand up Postgres + pgvector

The fastest path is the pgvector image:

```bash
docker run -d --name axiom-rag \
  -e POSTGRES_DB=axiom_db \
  -e POSTGRES_USER=axiom \
  -e POSTGRES_PASSWORD=changeme \
  -p 5432:5432 \
  pgvector/pgvector:pg16

# Wait until it accepts connections:
docker exec axiom-rag pg_isready -U axiom
```

Create the `vector` extension (one time; the store creates its own tables on
first connect, but the extension must exist first):

```bash
docker exec -it axiom-rag \
  psql -U axiom -d axiom_db -c "CREATE EXTENSION IF NOT EXISTS vector;"
```

If you run your own PostgreSQL, do the equivalent: a database, a role that can
create tables, and `CREATE EXTENSION vector;` as a superuser.

---

## 2) Point Axiom at the database

The store reads its connection string from `DATABASE_URL`, or from the
`rag.database_url` setting. Either works:

```bash
# Per-shell:
export DATABASE_URL="postgresql://axiom:changeme@localhost:5432/axiom_db"

# Or persisted in axi settings:
axi settings set rag.database_url "postgresql://axiom:changeme@localhost:5432/axiom_db"
```

Confirm the store connects and the schema is created:

```bash
axi rag status
# RAG Index Status
#   Total documents: 0
#   Total chunks:    0
#   ...
```

---

## 3) Choose an embedder (or skip it)

Vector search needs an embedder. The provider chain, in order:

1. `NEUT_EMBED_URL` — any OpenAI-compatible `/v1/embeddings` server.
2. `OPENAI_API_KEY` — OpenAI `text-embedding-3-small`.
3. Local Ollama with `nomic-embed-text` (free, offline).
4. None — ingest stores text only; search falls back to keyword matching.

Local Ollama is the simplest self-contained option:

```bash
ollama pull nomic-embed-text
# Ollama listening on localhost:11434 is auto-detected.
```

To do a **dry run with no real provider**, set none of the above. Ingest will
log "no embedding provider" and index text-only — useful for checking the
ingest/provenance/audit flow before you wire up embeddings.

A note on durability: if a provider **is** configured but the call fails (e.g. a
network drop), that document is **not** committed and is counted as a failure,
so a later run retries it. It is never stored half-indexed with its checksum
recorded — that would make it look "unchanged" and skip it forever.

---

## 4) Write a provenance rule file

Provenance rules gate and route files **by where they come from**, before any
bytes are read. A rule's `pattern` ending in `/` matches a directory subtree;
otherwise it is a glob (e.g. `*.zip`). First matching rule wins.

`rules.toml`:

```toml
# Never ingest a licensed/controlled source folder.
[[rule]]
pattern     = "vendor/"
disposition = "exclude"
reason      = "licensed vendor docs"

# Hold opaque archives for a human to look at; do not ingest.
[[rule]]
pattern     = "*.zip"
disposition = "quarantine"
reason      = "archive — review before ingest"

# Route a public subtree to the shared community tier.
[[rule]]
pattern     = "public/"
disposition = "allow"
tier        = "rag-community"
reason      = "cleared for sharing"
```

Dispositions: `exclude` (never ingest), `quarantine` (hold for review, do not
ingest), `allow` (ingest into `tier`, or the run's `--corpus` if no tier).

Shared tiers are gated by default: ingesting into `rag-org` or `rag-community`
**without** rules is refused, because an ungated bulk ingest can pull in
controlled or proprietary content. Pass `--rules <file>`, set `AXIOM_RAG_RULES`,
or pass `--no-rules` to acknowledge.

---

## 5) Preflight (dry run) first

See what the run would do without writing anything:

```bash
axi rag ingest ./sample-corpus --rules rules.toml --dry-run
```

The preflight panel reports the supported / unsupported / already-indexed file
counts, raw size, predicted chunk count and destination size, free disk space,
and whether the destination is reachable. If it prints an `ABORT:` line, fix the
cause (commonly: `DATABASE_URL not set`, or not enough free space) before the
live run.

---

## 6) Ingest for real

```bash
axi rag ingest ./sample-corpus --rules rules.toml --corpus rag-internal
```

The summary line is honest about what entered the index versus what was dropped
and why:

```
Indexed: 12 files, 48 chunks  (0 unchanged)  [corpus: rag-internal]
Dropped (not indexed): 3 unsupported (.png=2, .xlsm=1); 2 excluded by provenance (vendor/=2); 1 quarantined for review
```

Read the **drop report** line: it tells you what curation gaps exist (unsupported
file types), and confirms the provenance rules kept controlled and unvetted
content out of the index. "Unchanged" files are not drops — they were already
indexed with a matching checksum, so re-running is cheap and does no double work.

---

## 7) Search a known fact

Pick a phrase you know is in one of the ingested docs:

```bash
axi rag search "negative feedback loop"
```

Each result shows its combined score, corpus, source path and chunk index, the
title, and a text snippet. With an embedder configured you get hybrid
vector + keyword ranking; without one, keyword ranking only. If a fact you
expect is missing, check the drop report from step 6 — the doc may have been
unsupported, excluded, or quarantined.

---

## 8) Audit the corpus

Auditing finds controlled or proprietary content that is **already** in a corpus
— for example, content ingested before the rules existed. It is read-only unless
you pass `--purge`.

```bash
# Read-only: list what a current rule set would flag.
axi rag audit --corpus rag-org --rules rules.toml

# Audited 5 documents in rag-org: 1 excluded, 1 quarantined
#   EXCLUDE    vendor/manual.md      (licensed vendor docs)
#   QUARANTINE incoming/bundle.md    (review before ingest)
```

To remove the EXCLUDE-flagged documents (quarantined and allowed docs are left
alone):

```bash
# Shows what it would delete, but does not delete yet:
axi rag audit --corpus rag-org --rules rules.toml --purge

# Actually delete the EXCLUDE-flagged documents:
axi rag audit --corpus rag-org --rules rules.toml --purge --yes
```

`--purge` only deletes documents flagged `exclude`. Quarantined documents stay
in place — quarantine is a "hold for a human", not a delete.

---

## 9) Tear down (optional)

```bash
docker rm -f axiom-rag
```

---

## Quick reference

| Step | Command |
|---|---|
| Status | `axi rag status` |
| Preflight | `axi rag ingest <path> --rules rules.toml --dry-run` |
| Ingest | `axi rag ingest <path> --rules rules.toml --corpus <tier>` |
| Search | `axi rag search "<known fact>"` |
| Audit | `axi rag audit --corpus <tier> --rules rules.toml` |
| Purge | `axi rag audit --corpus <tier> --rules rules.toml --purge --yes` |

Corpus tiers: `rag-internal` (personal workspace), `rag-org`
(organization-managed), `rag-community` (shared/pre-indexed). Shared tiers
require provenance rules unless you pass `--no-rules`.
