Metadata-Version: 2.5
Name: textbook-rag
Version: 0.1.0
Summary: Grounded question answering over technical books, with citation and code verification.
Project-URL: Homepage, https://github.com/karthikraja/textbook-rag
Project-URL: Issues, https://github.com/karthikraja/textbook-rag/issues
License-Expression: MIT
License-File: LICENSE
Keywords: bedrock,cli,embeddings,pdf,rag,retrieval
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Indexing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: boto3>=1.40
Requires-Dist: numpy>=1.26
Requires-Dist: platformdirs>=4.2
Requires-Dist: typer>=0.15
Provides-Extra: dev
Requires-Dist: boto3-stubs[bedrock-agent-runtime,bedrock-runtime]>=1.40; extra == 'dev'
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: rapidfuzz>=3.9; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: ingest
Requires-Dist: marker-pdf>=1.6; extra == 'ingest'
Requires-Dist: pymupdf>=1.24; extra == 'ingest'
Requires-Dist: rapidfuzz>=3.9; extra == 'ingest'
Description-Content-Type: text/markdown

# textbook-rag

Ask questions about technical books you own. Get a prose answer with citations
to the page it came from — and a check that the model did not make it up.

```console
$ tb ask "how do I avoid callback hell"

Q: how do I avoid callback hell

The book identifies the problem as code whose nesting grows with each
asynchronous step until it becomes unreadable [1]. Three remedies are
offered: early return from error branches so the happy path stays flat [1],
naming and hoisting callbacks out of the nested position [2], and using the
sequential-execution pattern to run a known set of tasks in order [3].

------------------------------------------------------------------------------
Sources:
 *[1] nodejs-design-patterns · Callbacks and Events > The callback hell · p104
 *[2] nodejs-design-patterns · ... > Callback discipline · pp106-107
 *[3] nodejs-design-patterns · Asynchronous Control Flow > Sequential execution · p115
  [4] nodejs-design-patterns · ... > The problem with callbacks · p103

tokens: 3412 in / 198 out · top score 0.612
```

## Why this is not another RAG wrapper

Two checks run on every answer, because **a language model's confidence carries
no information about its correctness.**

1. **Every `[n]` must refer to a passage the model was actually given.** An
   invented `[7]` against six passages is a fabricated source.
2. **Every line of code inside a fence must appear in those passages.**

The second check exists because the first was not enough. Asked for a
concurrency limiter, the model emitted a `TaskQueue` class and cited `[3]` — a
real, valid passage number. No retrieved passage contained that class. Check 1
passed it. The number was valid and the content was invented.

For a programming book that is the dangerous failure, because a reader cannot
tell fabricated code from quoted code by looking at it.

> **Treat every code block in an answer as unverified until the verifier clears
> it.** Prose answers are reliable; generated code is not. The tool tells you
> which is which rather than pretending the distinction does not exist.

It also **refuses**. Ask about something the corpus does not cover and you get a
decline and exit code `2`, not a confident answer built on the nearest
unrelated chunk.

```console
$ tb ask "how do I train a convolutional neural network"
I can't answer that from this corpus. (best match scored 0.196, below the
0.25 floor; nearest section was "Running CPU-bound tasks")
$ echo $?
2
```

## Install

```console
pip install textbook-rag
```

That is the query path. To add books of your own you also need the parser
stack, which pulls PyTorch and is about 2.5 GB:

```console
pip install 'textbook-rag[ingest]'
```

## Setup

**You configure credentials by not configuring them.** The tool never reads your
access key; boto3 resolves credentials through its own chain, so `aws
configure`, `AWS_PROFILE`, SSO and instance roles all work untouched.

```console
aws configure          # or: aws sso login --profile you
tb doctor
```

`doctor` checks everything in dependency order and names the fix for whatever
is broken:

```console
$ tb doctor
✓ config           /Users/you/.config/textbook-rag/config.toml
✓ credentials      account 123456789012, region us-east-1
✓ bedrock embed    amazon.titan-embed-text-v2:0, 1024 dims
✓ bedrock generate us.amazon.nova-pro-v1:0
✗ books            none ingested (looked in ~/.local/share/textbook-rag)
                   → tb ingest path/to/book.pdf
! ingest extra     not installed — querying works, ingesting does not
                   → pip install 'textbook-rag[ingest]'
```

You need Bedrock **model access** enabled for the embedding, generation and
rerank models in your region — that is a separate switch from IAM, in the
Bedrock console under *Model access*. `doctor` tells you if it is missing.

Then ingest a book you own:

```console
tb ingest ~/books/nodejs-design-patterns.pdf
```

No corpus ships with this package, and none can: book text is copyrighted. You
point it at PDFs you have.

## Commands

| | |
|---|---|
| `tb ask "..."` | answer with citations and verification |
| `tb search "..."` | retrieve only — one embedding call, no generation, no cost |
| `tb ingest book.pdf` | parse, chunk and embed a book |
| `tb books list` | what is in the corpus |
| `tb books info <slug>` | pages, chunks, which embedding model built it |
| `tb books remove <slug>` | delete a book's index |
| `tb config get/set/path` | region and model ids |
| `tb doctor` | diagnose setup |
| `tb eval --gold gold.json` | score retrieval against a gold set |

Every command takes `--json`. Exit codes: `0` answered, `1` failed, `2` refused.

## How retrieval works

```
question ──► decompose ──► embed each part ──► rerank each against ITSELF ──► interleave
                                                                                  │
                                           refusal gate ◄── best cosine ──────────┤
                                                                                  ▼
                                                        parent sections ──► model ──► VERIFY
```

Four strategies, selectable with `--strategy`. Measured on 41 questions against
one book:

| | recall@3 | recall@6 | multi-hop | zero-leak recall |
|---|---|---|---|---|
| `bi-encoder` | 0.722 | 0.889 | 8/11 | — |
| `reranked` | 0.778 | 0.944 | 9/11 | — |
| `per-subquery` *(default)* | **0.917** | **0.972** | **10/11** | **1.000** |

**"Zero-leak recall" is the number that matters.** Questions are split by
whether they share vocabulary with the section that answers them. "What is the
reactor pattern" against a section called *Reactor pattern* is answerable by
keyword matching — BM25 passes it, and it proves nothing about semantic search.
The zero-overlap questions are the real test: *"the writer produces data faster
than the reader can consume it"* → **Backpressure**, sharing no content word
with its answer. There is no lexical path to that.

A single aggregate recall number hides this split. A purely lexical system would
score 10/10 on the leaky half, 0/10 on the rest, and still report ~0.55 —
reading as mediocre rather than fundamentally broken.

> Given any RAG evaluation with one recall number, the first question to ask is
> what fraction of its questions share vocabulary with their answers.

## Ingestion, and why it runs two parsers

**Every ML layout parser reflows code as prose.** Marker, Docling and
PyMuPDF4LLM all collapse a multi-line listing onto one line, and two of them
corrupt it doing so — `cb(null, filename, false)` came out as `cb(null filename
false)`. That is a category behaviour, not a bug in one tool, so no choice of
parser avoids it.

So two extractors run over the same pages:

```
Marker    headings, hierarchy, prose      9/9 sections, 0 false positives
PyMuPDF   anything in a monospace face    verbatim, never reflowed
          └── merged by locating Marker's collapsed rendering and swapping it back
```

Font is the discriminator because it is ground truth from the PDF, not a guess
from indentation — indentation also matches block quotes, ASCII tables and
poetry.

Chunking then follows three rules, in priority order: cut on headings; **never
split a code fence**; give every chunk its section path, which is both what
makes a chunk full of pronouns findable and what becomes the citation.

## Configuration

Settings resolve in one place: **CLI flag → environment → config file →
default.**

```console
tb config set region eu-west-1
tb config set generate_model anthropic.claude-sonnet-4-5-20250929-v1:0
```

| setting | default |
|---|---|
| `region` | `us-east-1` |
| `generate_model` | `us.amazon.nova-pro-v1:0` |
| `embed_model` | `amazon.titan-embed-text-v2:0` |
| `rerank_model` | `cohere.rerank-v3-5:0` |
| `refusal_threshold` | `0.25` |
| `top_n` | `6` |
| `wide_k` | `25` |
| `rerank_depth` | `10` |

Those last four are measured, not chosen by taste. `top_n = 6` because coverage
plateaus there and `k=10` costs 54% more tokens for no gain — and measurably
*degrades* code fidelity, as more code passages make the model blend and
reconstruct across them instead of quoting one. `wide_k = 25` because
recall@10 equals recall@6 (widening finds nothing new) while recall@25 is 1.000.

**Changing `embed_model` is a migration, not a flag.** An index is a card
catalogue written in one model's language; vectors from a different model cannot
be compared with existing ones even when the dimensions match. The model id is
recorded in every index, so a mismatch is reported rather than silently
returning nonsense:

```console
! books   1 ingested, but these use a different embedding model than the
          current setting: nodejs-design-patterns (embedded with titan-v1)
          → tb ingest --reembed nodejs-design-patterns
```

## Where things live

| | |
|---|---|
| config | `~/.config/textbook-rag/config.toml` |
| corpus | `~/.local/share/textbook-rag/books/<slug>/` |

Both follow each OS's convention. `TEXTBOOK_RAG_HOME` overrides the data root.

## Requirements

Python 3.11+, an AWS account with Bedrock model access.

## License

MIT
