# Full-document AI verification (no-hallucination / no-text-loss gate)

> Loaded on demand from SKILL.md Phase 5d. Mandatory; never skipped.

This is the keystone meta-science gate. Char-ratio + Jaccard verifiers are blind to "right words in wrong order under wrong heading"; a 30-line eyeball is blind to mid-document drift; only a full-document structured comparison against the PDF itself catches the two failure modes that make this library scientifically unusable:

- **Text disappearing.** A real paragraph from the PDF is missing from the rendered output. This is the biggest nono — it silently corrupts downstream meta-science.
- **Hallucinated text.** The rendered output contains a sentence, citation, or value that does not appear in the source PDF. Equally fatal.

Plus structural correctness: sections at the right boundaries, tables with the right cells, figure captions paired to the right figures, footnotes preserved, no front-matter / running-header / acknowledgment leaks mid-body.

---

## Ground truth is OWNED by article-finder. docpluck-iterate NEVER generates it.

**The single most important rule of this gate: docpluck-iterate does NOT produce ground truth. It consumes it.**

Ground truth ("gold") for any PDF is generated by exactly ONE skill — `article-finder` — through exactly ONE shared protocol — `~/.claude/skills/article-finder/gold-generation.md`. Every consumer skill (docpluck-iterate, escicheck-iterate, CitationGuard, …) reads the gold article-finder produced; none improvises its own.

**Why this rule exists (2026-05-16 user directive).** docpluck-iterate used to carry its own gold-extraction prompt, and escicheck-iterate carried another. The result: the same paper was extracted twice, by two divergent prompts, producing two different `reading` golds (one 981 lines / 97 headings, the other 617 lines / 77 headings). Two "ground truths" for one paper is not ground truth — it is a fork. The fix is structural: there is one producer (`article-finder`), one protocol (`gold-generation.md`), one gold per paper. A consumer that hand-rolls extraction re-creates the fork.

**Hard prohibitions for docpluck-iterate:**
- NEVER write, paste, or embed a gold-extraction prompt in this skill, in SKILL.md, or in a subagent brief.
- NEVER dispatch a `general-purpose` subagent to "read the PDF and produce a gold."
- NEVER use pdftotext, Camelot, pdfplumber, or any deterministic extractor as ground truth — they have the exact flaws this library exists to fix (memory `feedback_ground_truth_is_ai_not_pdftotext`).
- If the shared `reading` gold lacks detail this skill needs (e.g. cell-by-cell tables for the TABLE check), that detail is added to `gold-generation.md` itself — coordinated with the `article-finder` owner — NEVER restored as a docpluck-local prompt.

The PDF is the only authoritative source; the gold is what a careful human reader would write down from the visual PDF. Diagnostic extractors (pdftotext/Camelot) are used ONLY *after* a finding, to pinpoint which library layer is at fault — never as the verdict source.

---

## The protocol (every affected paper, every cycle, no exceptions)

For each paper the cycle's fix targets (typically 3–8 papers):

### Step 1 — Obtain the AI gold (resolve key → check shared cache → on miss, delegate to article-finder)

The gold lives in the shared article repository's multi-view cache. docpluck-iterate consumes the **`reading`** view — the full-document AI multimodal read. The cache layout is a per-key directory of named views:

```
ai_gold/<canonical_key>/
  ├── reading.md          ← the full-document gold view docpluck-iterate consumes
  ├── reading.meta.json
  ├── citations.json      ← other producers' views
  └── stats.json
```

**Step 1a — Resolve the canonical key.** A paper has exactly ONE canonical record. Derive the DOI (page 1, pdftotext, or the filename — `jdm_.2023.10` → `10.1017/jdm.2023.10`), then `python ~/.claude/skills/article-finder/ai-gold.py resolve "<doi-or-stem>"`. Use the returned canonical key for every subsequent call. A stem key for a paper that has a DOI fragments the record (memory `feedback_gold_canonical_key`).

**Step 1b — Check the shared cache.**

```bash
AI_GOLD="~/.claude/skills/article-finder/ai-gold.py"
if cached=$(python "$AI_GOLD" check "$KEY" --view reading); then
    cp "$(python "$AI_GOLD" get "$KEY" --view reading)" "tmp/<paper>_gold.md"   # HIT
else
    : # MISS — go to Step 1c
fi
```

**Step 1c — On a cache miss, DELEGATE generation to article-finder. Do NOT generate it here.**

Invoke the **`article-finder` skill** to produce the gold:

```
generate-gold <absolute-path-to-PDF>
```

`article-finder`'s `generate-gold` runs `gold-generation.md` end to end — the canonical extraction prompt(s), anti-hallucination rules, cross-check, schema validation, and `register-view` storage under the canonical key. It produces (and registers) the `reading`, `citations`, and `stats` views in one pass. This is the ONLY sanctioned way for docpluck-iterate to obtain a gold that does not already exist.

> **Prerequisite — the `codex` CLI must be installed AND authenticated.** `gold-generation.md` runs an independent Codex / GPT-5.5 cross-model audit of every gold before storage, and `generate-gold` **blocks** (rather than shipping unverified gold) if `codex` is missing or unauthenticated. Verify ahead of time with `codex --version`. If a `generate-gold` call fails on the cross-model audit step, the user must run `codex login` — it is interactive, so the skill cannot do it; surface this to the user as a blocker rather than falling back to a local extraction.

After `generate-gold` completes, the `reading` view is in the shared cache — copy it to the working path exactly as in the cache-HIT branch:

```bash
cp "$(python "$AI_GOLD" get "$KEY" --view reading)" "tmp/<paper>_gold.md"
```

`tmp/<paper>_gold.md` is the ephemeral working copy; the shared cache is the durable artifact. docpluck-iterate never calls `register-view` itself — `generate-gold` already registered, with the correct producer / schema / source-PDF metadata.

If `generate-gold` is unavailable or fails for a paper (e.g. the Anthropic content filter blocks the extraction subagent — observed on some JESP social-psych PDFs), treat the paper as **gold-blocked**: render-pass verified only, recorded as such in the coverage matrix. Do NOT fall back to a local extraction.

### Step 2 — Capture the library's rendered output

```bash
python -u -c "
from pathlib import Path
from docpluck.render import render_pdf_to_markdown
md = render_pdf_to_markdown(Path('<pdf>').read_bytes())
Path('tmp/<paper>_v<version>.md').write_text(md, encoding='utf-8')
" 2>&1 | awk '{print; fflush()}'
```

### Step 3 — Capture diagnostic artifacts (optional, used only when gold ↔ rendered.md disagree)

These are NOT ground truth. They are diagnostics — they help pinpoint which library layer caused a finding once the gold ↔ rendered.md comparison has already identified the finding.

```bash
# DIAGNOSTIC ONLY — not ground truth
pdftotext -enc UTF-8 <pdf> tmp/<paper>_pdftotext.txt
python -u -c "
import json
from pathlib import Path
from docpluck.extract_structured import extract_pdf_structured
r = extract_pdf_structured(Path('<pdf>').read_bytes())
Path('tmp/<paper>_structured.json').write_text(json.dumps(r, indent=2, default=str), encoding='utf-8')
"
```

### Step 4 — Dispatch the verifier subagent (compares rendered.md against the AI gold)

The verifier is docpluck-iterate's own consumer-side check: it compares the library's *output* against the shared gold. It does NOT generate ground truth — it only reads the gold article-finder produced. Use the `Agent` tool with `general-purpose`. Prompt template:

```
You are auditing the docpluck library's PDF→Markdown rendering for a meta-science
project that demands zero text loss and zero hallucinations.

GROUND TRUTH: tmp/<paper>_gold.md — the shared AI multimodal gold (the `reading`
              view produced by article-finder's gold-generation protocol). This
              is what a careful human reader would write down. It preserves Greek
              letters, comparison operators, thousands separators, and dashes
              exactly as the PDF prints them.
RENDERED:    tmp/<paper>_v<version>.md — the library's output (what is being
              audited).
DIAGNOSTIC (optional, do NOT use as truth):
             tmp/<paper>_pdftotext.txt — pdftotext output (the library's
             TEXT-channel input).
             tmp/<paper>_structured.json — Camelot/pdfplumber LAYOUT-channel
             extraction (the library's table/figure input).
             Use these ONLY after you have already identified a finding via
             gold ↔ rendered comparison, to pinpoint which library layer is
             responsible (e.g. "rendered says 'beta', gold says 'β', pdftotext
             also says 'beta' → bug is in pdftotext upstream, not in
             normalize.py").

YOUR JOB: read BOTH gold and rendered.md IN FULL. Then produce a structured
verdict. Be conservative — when in doubt, flag it.

CHECKS (in order; each must pass):

1. TEXT-LOSS check.
   For every substantive paragraph in the gold (sentence-shaped, ≥60 chars),
   verify the same paragraph appears in the rendered .md. Allowed normalizations
   the library may apply: U+2212→hyphen, soft-hyphen rejoin, line-wrap reflow,
   NFKC composition, smart-quote→straight-quote. Allowed omissions ONLY:
   running headers, page numbers, copyright lines, Crossref boilerplate,
   watermark strips, ORCID lines, DOI banner lines. Anything else missing is a
   TEXT-LOSS finding.

2. HALLUCINATION check.
   For every substantive paragraph in the rendered .md (≥60 chars,
   sentence-shaped), verify the same content appears in the gold. Structured
   wrappers added by the renderer (`### Table N`, italic captions, fenced
   `unstructured-table` blocks, `## ` heading markers) are renderer markup,
   NOT hallucination. Sentences / numbers / claims not traceable to the gold
   are HALLUCINATIONs. Glyph hallucinations count: if the gold has β and the
   rendered .md has "beta" and Greek→ASCII was not an authorized normalization,
   "beta" is hallucinated character content.

3. SECTION-BOUNDARY check.
   List every `## ` heading in the rendered .md. For each: does the heading
   appear at a plausible boundary as defined by the gold's section structure?
   Flag headings mid-paragraph, content under the wrong canonical name,
   sections in the gold missing from the rendered .md, hallucinated headings
   not in the gold, and endmatter routing errors.

4. TABLE check (for each `### Table N` heading in the rendered .md).
   Compare against the gold. Flag phantom empty columns, concatenated cells,
   caption welded into thead, missing rows, empty `<table>` shells with cells
   dumped to the body stream, body prose absorbed into the table, two tables
   merged into one.

5. FIGURE check (for each `### Figure N` heading).
   Compare caption against the gold; flag truncation mid-sentence, off-by-one
   pairing, and double-emission.

6. METADATA-LEAK check.
   The body of any section should NOT contain orphan metadata: affiliations,
   corresponding-author contact lines, supplemental-data notices,
   acknowledgments / funding outside their sections, journal running headers,
   running-header text wedged inside `<table>` or fenced blocks.

OUTPUT FORMAT (markdown):

## Verdict: PASS / FAIL

## Findings (severity-ordered)
### TEXT-LOSS (critical — blocks ship)
- <gold paragraph excerpt> · gold §<section> · NOT FOUND in rendered
### HALLUCINATION (critical — blocks ship)
- <rendered sentence> · NOT TRACEABLE to gold
### SECTION-BOUNDARY (high)
- ...
### TABLE / FIGURE / METADATA-LEAK (medium)
- ...

## Confirmed matches (one line each — proves you actually read both files)
- ... etc

Be specific. Quote excerpts. Cite paragraph/line references in BOTH files.
A vague "section boundaries look OK" is NOT acceptable — list every ##
heading in BOTH files and rule on each.
```

### Step 5 — Adjudicate

- **All checks PASS** → record `5d: AI-verified <paper> against gold` in cycle report. Proceed.
- **Any TEXT-LOSS or HALLUCINATION finding** → revert the cycle's library edit (if newly introduced) AND queue a new cycle to fix the root cause (if pre-existing). These are uncategorical-blockers. Append a LEARNINGS entry explaining what slipped through Phase 5a–c. Do NOT patch the renderer around the AI's complaint — root-cause it in the layer of origin.
- **SECTION-BOUNDARY or TABLE / FIGURE / METADATA-LEAK finding** → if newly introduced this cycle → revert. If pre-existing → DO NOT ship around it: **queue an immediate-subsequent cycle in the same run to fix it** (rule 0e in CLAUDE.md).

**Pre-existing ≠ deferrable.** A finding that existed in the prior release and was missed by an earlier cycle's verification is even MORE reason to fix it now.

**Critical rule:** do not negotiate with TEXT-LOSS or HALLUCINATION findings. A meta-science user pulling data from this library cannot be told "we silently dropped one paragraph."

### Step 6 — Cross-paper sweep (every 3rd cycle)

The single-paper AI verify catches cycle-specific regressions. Systematic patterns need a corpus-wide view. Every 3rd cycle, dispatch ONE subagent given the rendered .md + gold files for 5 randomly-sampled papers, asking for a corpus-level findings list ranked by paper-count. This drives the TRIAGE update.

---

## What NOT to do

| Anti-pattern | Why it's wrong |
|--------------|----------------|
| "Write a gold-extraction prompt and dispatch a subagent to make the gold" | docpluck-iterate does NOT produce ground truth. Generation is delegated to `article-finder generate-gold` — one producer, one protocol. A local prompt re-forks the gold (2026-05-16 directive). |
| "The shared `reading` gold lacks table cell detail — I'll extract a richer one myself" | The richness goes INTO `gold-generation.md` (coordinated with article-finder), never into a docpluck-local prompt. One protocol. |
| "Use pdftotext output as source of truth" | Pdftotext has its own flaws (Greek glyphs dropped, column interleave, tables invisible) — the bug class the library exists to fix. Memory `feedback_ground_truth_is_ai_not_pdftotext`. |
| "Use Camelot's structured output as table truth" | Camelot phantom-emits empty columns and fuses cells — the bug class we catch. Cell truth comes from the AI gold. |
| "Read first 30 lines of rendered.md" | Mid-document drift only shows mid-document. A 30-line skim is the baseline before AI verify, not a substitute. |
| "Trust char-ratio + Jaccard" | Both blind to "right words wrong order under wrong heading." |
| "Subagent costs tokens — skip on small fixes" | A "small fix" that silently drops one paragraph is a catastrophic meta-science failure. |
| "26-paper baseline passed, so it's fine" | Baseline checks structure metrics, not content correctness. AI verify against gold is the content gate. |

## How this composes with other phases

- **Phase 5a–c (unit tests + baseline)** check that the LIBRARY CODE works as intended on its own terms.
- **Phase 5d (this protocol)** checks that the OUTPUT IS CORRECT against the source PDF (via the shared AI gold).
- Both are required. Neither substitutes for the other.

## Cost management

- Gold generation is article-finder's cost, paid once per paper (PDFs are immutable), shared across every consumer project.
- Verification (Step 4): ~1 subagent call per paper per cycle, ~1-2 minutes.
- Per-3rd-cycle corpus sweep: + ~3-4 min for one subagent given 5 paper gold+rendered pairs.

## Gold extraction internals are article-finder's concern

Extraction failure modes (image-dimension limits on high-DPI pages, chunk sizing, `pypdfium2` fallback when `pdftoppm` is unavailable, subagent tool-call budgeting) are handled inside `gold-generation.md` and `article-finder`. docpluck-iterate does not need to know them — it only invokes `generate-gold` and consumes the `reading` view. If `generate-gold` reports a systematic extraction problem, raise it with the `article-finder` owner; do not work around it locally.

## Gold lifetime + invalidation

- Golds are durable: cached in the shared repository, keyed by canonical DOI, reused across cycles and across projects. A library version bump does NOT invalidate a gold (the gold is about the PDF, not the library).
- A gold is regenerated (via `generate-gold`) ONLY when: (a) a prior gold is found inaccurate, or (b) the source PDF's SHA256 changed. Both are rare — test-corpus PDFs are immutable.
- `tmp/<paper>_gold.md` is the ephemeral working copy; the shared cache is the source of truth.
