# Instruction — docpluck: AI-gold via article-finder (canonical keys, shared protocol)

**Date:** 2026-05-16
**From:** article-finder / cross-project AI-gold coordination
**To:** docpluck-iterate maintainer / next docpluck session
**Status:** Action required. article-finder's side is fixed and committed; the items
below are docpluck's to do in its next iteration.

---

## TL;DR

docpluck must change three things:

1. **Stop using docpluck's private extraction prompt.** Generate all AI gold through
   the shared protocol `gold-generation.md` only.
2. **Key every gold under the paper's canonical DOI.** Bare local stems
   (`chen_2021_jesp`, `efendic_2022_affect`) are now *rejected* by the cache.
3. **Regenerate docpluck's existing `reading` golds.** They were produced by the old
   private prompt and diverge from the shared protocol's output (the "981-line vs
   617-line" divergence). They are not trustworthy shared ground truth as-is.

**What article-finder already shipped for you** (skill repo, committed):

- `generate-gold <pdf>` is now a *routed* invocation — it skips the download cascade
  and runs `gold-generation.md` directly. Previously it was advertised but unrouted.
- `gold-generation.md`'s `reading` prompt now transcribes every table **in full,
  cell-by-cell**, as a markdown grid — not just the caption. This closes the gap
  that originally justified docpluck's private prompt: the shared `reading` view is
  now rich enough for docpluck's TABLE verifier.
- `register-view` and `migrate` now **reject a non-canonical key** with an
  actionable error.
- `gold-generation.md` enforces a **100%-accuracy, zero-hallucination policy**:
  dual independent extraction + cross-check, verbatim anchors, schema validation.
  (An independent cross-vendor verification pass is planned but not yet live — see
  article-finder's `TODO.md`.)

---

## Why this matters

The same paper was landing in the cache under two keys — docpluck's short stem
`chen_2021_jesp` and ESCIcheckapp's PDF-stem `Chen_et_al-2021-JESP-...`. A paper
split across two keys fragments its record: every project that reads it sees only
its own slice, `ai-gold.py gaps` reports phantom gaps, and nobody can reuse anyone
else's work. The cause was every project keying papers its own way. The fix is one
canonical key per paper, enforced.

Separately, docpluck's `reading` golds were generated by a docpluck-private prompt
(`references/ai-full-doc-verify.md`, Step 1b), not the shared `gold-generation.md`.
Two prompts produce two different "ground truths" for the same PDF. Ground truth
must be single-source.

---

## Rule 1 — Ground truth ONLY through article-finder

docpluck never re-implements PDF extraction and never carries its own extraction
prompt. To obtain AI gold for a paper:

- **Consume first.** Before extracting anything, check the cache:
  ```
  python ~/.claude/skills/article-finder/ai-gold.py check <key> --view reading
  python ~/.claude/skills/article-finder/ai-gold.py get   <key> --view reading
  ```
  On a hit, use the cached gold. Zero tokens.
- **Generate on a miss.** Invoke the `article-finder` skill as `generate-gold <pdf>`.
  It runs `gold-generation.md` (dual stats extraction → reading+citations carrier
  pass → cross-check → schema-validated registration) and registers the views under
  the canonical key. You do not write the extraction logic.
- **Retire `references/ai-full-doc-verify.md` Step 1b** as a gold *producer*. If
  docpluck needs an extra verification pass for its own pipeline, that is fine — but
  the *ground truth* it verifies against comes from the cache, not from that prompt.

The shared `reading` prompt now captures full cell-by-cell tables, so docpluck's
table verifier has the detail it needs. If you find it still insufficient, raise it
with article-finder — do **not** fork the prompt. `gold-generation.md` is owned by
article-finder and is the single source of extraction rigor.

## Rule 2 — Canonical keys only

A cache key must be a **DOI-stem** (`10.1016__j.jesp.2021.104154`) or a
`fixture__<producer>__<name>` key. Nothing else.

- Pass the paper's **DOI** as the key to `register-view` / `generate-gold`; the CLI
  canonicalizes it (`10.1016/j.jesp.2021.104154` → `10.1016__j.jesp.2021.104154`).
  The DOI is in every gold's `article_metadata.doi` and the `reading` gold's
  `**DOI:**` line.
- If a paper genuinely has no DOI, use `fixture__docpluck__<pdf-stem>`.
- `register-view` and `migrate` now HALT on a bare stem. If docpluck-iterate's
  autonomous run keys by a local stem, it will fail loudly — fix the iterate skill
  to resolve and pass the DOI.

`docpluck.yaml` (the producer manifest) is reflexive — it globs `ai_gold/*/reading.md`
and keys by the parent directory. Once golds live in canonically-named directories,
the manifest follows automatically. No manifest edit is needed; the fix is in the
docpluck-iterate skill that *writes* the golds.

## Rule 3 — Regenerate the stale `reading` golds

docpluck's existing `reading` golds in the cache were produced by the old private
prompt. Regenerate them through `generate-gold` so the cache holds one
protocol-consistent `reading` view per paper. Priority order: papers other projects
consume (the fragmented three below) first, then the rest of docpluck's set.

When you regenerate, the new gold registers under the canonical DOI key. A second
`reading` gold at the same key supersedes the old one (the cache archives the old
copy automatically).

---

## The three fragmented papers — fix these first

Each currently has docpluck's `reading` under a short stem AND ESCIcheckapp's
`reading`+`stats` under a PDF-stem. Regenerate each via `generate-gold` and register
under the canonical DOI key; the duplicate keys then collapse to one record.

| Paper | docpluck's current key | Canonical DOI key |
|---|---|---|
| Chen et al. 2021, JESP — hindsight bias | `chen_2021_jesp` | `10.1016__j.jesp.2021.104154` |
| Xiao, Zeng & Feldman 2021, CRSP — decoy effect | `xiao_2021_crsp` | `10.1080__23743603.2021.1878340` |
| Efendic et al. 2022, SPPS — affect heuristic | `efendic_2022_affect` | `10.1177__19485506211056761` |

After regeneration, the old short-stem directories (`chen_2021_jesp/` etc.) can be
removed — coordinate the cleanup with article-finder so `index.json` stays consistent
(`ai-gold.py audit` must report 0 issues).

---

## docpluck's next iteration — step by step

1. Update docpluck-iterate so its gold step is "invoke `article-finder
   generate-gold <pdf>`", not the private prompt.
2. Make docpluck-iterate resolve the paper's DOI and pass it as the key (or let
   `generate-gold` do it — `gold-generation.md` reads `article_metadata.doi`).
3. Regenerate `reading` for Chen / Xiao / Efendic; verify each lands under the DOI
   key with `ai-gold.py views <doi>`.
4. Regenerate the remaining docpluck `reading` golds through the shared protocol.
5. Run `ai-gold.py audit` — expect 0 issues. Run `ai-gold.py gaps` to confirm no
   phantom fragmentation remains.
6. Commit docpluck's skill change in the docpluck repo; do not commit cache data
   from docpluck (article-finder owns the cache repo's commits).

## Command cheat-sheet

```
# Resolve a paper to its canonical key
python ~/.claude/skills/article-finder/ai-gold.py resolve "10.1016/j.jesp.2021.104154"

# Is the view already cached?
python ~/.claude/skills/article-finder/ai-gold.py check <key> --view reading
python ~/.claude/skills/article-finder/ai-gold.py get   <key> --view reading

# What views does a paper have?
python ~/.claude/skills/article-finder/ai-gold.py views <key>

# Generate gold for an uncovered PDF — invoke the article-finder skill:
#   article-finder generate-gold <absolute-pdf-path>

# Consistency check (run after any cache change)
python ~/.claude/skills/article-finder/ai-gold.py audit
```

## Definition of done (docpluck)

- [ ] docpluck-iterate generates gold only via `article-finder generate-gold` /
      `gold-generation.md`; the private prompt is no longer a gold producer.
- [ ] Every docpluck gold is registered under a canonical DOI key.
- [ ] Chen / Xiao / Efendic regenerated and co-located under their DOI keys.
- [ ] docpluck's other `reading` golds regenerated through the shared protocol.
- [ ] `ai-gold.py audit` clean.
