# Pinned corpus for the Tier 1 retrieval-quality gate (scripts/eval_retrieval.py).
#
# Format: `owner/repo <40-char commit SHA> <entry count>`, one per line.
# Materialise it with
#   bash scripts/ensure_eval_corpus.sh
# which taps each repo, checks it out at the pinned commit, and verifies it
# scans into the recorded number of entries (scripts/eval_corpus.py). Re-measure
# the counts with `python3 scripts/eval_corpus.py --relock`; check the
# distribution without a network with `--audit`.
#
# The golden set (golden.jsonl) grades real catalog items by NAME, so the gate
# needs the repos that actually contain those items tapped before it can score.
# The first six repos are the minimal set covering all 77 golden targets across
# the skill / rule / workflow kinds. The rest exist to make the corpus a
# realistic size, which is the whole point: over the six alone BM25 scored
# recall@10 1.000 / hit@1 0.791, and the same gate over these twenty scores
# 0.863 / 0.473. A user with twenty taps gets the second number, so flooring
# against the first was measuring the corpus rather than the retrieval.
#
# WHY THE SHAs. This list pinned repo *names* only, and tapping shallow-clones
# whatever the default branch points at, so the corpus was reproducible only for
# as long as twenty third parties happened not to push. Nothing had drifted when
# this was written — today's scores match the committed baseline exactly — but
# the size is worth stating, because it is not what anyone assumed: these twenty
# repos are **10,152 entries**, and ONE of them, sickn33/antigravity-awesome-
# skills, is 6,309 of those — 62% of the gate's corpus in a single third-party
# repository. affaan-m/ECC is another 1,616 (16%).
#
# The margin that protects is thin. BM25 scores recall@10 0.863 over this corpus;
# CI floored it at 0.85, which is a buffer of 1.15 queries out of 91 — one golden
# query can drop out of the top 10 and the gate still passes, two cannot. A
# single upstream push to the repo holding 62% of the corpus could therefore turn
# a required check red on a pull request that touched nothing to do with
# retrieval. Pinning removes that coupling, and makes every number in this header
# reproducible rather than "true on the day it was measured".
#
# WHY THE COUNTS. A SHA fixes the tree; it does not fix what this project makes
# of that tree, and it does nothing when the tree cannot be fetched at all. Both
# gaps end in the gate scoring a corpus its floors were never set on, and the
# direction of the error is the surprise. Measured over the 91-query required
# set, one repo removed and everything else identical:
#
#   all 20 repos           10,152 entries   0.852 / 0.473 / 0.605 / 0.657
#   minus sickn33 (62%)     3,843 entries   0.885 / 0.593 / 0.711 / 0.746
#   minus that and ECC      2,227 entries   0.967 / 0.659 / 0.769 / 0.814
#   minus LessUp (targets)  9,682 entries   0.676 / 0.374 / 0.483 / 0.523
#   floors                                  0.780 / 0.400 / 0.520 / 0.580
#
# Losing a *scale* repo makes the gate EASIER: every metric rises and the check
# goes green having measured a third of the intended corpus. Losing a repo that
# holds golden targets fails all four floors, indistinguishably from "this pull
# request broke retrieval". So skipping whichever repos are missing today is
# unsafe in both directions — the counts are what make the first case detectable
# before anything is scored, and an unreachable repo now exits 75 (EX_TEMPFAIL)
# so a third party's outage does not read as a regression here.
#
# The same table is why the 62% is not fixed by trimming: dropping the big repo
# raises every number. Diluting the concentration means adding breadth. The
# ceiling in eval_corpus.py (MAX_SHARE) is a ratchet against making it worse.
#
# NOTE: `boost tap --defaults` is NOT sufficient — it taps only skill repos and
# scores 0.000 on every rule and workflow query (overall recall ~0.33). That is
# why this explicit list exists.
#
# If you edit this list — including moving a pin — re-measure and re-baseline:
#   BOOST_HOME=/tmp/evalhome python3 scripts/eval_corpus.py --relock
#   BOOST_HOME=/tmp/evalhome python3 scripts/eval_retrieval.py --build --save-baseline
#
# Cost: tapping is ~1 s/repo (measured 6 in 6.3 s, 14 more in 20.4 s), so this
# adds roughly 20 s to a cold CI run and nothing to a warm local one (the
# sentinel in ensure_eval_corpus.sh skips re-tapping, and is keyed on this
# file's digest so editing it can never leave a stale corpus behind). Pinning
# adds at most one `git fetch --depth 1 <sha>` per repo, and none when the
# shallow clone already carries the pinned commit. CI caches the materialised
# corpus on the same digest, so a repo that disappears upstream stops being able
# to redden every open pull request at once.
anthropics/skills                       b29e7cf65e5cb78a5ac33d582270551bc74a14eb    18
0xfurai/claude-code-subagents           9236d10702cdbba37eaa34515f1e1dbff8452506   138
NeoLabHQ/context-engineering-kit        555b95263ecf13d2ad9be14df676a1263d5a62b2    92
Aaronontheweb/dotnet-cursor-rules       9be2d201482bdafca8c687359f0ef0a71895d7af    11
ChrisWiles/claude-code-showcase         a95518f0cb67e86230119da40429169bc4c35a6f    14
LessUp/awesome-cursorrules-zh           cd6a1f34f4f9eed7a01084751b9f6514eee1882d   470

# --- scale: not golden targets, present so the corpus is a realistic size ---
aliarghyani/vue-cursor-rules            92609798e192312ce16edb78aa5424a8caaa415b    14
affaan-m/ECC                            e4e4163101f162881e628f300a9ca4e6a940bcea  1616
BagelHole/DevOps-Security-Agent-Skills  0365f57a079b1332f95cf26e31dd2d5332a8399f   163
composio-community/awesome-codex-skills 0930e1373789d2eda449039f7ac154b33031de89   880
first-fluke/oh-my-agent                 3016e9df463f28f8c9c33f4e810a1901425a8bb0   136
iktakahiro/python-fastapi-ddd-skill     aa498d4c81acae475e05528520009c275cbe554f     6
langchain-ai/skills-benchmarks          064a5087fda3472a6b2883be7841441d038d2e2f    21
minio/skills                            d54382989a5ccc5fcc7de2905f7454f944a28d76     3
OneWave-AI/claude-skills                27f8cdcea225e627a73de89b9a3d477af9e249da   204
quarkusio/quarkusdev-skills             b7c025b996bb4b9c3054740b242d1f368ae0eadd     5
sickn33/antigravity-awesome-skills      d43065e483d9057e5c5dd50ca3647652ca7c0b43  6309
tuanhaviet22/magento-skills             32d21015bd8eb46b8351fc87a8988ddca44de0f7     5
anthropics/claude-agent-sdk-python      f8b9ec923982082a02c485924e0f60367949c3a1     6
eclosion-labs/terraform-cursor-plugin   bcaad7033598c7bbb4efd73483b49962bd3398d9    41
