# Phase 3D: full TRAIN/VAL candidate-pool generation (Issue #101)

Reached only because Phase 3C (`data/phase3c_500_target_feasibility/findings.md`)
was GO on every criterion. **Formal TEST is still not generated or evaluated
in this document** -- per the user's explicit sequencing rule, that only
happens after full train+val are confirmed, feature schema/hyperparameters/
early-stopping/model-selection are frozen, and GATE_THRESHOLDS are fixed.

Both pools use the same fixed configuration as Phase 3C: `Exhaustive`
proposal mode, `data/templates_extracted_500.smi` (500 rules), `stock=None`
(Phase 3C-6's stock-agnostic-first design, carried through unchanged),
generated by the post-Phase-3C-2 `renkin-pool-gen` build that fails loudly
(and drops candidates, counted) on any `target_id` mismatch between the
caller's group input and `propose_one_step`'s internally re-derived
canonical form -- see Phase 3C's findings for the root cause and fix.

## Full VAL (4,931 groups)

| metric | value |
|---|---|
| groups requested | 4,931 |
| parse failures | 0 |
| target_id mismatches (dropped, counted) | 14 (0.28%) |
| zero-candidate groups | 1 |
| candidate rows | 135,641 |
| candidates/group p50 / p90 / p95 | 26 / 45 / 51 |
| wall clock | 437.35 s (7.3 min) |
| peak RSS (`maximum resident set size`) | 147,177,472 bytes (140.4 MB) |
| pool file size | see `manifest_val_full.json` |
| candidate_jsonl_sha256 | `sha256:2770047e6103ffee9d63b9122996739cb8f0fa17b1fc42312eed742ee38fc1e1` |
| target_group_index_sha256 | `sha256:bb93a9b46c7eafd6b9dc2f138e020f2e9407a6e11bf73e8c53405b9cb26fe6ca` |

Matches Phase 3C-4's extrapolation closely: projected ~134,550 rows / ~7.2 min
/ ~160 MB peak RSS vs. actual 135,641 rows / 7.3 min / 140.4 MB.

**Real-loader validation** (`scripts/phase3c_coverage_diagnostics.py`, same
machinery, reused unmodified for full-scale):

| check | result |
|---|---|
| unlabeled groups | 0 |
| split assignment mismatch | 0 |
| candidate/group index mismatch | 0 (validate_manifest/validate_pool_rows pass) |
| zero-positive groups dropped? | no -- retained (1,678 / 4,931) |

**Coverage/rank (pipeline feasibility only, not a reranker-accuracy gate)**.
`n_groups_zero_positive` as reported by the loader (1,678) conflates two
different things and must not be read as one coverage number -- the 14
`TargetIdMismatch` groups are a data-generation defect (excluded before
proposal ever produced candidates under a usable identity), not a real
"the rules found nothing" coverage gap. Split out:

| metric | value |
|---|---|
| groups with >=1 positive | 3,253 / 4,931 (65.97% of all groups) |
| groups excluded by the target_id-mismatch guard | 14 (0.28%) -- data-generation defect, not a coverage gap |
| groups with genuine zero positive (real proposal attempt, no match) | 1,664 / 4,917 (33.84% of proposal-attempted groups) |
| **coverage rate over groups that actually got a proposal attempt** | **3,253 / 4,917 = 66.16%** |
| positive count / group (mean, all groups) | 0.662 |
| best-positive rank p50 / p90 / p95 / max | 3 / 16 / 22 / 53 |
| rank_1 | 849 / 3,253 (26.1%) |
| rank_2-10 | 1,778 / 3,253 (54.7%) |
| rank_11-50 | 624 / 3,253 (19.2%) |
| rank_over_50 | 2 / 3,253 (0.06%) |

Consistent with Phase 3C's 500-target val sample (64.6% coverage) and
Phase 3B's 100-target test sample (67%) -- coverage and rank-headroom
characteristics hold up at full scale, not an artifact of small-sample
noise.

## Full TRAIN (39,927 groups)

| metric | value |
|---|---|
| groups requested | 39,927 |
| parse failures | 0 |
| target_id mismatches (dropped, counted) | 109 (0.273%) |
| zero-candidate groups | 20 |
| candidate rows | 1,095,467 |
| candidates/group p50 / p90 / p95 | 26 / 45 / 51 |
| wall clock | 2,546.79 s (42.4 min) |
| peak RSS (`maximum resident set size`) | 1,069,072,384 bytes (1.02 GB) |
| pool file size | 1,105,964,456 bytes (1.03 GiB) |
| group index file size | 8,417,640 bytes (8.0 MiB) |
| candidate_jsonl_sha256 | `sha256:43d0f5c97a52314262e3aa421bffe7b5c1982b92e9d75d444c17acaaea2183b4` |
| target_group_index_sha256 | `sha256:5c3d135703daaeecf01ee2452e551ea2637fa597a4f8f3bba7dd9a90cc026fc0` |

Faster and lighter than Phase 3C-4's extrapolation in every dimension:
projected ~64 min / ~1.14 GB pool / ~1.25 GB (point estimate) peak RSS vs.
actual 42.4 min / 1.03 GiB pool / 1.02 GB peak RSS. The target_id-mismatch
rate (109/39,927 = 0.273%) is consistent with what 500-scale (1/500 = 0.2%)
and full-val (14/4,931 = 0.28%) already showed -- a real, narrow,
bounded-rate phenomenon, not one that grows disproportionately at scale.
Every one of the 109 was caught and excluded by the new fail-loud guard
rather than silently corrupting the pool.

**Real-loader validation** (same `scripts/phase3c_coverage_diagnostics.py`):

| check | result |
|---|---|
| unlabeled groups | 0 |
| split assignment mismatch | 0 |
| candidate/group index mismatch | 0 |
| zero-positive groups dropped? | no -- retained (13,279 / 39,927) |

**Coverage/rank (pipeline feasibility only, not a reranker-accuracy gate)**.
Same conflation to split out as val: the loader's `n_groups_zero_positive`
(13,279) includes the 109 `TargetIdMismatch` groups, which are a
data-generation defect, not a real coverage gap.

| metric | value |
|---|---|
| groups with >=1 positive | 26,648 / 39,927 (66.74% of all groups) |
| groups excluded by the target_id-mismatch guard | 109 (0.27%) -- data-generation defect, not a coverage gap |
| groups with genuine zero positive (real proposal attempt, no match) | 13,170 / 39,818 (33.08% of proposal-attempted groups) |
| **coverage rate over groups that actually got a proposal attempt** | **26,648 / 39,818 = 66.92%** |
| positive count / group (mean, all groups) | 0.670 |
| best-positive rank p50 / p90 / p95 / max | 3 / 16 / 22 / 90 |
| rank_1 | 6,678 / 26,648 (25.06%) |
| rank_2-10 | 14,581 / 26,648 (54.72%) |
| rank_11-50 | 5,363 / 26,648 (20.13%) |
| rank_over_50 | 26 / 26,648 (0.10%) |

## Cross-scale consistency

| split / scale | groups | coverage rate | rank_1 share | rank_2-10 share | rank_11-50+ share |
|---|---|---|---|---|---|
| test-100 (Phase 3B) | 100 | 67% | -- | -- | -- |
| train-500 (Phase 3C) | 500 | 65.2% | 22.7% | 55.8% | 21.5% |
| val-500 (Phase 3C) | 500 | 64.6% | 23.8% | 53.3% | 22.9% |
| val-full (Phase 3D) | 4,931 | 65.97% | 26.1% | 54.7% | 19.3% |
| train-full (Phase 3D) | 39,927 | 66.74% | 25.1% | 54.7% | 20.2% |

The coverage rate (~65-67%) and rank distribution (roughly 1-in-4 positives
already at rank 1, just over half at rank 2-10, the rest at rank 11-50+)
hold essentially unchanged from 100 groups to the full 39,927-group corpus.
This is the number the user flagged as decisive going in: coverage is real
but bounded (~1/3 of groups have no positive candidate at all -- a proposal-
coverage gap reranking cannot fix), and among covered groups, roughly 3 in 4
correct answers are *not* already first under `propose_one_step`'s own
rule-firing order -- consistent with substantial reranker headroom on the
covered subset. No development decision (hyperparameters, features, gate
thresholds) has been made from this observation; per the user's explicit
instruction, that is out of scope for this document.

## 123 targets are now permanently unusable label rows -- open decision

109 train + 14 val = **123 groups (0.27% of the combined 44,858-group
corpus) are excluded by the target_id-mismatch guard, not by a real
coverage gap.** Each of these 123 targets still exists in
`reranker_labels_uspto50k_{train,val}.jsonl` and
`reranker_split_manifest.jsonl` under its *original* canonical form, but no
pool row will ever be produced under that identity by the current binary --
the guard drops the group rather than exporting candidates under a
mismatched identity (see Phase 3C's findings for the root cause: RENKIN's
canonical-form tie-break is sensitive to whether the parsed molecule still
carries atom-map annotations, so the original label-generation pass and the
current binary disagree on the canonical form for these 123 molecules
specifically). **These 123 labeled targets are unusable for training until
someone decides what to do about it.** Two options, not decided here:

1. Regenerate the label/groups/split-manifest corpora under the current
   canonicalizer. This would invalidate the already-verified quarantine and
   decontamination chain, the split manifest, and both full pools just
   generated in this round -- a new round of work, not a small fix.
2. Accept the 123 lost targets (0.27% of the corpus) and proceed to
   training on the 44,735 that remain usable.

Not decided in this document -- this is the user's call, not a default this
round should silently take. The underlying `chematic`/`to_canonical`
atom-map-sensitivity itself also remains unfixed and unfiled; a minimal
reproducer exists in `data/phase3c_500_target_feasibility/findings.md`.
Filing it against `chematic` (the project has done this before for a
related canonicalization issue) is recommended but is an outward-facing
action, so it is not done automatically here.

## Status

Phase 3D complete for TRAIN and VAL. **Formal TEST remains untouched** --
no test pool generated, no test evaluation run, consistent with the
project's "formal TEST opens only after model freeze" rule. PR #104
untouched; PR #105 remains draft, not merged. No adaptive beam, timeout, or
runtime reranker-integration changes made. Nothing posted to Issue #101.
No tag/release.
