evaluate op• Data kinds: signal × signal × signal × signal → table
• Call: import fullseye as fs; fs.ledger.holdout_threshold(train_pos, train_neg, test_pos, test_neg, target_fpr: 'float' = 0.05) -> 'dict' (to call the implementation directly, import emproof; emproof.holdout_threshold(train_pos, train_neg, test_pos, test_neg, target_fpr: 'float' = 0.05) -> 'dict'; from the registry, opsemproof.get("holdout_threshold"))
Chooses the threshold on the training side, and measures the evaluation on the other side (a higher score means "more suspicious").
tau = the smallest threshold at which the false-positive rate of the training negatives (components and pairs not planted) is at most target_fpr (the upper target_fpr quantile of the negative scores). Returned are tau and, for training and evaluation each, AUC, TPR, FPR and counts. Only the evaluation-side numbers are used for claims -- a TPR measured on the side where the threshold was chosen is always optimistic (the holdout discipline of evolve carved out as an op). AUC is Mann–Whitney (ties 0.5).
>>> r = holdout_threshold(tr_pos, tr_neg, te_pos, te_neg, target_fpr=0.05)
>>> r["tau"], r["test_auc"], r["test_tpr"], r["test_fpr"]
• Sample-data catalog (download URLs / licences) — 2-D uses skimage.data (BSD/public domain) plus synthetic images; 3-D lists download URLs for real data sources (Stanford, PDS, …).
• Operator provenance and references — the sources of the research/methods this op family came from.
• The canonical algorithm (author, year) and its uses are named in the family usage guide above.
• poc_em_second_opinion — py -3.11 examples/poc_em_second_opinion.py
table as input)seg_synapse_partners · seg_wiring_variation · seg_synapse_nri · seg_wiring_exposure
evaluate)—
*Provenance: emproof.py — EMPROOF operator registry. This per-op note is generated by tools/opdocs.py md (do not hand-edit).*
© 2026 Kazufumi Furuse — Fullseye operator documentation. Licensed under Apache-2.0.