Review of Submission 1621 (CoUS: Behavior-Equivalence-Based Candidate Selection)

Summary

CoUS picks one program out of 50 LLM samples using 100 LLM-generated tests. Tests that split the pool into the same groups are merged into one unit; each unit is weighted by how often its majority output matches the generated expected outputs; candidates score the weighted units whose majority they belong to. On CodeContests and LiveCodeBench with four 2026 models it beats five baselines by 1.2 to 6.7 points.

Strengths

- The observation is right: two tests that induce the same partition carry the same ranking evidence, and vote-counting methods count it twice.
- The method is cheap, and Property 1 makes it invariant to test duplication, which no baseline is.
- The failure analysis separates unrecoverable cases (no correct sample) from recoverable ones and is honest about the headroom left.

Weaknesses

- All of the gain comes from merging, yet merging is never applied to the baselines.
- The strongest related methods (B4, ACES, AlphaCode clustering, S*) are absent.
- The repository reveals undisclosed pipeline steps, including filtering candidates on public example tests.
- Property 2's independence assumption contradicts Limitation 3.
- Two worked examples are wrong or uninformative.
- One sampling seed, no variance, no significance tests.

Detailed review

Novelty. Grouping tests by identical candidate partition is the transpose of CodeT's consensus sets (candidates with identical pass vectors), and it is column deduplication of a mutation kill matrix with candidates as mutants (Ammann, Delamaro, Offutt, ICST 2014; Kurtz et al. on dynamic subsumption). Neither is cited. The idea is worth publishing, but "a new evidence granularity" oversells it. Say instead: we apply test-suite reduction to the selection matrix.

The decisive experiment is missing. Table II shows CoUS without merging (55.00 / 52.89) is below CodeT (56.97 / 56.20) and MBR-EXEC. So the scoring design does not win; merging does, and merging is preprocessing. Run MBR-EXEC, CodeT and DS3 on merged units. If they close the gap, this is a paper about a preprocessing step, which is fine but different. Also report K/m' per problem; it is asserted, not measured.

Missing baselines. B4 (Chen et al., ASE 2024) is a Bayesian selector built for unreliable tests and beats CodeT by a wide margin. ACES (Sun et al., arXiv, April 2026, from the DS3 group) weights tests by leave-one-out AUC. AlphaCode's behaviour clustering is the original selector on CodeContests. S* (Li et al., 2025) synthesizes distinguishing inputs, your Type 2 failure. VRank is a Verilog paper, an odd stand-in for LLM judges.

Undisclosed pipeline. The repository's CodeContests config sets filter_candidates_on_public_examples: true (keep ratio 0.6, minimum 30 kept), the README prunes duplicate candidates, and there are separate LLM prompts for test validation and expected-output generation. None of this appears in Section IV-E. Public-example filtering is a ground-truth oracle applied before selection; duplicate removal changes MBR-EXEC and CodeT vote counts. State exactly what the pool is, and rerun unfiltered. The repository holds no baseline code, candidate dumps or results, so the data-availability claim is not yet true.

The Gmax restriction. A correct minority group gets zero from a unit even when the oracle agrees with it; that evidence only enters the tie-break. Score every group by its own oracle agreement: a one-line change and the obvious ablation.

Property 2 versus Limitation 3. Tests that induce the same partition usually probe the same input region (in Fig. 1 every input is 1 mod 4), so their oracle errors are correlated and averaging attenuates nothing. Measure how often oracles within a merged unit are wrong together.

Examples. In Fig. 2, Eq. 6 gives S(C2)=3 and S(C1)=S(C3)=2, not 0.783, 0.580 and 0.591. In Section III-E, c1 also wins under plain pass rate and MBR-EXEC, so the example shows nothing CoUS adds. Fig. 1's scores follow from no method in the paper.

Statistics and data. The 1.21-point minimum gain is two problems out of 165. Report paired McNemar or a bootstrap over problems, with at least three sampling seeds. LiveCodeBench release_v5 has 880 problems, not 121; explain the subset. All four models postdate January 2025, so "contamination-free" does not hold; keep only problems released after each model's cutoff. Add the pool's Pass@50 to Table I.
