Review: Paper 554 — BEACON: Behavior-Grounded Evidence Aggregation for LLM-Generated Code Selection

Summary
-------
BEACON picks one program from a pool of LLM-sampled candidates. It clusters candidates by their outputs on generated inputs, extracts a minimal set of inputs that tell the clusters apart, and asks LLM agents to judge which cluster's behavior matches the specification, aggregating rankings by Borda count. It beats CodeT, SRank, and DebateCoder on LiveCodeBench and CodeContests, most clearly on hard tasks.

Strengths
---------
+ The diagnosis is right and well put: consensus tells you who agrees, not who is correct.
+ Clustering plus input minimization keeps the judge's prompt small. The ablation showing that adding source code HURTS the judge is the most interesting result in the paper.
+ Multi-model BEACON is both more accurate and about half the cost of DebateCoder.
+ Working artifact; the abc329_e case study is instructive.

Weaknesses
----------
- The core idea is older than the paper allows. Distinguishing inputs go back to oracle-guided synthesis (Jha et al., ICSE 2010). S* (arXiv:2502.14382, Feb 2025) already has an LLM choose between candidates by inspecting execution outputs on generated distinguishing inputs, evaluated on LiveCodeBench. B4 (ASE 2024) and ALGO (NeurIPS 2023) attack the same selection problem. None is cited or compared.
- No oracle ceiling. Without pass@10 of the pool per subset, I cannot tell how much available headroom BEACON captures.
- No statistical tests. CodeContests Advanced has 74 tasks; the 14.86pp headline is eleven tasks.
- With 10 samples, 85.8% of Advanced classes are singletons. Consensus barely exists in that regime, so beating CodeT and SRank there is a low bar; those methods were built for pools of 100+.
- The judges are the models that wrote the candidates; misconceptions correlate.
- Exact output equality mislabels tasks that admit several valid answers.

Detailed comments
-----------------
The composition is new; the parts are not. Say so, and compare against S* and B4 directly — both run on your benchmarks. Building on old ideas is fine; leaving the reader to think the voting-to-judging shift is unprecedented is not.

To make the results convincing: 

(1) report the pool ceiling and selection efficiency; 

(2) add McNemar or bootstrap intervals over per-task outcomes; 

(3) vary the sample count (10, 50, 100); if BEACON's edge shrinks as consensus strengthens, that is the honest scope of the method, and worth knowing; 

(4) give a flip table: tasks where BEACON is wrong and SRank right, and the reverse; 

(5) DebateCoder starts from one candidate per model while BEACON gets ten, so equalize the pools or flag it; 

(6) try a third model as judge to break the generator-judge correlation; cheap, and it would strengthen the multi-agent claim; 

(7) your input cost function favors short outputs, so a trivial error-versus-output divergence can crowd out the telling one; check this; 

(8) count special-judge tasks, which corrupt clustering; 

(9) ship baselines, seeds, and per-task results with the artifact.
