Review: Paper 7689, ICT-Eval: Cost-Effective Reference-Free Correctness Prediction for LLM-Generated Code

Summary
-------
ICT-Eval fine-tunes two GraphCodeBERT encoders, one on the task description and one on the candidate code plus a value-level execution trace from half of the test inputs, and trains an MLP to predict full-suite pass. On HumanEval(+), MBPP, BigCodeBench and held-out LiveCodeBench it reports higher Kendall tau than CodeJudge, ICE-Score and a vanilla prompt on six judge backbones, in under a second.

Strengths
---------
+ The trace collector is careful: assertions become bare calls before tracing, so expected outputs never reach the model.
+ Reporting a crash-detector floor (Table IX) and bootstrap intervals on silent subsets is more honest than most papers here.
+ Full artifact: code, caches, checkpoints, judge verdicts.

Weaknesses
----------
- The generators are broken, so the labels are not what the paper says. Table I gives MBPP pass rates of 5-7%, and 7.4% on LiveCodeBench for DeepSeek-R1-Distill-32B; published figures are roughly 60-80% and 57%. The notebook shows why: MBPP prompts are a bare comment with no signature while the tests call the canonical function name, and generators run 4-bit with max_new_tokens=512, which cuts off R1's thinking before any code. Most "crashes" are NameErrors and truncation.
- "HumanEvalPlus" is plain HumanEval; the code says "EvalPlus oracle removed".
- BigCodeBench has no execution trace. run_bcb_test records only IMPORT_OK or IMPORT_FAIL:<type>, so BCB claims about value-aware traces rest on text alone.
- Table IX shows the trivial crash detector beating ICT-Eval on MBPP (0.927 vs 0.528) and LiveCodeBench (0.474 vs 0.253) and tying elsewhere. It is missing from Table II.
- Design choices were picked on the test split (Ablation 1 selects by zero-shot test tau, Ablation 2 by "best mean test tau"). The split is 80/10/10, not 80/20, so the HE+ test set is about 17 problems and MBPP about 50; GPT-5.5's tau of 1.000 rests on 17 problems.
- LLM judges see intent and code; ICT-Eval also sees execution on half the tests. No judge got that trace.

Detailed comments
-----------------
Novelty is modest. LEVER (Ni et al., ICML 2023) trains a verifier on exactly this triple: natural language, program, execution result. TRACED (Ding et al., ICSE 2024) pre-trains encoders on traces with variable values. Neither is cited. CodeScore's NL-only mode is reference-free, contrary to Sec. VI; CodeCircuit (arXiv 2602.07080, Feb 2026) is another test-free learned predictor. New here: the partial-budget framing and the silent-failure analysis. Build the paper on those.

The evidence does not yet support the trace story. In your ablation the trace adds 0.048 mean tau over NL|code (0.476 vs 0.428), and value-aware beats control-flow-only by 0.009. Yet the same final configuration reads 0.476, 0.480, 0.495 and 0.508 in Tables VI, VII, VIII and II, so those gaps sit inside unmeasured seed noise. GPT-5.5 calling most MBPP solutions correct while the harness fails them says the labels are wrong, not that GPT-5.5 "collapsed".

Fixes, in priority order:

(1) Regenerate solutions with proper prompts (signature plus example tests for MBPP, the EvalPlus harness for HE+, R1 with at least 8k tokens). Print pass rates beside published ones and recount silent failures; expect hundreds, not 2 and 3.

(2) Add three rows to Table II: crash detector, pass rate on the exposed 50% of tests, and each LLM judge with the trace pasted into its prompt. Only the last fairly tests "grounding beats scale".

(3) Trace BCB for real (call task_func on test inputs under settrace) or drop its silent-failure claims.

(4) Freeze design on validation; use the full 20% as test or do 5-fold by task; report mean and std over seeds.

(5) Ship the code that built the cached parquet files. The notebook never binds `candidate` for HumanEval asserts, so rerunning it would fail every HE solution; the cached labels came from elsewhere.

(6) Table V: say the judges run 4-bit (a 32B model in 8.5 GB while an 8B needs 12.1 GB makes no sense otherwise), and note Vanilla·Qwen is faster than ICT-Eval (0.82 vs 0.89 s) and beats it on LCB.

(7) Small things: "all three as baselines" after naming two; "wildw"; "are follow from"; Qwen2.5-Coder-7B is 7.6B, not 8B.
