Review: TriConList — Learning Tri-tier Contract Consistency for Full-Stack Repository Generation

Summary

The paper tackles frontend/backend/database contract mismatches in full-stack code generation. It builds TriConList, 1,246 ranked candidate lists made by injecting graded defects into gold repositories, trains a listwise Consistency Reward Model (CRM) to drive GRPO fine-tuning of Qwen2.5-Coder-7B, proposes the CCS metric, and adds a prompted deployment-repair pass (SR).

Strengths
- The problem framing is genuinely new. I found no prior work that learns a reward for cross-tier contract alignment; graded defect injection is a clever way to manufacture ranking supervision.
- Good ablations: reranking vs. training, tier-by-tier generation, SFT, generic RM, and a second backbone.
- CCS's API/field/type decomposition is interpretable and diagnostic.

Weaknesses
- "Linear-BT" is not this paper's invention. LiRE (Choi et al. 2024, which you cite) already replaces exp(x) with the linear score x in the Bradley-Terry model, ties included (labels {0,0.5,1}). Your own repo ships train_lire.sh.
- The evaluation is circular: an LLM extracts CCS, and the COV judge is handed the CCS bundle and told to use it — so FCS = COV x CCS counts CCS twice. The judge model is never named.
- No external benchmark. WebGen-Bench and FullStack Bench are cited but unused, so we cannot tell whether GRPO gains transfer or merely fit a metric that mirrors the training defect taxonomy.
- The CRM's pairwise accuracy is 0.68 (Table 7) on candidates a full severity level apart, yet it drives large policy gains. Reconcile this.
- Table 6 contradicts "identical relative rankings": on Stype, TriConCoder beats GPT-4o automatically (0.9474 vs. 0.8944) but loses under humans (0.8864 vs. 0.8933).
- Code/paper mismatches: the repo README says NeurIPS 2026, LoRA r=64 (paper: 16), five candidates per list (paper: four), and a ListMLE+ListNet+RankNet+NDCG loss, not Linear-BT.
- EXE at or below 0.10 for every base model, with GPT-4o-mini beating GPT-4o, points at a broken protocol, not weak models. Who hand-deployed 500 repos times nine systems?

Detailed review

First, make CCS deterministic: Express routes, request/response keys, and Prisma types are parseable, which removes extractor noise and yields a verifiable reward. Then run GRPO with deterministic CCS as the reward. If the learned CRM cannot beat that baseline, it earns nothing. Add a pairwise DPO baseline on the same lists — "listwise beats pairwise" is asserted, never tested at the policy level. Second, evaluate on WebGen-Bench. Third, hide the CCS bundle from the COV judge, name the judge, and report inter-annotator agreement for the human study. Fourth, the CRM context is 4k-7k tokens; report how often candidate repositories are truncated. Finally, cite Self-Refine/Reflexion (SR is a checklisted variant), AceCoder (RM+RL on the same base model), BaxBench, WebApp1K, and consumer-driven contract testing (Pact) as the software-engineering lineage of CCS.

Novelty: framing and dataset, yes; the training objective and SR, no. Reposition accordingly.
