Choosing among code drafts rivaled more elaborate methods
On Gemma 4 31B, one of 66 techniques beat budget-matched code selection on its primary target beyond the 95 percent uncertainty interval, consistent with chance.
Edited by Boris Djordjevic · Paperfoot
A systematic literature review and controlled re-evaluation examined techniques for improving model-generated code, with benchmark tests of correctness, ease of maintenance, efficiency and security. The review covered 628 studies. Most measured only the quality they targeted. The tests compared ten techniques across four models at matched computing budgets, then extended to thirteen models and tasks involving code repositories.
Selecting among several generated answers using a ranking system that weighted automated code-checker results matched or beat quality-focused instructions, planning and model self-revision. That result held on measures the ranking system did not optimize. Feeding checker feedback back to the model, and additional model training, beat selection only on checker issue counts. In a separate comparison of 66 techniques on Gemma 4 31B, one beat selection on its primary target beyond the 95 percent interval, consistent with chance.
Why it matters
The controlled tests showed that generating and ranking alternatives can rival more elaborate ways to improve code. The authors recommended comparisons at equal computing budgets using quality measures independent of each method’s own signals.
Caveats
The preprint’s 66-technique comparison used just one model. Large correctness losses on HumanEval+, a likely contaminated code-correctness benchmark near its ceiling, did not appear on contests held after the models’ training cutoffs.
In this paper
improving the quality of LLM-generated code
- Methods
- best-of-n (baseline)T7 (baseline)T0 (baseline)quality instructions (T1) (evaluated)planning (T2) (evaluated)analyzer loop (T3) (evaluated)Self-Refine (T4) (evaluated)reviewer agent (T5) (evaluated)retrieval (T6) (evaluated)RL (T10) (evaluated)AutoSafeCoder (evaluated)Gemma 4 31B
- Datasets
- HumanEval+ (evaluated)CWEval (evaluated)BaxBench (evaluated)PythonSecurityEval (evaluated)EvalPerf (evaluated)LiveCodeBench (evaluated)Mercury Beyond (evaluated)EffiBench-X (evaluated)LiveCodeBench Pro (evaluated)DeepSWE (evaluated)
Results
- T0, maintainability issues per kLOC, on Op: 21.37; T0, maintainability issues per kLOC, at T0 on oss: 29.22
H4 therefore holds for absolute gains, which partly reflect that the weaker models start with more findings ( 29.22 maintainability issues per kLOC at T0 on oss against 21.37 on Op), and not for the relative gains of the analyzer loop.
16.1. Results of the main re-evaluation
- T0, pass@1, on HumanEval+, single-shot on Op: 96.40; T0, pass@1, on HumanEval+, single-shot on G31: 96.06
No technique is more correct than T7 on every model, and on the two models with the highest single-shot HumanEval+ pass@1 (Op, 96.40; G31, 96.06 ) almost every technique is below the single shot: T2 to T6 by 0.59 to 6.90 points on Op and by 3.81 to 11.08 on G31, and the fine-tuned G31 models by 7.72 to 10.59; only the quality instruction (T1, by 2.70 on Op and 1.90 on G31) and, on Op, best-of- n are above it …
16.1. Results of the main re-evaluation
- T7, score, on CWEval: 74.85; T7, score, on CWEval: 78.17; T7, score, on CWEval: 74.43; T7, score, on CWEval: 86.29
The curves are not smooth (on CWEval, T7 on Op scores 74.85, 78.17, 74.43 and 86.29 at B=1, 4, 8 and 16 ): the per-seed values show that these steps recur in every rerun rather than reflecting seed noise, and we read only steps beyond the half-width, here only the gain at B=16 (Supplementary Material …
16.1. Results of the main re-evaluation
- AutoSafeCoder, pass@1, on HumanEval: 0.8780; AutoSafeCoder, pass@1, on HumanEval: 0.9085
… AutoSafeCoder lowered HumanEval pass@1 from 0.9085 to 0.8780 while reducing Bandit findings, whereas the fitness-gated Pylint and Bandit loop of raised functional correctness from 38.16% to 47.81 …
RQ3: effects on correctness and on non-targeted dimensions.
What it shows
- Across thirteen models, maintainability and security increase alongside model capability while efficiency does not.
Across the thirteen current models, maintainability and security rise with capability while efficiency does not (§16.2), so the choice of model moves these dimensions about as much as any single technique, and efficiency needs its own measurement.
For practitioners.
- At matched call budgets, sampling and ranking match or exceed iterative refinement techniques across evaluation measures.
This supports H1: at matched calls, sampling and ranking match or beat iterative refinement, as found for self-repair and and for self-refinement and post-hoc operators on small models, and the budget curves show it is not an artefact of B=4: on maintainability issues per kLOC, the only inference-time techniques better than T7 beyond the interval at B=8 or B=16 are the analyzer loop and, on oss at B=16, retrieval (Fig. 4 …
16.1. Results of the main re-evaluation
- At a single generation call, quality instructions fail to lower static analyzer issue counts across evaluated models.
A single call with a quality instruction or a plan does not lower maintainability issues on any of the four models (§16.1); its apparent gains at larger budgets come from the ranker it shares with best-of- n.
For practitioners.
- Frontier code-generation agents generate functional but insecure code on roughly four in ten SusVibes repository tasks.
Security techniques lower rule-based detector hits far more often than they raise the share of code that is both functional and secure (§16.3), and current frontier agents produce working but insecure code on about four in ten SusVibes tasks (§16.2).
For practitioners.
Quoted word for word from the full text on arXiv.
The paper
arXiv, Oct 28, 2026, Open access, Preprint, not peer-reviewed
doi.org/10.48550/arxiv.2610.38977
arXiv:2610.38977v1 in cs.SE, also cs.CL, cs.LG
Comments: 55 pages, 5 figures, plus 120 pages of supplementary material
Cite as BibTeX
@misc{kadam2026beyond,
title = {Beyond Passing Tests: A Systematic Review and Controlled Re-evaluation of Techniques for Improving the Quality of LLM-Generated Code},
author = {Kadam, Prashank},
year = {2026},
eprint = {2610.38977},
archivePrefix = {arXiv},
primaryClass = {cs.SE},
doi = {10.48550/arxiv.2610.38977},
url = {https://arxiv.org/abs/2610.38977}
}