# LLM code generation

ChatGPT, Claude, Gemini and Grok each generated PyTorch helmet-compliance training scripts from three prompt tiers of increasing specificity, run 20 times on 972 balanced images: ten of twelve configurations converged to a 0.957–0.967 accuracy plateau, but the most detailed tier left Gemini and Grok near 0.58, with scripts that ran yet failed to converge. A backbone-by-scheduler ablation traced the failure to the prescribed from-scratch multi-scale network, which collapsed under both schedulers while a pretrained ResNet18 converged under either.

ChatGPT, Claude, Gemini and Grok each generated PyTorch helmet-compliance training scripts from three prompt tiers of increasing specificity, run 20 times on 972 balanced images: ten of twelve configurations converged to a 0.957–0.967 accuracy plateau, but the most detailed tier left Gemini and Grok near 0.58, with scripts that ran yet failed to converge. A backbone-by-scheduler ablation traced the failure to the prescribed from-scratch multi-scale network, which collapsed under both schedulers while a pretrained ResNet18 converged under either.

Reasoning models draft code early inside their chain of thought, and SecReason scans the streaming tokens before delivery and inserts a warning so the model repairs the draft within the same reasoning trace, raising the share of runs that are both secure and functionally correct from below 15% to 72.8%–98.1%.

Reasoning models draft code early inside their chain of thought, and SecReason scans the streaming tokens before delivery and inserts a warning so the model repairs the draft within the same reasoning trace, raising the share of runs that are both secure and functionally correct from below 15% to 72.8%–98.1%.

Quantum programs are judged by probabilistic outputs and domain-specific checkers, so a syntactically valid test can still disagree with the benchmark verdict. QTestBench has models write syntactic, semantic, and behavioral pytest modules per task, injects candidate solutions, and compares each test verdict against the benchmark checker rather than the tests themselves across three validation cohorts. On eight PennyLane QHack challenges and ten Qiskit HumanEval tasks, generated suites almost always execute, yet discrimination lags: the best test quality is 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality.

Quantum programs are judged by probabilistic outputs and domain-specific checkers, so a syntactically valid test can still disagree with the benchmark verdict. QTestBench has models write syntactic, semantic, and behavioral pytest modules per task, injects candidate solutions, and compares each test verdict against the benchmark checker rather than the tests themselves across three validation cohorts. On eight PennyLane QHack challenges and ten Qiskit HumanEval tasks, generated suites almost always execute, yet discrimination lags: the best test quality is 0.71 on QHack and 0.34 on Qiskit HumanEval, so executability alone overstates test quality.

Coding agents often fill in underspecified requirements with their own assumptions, producing code that looks correct but behaves in ways the user never intended, and fixing that later grows costly. CONTRA splits clarification into broad question generation followed by semantic filtering and execution-based qualification, keeping only questions whose two plausible answers yield stable behavioral differences, then uses interaction history to ask or stop. On ClarifyCodeBench it reaches the highest F1 with all four coding agents, 13.88 percentage points above the best baseline macro-average.

Coding agents often fill in underspecified requirements with their own assumptions, producing code that looks correct but behaves in ways the user never intended, and fixing that later grows costly. CONTRA splits clarification into broad question generation followed by semantic filtering and execution-based qualification, keeping only questions whose two plausible answers yield stable behavioral differences, then uses interaction history to ask or stop. On ClarifyCodeBench it reaches the highest F1 with all four coding agents, 13.88 percentage points above the best baseline macro-average.

The work proposes dynamic context adaptation: a validation-generation loop in which a validation agent extracts structured diagnostic feedback from execution traces, a generation agent proposes multiple candidates per iteration, a knowledge graph supplies semantic constraints, and simulated annealing performs non-greedy selection, targeting the LLM code-generation limitation the authors call static binding; across eight problems the method outperforms zero-shot, Reflexion, and OpenEvolve on seven of eight problems at both 300 and 600 evaluations (p < 0.01), achieves the best score at 1000 evaluations on the primary motivating problem of cross-coupled optimization (0.694 vs. 0.681, p = 0.019, d = 0.52), and ablations identify structured execution feedback as the primary driver.

The work proposes dynamic context adaptation: a validation-generation loop in which a validation agent extracts structured diagnostic feedback from execution traces, a generation agent proposes multiple candidates per iteration, a knowledge graph supplies semantic constraints, and simulated annealing performs non-greedy selection, targeting the LLM code-generation limitation the authors call static binding; across eight problems the method outperforms zero-shot, Reflexion, and OpenEvolve on seven of eight problems at both 300 and 600 evaluations (p < 0.01), achieves the best score at 1000 evaluations on the primary motivating problem of cross-coupled optimization (0.694 vs. 0.681, p = 0.019, d = 0.52), and ablations identify structured execution feedback as the primary driver.