Control policies without gradient training: an LLM rewrites the program, and four tasks improve on every held-out episode

Improvement in a control policy can happen during program search rather than only by training neural-network weights. EliteTable has an LLM repeatedly propose, edit, and remix policy source code, keeps the highest-scoring programs in a table each generation, and makes new candidates compete with them. Earlier program-search systems each retain something different: FunSearch keeps an island-based database of multiple populations, ShinkaEvolve combines parent sampling, code-novelty rejection, and bandit-based model selection, and Evolution of Heuristics evolves heuristic descriptions together with their executable implementations. Ten independently initialized searches per task across six Gymnasium control tasks, each with ten generations and fifty candidate slots per generation; between the first and final generations, newly evaluated candidate-score medians rose in all six tasks and selected-winner search-score medians rose in five, with CartPole-v1 winners already at the search-score ceiling in generation one.

The winners selected during search transfer their returns to new episodes in some tasks. Rising search scores alone do not show generalization, because winners are chosen only by their score on the search episodes and may simply fit the selection panel. Each search's first- and final-generation winners were evaluated in pairs on the same 100 held-out episode seeds, disjoint from the search panel; all ten endpoint pairs improved in MountainCar-v0, LunarLander-v3, BipedalWalker-v3, and CarRacing-v3, CartPole-v1 stayed at the ceiling, and six of ten Acrobot-v1 searches declined with a negative median paired change.

Generation, validation, and repair form one pipeline that lets LLM-written policy code actually run. Code produced directly by a language model often has syntax or runtime errors, so it has to be checked before it is executed. Candidates are checked for interface compliance, valid source edits, and parseable Python syntax via the abstract syntax tree, with duplicate detection comparing AST structures rather than source text; validation or execution failures send diagnostics to a repair loop with up to five repairs per candidate slot, timeouts count as failures, and of 30,000 candidate slots 29,959 (99.9%) reached evaluated status while 41 were discarded, with 38,289 logged model calls including 8,287 repair calls.

The report is for readers working on program search, reinforcement learning, or agent control. The demonstrated scope is six Gymnasium control tasks, ten independently initialized searches per task, ten generations per search, fifty candidate slots per generation, an elite table of size ten, and held-out evaluation on 100 episode seeds disjoint from the search panel. The author's stated next step is budget-matched comparisons with other program-search algorithms, including a history-independent generation baseline, followed by complete-search ablations and comparisons across language models, with longer budgets, new episode panels, and other task families testing the limits of the observations.

The author states that the observed improvements cannot yet be attributed specifically to reuse, editing, remixing, retention, or repair, and that ablations and comparisons with other systems are needed. Ten searches per environment quantify variation under fixed episode panels but do not establish robustness across episode distributions, language models, prompts, or search configurations; the tasks are well known, so the model may have encountered descriptions or potential solutions during training. Time-based deadlines can produce different outcomes on different hardware, and hosted-provider variation can change model outputs and search trajectories. Acrobot-v1's search scores rose while its median paired held-out change was negative, and the cause needs further examination; held-out evaluation covers each generation's selected winner, not every retained elite. BipedalWalker-v3 and CarRacing-v3 improved without reaching their reporting targets, and the tested budget does not establish whether additional generations would eventually solve these tasks.