Frozen models improve without weight updates: TextualRL turns repeated same-task attempts into context patches, gaining up to 24.4 points on a six-benchmark average

An agent can keep improving from execution rewards by revising only the textual context that guides its behavior, without changing model parameters; TextualRL organizes this as a gradient-free policy optimization loop. Earlier context optimization mostly extracted lessons from individual trajectories or small trajectory minibatches, and a single trajectory offers limited comparative evidence, while task-specific fixes folded into a shared context can overgeneralize. On five frozen target models and six benchmarks, TextualRL achieves the highest six-benchmark average for every target, gaining 11.4 to 24.4 percentage points over the no-skill baseline and 0.9 to 3.2 points over the strongest baseline SkillOpt, and beats all baselines in 29 of 30 target-benchmark settings while tying SkillOpt in the remaining one.

The method compares a group of executions of the same task under the same context: mixed-outcome groups are contrasted directly, while all-success or all-failure groups draw on cross-task evidence to decide what to preserve or to form a repair hypothesis. Within-task contrast carries information only when outcomes are mixed; when every rollout succeeds or every rollout fails that contrast disappears, so trajectories must be routed by outcome structure rather than pooled. On Qwen3.8-27B, removing outcome routing lowers the six-benchmark average by 1.5 percentage points, and reducing each task to a single rollout lowers it by 2.5 points.

A review stage checks each group's proposed edits against independently pooled evidence cards, and cards are kept even when they produce no edit so they can limit the applicability of other groups' proposals. Merging proposals alone cannot reveal every exception: a successful execution may need no edit yet still contradict a rule proposed by another group. In the same ablation, restricting review to cards from the originating groups lowers the average by 1.2 points, removing cards that produced no edit lowers it by 1.7 points, and disabling cross-group proposal review lowers it by 2.7 points.

Reviewed edits form a bounded textual patch, and a candidate context is adopted only when its reward on held-out validation exceeds the recorded reference score. Adoption is decided by held-out validation reward rather than by the rollout group that proposed the edit. Repeating the full optimization three times on Qwen3.8-27B gives TextualRL an overall mean of 70.68% versus 67.43% for SkillOpt, with a higher mean on all six benchmarks and an overall advantage of 3.18 to 3.36 points across the three runs.

The demonstrated scope is five frozen target models and six benchmarks, with a separately optimized context for each target-benchmark pair and the SkillOpt-released data splits for training, validation and test; GPT-5.5 was also tested inside the Codex and Claude Code harnesses on five benchmarks.

Training dynamics show most test gains arrive early, DocVQA and LiveMath validation scores then plateau while test scores fluctuate within a band, and OfficeQA shows no sustained improvement after the first epoch; on OfficeQA and LiveMath validation scores rise far more than test scores, so instructions favored by a fixed validation set may be less effective on unseen tasks.