Prompt optimization lifts a deep-research agent's citations: on unseen topics, reward still rises 0.079

Prompt optimization can improve a deep-research agent's citation quality on topics it never saw, without updating any model weights. Prior studies of this kind reported mostly in-distribution gains, and an optimizer that reads the training examples can write their answers into the prompt, so an in-distribution gain cannot separate behaviour change from memorization. On a held-out set of 500 tasks over 254 topics with zero topic overlap against train and validation, the optimized prompt beats the seed by 0.079, with a bootstrap 95% confidence interval of [0.051, 0.107] and a significant Wilcoxon signed-rank test; 287 tasks improved, 147 regressed, and 66 were unchanged.

The gain comes from transferable citation discipline, not from copying training answers into the prompt. Nearly half of the winning prompt by word count is a memorized answer key, which invites the reading that this key is what inflates the validation score. On the 24 exact training questions the optimized prompt beats the seed by only 0.090, statistically indistinguishable from the 0.079 held-out gain and far below the 0.202 validation gain; on 96 fresh same-topic questions the memorized families gain 0.071 against 0.154 for the non-memorized families.

What makes the gain transfer is decomposed per-dimension feedback, not finer or repeated scoring. Collapsing the per-dimension textual feedback into a single scalar reward drives held-out transfer negative, while coarsening the reward resolution or repeating scoring three times does not raise it. Re-scored by one canonical verifier on 120 topic-disjoint tasks, the fine single-pass decomposed configuration transfers 0.079, the coarse one 0.075, the three-times-repeated one 0.059, and the scalar configuration is negative; repetition only tightened the seed-to-seed spread to 0.018.

The rules the optimizer found can be written by hand, and the search itself adds only a small residual. If the gain came mainly from the search machinery, a hand-written rule set should recover only a small part of it. On 100 held-out tasks, a hand-written recipe with all memorized content removed recovers 0.047 of the 0.083 total gain, about 57%, and the further 0.036 added by the full optimized prompt is not individually significant at that sample size.

The protocol covers one multi-agent deep-research system, one citation-quality task, and one frozen news corpus, with a held-out set of 500 tasks over 254 topics and train and validation sets of 24 tasks over 12 topics each. It suits researchers and engineers who are evaluating or deploying prompt optimization and need to know how much of a reported gain survives on unseen topics. The authors propose replicating the protocol in domains split by entity or document source rather than by news topic, and constraining the optimizer's update step with generalizability requirements and a validation gate.

The reward is scored by language-model judges, and the human check covers only 48 pairs with a single annotator, which the authors say a larger multi-annotator study would strengthen. The policy, the reflector, and the citation judges come from the same provider family; an out-of-family judge preserves the gain, but that check uses one alternative model on a small sample. The verifier ablation runs only two seeds per configuration, so the authors read it as directional rather than precise. The held-out gain replicates across three seeds, and the authors treat the validation-minus-held-out gap as a diagnostic for overfitting rather than a calibrated measurement of memorization.