Prompt optimization lifted citation reward by 0.079, but nearly half the winning prompt was a memorized answer key that earned nothing

Citation quality in a deep-research agent can improve by rewriting prompts alone, with no weight updates. Held-out citation reward rose from 0.702 to 0.781, a gain of 0.079 with a bootstrap 95% confidence interval of [0.051, 0.107]. Prior prompt-optimization studies typically report in-distribution gains, and an optimizer that reads training examples can copy their answers into the prompt, so such a gain cannot separate a transferable behavior change from memorization. The test set holds 500 tasks over 254 topics with zero topic overlap against train and validation; three independent seeds all produced positive held-out gains whose confidence intervals exclude zero, with a mean of 0.107 and a range of 0.079 to 0.130.

The gain comes from transferable citation-discipline rules rather than from the search machinery. The optimizer moved the answer to the front of the report, split multi-claim sentences so one claim carries one citation, required verifying that the cited page's content supports the specific claim, and restricted citations to corpus URLs. Each rule answers a specific diagnostic the decomposed reward surfaces, that a link failed to resolve, that a cited page was off-topic, or that a page did not support the claim; collapsing the reward to a single scalar removes that diagnostic channel and turns held-out transfer negative. The verifier ablation holds data, policy, reflector, and budget fixed and varies only the reward signal, running 4 configurations at 2 seeds each for 8 optimization runs, then re-scoring every prompt with one canonical verifier on 120 topic-disjoint tasks; the fine, single-pass, decomposed reference transfers positively while the scalar variant does not.

Nearly half of the winning prompt is copied training answers, and it buys no reward. By word count the memorized answer key is 52% of the orchestrator prompt and 39% of the researcher prompt. If memorization drove the validation gain, the best prompt's improvement on those 24 exact training questions should approach the 0.202 validation gain. On the 24 exact training questions the best prompt beats the seed by only 0.090, statistically indistinguishable from the 0.079 held-out gain; on 96 fresh same-topic questions the 12 memorized families gain 0.071 against 0.154 for non-memorized families.

A hand-written version recovers most of the gain, so the improvement is largely a describable rule set rather than a property of the search. With all memorized answer content removed, the recipe recovers 0.047 of the 0.083 total held-out gain on 100 tasks, about 57%. The full optimized prompt adds a further 0.036, but that increment is not individually significant at 100 tasks, consistent with the memorized answer key being inert. Recipe, seed, and best prompt are scored under the same policy, reward, and corpus; the total seed-to-best gain of 0.083 is significant and matches the 0.079 on the full 500-task set.

The protocol is demonstrated on one multi-agent deep-research agent, one citation-quality task, and a single news corpus, with a language-model-judge reward. The held-out gain of 0.079, the mean of 0.107, and the roughly 57% recovery by the hand-written recipe all hold within that setting. The judge-versus-human check covers 48 annotated pairs with a single annotator, and the authors propose a larger multi-annotator study plus more seeds and an out-of-family verifier as natural next steps.

The validation-minus-held-out gap swings from 0.034 to 0.160 across three seeds, about a factor of five, so it is treated as a diagnostic that flags overfitting rather than a calibrated measurement of memorized reward. The verifier ablation runs each configuration at only 2 seeds, so the authors read the cell means as directional rather than precise. The policy, reflector, and citation judges all come from the same provider family, raising a self-preference risk, and the out-of-family check uses a small sample. Memorization pressure is expected to be larger in narrow domains with recurring entities, which has not yet been tested on other tasks or domains.