Dynamic product ads get an offline judge: ranking six retrieval sources before any A/B test, without click logs
Synopsis
Retrieval quality in dynamic product advertising could previously be compared only through sparse, biased engagement logs or costly online experiments. The new method compresses a user's engagement history into a small set of intent representatives grouped by semantic ID and stratified by event type, then has a large language model judge each candidate product against every intent and aggregate the results into a user-level score. On 30 days of production traffic, the score ranked six retrieval sources consistently with conversion precision and recall, and moved in the same direction as online CTR and CVR across four A/B tests.
Interpretation
The retrieval stage of dynamic product advertising now has an offline relevance score that needs no explicit query, so different retrieval algorithms can be compared on how well they surface relevant products. Such systems previously relied on offline metrics such as future-engagement recall, or on implicit engagement logs of clicks and likes; the former suffers from selection bias and data sparsity, the latter from presentation and attention effects. Over a 30-day production window the score ranked six retrieval sources consistently with conversion precision and recall, with MicroRel showing the highest cross-source Pearson correlation with conversion precision, while MAP@15 and NDCG@15 computed on the same judge labels were weaker proxies.
The score scales because long, heterogeneous engagement histories are first compressed into a few weighted intent representatives, and a fine-tuned small model then serves as the judge. Sending every event-candidate pair to a frontier model at production volume is prohibitively expensive, so the intent set must be compressed and the frontier judgments distilled into a smaller model. The semantic-ID-grouped, event-stratified intent set reached a Spearman correlation of 0.95 against the full sequence at only a few percent of its pairwise call cost, and beat the full sequence and every other variant on CVR, CR and CP.
The judging guideline is not hand-written; an agentic loop iteratively discovers sub-rules from labeled examples, and human reviewers then steer the rule set toward business priorities. A loose prompt carrying only the five-tier instruction led the frontier model to systematic errors on rule classes such as demographic mismatch and brand-versus-category confusion. On 10,000 human-audited pairs, Gemini-2.5-Pro with the guideline was about 4 percent better in both precision and recall than without it, and the fine-tuned Gemini-2.5-Flash student reached an F1 of 0.869, matching the teacher.
Perspective
The report covers offline evaluation of the retrieval stage in dynamic product advertising, validated on 30 days of production traffic, six retrieval sources and four A/B tests, with judge accuracy checked against 10,000 human-audited pairs. The intended readers are engineers and researchers who own retrieval evaluation and model iteration. The source positions the score as an offline gate before online deployment and reports no validation on other ad platforms or product catalogs.
The source notes that engagement-based retrieval iterations mix ranker alignment with engagement signals, so their offline-to-online deltas are smaller and noisier and the correlation is weaker. Demographic-mismatch rules are too rare in the calibration set for the agentic loop's statistical gate, so the loop repeatedly deprioritized them and human review promoted them to a hard irrelevant override. A threshold sweep shows all three correlations peak near 0.5, and beyond that conversion precision and recall decay as the candidate pass rate falls.