One 9B backbone reranks and answers: Glance lifts HotpotQA answer F1 by 0.045 over cross-encoder-selected context
Synopsis
Retrieval-augmented generation usually hands reranking and answering to two models, so candidate passages are processed twice. Glance adds a lightweight decision adapter and pointer head to the generator, prefills the candidates once, and reads one short probe per passage for a relevance score. After 39 minutes of MS MARCO fine-tuning it reaches nDCG@10 of 0.741 on SciFact and 0.345 on NFCorpus, against 0.721 and 0.326 for bge-reranker-v2-m3; on 300 HotpotQA questions the same base generator answers with F1 0.045 higher from its selected context than from cross-encoder-selected context.
Interpretation
The generator can serve as its own reranker, so candidate passages need not pass through a second model before the generator reads them. Prior practice encodes each query-passage pair with a cross-encoder, or judges passages one at a time with a large model; either way an extra model is served and the candidates are processed before the generator sees the selected context. On the two BEIR datasets SciFact and NFCorpus, after 39 minutes of MS MARCO fine-tuning, Glance's packed_all single prefill scores nDCG@10 0.741 and 0.345, against 0.721 and 0.326 for bge-reranker-v2-m3; these BEIR differences are not significance-tested.
One resident 9B backbone handles both reranking and answering, and switching modes changes no weights. When reranking and generation are separate models, both sets of weights stay resident and passages are encoded twice. On the first 300 HotpotQA distractor validation questions, the same base generator with the adapter off answers from Glance's top-3 at F1 0.718 versus 0.683 from the cross-encoder's top-3; over 20 alternations of reranking and generation, entering adapter-off took a median 11 ms and leaving it 2.8 ms, with allocated GPU memory held at 16.77 GiB.
Candidates are prefilled once, each passage branches a short probe from the shared cache, and a pointer head turns each probe into a relevance score. Pairwise encoding prefills each query-passage pair separately, and per-passage judging runs the model once per passage. Zero-shot, packed_all scores 0.715 nDCG@10 on SciFact, close to 0.729 from 20 separate decision calls to the same 9B model, at 614 ms versus 2210 ms per query; scaling to 200 candidates (about 43k tokens) the trained-long adapter reaches 0.717 against 0.659 for the cross-encoder on the same 100 queries.
Perspective
The demonstrated scope is two BEIR retrieval datasets (300 SciFact test queries and 323 NFCorpus test queries), the first 300 HotpotQA distractor validation questions, and one hybrid 9B backbone family; the authors frame this as a proof of concept and name as next questions how the benefits extend across larger benchmarks and backbone families.
Evaluation sets are small, covering two BEIR datasets and one QA dataset on a single hybrid 9B backbone family; BEIR differences are not significance-tested and the per-passage baseline was not re-run with the trained adapter. The zero-shot adapter degrades beyond 50 candidates and runs out of memory in a single call at 200 candidates, and memory currently grows with context length; the activated-adapter and cache-reuse results are single-seed on 100 questions, the bitwise cache check used a 0.8B model, and with cache reuse the generator sees all passages rather than the reranked subset. Latencies come from different, partly shared GPUs, and a dedicated cross-encoder remains faster per query.