Blood-borne virus prediction models get a retrieval-grounded bias check: ARBITER matches a human statistician and a general LLM against expert consensus

Risk-of-bias appraisal of blood-borne virus prediction models can now be produced question by question by a retrieval-augmented pipeline that attaches traceable evidence to each judgement. Such appraisal previously meant a reviewer reading a manuscript and judging each PROBAST+AI signalling question across four domains; in the authors' earlier systematic review, 64 of 71 studies (90%) were judged at high risk of bias, and the appraisal itself is time-consuming and varies by reviewer. Across 200 signalling-question observations from nine studies, ARBITER's weighted kappa against expert consensus was 0.701 (95% CI 0.650–0.754), the manual statistician reviewer's 0.724 (0.674–0.774) and Claude Sonnet 4.6's 0.714 (0.617–0.794), with overlapping intervals.

Each judgement is driven not by model memory but by a prompt assembled at run time, combining that question's criteria, retrieved statistical-method passages and the pathway-specific text of the study. General-purpose models given the guidance and source text tend to produce fluent assessments without methodological support, and earlier retrieval systems drew evidence only from the paper under assessment. Criteria files cover 34 signalling questions across the development and evaluation pathways plus six applicability assessments, and the methodological knowledge base combines the PROBAST and PROBAST+AI guidance, a statistics textbook and methods papers, retrieved semantically per question.

The three assessors fail in different places, so human review can catch the pipeline's slips and the pipeline can catch the reviewer's. Agreement coefficients alone do not show where or why assessors diverge. The 101 discrepant observations yielded 104 failure-type components: reasoning failure 39 (37.5%), retrieval failure 17 (16.3%), criterion ambiguity 17 (16.3%), genuine disagreement 15 (14.4%) and threshold miscalibration 15 (14.4%); ARBITER had the most reasoning failures (21), while the manual reviewer's retrieval failures concentrated on development SQ 2.2, missing brief pre-processing descriptions in the Methods.

Validation covers nine HIV, HBV and HCV prediction-model studies plus one paper used for iterative refinement; input is limited to title, abstract, methods and results, with tables and figures excluded. The authors propose next validating on more blood-borne virus prediction models and other clinical domains, and exploring multimodal retrieval over rendered pages to include tables and figures.

The authors describe this as an early proof-of-concept: refinement was done on a single paper and does not show how broadly such improvements apply; signalling-question ratings are nested within studies and, despite study-level clustered bootstrapping, precision remains limited; quality-tier assignments were reviewed by the first author without independent verification by all three assessors; and the statistical knowledge base leans on traditional methods, with sample-size adequacy judged by events per variable rather than the minimum sample-size criteria of Riley et al.