# Risk-of-bias assessment

Risk-of-bias appraisal of blood-borne virus prediction models has depended on reviewers working through PROBAST+AI signalling questions by hand, which is slow and varies between assessors. ARBITER grounds each judgement in retrieved PROBAST+AI criteria, statistical methods and disease-specific knowledge, and separates model-development from model-evaluation pathways. Across 200 signalling-question observations from nine HIV, HBV and HCV prediction-model studies, ARBITER's weighted kappa against expert consensus was 0.701, versus 0.724 for a manual reviewer and 0.714 for Claude Sonnet 4.6, with overlapping confidence intervals.

Risk-of-bias appraisal of blood-borne virus prediction models has depended on reviewers working through PROBAST+AI signalling questions by hand, which is slow and varies between assessors. ARBITER grounds each judgement in retrieved PROBAST+AI criteria, statistical methods and disease-specific knowledge, and separates model-development from model-evaluation pathways. Across 200 signalling-question observations from nine HIV, HBV and HCV prediction-model studies, ARBITER's weighted kappa against expert consensus was 0.701, versus 0.724 for a manual reviewer and 0.714 for Claude Sonnet 4.6, with overlapping confidence intervals.

This study designed a zero-shot prompted LLM pipeline as a virtual mimic of a human reviewer for the QUIPS framework and applied it to 298 articles from previously published systematic reviews of prognosis studies in neurology across three domains (epilepsy, traumatic brain injury, stroke), finding limited LLM-human agreement (weighted kappa = 0.22, 95% CI 0.12-0.33) while tentatively suggesting, based on a small sample (n=5), that it may not be inferior to human-human agreement (weighted kappa = -0.25, 95% CI -1.04-0.54), with Wilcoxon signed-rank tests statistically significant (p < 0.05) across four bias domains and overall risk scores, and rank-biserial correlations demonstrating human raters' tendency to assign higher risk scores than LLM counterparts.

This study designed a zero-shot prompted LLM pipeline as a virtual mimic of a human reviewer for the QUIPS framework and applied it to 298 articles from previously published systematic reviews of prognosis studies in neurology across three domains (epilepsy, traumatic brain injury, stroke), finding limited LLM-human agreement (weighted kappa = 0.22, 95% CI 0.12-0.33) while tentatively suggesting, based on a small sample (n=5), that it may not be inferior to human-human agreement (weighted kappa = -0.25, 95% CI -1.04-0.54), with Wilcoxon signed-rank tests statistically significant (p < 0.05) across four bias domains and overall risk scores, and rank-biserial correlations demonstrating human raters' tendency to assign higher risk scores than LLM counterparts.

This study evaluated three large language models (GPT-5, OpenAI o3-mini, and GPT-3.5) on risk-of-bias (RoB 2) assessment across 97 randomized controlled trials, retrieving relevant passages from trial reports with Okapi BM25 and then assigning each generated claim an evidence verdict of supported, contradicted, not found, or out of scope with verbatim quotations; binary accuracy of AI-generated judgments was high (90%-98%), yet evidence support rates were only 60%-65% with conservative hallucination rates of 34%-37%, showing that high decision-level accuracy does not guarantee documentary support.

This study evaluated three large language models (GPT-5, OpenAI o3-mini, and GPT-3.5) on risk-of-bias (RoB 2) assessment across 97 randomized controlled trials, retrieving relevant passages from trial reports with Okapi BM25 and then assigning each generated claim an evidence verdict of supported, contradicted, not found, or out of scope with verbatim quotations; binary accuracy of AI-generated judgments was high (90%-98%), yet evidence support rates were only 60%-65% with conservative hallucination rates of 34%-37%, showing that high decision-level accuracy does not guarantee documentary support.