# Inter-rater agreement

Risk-of-bias appraisal of blood-borne virus prediction models has depended on reviewers working through PROBAST+AI signalling questions by hand, which is slow and varies between assessors. ARBITER grounds each judgement in retrieved PROBAST+AI criteria, statistical methods and disease-specific knowledge, and separates model-development from model-evaluation pathways. Across 200 signalling-question observations from nine HIV, HBV and HCV prediction-model studies, ARBITER's weighted kappa against expert consensus was 0.701, versus 0.724 for a manual reviewer and 0.714 for Claude Sonnet 4.6, with overlapping confidence intervals.

Risk-of-bias appraisal of blood-borne virus prediction models has depended on reviewers working through PROBAST+AI signalling questions by hand, which is slow and varies between assessors. ARBITER grounds each judgement in retrieved PROBAST+AI criteria, statistical methods and disease-specific knowledge, and separates model-development from model-evaluation pathways. Across 200 signalling-question observations from nine HIV, HBV and HCV prediction-model studies, ARBITER's weighted kappa against expert consensus was 0.701, versus 0.724 for a manual reviewer and 0.714 for Claude Sonnet 4.6, with overlapping confidence intervals.

In this exploratory cross-sectional study, 68 investigator-developed, evidence-mapped English prompts, each containing a false, absolute, unsafe, or contested premise, were submitted once in independent single-turn conversations to ChatGPT (GPT-5), Gemini 2.5 Pro, Microsoft Copilot, DeepSeek-V3.2-Exp, and Doubao-Seed-1.6, yielding 340 complete responses that five clinicians independently scored, showing that 307 responses (90.3%) were classified safe and 33 (9.7%) unsafe, with safe-response rates from 85.3% to 94.1% and a non-significant omnibus safety comparison (P = 0.203) that was not interpreted as equivalence, while all six graded outcomes differed across models (all P < 0.

In this exploratory cross-sectional study, 68 investigator-developed, evidence-mapped English prompts, each containing a false, absolute, unsafe, or contested premise, were submitted once in independent single-turn conversations to ChatGPT (GPT-5), Gemini 2.5 Pro, Microsoft Copilot, DeepSeek-V3.2-Exp, and Doubao-Seed-1.6, yielding 340 complete responses that five clinicians independently scored, showing that 307 responses (90.3%) were classified safe and 33 (9.7%) unsafe, with safe-response rates from 85.3% to 94.1% and a non-significant omnibus safety comparison (P = 0.203) that was not interpreted as equivalence, while all six graded outcomes differed across models (all P < 0.