AI-RADS let 5 radiologists grade 350 AI outputs, reaching interreader agreement of α=0.87 for image tasks and α=0.93 for generative tasks
The study developed and multireader-evaluated AI-RADS, a structured framework for case-level assessment of radiology AI output reliability, clinical utility, and recommended actions, in which 5 board-certified radiologists independently evaluated 350 cases processed by 7 representative AI applications, assigning each case one of 5 AI-RADS categories, applicable modifiers, and an independent correctness rating as a reference; substantial interreader agreement was observed for core categories in image-based tasks (Krippendorff's α=0.87; 95% CI: 0.83-0.91) and generative AI tasks (α=0.93; 95% CI: 0.91-0.