LLM-assisted chart review lifts skin-toxicity F1 from 0.77 to 0.88 and cuts reported review time by about a third

Non-dermatologist reviewers using an LLM-assisted workflow judged cirAEs in outpatient notes more accurately and more consistently than reviewers working manually. Earlier automated irAE detection was framed mainly as a fully automated task without linked supporting evidence, so an expert verifying a prediction still had to repeat the full chart review; manual review itself is time-consuming, variable across reviewers and hard to scale. On records from 160 patients treated with immune checkpoint inhibitors, with ground truth set by a board-certified oncodermatologist, the assisted group reached F1 0.88 versus 0.77 and Cohen's κ 0.82 versus 0.50, a κ difference of 0.32 with a 95% bootstrap CI of 0.25 to 0.39.

The assisted workflow hands retrieved note passages to two model roles with separate jobs: a primary agent synthesizes an initial call, and a critic agent checks its internal consistency, after which the reviewer sees both the binary recommendation and the note excerpts behind it. The pipeline uses retrieval-augmented generation, meaning it first pulls relevant text from the note store and then has the model answer from it, combining dense semantic search with sparse keyword search whose terms were chosen with domain-expert input to cover common descriptions and frequent negations. The reasoning core was chosen on a separate 100-patient development set that did not overlap the 160-patient evaluation cohort; there the GPT-4o agentic pipeline reached sensitivity 0.82 and specificity 0.88, above LLaMa-3.1 at 0.55 and 0.63, Gemma-2 at 0.68 and 0.64, and Qwen-2.5 at 0.23 and 0.94.

Reviewers self-reported average review time of 10 minutes per chart with assistance versus 15 minutes manually, about one-third shorter, and rated the model helpful or very helpful in 70% of cases. Review time came from a retrospective post-study survey rather than prospective timing, and the helpfulness ratings are subjective. Two pairs of non-dermatologist reviewers, each trained in cirAE adjudication and blinded to ground truth, were fixed to their assigned workflow with no crossover between groups.

The results come from retrospective records of 160 patients treated with immune checkpoint inhibitors across Massachusetts General Hospital, Brigham and Women's Hospital and Dana-Farber Cancer Institute, with ground truth set by one board-certified oncodermatologist, and the evaluation is limited to cirAE as the skin-toxicity endpoint.

Reviewer pairs were fixed to their workflow with no crossover, so reviewer-specific differences may have contributed to the observed differences in accuracy, agreement and review time; review time was retrospectively self-reported rather than prospectively timed and helpfulness ratings were subjective; the authors state that external validation is still required.