Fact-checking articles can now be drafted automatically: QRAFT chains evidence, drafting and simulated editing, but experts still want a human in the loop
Lead
QRAFT splits fact-checking article writing into evidence gathering, drafting and a simulated editorial review run by multiple LLM agents, beating prior text-generation methods on automatic metrics across 1,227 examples while still scoring well below expert-written articles with professional fact-checkers.
Story
Automatic fact-checking gains a new step: beyond judging whether a claim is true, the system now produces a full article for the public. Earlier pipelines stopped at brief explanations that justify the verdict but leave out the claim's origins, context and counter-arguments. The task definition comes from interviews with experts at four leading fact-checking organizations, who listed accurate clarification of the claim, origin details, transparent sourcing and a clear structure.
QRAFT assigns three LLM agents distinct roles: a Planner extracts evidence nuggets and proposes an outline, a Writer expands it into a first draft, and an Editor questions the Writer to produce a list of edits for iterative refinement. Comparisons include naive sentence concatenation, a summarization model fine-tuned on evidence-to-article pairs, JustiLM, and the LLM-based Vanilla-GPT and Storm. On 1,227 examples merged from ExClaimCheck and AmbiguousSnopes, QRAFT leads on 6 of 7 automatic measures, improves FactScore by 11 points over the next-best method, and cites more than 90% of available evidence sources.
Leading on automatic metrics does not mean the articles are usable: professional fact-checkers rated the generated articles in the 3-to-4 range. Expert-written articles scored highest on relevance, comprehensibility, evidence presentation and publishability, with QRAFT usually second and Storm lowest. In a blind evaluation of 12 climate-change claims, experts said QRAFT often presented data without building the argument for why the claim does or does not hold.
What to watch
The work targets professional fact-checkers and takes the claim, its veracity and a set of evidence documents as given, so automatic evidence retrieval and claim verification are out of scope. Evaluation covers 987 examples from ExClaimCheck and 240 from AmbiguousSnopes, 1,227 in total with 22 veracity labels, plus a blind expert evaluation of 12 climate-change claims. The authors propose compiling more detailed article requirements, developing a more robust evaluation suite, and benchmarking stronger reasoning models.
The authors note that evidence-nugget extraction relies on the underlying LLMs to preserve context, which may drop indirectly related information for complex claims; experiments used only GPT-4o and GPT-4o-mini, without a systematic comparison of other model combinations; experts also found the generated articles overly elaborate and repetitive, and sometimes presented context-specific information as general fact.