Noah L. Schroeder

Through a pilot study and a main study, this work used LLMs including Claude 2.1, ChatPDF, GPT-4, Gemini 1.5 Flash, Gemini 1.5 Pro, and Mistral Large 2 to extract explicit and derived variables from 112 studies in a published education review and compared them with human coding, finding higher agreement for explicitly stated data but markedly lower agreement and generally low Cohen's Kappa for categorizing data into predefined categories, and on that basis proposed and developed the open-source human-in-the-loop (HIL) data extraction tool AIDE that enforces per-item human validation.

Through a pilot study and a main study, this work used LLMs including Claude 2.1, ChatPDF, GPT-4, Gemini 1.5 Flash, Gemini 1.5 Pro, and Mistral Large 2 to extract explicit and derived variables from 112 studies in a published education review and compared them with human coding, finding higher agreement for explicitly stated data but markedly lower agreement and generally low Cohen's Kappa for categorizing data into predefined categories, and on that basis proposed and developed the open-source human-in-the-loop (HIL) data extraction tool AIDE that enforces per-item human validation.