ANA GVRITISHVILI

Using 903 open-ended responses across six variables from a European PhD student survey, this study had five human coders and GPT-5.4 each perform the same inductive content analysis procedure to produce codes and themes, and measured agreement with the Adjusted Rand Index (ARI), finding average human-LLM agreement of 0.61 for coding and 0.54 for themes, close to within-human consistency (0.68) and within-LLM consistency (0.76), with wide variation across variables and low within-entity consistency consistently accompanying low between-entity agreement.

Using 903 open-ended responses across six variables from a European PhD student survey, this study had five human coders and GPT-5.4 each perform the same inductive content analysis procedure to produce codes and themes, and measured agreement with the Adjusted Rand Index (ARI), finding average human-LLM agreement of 0.61 for coding and 0.54 for themes, close to within-human consistency (0.68) and within-LLM consistency (0.76), with wide variation across variables and low within-entity consistency consistently accompanying low between-entity agreement.