Decision Support in Publicly Available Patient Information Policies at U.S. Osteopathic Medical Schools: A Vignette-Based Document Analysis

The study designs and applies a vignette-level document analysis that separates whether relevant public guidance can be retrieved from whether it is sufficient to support a concrete course of action. Rather than asking whether policies exist or mention a topic, it moves the unit of assessment to the school-vignette pair with a four-level classification and three repeated runs. Exploratory, geographically diverse nonprobability sample of 20 schools, eight vignettes, 160 pairs, three AI-assisted retrieval-and-evaluation runs per pair, descriptive analysis.

Across the 160 pairs, final classifications were Explicitly Supported for 18 (11.3%), Inferable for 3 (1.9%), Ambiguous for 132 (82.5%), and Not Addressed for 7 (4.4%), giving 21 pairs (13.1%) with sufficient support after grouping. It reports the distribution of support strength for specific scenarios rather than a qualitative description of policy content. Descriptive counts; retrieval succeeded for 159 of 160 pairs (99.4%), indicating the low sufficiency rate is not explained by retrieval failure.

Sufficiency varied by scenario: generative AI-assisted reflective writing was most often judged sufficient (7/20 schools, 35%), while personal cloud notes and official clinical logs had no sufficiently supported pair at any school. It locates policy gaps in specific use scenarios instead of stating generally that policies are incomplete. Scenario-level counts; ten schools had no sufficiently supported vignette and the maximum was four of eight.

The evaluation process itself was quantified: three-run ratings were unanimous for 113 pairs (70.6%) with 80.2% pairwise exact agreement, multiple documents contributed to 111 evaluations (69.4%), and investigator review retained 44 of 47 selected discordant ratings while revising three upward. Reporting AI-assisted evaluation consistency alongside human review lets readers judge the reproducibility of the classifications. Three repeated runs plus single-investigator review; the two-school model-investigator comparison was exploratory, with the investigator more often judging evidence sufficient than the models did.

The study applies to publicly available patient-information policies at U.S. osteopathic medical schools, with the school-vignette pair as the unit of assessment, and its conclusions are limited to classifications within this evaluation framework. It can be used directly by institutions to identify which scenarios need clarification and as a starting point for reusing the vignette-based review process in other institution types or policy domains; the authors note that evaluation with learners and assessment of internal and clinical-site guidance would help determine how these findings translate into students' decisions.

About three in ten pairs were not unanimous across the three runs, and discordant ratings were reviewed by a single investigator, so different reviewers might reach different results; the two-school model-investigator comparison was exploratory and its directional difference needs confirmation in larger samples. In addition, the available text here is the abstract and declarations, without the main-text tables, full vignettes, or supplementary details, so the specific basis for each scenario's classification and the coding rules remain to be confirmed against the original.