M3: Conversational LLMs simplify secure clinical data access, understanding, and analysis

M3 provides a software framework for MIMIC-IV that, with a single command, retrieves the data from PhysioNet, launches a local SQLite instance or connects to hosted BigQuery, and lets researchers pose questions in plain English. Accessing MIMIC-IV previously required SQL proficiency and clinical domain knowledge, or reliance on visual query builders and curated SQL templates; M3 integrates natural-language-to-SQL translation with database execution in one deployable system. The paper grounds this in a described system architecture (data access layer, security middleware, FastMCP-based MCP client) and a dual-backend design, with an open code repository; no end-to-end deployment performance benchmark is reported.

M3 integrates an LLM tool layer with MIMIC-IV through the Model Context Protocol, providing standardized tool exposure, execution logging, and access-control boundaries, which the authors frame as an engineering integration of existing technology rather than a novel protocol contribution. The authors state that, to their knowledge, none of the prior solutions is currently integrated in a desktop generative AI application such as Claude Desktop; M3 achieves an auditable integration through two tiers of tools (core database tools and domain-specific clinical tools). Evidence is architectural and design-level, including OAuth 2.0 with JWT token authentication, sqlparse-based read-only query validation, output-size limits, and per-user rate limiting; the authors explicitly describe these as design features and list formal security evaluation (adversarial SQL-injection testing, penetration testing, authentication audits) as future work.

On 100 answerable EHRSQL 2024 questions, Claude Sonnet 4 answered 94 correctly (94.0%; Wilson 95% CI [87.5%, 97.2%]) and gpt-oss-20B answered 93 correctly (93.0%; Wilson 95% CI [86.3%, 96.6%]), with a paired McNemar test on 7 discordant pairs yielding p=1.00. The result provides empirical evidence for the feasibility of natural-language-to-SQL translation in a clinical context and shows that an open-weights model runnable locally on a MacBook M1 Max with 32GB RAM performs comparably to a proprietary model. The sample is 100 questions randomly drawn from the EHRSQL 2024 MIMIC-IV test set (1,167 questions, of which 934 are answerable and 233 unanswerable); correctness was adjudicated by four evaluators (two technical, two clinical), with each question reviewed independently by at least two evaluators including at least one technical and one clinical expert, and disagreements resolved by consensus; the authors also note that pretraining exposure to benchmark artifacts cannot be fully excluded.

On 100 unanswerable questions, gpt-oss-20B correctly abstained on 69 (69.0%; Wilson 95% CI [59.4%, 77.2%]), with the remaining failures split into 12 SQL-backed answers that misrepresent the data and 15 general-knowledge answers that do not abstain. This evaluation reports reliability (abstaining on unanswerable questions) as a separate axis and identifies the dominant correctable failure mode as the model committing to a natural-language-to-schema mapping that does not semantically match. Evaluated only under the gpt-oss-20B condition, justified by the deterministic reproducibility of open frozen weights; outcomes were adjudicated per question by the evaluator panel into three categories, with all conversation transcripts and per-question classifications released.

The work targets researchers who want to query MIMIC-IV but lack SQL or schema-level familiarity, particularly teams constrained by data privacy, regulatory requirements, or limited connectivity that need local deployment; the empirical results are scoped to the 100-patient MIMIC-IV demo subset (SQLite backend) and EHRSQL 2024 benchmark questions, while the full MIMIC-IV v3.1 BigQuery backend is architecturally supported but not empirically benchmarked. By design the system exposes generated SQL and tool-call traces, making expert review possible, so its intended setting is exploratory data analysis paired with human oversight rather than treating results as authoritative in isolation. The authors also recommend phased deployment, beginning with supervised use in educational settings alongside training and governance frameworks.

Several open questions remain for a careful reader: whether scale-dependent differences in query complexity, schema coverage, and runtime behavior appear on the full MIMIC-IV v3.1 dataset; how the interface performs in the everyday analytical workflows of practicing clinical researchers and how users interpret generated queries and results; and whether adding an explicit ambiguity-detection or clarifying-question step reduces reliability failures when questions admit multiple reasonable interpretations. The authors also note that some temporal-reasoning failures may reflect prompt sensitivity to the benchmark's fixed 'current time' instruction, and that pretraining exposure to public benchmark artifacts cannot be fully excluded; the security controls (OAuth2/JWT, read-only validation, rate limiting) are currently design features whose effectiveness under adversarial conditions has not been formally evaluated.