Fluent definitions are not ontology-ready: an agentic QC loop cuts DO 2024 entity errors from 67.0% to 37.8%
Synopsis
Biomedical ontologies need new textual definitions continuously, and a large language model can write definitions that read fluently and resemble curated text while still naming entities absent from the ontology or violating its logical constraints. A quality-control agentic workflow turns definition generation into an iterative process: draft a definition, extract predicate-object axioms, check whether entities resolve in the target ontology, and test logical consistency with a reasoner, then rewrite using the diagnostics. On Disease Ontology DO 2024, the Qwen2.5-14B base model's entity error rate fell from 67.0% to 37.8% after the existence-repair loop, while text-similarity scores stayed roughly level.
Interpretation
Ontology definition generation is treated as a knowledge-compatibility problem rather than only a text-generation problem: for a definition to enter an ontology library, its entities must map to ontology identifiers, its implied relations must resolve, and the resulting axioms must be compatible with existing constraints. Prior evaluation mainly measured lexical or semantic closeness to a reference definition, using ROUGE, BLEU, BERTScore and sentence-embedding cosine similarity, which do not show whether the content can be grounded in the target knowledge system. On Disease Ontology DO 2024, the Qwen2.5-14B base model reached Token F1 0.283 and cosine similarity 0.664 under direct generation, yet its entity error rate was 67.0%, showing that text scores and ontology compatibility can diverge sharply.
The workflow splits definition generation into generation, axiom extraction, entity existence checking, reasoning-based validation and targeted repair, with each step's tool output feeding the next. Direct decoding produces one piece of text and offers no chance to catch ungrounded entities or conflicting axioms before merging into the ontology. Across the DO 2024, DO 2025 and Uberon-Constraint-Stress test settings, the Qwen2.5-14B base model's entity error rate fell from 67.0% to 37.8%, from 55.1% to 31.8%, and from 52.3% to 2.5%, respectively, after the existence loop.
A separate logical inconsistency rate counts candidates whose entities are grounded but which make a class unsatisfiable after merging, exposing ontology-level failures that text similarity cannot see. Passing the entity check does not mean the axioms are compatible with disjointness, partonomy or domain/range constraints, and such failures are sparse and offer little local diagnostic signal. On Uberon-Constraint-Stress, Qwen2.5-14B with LoRA had a logical inconsistency rate of 2.95% (28/950) with only the existence loop and 0.42% (4/950) with the logic loop added, a smaller reduction than seen for entity errors.
Perspective
Evaluation covers the 2024 and 2025 incremental releases of the Disease Ontology DO, the 2024 and 2025 incremental releases of Uberon, and Uberon-Constraint-Stress, a 973-term set drawn from the Uberon 2023 release that involves strong constraints; reasoning uses ELK, and entity resolution uses a local ontology index with optional OLS4 lookup.
The logic loop reduces logical inconsistency less and less consistently than the existence loop reduces entity errors, and LoRA models retain non-zero entity error rates after repair in some settings, for example 5.2%, 8.9% and 6.1% for Llama-3.1-8B on Uberon 2024, Uberon 2025 and Uberon-Constraint-Stress; only 26 of 11,454 active terms in the DO 2023 release involve strong constraint axioms such as DisjointFrom, so DO offers limited power to test logical inconsistency.