Fixing an LLM's weak spot is not enough: judging its knowledge state first lifts two benchmarks by 18.41 points on average
Synopsis
Knowledge improvement for large language models is moving from predefined incorporation pipelines to evaluation-guided updating, yet locating a weak spot does not say which intervention fits. Recursive Knowledge Improvement (RKI) organizes knowledge into graph-structured units and, each round, localizes weaknesses, estimates the model's operational knowledge state from diagnostic evidence and intervention history, selects an intervention, and re-evaluates the updated model. On HotpotEval and SafeInternalEval, RKI outperforms existing knowledge incorporation methods by 18.41 points on average, and analyses indicate knowledge states add guidance beyond weakness localization and can change across updates while the observed weakness persists.
Interpretation
Knowledge improvement for large language models now uses evaluation as the basis for each round of updating, rather than following a predefined knowledge incorporation pipeline. Earlier methods used evaluation to localize model-specific knowledge weaknesses and then guided targeted post-training from that localization, so the diagnosis decided where to change the model. The same localized weakness can be compatible with distinct knowledge states, where the relevant knowledge is unavailable, available but hard to access reliably, or accessible but ineffective downstream; this ambiguity is called knowledge-state aliasing.
RKI formulates knowledge improvement as a sequential, state-aware intervention process that re-decides what to do at every round. It organizes knowledge into graph-structured units so that diagnostic evaluation and targeted intervention share one basis across rounds instead of being handled separately. Each round localizes weaknesses, estimates the model's operational knowledge state from diagnostic evidence and intervention history, selects a corresponding intervention, and re-evaluates the updated model before the next choice; on HotpotEval and SafeInternalEval, RKI outperforms existing knowledge incorporation methods by 18.41 points on average.
Model updates can change the underlying knowledge state before the observed weakness disappears. An intervention appropriate in one round may no longer address the bottleneck in the next, so greater diagnostic resolution alone is insufficient for intervention selection. Analyses show knowledge states provide additional guidance for intervention selection beyond weakness localization and can change across model updates even when the observed weakness persists.
Perspective
The demonstrated scope is the knowledge-improvement comparison on HotpotEval and SafeInternalEval plus the analyses relating knowledge states to weakness localization; the summary proposes no follow-up experiment or deployment step.
The summary presents knowledge-state aliasing as a problem to be addressed and notes that knowledge states can change across model updates while the observed weakness persists; the scope of these claims is not further bounded at the summary level.