LLMs quietly flip constraints when they write optimization models: post-hoc repair lifts zero-shot accuracy from 44.8% to 64.9% without retraining the generator
Lead
OPT-Kraken treats an LLM's optimization model and solver code as frozen artifacts and repairs them after the fact, lifting zero-shot accuracy from 44.8% to 64.9% across six benchmarks while making only 0.6% of initially correct outputs incorrect.
Story
When a large language model translates a natural-language operations research problem into a mathematical model and solver code, the artifacts often disagree with the original problem; the work calls these optimization modeling hallucinations, covering unsupported additions, omitted requirements, incorrect formulations, model-code inconsistencies, data-binding faults, and result-extraction errors. Earlier general hallucination methods check whether generated statements are supported, or rely on whether code executes successfully; but supported statements do not establish that every problem requirement is represented, and running code does not rule out an error in the mathematical model itself. On six benchmarks, MAMO Complex, IndustryOR, ComplexOR, OptMATH, NLP4LP, and BWOR, paired before-and-after comparisons of frozen artifacts from four generation pipelines and four fine-tuned generators raise zero-shot accuracy from 44.8% to 64.9%, with 36.9% Rescue, 0.6% Regression, and 94.1% Error Coverage.
Repair does not regenerate a model and code; it extracts requirements and instance data from the original problem, localizes errors, picks the minimum sufficient repair granularity for the affected scope, constrains local edits with contracts, and commits only after artifact-level auditing. Generative methods treat modeling errors as failures to avoid during generation or correct them inside a refinement workflow; here repair is an independent post-hoc transformation of frozen artifacts, and an unnecessary edit to an initially correct model-code pair is itself a failure. Blind paired manual review of 339 selected DeepSeek V4 Flash zero-shot artifacts shows source-level fidelity rising from 2.7% to 51.8% across 330 adjudicable pairs, with 50.8% Artifact Rescue; on 140 frozen artifacts, evidence-constrained local repair rescues 11/40 versus 1/40 for SAC-Opt semantic-anchor correction, with 0.0% Regression for the former.
What to watch
The repair targets natural-language optimization problems together with the mathematical model and solver code produced by an upstream generator, and runs as a plug-and-play post-processor without retraining or changing that generator. Evaluation covers six benchmarks, MAMO Complex, IndustryOR, ComplexOR, OptMATH, NLP4LP, and BWOR, across four generation pipelines, four fine-tuned generators, and three repair backbones, with Gurobi as the solver. Detection reads only the original problem and instance data, excluding upstream solve status, error messages, objective values, and reference answers; execution checks follow candidate generation, and references are used only for offline evaluation.
Accuracy is scored by objective value or required solve status, which is not equivalent to complete fidelity of the modeling artifact to the original problem: among 359 numerically correct DeepSeek V4 Flash zero-shot results, manual review found 4 artifacts (1.1%) with modeling hallucinations that outcome-level evaluation did not expose. Error Coverage measures coverage of potential repair opportunities, and Regression is not a detector false-positive rate; neither establishes calibrated binary hallucination detection. Minimum sufficient scope is relative to supported routes and evidenced dependencies, not a certificate of global edit minimality or complete semantic dependency discovery; when evidence is insufficient the framework returns Unresolved and retains the current artifacts rather than enlarging the repair scope.
