LLM explanations stop being written per question: one executable program reproduces model outputs and runs 6 to 100 times shorter than chain-of-thought
Researchers replace per-instance free-text explanations of large language models with a single executable program that runs over a dataset and is compared against the model's outputs, with consistency measuring how often it reproduces model behavior. Across three open-source code-capable models and eight tasks, 6 of 24 model-task settings exceed 0.75 consistency, code explanations are 6 to 100 times shorter than chain-of-thought, and they align better with model behavior in 9 of 24 settings; on tasks with more complex linguistic components, consistency is no higher than an uninformative baseline.