LLM explanations stop being written per question: one executable program reproduces model outputs and runs 6 to 100 times shorter than chain-of-thought

The way a large model's behavior is explained shifts from generating free text per instance to generating one program that runs over the whole dataset, so agreement between program outputs and model outputs can be computed automatically. Chain-of-thought and post-hoc rationales are tied to individual inputs, so judging whether an explanation is accurate requires separately specifying input perturbations or an input distribution that the explanation method itself does not provide. Across three open-source code-capable models and eight BBH tasks, 6 of 24 model-task settings exceed 0.75 consistency, showing that high consistency is reachable in some settings but is not the general outcome.

The procedure shows the model several input-output examples and asks it to write a Python function that reproduces its own outputs, then samples multiple candidate programs and selects by consistency on the training set. Consistency is the fraction of inputs on which the program produces the same output as the model, with a normalized version relative to an uninformative baseline that always answers the most common output, so settings with different output spaces can be compared. Selected programs are evaluated on a held-out test set and results are averaged over three random seeds; against per-instance chain-of-thought, code explanations are 6 to 100 times shorter on all tasks and achieve higher sufficiency in 9 of 24 settings.

The evaluation is limited to Python code explanations and fixed generation hyperparameters, with tasks drawn from eight BBH tasks and models Gemma3-27B, Qwen2.5-Coder-32B, Llama3.1-70B and Claude Opus 5.5; consistency measures behavioral reproduction and does not establish whether a program faithfully describes the model's internal computation.

On complex linguistic tasks, generated programs reproduce model outputs no more often than an uninformative baseline and rely more on longer hard-coded linguistic patterns; multi-turn refinement and giving Claude Opus 5.5 code execution plus consistency feedback improve consistency, but it remains substantially lower on complex tasks. Low consistency has two possible interpretations: current models cannot write code that accurately captures their own behavior, or some linguistic behaviors are inherently hard to represent with compact, human-interpretable programs, and the two are not yet distinguished.