One Space Decides a Multiple-Choice Score: Correct-Answer Likelihood Falls by Orders of Magnitude, Accuracy Usually Moves Less Than Half a Point
Synopsis
In multiple-choice evaluation, whether the space after the prompt's colon belongs to the prompt or to the option changes the probability the model assigns to the correct answer. Keeping the space with the option matches the tokenization seen in pre-training; keeping it with the prompt lowers the correct answer's log-probability by several orders of magnitude. Accuracy changes by less than half a point in six of nine model-benchmark pairs, but Qwen3.5 loses almost ten points on PIQA.
Interpretation
A multiple-choice score depends on which side of the prompt-answer boundary a single space is counted on, not only on whether the model knows the answer. The common convention ends the prompt with Answer: and reads next-token probabilities for labels A through D, so the space must belong to one side, and this detail had been treated as an irrelevant formatting choice. Across nine combinations of MMLU, ARC and PIQA with Olmo-3-7B, Qwen3.5-35B-A3B and Gemma-4-26B-A4B, Space-in-Answer scores at least as high in eight pairs, and the gap is below 0.5 percentage points in six.
With the space in the prompt, the correct answer's probability drops sharply, but the drop falls roughly evenly on all candidates, so rankings mostly survive. Correct-answer log-probabilities differ clearly between the two splits and consistently across models and benchmarks, with the Space-in-Prompt histograms shifted lower and spread wider. The exceptions are Qwen3.5 on MMLU at 2.6 points and on PIQA at 9.8 points, and Olmo 3 on MMLU at 1.5 points.
The root is tokenization: in pre-training text, Answer, the colon and the space-prefixed label form one token, so a prompt that hands the space to the prompt asks the model to predict a bare label it has almost never seen. All six modern tokenizers split Answer:␣B into Answer, : and ␣B, and on a one-million-document DCLM-Baseline sample this canonical form appears about 2,300 times per million tokens, while a standalone space followed by a bare B appears about 0.03 to 0.10 times per million. The two splits are equivalent only if tokenizing prompt and option separately and concatenating equals tokenizing the combined string, and every tokenizer examined breaks this condition at the boundary.
Perspective
The scope is likelihood-based scoring, reading next-token probabilities for option labels rather than generating text, with experiments on MMLU, ARC and PIQA and three base models. Quantities built directly on these likelihoods, such as bits-per-byte of the correct answer, are likewise affected by the split choice. The recommendation is to decide the split before inference by testing which one yields the canonical tokenization, using nothing but the tokenizer.
Accuracy changes are small in most pairs but not all, and Qwen3.5's nearly ten-point PIQA loss comes from label preference rather than a uniform drop. Models favor some bare labels over others, such as B for Qwen3.5 and D for Gemma 4, and this changes accuracy mainly when the model is less confident. Judged by another model, 90.25% of completions were semantically correct, about 1.4 points above Olmo 3's Space-in-Prompt ARC accuracy, so the missing probability mass did not move to a semantically different answer.