Pre-training scores stop tracking answer length: PriorBPB integrates a per-byte cost curve under one shared length prior, ranking tasks closer to accuracy

The bits-per-byte score used to judge whether a pre-training change helped varies systematically with how long the gold answer is, not only with how fluently the model predicts it. That score had been treated as a fine-grained signal largely independent of answer length, used to choose data and architectures across many small experiments. On three semantically identical answers to "What is the capital of France?", Qwen3-4B's BPB falls from 1.85 for "It is Paris." to 0.77 for "It is Paris, the capital of France.", and OLMo-3-7B from 1.38 to 1.22; within MMLU full text split into five gold-length groups, the shortest group (6 bytes on average) scores about 11 times worse in mean-of-ratios BPB than the longest (96 bytes on average).

Writing each item's bits as a fixed overhead plus a per-byte rate makes corpus BPB exactly the fixed overhead divided by the task's mean gold length plus the per-byte rate, so short-answer tasks score worse through the denominator rather than through difficulty. The standard practice divides total bits by total bytes without separating the part that does not scale with length from the part that does. Across 19 tasks and multiple models, the fixed-cost term correlates strongly with mean gold length while the per-byte rate barely does; within a single task split into gold-length quintiles, short items score worse on every task tested, with a median top-to-bottom quintile ratio of about 0.43.

PriorBPB averages a per-byte marginal cost curve, estimated from per-token surprisal, against one length prior shared by every task, removing the task's own length distribution from the weights while keeping the byte denominator. Setting those weights to a task's own length distribution makes PriorBPB reduce exactly to corpus BPB, so the difference is only whether the weights come from the data or are fixed in advance. The prior is geometric with median length 1024 bytes, set from the length distribution alone before any score is computed; on 19 length-varying tasks, PriorBPB's per-byte cost curve compared at matched content-byte positions between long- and short-gold quintiles has a median top-to-bottom ratio of 0.98 with a 95% credible interval of [0.94, 1.01].

PriorBPB is more usable than standard BPB both for ordering tasks against each other and for tracking accuracy, making it a better proxy for screening pre-training recipes. Corpus and mean-of-ratios BPB correlate strongly with mean gold length, and mean-of-ratios BPB additionally inflates reported scores. Across 62 base checkpoints the mean absolute correlation with mean gold length drops from 0.68 to 0.18; in a paired, clustered Bayesian comparison against mean-of-ratios BPB, the accuracy-correlation advantage is +0.021 on base checkpoints (95% credible interval [+0.004, +0.079]), +0.122 on instruct checkpoints ([+0.005, +0.571]), and +0.047 over all 81 checkpoints ([+0.011, +0.163]).

The report is aimed at model builders and evaluation engineers who compare pre-training checkpoints across tasks, in settings where gold answer length varies both between tasks and within them, such as multiple-choice and question-answering evaluations. PriorBPB keeps the byte denominator and therefore inherits BPB's tokeniser independence; the prior is geometric with median length 1024 bytes, set from the length distribution alone before any score is computed. The authors recommend using PriorBPB with a frozen, shared length prior and stating that prior when a single cross-task scalar is needed, and not naively averaging mean-of-ratios or corpus BPB across tasks.

PriorBPB's per-byte cost curve at position k averages only over items that reach byte k, so its late-position values describe a length-selected sub-population, and this survivorship bias may contribute to the residual length correlation; dropping short items outright does not remove it. The residual grows in magnitude with the prior mean, from about −0.10 at 256 bytes to about −0.30 at 4096 bytes, while cross-task score ordering agrees with that at the frozen 1024 bytes (Spearman ρ = 0.99), so the authors report the 1024-byte setting. The authors also suggest that the growing fixed cost across the Qwen3 instruct series, which appears as an inverse-scaling trend, may come from prompt format rather than model quality; re-scoring without a chat template attenuates but does not remove it, and a template-only control did not remove the anomaly, so format mismatch remains only a possible hypothesis.