# eval-framework

Bits-per-byte (BPB), a common pre-training score, divides a model's surprisal on the gold answer by that answer's UTF-8 byte length, yet part of the cost does not grow with answer length, so short-answer tasks and short items are penalized. Aleph Alpha's team fits each item's bits as a fixed overhead plus a per-byte rate, shows corpus BPB equals that fixed overhead divided by the task's mean gold length plus the per-byte rate, and releases PriorBPB in the open-source eval-framework: it integrates a per-byte cost curve against one length prior shared across tasks. Across five model families and more than 60 checkpoints, PriorBPB is far less correlated with mean gold length and tracks accuracy better than standard BPB.

Bits-per-byte (BPB), a common pre-training score, divides a model's surprisal on the gold answer by that answer's UTF-8 byte length, yet part of the cost does not grow with answer length, so short-answer tasks and short items are penalized. Aleph Alpha's team fits each item's bits as a fixed overhead plus a per-byte rate, shows corpus BPB equals that fixed overhead divided by the task's mean gold length plus the per-byte rate, and releases PriorBPB in the open-source eval-framework: it integrates a per-byte cost curve against one length prior shared across tasks. Across five model families and more than 60 checkpoints, PriorBPB is far less correlated with mean gold length and tracks accuracy better than standard BPB.