Byte order, not just byte count, sets boundary prediction: UBE cuts one-symbol balanced grapheme error from 8.76% to 0.06%
Synopsis
Byte-level language models group bytes into patches with a learned boundary predictor, and what that predictor can see at limited lookahead depends on the text encoding. In UTF-8 many scripts take three bytes per codepoint, and the first byte carries only 4 high codepoint bits, so after the same left context a mark extending a grapheme cluster and a letter starting the next can share their first byte. Training small causal predictors on matched FineWeb-2 text and evaluating on 17 FLORES-200 languages, balanced grapheme-boundary error at one-symbol lookahead is 8.76% with UTF-8 and 0.06% with UBE; the same-length UBE-BE variant gives 8.62%, so the gain follows byte order rather than stream length.
Interpretation
Whether a byte-level boundary predictor can tell “extend the current grapheme cluster” from “start the next one” at limited lookahead depends on which codepoint bits the encoding places in the first symbol. Under UTF-8 the first byte of a three-byte codepoint holds only 4 high codepoint bits, so after identical left context a cluster-extending mark and a letter starting a new cluster can share that byte, leaving the required decision indistinguishable. On a 17-language panel spanning 13 Brahmic scripts, balanced grapheme-boundary error at one-symbol lookahead is 8.76% for UTF-8 and 0.06% for UBE; UBE-BE, which keeps UBE's stream length, gives 8.62%, close to UTF-8.
UBE keeps one- and two-byte UTF-8 encodings and writes longer codepoints as two UTF-16LE bytes, low byte first, so its first symbol carries 8 low codepoint bits. Within a known 256-codepoint page that low byte identifies the codepoint, making the mark-versus-letter distinction visible at one-symbol lookahead; codepoints on different pages can still share a low byte. On 17,204 constructed pairs with identical left context whose next codepoints lie on the same page but require opposite decisions, the trained UBE predictors answer every pair correctly in each seed, while UTF-8 and UBE-BE cannot because their first symbols coincide.
On the task of retaining a parent tokenizer's subword cuts, UBE also lowers error, and the gain concentrates at one-symbol lookahead. Every evaluated Korean and Chinese cluster is a single codepoint, so grapheme boundaries are visible under all three encodings and the difference falls on selecting subword boundaries. At one-symbol lookahead UBE lowers errors per 1,000 scalars from 54.44 to 14.75 in Korean (72.91% reduction) and from 159.95 to 103.50 in Chinese (35.29%); eight more scripts fall by 67.32%–99.55%, and in the seven whose main Unicode block fits on one page UBE-BE stays within 3.0% of UTF-8.
Perspective
The demonstrated scope is small causal predictors trained on matched FineWeb-2 text and evaluated on the FLORES-200 devtest 17-language panel, plus retained-parent extensions in Korean, Chinese and eight more scripts; the released-model analysis covers two parent families and one family of conversion recipes.
The one-symbol guarantee covers only continuations within the same Unicode page, and cross-page collisions remain, for example U+200D and U+090D sharing low byte 0D; text exposure is matched across encodings but compute is not, and UBE-BE also changes the entire encoded history; the released checkpoints differ in architecture, size, data and training history, so their comparison does not isolate the effect of byte-level modeling.