# byteification

Byte-level language models group bytes into patches with a learned boundary predictor, and what that predictor can see at limited lookahead depends on the text encoding. In UTF-8 many scripts take three bytes per codepoint, and the first byte carries only 4 high codepoint bits, so after the same left context a mark extending a grapheme cluster and a letter starting the next can share their first byte. Training small causal predictors on matched FineWeb-2 text and evaluating on 17 FLORES-200 languages, balanced grapheme-boundary error at one-symbol lookahead is 8.76% with UTF-8 and 0.06% with UBE; the same-length UBE-BE variant gives 8.62%, so the gain follows byte order rather than stream length.

Byte-level language models group bytes into patches with a learned boundary predictor, and what that predictor can see at limited lookahead depends on the text encoding. In UTF-8 many scripts take three bytes per codepoint, and the first byte carries only 4 high codepoint bits, so after the same left context a mark extending a grapheme cluster and a letter starting the next can share their first byte. Training small causal predictors on matched FineWeb-2 text and evaluating on 17 FLORES-200 languages, balanced grapheme-boundary error at one-symbol lookahead is 8.76% with UTF-8 and 0.06% with UBE; the same-length UBE-BE variant gives 8.62%, so the gain follows byte order rather than stream length.