Functionalizer splits casing, diacritics and repetition into reversible opcodes, cutting vocabulary needs by up to 19.7% under corpus exhaustion and lifting a 98M-parameter GPT-2's Python syntax validity from 7.70% to 9.12%

The Functionalizer separates word-form variation from the root: a canonical base token (operand) is prefixed by parametric transformation operators (opcodes) encoded in the Unicode Private Use Area, covering casing (CAPITALIZE), 13 dedicated diacritic opcodes, and character repetition (REPEAT, MULTIREPEAT), and is fully reversible. Unlike treating hello, Hello, HELLO and Héllo as unrelated vocabulary entries (which fragments the embedding space) or discarding the variation through lossy normalization, and unlike learned, non-bijective subword codes such as the Factorizer, this framework is a deterministic, rule-based, fully bijective instruction-set-style scheme that needs no external dictionaries or frequency thresholds. The paper gives the full opcode table (CAPITALIZE U+E100, 13 diacritic opcodes U+E101–U+E10D, REPEAT U+E200, MULTIREPEAT U+E201) and worked Appendix B examples, e.g. Hello encoded as CAPITALIZE(0)+hello, Héllo as ACUTE(1) CAPITALIZE(0)+hello, and four spaces as REPEAT(0,4)+space, with decoding applying operators in reverse to recover the original text exactly.

Under unconstrained merge exhaustion, the Functionalizer lets each corpus be fully covered with a smaller vocabulary, reducing required vocabulary slots by 14.61% to 19.72%, averaging 17.16%, with the peak reduction on FineWeb-Edu. Where earlier work often focuses on tokenization quality or sequence length, this work measures the total vocabulary slots needed to cover a corpus, showing that collapsing formatting variants converts directly into vocabulary budget savings. BPE tokenizers with an unconstrained target vocabulary budget (4096k) were trained to merge candidate exhaustion on up to 100,000 sampled documents each from Wikitext, Python-Codes, FineWeb-Edu and GitHub-Code-Python; the table reports baseline versus Functionalizer vocabulary sizes, e.g. Wikitext 106,023 vs. 90,531 and FineWeb-Edu 1,214,684 vs. 975,169.

On 98M-parameter GPT-2 Small models (12 layers, 768 hidden dimension, 12 heads, context length 512, 16k vocabulary), the Functionalizer configuration raises Python syntax success on GitHub-Code-Python from 7.70% to 9.12% (an 18.4% relative improvement) and lowers code character-level perplexity (1.5328 vs. 1.5697). The paper attributes part of this to structured decomposition: standard BPE fragments indentation into arbitrary whitespace chunks, whereas REPEAT parameterizes indentation into an arithmetic relationship sharing a base character and differing only by an ordinal count, which together with unified casing across identifier conventions gives downstream models clearer structural representations. Five random seeds (1–5) trained for 50,000 steps, with greedy decoding on 1,000 validation prompts per dataset (KV caching, up to 256 new tokens), and substantially tighter variance across seeds for the Functionalizer configuration.

On natural language prose, the Functionalizer reduces duplicate word n-gram repetition on FineWeb-Edu from 66.0% to 55.8%, and on GitHub-Code-Python from 25.5% to 17.9%, while prose character-level perplexity stays essentially level with the baseline (2.2656 vs. 2.2662). This indicates that separating surface variation from lexical roots can mitigate repetitive degeneracy without degrading normalized modeling capacity; the authors hypothesize that reduced surface variation helps keep the model from locking into repetitive surface loops. Repetition is measured as the proportion of duplicate overlapping word n-grams (averaged over n), with generation halting early on a cycle of four consecutive repeated tokens; the paper also reports the decoding cost of standalone prefixes, including 7.1% empty sequences on prose, some of which are operator-only sequences.

The framework targets the pre-tokenization stage before standard tokenizers such as Hugging Face BPE, and it is meant to run after a regex splitter (e.g. Llama Split) that segments text into localized pieces so numeric parameters stay within a single-byte range; operators are emitted by default as standalone prefix tokens, or can be fused with base pieces before subword training. It suits corpora and code settings where one wants to share base embeddings without discarding orthographic information and to express structure such as indentation and casing parametrically; the authors identify fused configurations, prefix-aware attention optimizations, and larger-scale evaluation under equalized byte budgets as natural next steps.

Downstream evaluation was run at the 98M-parameter scale with a fixed 50,000-step budget, and because of standalone sequence expansion the Functionalizer processed 8–16% fewer raw bytes during pretraining than the baseline, so representational gains are not yet separated from sequence-length differences; at small scale standalone prefixes also produce empty sequences and orphaned control characters, for which the authors propose constrained decoding or prefix masking. Parameter addressing is currently bounded within pre-tokenized pieces and diacritics to 13 combining marks, leaving non-Latin scripts, morphological lemma folding for agglutinative languages, and numeric/date templates uncovered; the downstream experiments evaluate the composite CAPITALIZE + diacritics + REPEAT pipeline, so the individual contribution of each component remains to be disentangled. In addition, several numeric cells in the loaded result tables are empty, so some metrics can only be summarized from the prose rather than from table values.