А. Бочков

The study asks whether a decoder-only language model needs an independently trainable input vector per token: with V=65,536 and d_model=1024, it replaces the usual V×d_model input table with fixed 16-bit binary token codes plus a parameter-free tiled lift to model width, removing 67.1M trainable parameters (about 12.5% of the untied learned-input baseline), and across three training seeds 32-layer models trained on roughly 16-17B tokens obtain mean held-out perplexities of 2.44 for the learned-input baseline, 2.36 for canonical binary codes, and 2.39 for affine-recoded codes, with the authors stating these descriptive results do not establish statistical superiority or equivalence.

The study asks whether a decoder-only language model needs an independently trainable input vector per token: with V=65,536 and d_model=1024, it replaces the usual V×d_model input table with fixed 16-bit binary token codes plus a parameter-free tiled lift to model width, removing 67.1M trainable parameters (about 12.5% of the untied learned-input baseline), and across three training seeds 32-layer models trained on roughly 16-17B tokens obtain mean held-out perplexities of 2.44 for the learned-input baseline, 2.36 for canonical binary codes, and 2.39 for affine-recoded codes, with the authors stating these descriptive results do not establish statistical superiority or equivalence.