Minixhofer et al. introduce byteification, retrofitting token-based LLMs to run at the byte level so they can count individual letters in words
Minixhofer et al. report in Nature an approach called byteification that retrofits token-based large language models to operate at the byte level; the authors show that byteified models achieve competitive performance while retaining the ability to read individual characters, enabling tasks such as counting the letter 'r' in 'strawberry'.