# Tokenizer

Small models specialized for one domain usually inherit the multilingual vocabulary of a general base model, so domain text is split into many pieces and every generated token scores tens of thousands of entries the task never uses. Dharma-AI's VocabX changes the selection criterion from token frequency to inference cost weighted by field, rebuilding sub-word segmentation while merging repeated spans into macro tokens. On a Portuguese essay grader built on Qwen3-4B, with the same base model, data and training recipe, average response length fell from 350 to 101 tokens, measured inference cost dropped 77.0%, and the bench score rose from 0.703 to 0.810.

Small models specialized for one domain usually inherit the multilingual vocabulary of a general base model, so domain text is split into many pieces and every generated token scores tens of thousands of entries the task never uses. Dharma-AI's VocabX changes the selection criterion from token frequency to inference cost weighted by field, rebuilding sub-word segmentation while merging repeated spans into macro tokens. On a Portuguese essay grader built on Qwen3-4B, with the same base model, data and training recipe, average response length fell from 350 to 101 tokens, measured inference cost dropped 77.0%, and the bench score rose from 0.703 to 0.810.