VocabX rebuilds a small model's vocabulary around token cost, cutting measured inference cost 77% on a Portuguese essay grader while its bench score rises from 0.703 to 0.810
Synopsis
Small models specialized for one domain usually inherit the multilingual vocabulary of a general base model, so domain text is split into many pieces and every generated token scores tens of thousands of entries the task never uses. Dharma-AI's VocabX changes the selection criterion from token frequency to inference cost weighted by field, rebuilding sub-word segmentation while merging repeated spans into macro tokens. On a Portuguese essay grader built on Qwen3-4B, with the same base model, data and training recipe, average response length fell from 350 to 101 tokens, measured inference cost dropped 77.0%, and the bench score rose from 0.703 to 0.810.
Interpretation
A specialized small model's vocabulary can be rebuilt around the actual serving cost of each token instead of inherited from the general base model, lowering inference overhead without swapping the base model. The tokenizer is usually treated as a fixed property of the base model, inherited and left alone; existing vocabulary-transfer methods pick tokens by frequency and work on only one layer, either sub-word segmentation or multi-word merging. On the production Portuguese essay grader, seven versions were trained with the same base model, data, recipe and seed, differing only in tokenizer, and all ran the same 500 requests on the same RTX 5090 with the vLLM server, with cost computed from measured runtime; cost fell 77.0%.
The method prices each candidate token's saving by whether it appears in the system prompt, the user message or the model response, because generating a token costs more than reading one. In the measured grader, a generated token costs 7.4 times a read token; that ratio varies with hardware, batch size and context length, but the direction that generation costs more holds. The vocabulary is rebuilt at the sub-word layer and, at the multi-word layer, five miners find repeated spans that become single macro tokens, both layers chosen under the same cost criterion with their own budgets; the grader's output layer went from about 152,000 rows to about 65,000.
Compression must not swallow the decision the model exists to make, and fusing the grade with the surrounding JSON into one macro token was worse. The first vocabulary version fused each competency's grade with the surrounding JSON into a single macro token, the most compact option, but its bench score was 0.689, below the production grader's 0.703. Giving the grade its own token cost 2 extra tokens per response, raised the bench score to 0.810 and measured a 77.0% cost reduction, against 73.3% for the fused version.
Perspective
The demonstrated scope is Dharma-AI's own production Portuguese essay grader plus comparisons on five public domains (contracts, medical papers, scientific papers, news and Python code); the authors state that the full method, per-experiment tables and discarded interventions will appear in the forthcoming VocabX paper.
The numbers come from the current version of a paper still under review, which the authors say may change before publication; the 7.4-times cost ratio between generation and reading varies with hardware, problem, batch size and context length and can be larger; VocabX's selection criterion does not include the cost of the output-layer matrix itself, and vocabulary size is instead controlled in two steps, the sub-word merge and a fixed budget for macro tokens.