MinimalLM cuts LLM output by 47.6% without retraining, and none of 990 answers ends mid-sentence
Synopsis
Long LLM answers raise inference latency and compute cost, yet prompt-only conciseness is hard to control and early stopping can cut sentences in half. MinimalLM separates when stopping becomes desirable from whether stopping is structurally appropriate: during decoding it adjusts stop-token scores by length and uncertainty, and only lets them take effect at sentence-ending punctuation, closed code fences or balanced brackets. Across three open-source models and 330 questions, generated tokens fell 47.6% with an average quality score of 8.04/10 versus 8.42/10 for unmodified models, and no structurally truncated outputs appeared.
Interpretation
MinimalLM makes open-source LLMs generate nearly half as many tokens at inference time while keeping every answer ending at a complete sentence or structural block. Prompting the model to be concise is free but loose and model-dependent, while a token cap or a boosted stop-token score saves tokens by cutting answers wherever it stops. On Mistral-7B-Instruct-v0.2, Llama-3.1-8B-Instruct and Qwen2.5-3B-Instruct over 330 questions, MinimalLM reduces generated tokens by 47.6% (95% CI 45.7%–49.5%), yields zero structurally truncated outputs among 990 responses, and scores 8.04/10 on average against 8.42/10 for unmodified models.
The saved tokens come from earlier termination at valid boundaries rather than from trimming text afterward. Without the completeness guardrails, the same stopping mechanism cuts 32.6% of tokens but leaves 59.3% of responses structurally truncated; on MT-Bench, 82% of shortened responses terminate prematurely. Only 5 of 990 responses require trimming at the generation limit; against baseline answers truncated to the same token budget, MinimalLM scores 8.04 versus 6.46 for truncation and 6.82 after trimming to the last complete sentence, with 77%, 11% and 40% judged complete respectively.
The stopping decision combines length pressure, an uncertainty signal and a structural gate, with no parameter changes or auxiliary models. A system instruction of about 200 words asks the model to cover the request and finish sentences and code blocks, and decoding then adjusts only stop-token scores while leaving other token choices unchanged. The stop score rises with generated length and is modulated by the entropy of the next-token distribution; the gate admits stopping only at sentence-ending punctuation, closed code fences or balanced brackets, with thresholds near the 25th and 50th percentiles of development-set response lengths, taking effect after roughly 380 generated tokens.
Perspective
The evaluation covers three open-source models of 3B–8B parameters with greedy decoding on a single NVIDIA A40, using first-turn questions from MT-Bench, AlpacaEval and Arena-Hard-Auto plus one representative question in each of eight domains: health, history, economics, programming, science, mathematics, writing and law.
Structural completeness is not information coverage: the automatic judge flags missing requested information in 22.9% of MinimalLM responses versus 12.6% for the baseline, though its agreement with manual assessment is limited, and none of 24 manually inspected MinimalLM responses was judged incomplete. A fixed budget of about 200 words increases tokens by 42% on already-brief mathematics answers, and larger models, other decoding strategies and hardware configurations remain unexplored.