# Inference acceleration

Computer-use agents encode each screenshot into thousands of visual tokens, and those tokens pile up into long contexts on long-horizon tasks and in real-time games. VTR trains a vision-language model jointly with a lightweight spatio-temporal router that inspects the same image region in the current and previous screenshots and decides whether that token enters the language model, using an adaptive loss to keep as few tokens as possible without losing action prediction. After only 2,000 training steps, VTR matches the full-resolution model on OSWorld, CUA-World and video-game action prediction while using 2.5–5.8× fewer visual tokens, cutting inference cost by up to 3.3× and speeding prefill by up to 2.9×.

Computer-use agents encode each screenshot into thousands of visual tokens, and those tokens pile up into long contexts on long-horizon tasks and in real-time games. VTR trains a vision-language model jointly with a lightweight spatio-temporal router that inspects the same image region in the current and previous screenshots and decides whether that token enters the language model, using an adaptive loss to keep as few tokens as possible without losing action prediction. After only 2,000 training steps, VTR matches the full-resolution model on OSWorld, CUA-World and video-game action prediction while using 2.5–5.8× fewer visual tokens, cutting inference cost by up to 3.3× and speeding prefill by up to 2.9×.

PowerMH must decide each accept/reject before it can generate the next proposal; PreSTO prefetches several future proposals in one batched call, giving a median 1.71× speedup over PowerMH across seven datasets and five base LLMs while leaving its transition kernel unchanged.

PowerMH must decide each accept/reject before it can generate the next proposal; PreSTO prefetches several future proposals in one batched call, giving a median 1.71× speedup over PowerMH across seven datasets and five base LLMs while leaving its transition kernel unchanged.

Visual token selection in multimodal LLMs now propagates scores over a visual graph and corrects for neighborhood degree: across four backbones, ten benchmarks and seven pruning ratios, LLaVA-NeXT retains 98.7% accuracy while keeping 30% of visual tokens, with a net prefill speedup.

Visual token selection in multimodal LLMs now propagates scores over a visual graph and corrects for neighborhood degree: across four backbones, ten benchmarks and seven pruning ratios, LLaVA-NeXT retains 98.7% accuracy while keeping 30% of visual tokens, with a net prefill speedup.

Speech recognition decoding no longer has to wait token by token: a lightweight draft module that reuses the target model's acoustic and decoder features delivers up to 1.9x and 2.4x lossless speedups on Whisper and Conformer.

Speech recognition decoding no longer has to wait token by token: a lightweight draft module that reuses the target model's acoustic and decoder features delivers up to 1.9x and 2.4x lossless speedups on Whisper and Conformer.

The authors propose V-CoLA, a training-free vision token compression framework for hybrid vision-language models with linear attention, deriving a uniqueness-aware importance criterion from the retention preferences of the bounded recurrent state, combined with adaptive chunk-wise merging and early exiting of deep-layer vision tokens; on Qwen3.5-9B and related models it reaches 99.5% of original performance at 50% vision tokens and over 88.0% at 12.5%, with 1.86 to 6.15 prefill speedup.

The authors propose V-CoLA, a training-free vision token compression framework for hybrid vision-language models with linear attention, deriving a uniqueness-aware importance criterion from the retention preferences of the bounded recurrent state, combined with adaptive chunk-wise merging and early exiting of deep-layer vision tokens; on Qwen3.5-9B and related models it reaches 99.5% of original performance at 50% vision tokens and over 88.0% at 12.5%, with 1.86 to 6.15 prefill speedup.

The authors argue that existing sparse-attention selectors keep large or high-mass entries of the attention matrix even though Transformers use that matrix through multiplication with value vectors, and propose MASA, which replaces attention-mass ranking with a closed-form score measuring each sparse unit's reduction in matrix-product approximation error; plugged into MInference, SeerAttention, and FlexPrefill without changing their sparse kernels, candidate units, or budgets, it yields consistent accuracy gains on RULER, InfiniteBench, and LongBench with negligible latency overhead.

The authors argue that existing sparse-attention selectors keep large or high-mass entries of the attention matrix even though Transformers use that matrix through multiplication with value vectors, and propose MASA, which replaces attention-mass ranking with a closed-form score measuring each sparse unit's reduction in matrix-product approximation error; plugged into MInference, SeerAttention, and FlexPrefill without changing their sparse kernels, candidate units, or budgets, it yields consistent accuracy gains on RULER, InfiniteBench, and LongBench with negligible latency overhead.