Cutting three-quarters of visual tokens, APS-VLA keeps 95.75% robot-policy success and drops inference to 68.75 ms
Synopsis
Vision-language-action policies process hundreds of visual tokens per step, which drives up inference latency. APS-VLA keeps the backbone frozen and uses a lightweight scorer conditioned on the current observation, instruction and proprioception to pick tokens, training the retained representation both to reproduce policy outputs and to predict the subsequent visual change. On LIBERO with 128 tokens retained, mean success is 95.75%, 1.25 points below the unpruned 97.00%, and inference latency falls to 68.75 ms.
Interpretation
The visual input of a robot manipulation policy can be compressed heavily while action generation changes only modestly. Earlier pruning relied largely on static spatial redundancy, immediate instruction relevance or heuristic importance, cues that do not necessarily preserve the visual information manipulation needs as it unfolds. On LIBERO with OpenVLA-OFT as the backbone, pruning 75% of visual tokens gives 95.75% mean success against 97.00% unpruned, while inference latency drops from 107.95 ms to 68.75 ms.
Which visual tokens to keep is decided by a small scorer that sees only the current frame's inputs. The scorer, called the Decision Influence Estimator, combines each image patch's features with whole-view visual context, the language instruction and normalized robot proprioception into a score. Training uses differentiable budget-constrained gates so gradients pass through the frozen policy back to the scorer; at deployment each camera view gets one hard Top-k selection packed in original order, and the auxiliary predictor is removed.
Besides action supervision, training requires the retained representation to predict the visual change that follows an action. Action supervision aligns both the frozen policy's full-input prediction and the demonstrated action, while a visual transition objective uses demonstrated actions to predict a feature summary of a later observation from the selected current representation. On LIBERO, adding future visual supervision raises success by 1.41, 1.17 and 1.17 percentage points at budgets of 256, 128 and 64 tokens.
Perspective
The demonstrated scope is simulation evaluation on LIBERO and RoboCasa365 plus five tasks on the YHRG THESEUS-S1 and dual-UR5 dual-arm platforms, 40 trials per task for 400 real executions; latency is measured on a single A800 at batch size one.
At 87.5% pruning, retaining only 64 tokens, the latency gain shrinks while success drops more sharply, indicating fine-grained spatial cues begin to be lost beyond 75%; the real-world 61% versus 60% mean success gap is observed over 40 trials per task.