Robota-Embed puts motion into the retrieval key: 47.33% clip-level R@1 on an 8,000-clip robot-experience gallery

Robot retrieval shifts from matching a similar task or scene to matching the recorded continuation that should follow, with the current state and the future continuation compared in one vector space. General-purpose embeddings aligned vision with language, while action-aware methods mostly placed action in the candidate content or the returned result rather than in the query key. On Robota-Embed-Bench, where 4,000 queries search a shared 8,000-clip gallery, Robota-Embed reaches 47.33% clip-level and 81.77% episode-level R@1; Section 4.1 lists the two strongest baselines, Gemini Embedding 2 and Omni-Embed-Nemotron-3B, at 25.24% and 48.59% and reports improvements of 22.09 and 33.18 percentage points, while the abstract states the episode-level gap as 33.47 points.

The query side fuses an instruction, an interaction-region image and synchronized action into one vector, and the candidate side fuses the same instruction, a four-second future video and its actions into another, ranked by cosine similarity. The model builds on Cosmos3-Edge 4B: query and candidate share the Transformer, video tokenizer and output head, each carries a bias-free residual adapter, and source-specific projectors map native actions into the backbone. Training uses 1,695,684 filtered current–future pairs from EgoDex, DROID, LIBERO and RoboMIND–Franka-dual, with an objective combining predictive alignment, multimodal grounding and discrimination, and representation regularization.

Action is not an optional extra input: removing it hurts retrieval more than removing vision. In a DROID control with matched backbone, data and training budget, the full vision–language–action input reaches 78.7% clip-level R@1, dropping to 59.8% without action and 76.8% without vision. With the final model and gallery fixed, masking query action or vision lowers clip-level R@1 by 27.25 and 24.33 percentage points, and the two interventions answer different questions.

The demonstrated scope is retrieval evaluation on four sources (EgoDex, DROID, LIBERO and RoboMIND–Franka-dual) plus downstream experiments on DROID-Sim, the LIBERO difficult subset, five ManiSkill tasks and real robots; the authors propose extending Robota-Embed to more robots with diverse kinematics, native action spaces and control interfaces, and exploring variable-length retrieval that adapts the temporal extent of recorded continuations to the task and execution stage instead of a fixed four-second window.

The authors note that retraining measures what the remaining modalities can relearn while masking measures a fixed model's reliance on its inputs, so the small no-vision retraining gap does not imply the final model ignores vision; the Embed2Evolve comparison with Omni-Embed-Nemotron-3B contrasts two independently evolved memory and context recipes rather than an encoder swap inside one fixed harness; execution time includes API waits and is affected by caller and scheduling differences, so the time savings are observational rather than an isolated method speedup; and fewer policy calls need not reduce physical control ticks. The abstract and Section 4.1 also disagree on the episode-level improvement, stating 33.47 and 33.18 percentage points respectively.