Mobile robots skip the prebuilt semantic map: Feijoa grounds instructions at query time and hits 94.2% exact match on 103 queries
Synopsis
When a mobile manipulator is told to put the toys away in the toy box, it usually relies on a task-agnostic semantic map that may lack the attributes, affordances or concepts the instruction needs. Feijoa has a language-model agent query a posed RGB-D scan after the instruction arrives, inspect the returned candidates and rewrite its queries until the reference resolves. On 103 open-ended grounding queries it reaches 94.2% exact match, against 68.0% for the best prebuilt representation, TextMap; on 50 real Spot trials it completes 35 tasks, 17 of them after recovering from a failed tool call or verification check.
Interpretation
The robot's understanding of the scene is now produced on demand after the instruction arrives, rather than built once while the instruction is still unknown. Earlier open-vocabulary 3D object maps, 3D scene graphs and 3D feature fields finish perception before any query and reduce the scene to embeddings or captions, so they may lack the fine-grained attributes, affordances and concepts a given instruction requires. Across 103 grounding queries covering category, attribute, spatial relation, sets and reasoning, Feijoa reaches 94.2% exact match, the best prebuilt representation TextMap reaches 68.0%, and the find_objects tool without the agent reaches 47.6%.
The agent links grounding, planning and robot skills into a closed loop, choosing each next step from the result of the previous one. Prebuilt representations do the expensive perception once before the query and only retrieve at query time, so they cannot ask again when the result is wrong. At each turn the agent selects from a tool library containing find_objects, estimate_cost, update_plan and the skills navigate, pick, place, push and wipe, while the harness assembles the context, dispatches calls and records results.
The execution harness checks a skill's preconditions and samples feasible base poses before execution, then verifies the effect with a visual judge afterwards. Without these checks the agent may drive to a pose from which the following skill fails, or mark a failed step as done and move on. Among 20 hand-confirmed failed place and wipe executions, the agent marked the failed step done and moved on 9 times with status text only, 6 times with before and after images added, and 2 times with the verdict added; removing feasibility-aware sampling left 2 of 5 and 1 of 5 trials reaching a feasible pose in two scenarios, against 5 of 5 for Feijoa.
Perspective
The demonstrated setting is mobile manipulation in a previously scanned workspace: the agent receives a posed RGB-D recording as observation memory and acquires fresh observations during execution; the real trials ran on a Spot quadruped with an arm over 50 tasks spanning short, medium and long horizons. Grounding was evaluated on 103 queries across four scenes, with predictions judged manually.
The authors identify robot physical skills as the main remaining bottleneck: 13 of the 15 failed trials came from physical skills and only 2 from grounding or the agent's reasoning, and grounding at query time plus language-model inference add latency. The appendix reports that the best standalone vision-language model, Gemini 3.8 Flash, reaches 96.1% exact match on the same queries at lower latency and similar token cost, but returns one 3D point per object without the extent and views skills need, and requires the whole scan to fit in its context window.