Aditya Agarwal

When a mobile manipulator is told to put the toys away in the toy box, it usually relies on a task-agnostic semantic map that may lack the attributes, affordances or concepts the instruction needs. Feijoa has a language-model agent query a posed RGB-D scan after the instruction arrives, inspect the returned candidates and rewrite its queries until the reference resolves. On 103 open-ended grounding queries it reaches 94.2% exact match, against 68.0% for the best prebuilt representation, TextMap; on 50 real Spot trials it completes 35 tasks, 17 of them after recovering from a failed tool call or verification check.

When a mobile manipulator is told to put the toys away in the toy box, it usually relies on a task-agnostic semantic map that may lack the attributes, affordances or concepts the instruction needs. Feijoa has a language-model agent query a posed RGB-D scan after the instruction arrives, inspect the returned candidates and rewrite its queries until the reference resolves. On 103 open-ended grounding queries it reaches 94.2% exact match, against 68.0% for the best prebuilt representation, TextMap; on 50 real Spot trials it completes 35 tasks, 17 of them after recovering from a failed tool call or verification check.