Robots stop taking commands literally: MIT's FLEX uses an LLM to turn "don't spill on the laptop" into avoiding every device
Synopsis
Researchers at MIT and New York University introduce FLEX, a flexible feature-editing pipeline for robot motion planning that lets a robotic arm go beyond the literal wording of a natural-language instruction and infer a higher-level intent the person never stated. Earlier methods added one object to an avoidance list per correction, so the list grew long and went stale as the environment changed. FLEX uses a large language model to generalize across the features already named and a vision system to drop objects no longer present; in three simulated tasks, serving coffee, moving a candle and carrying a bowling ball, the robot followed the literal command and extended avoidance to electronics, flammable objects and fragile items.
Interpretation
Told "don't spill on the laptop," the robot avoids the laptop and also steers around a nearby phone and other electronics that the person never named. Earlier approaches added one object to an avoidance feature list per correction, so a person had to name every object to avoid, and the list kept growing and picking up irrelevant entries as the environment changed. In three simulated scenarios, serving coffee, moving a candle and carrying a bowling ball, volunteers typed natural-language commands and the FLEX-controlled arm met the literal request while extending avoidance to electronics, flammable objects and fragile things.
The pipeline runs in three steps: add the object the person mentions to the task's feature list, let a large language model find commonalities among the listed features and generalize them into a broader category, then use the robot's own vision system to delete objects no longer in the scene. The generalization step is carried by the large language model GPT-4o, whose common sense absorbed from training text is used to relate liquids and electronics, turning a single object into a class of objects. The full three-step process takes about five seconds in simulation, tested on a robotic arm performing three carrying tasks with a person typing natural-language commands.
Perspective
The demonstrated scope is simulation: volunteers type natural-language commands to a virtual robotic arm in three tasks, serving coffee, moving a candle and carrying a bowling ball, and the robot extends its avoidance accordingly. The intended readers are researchers and engineers working on robot learning and human-robot interaction.
The reported results are from simulated scenarios; behavior on a physical robot or in a real home is not described, and the material gives no evaluation of whether the inferred categories always match the user's intent.