Visual assistant evaluation moves into closed-loop simulation: top model Gemini 4 Argon reaches only 53.4% task success within the time limit
Synopsis
Salesforce AI Research released Interactive Visual Gym (IVG), which moves visual AI assistant evaluation from passively watching pre-recorded video to closed-loop simulation: the assistant watches the user's live feed and speaks guidance, and the simulated user changes its actions in response. The accompanying IVG-Bench covers 315 complex scenarios across three operational environments and evaluates 14 visual assistants; the strongest, Gemini 4 Argon, reaches only 53.4% task success within the time limit, averaged across the three environments. In an early proof of concept, an open-source model trained with the simulator, Gwen 3.5-4B, rose from 33.3% to 42.2% success on entirely new tasks it had never seen during training.
Interpretation
The object of evaluation shifts from text answers about a pre-recorded video to a live exchange between the assistant and a simulated user: the assistant analyzes the user's feed and speaks an intervention, and the simulated user then adjusts its actions, changing the state of the digital twin. Most earlier evaluation frameworks relied on passive, pre-recorded video clips and graded an assistant's text outputs against a static answer key, so cause and effect were completely decoupled in static testing: the assistant issues a warning and the person in the video makes the mistake anyway. IVG-Bench sets 315 complex scenarios across three operational environments and evaluates 14 leading visual assistants; the strongest, Gemini 4 Argon, reaches only 53.4% task success within the time limit, averaged across the three environments.
The simulated environment does more than score: an assistant is trained to prioritize task completion and error prevention, and its success rate rises on new tasks it never saw during training. In an early proof of concept, the open-source model Gwen 3.5-4B rose from 33.3% to 42.2% success and became better at stepping in proactively before mistakes happened. The gain was measured on entirely new tasks not seen during training, and it is an early proof of concept rather than a conclusion about a full training pipeline.
IVG differs from conventional robotic simulators in its level of interaction: it operates at the high-level semantic layer where humans naturally communicate, directing an assistant to move to the prep station with the ingredients rather than computing joint velocities or degree coordinates. Standard robotic simulators mandate micro-level physics controls, while IVG places operational instructions at the semantic layer and lets the harness schedule out-of-plan disruptions at precise workflow milestones. The disruptions include procedural missteps, mid-task plan alterations, clarification questions and memory check-ins, used to stress-test an assistant's ability to detect anomalies and course-correct a live user under pressure.
Perspective
The demonstrated scope is 315 complex scenarios across three operational environments, plus a comparison with ratings from human participants on cooking tasks; the material addresses enterprise leaders preparing to deploy vision-language assistants. The material states no next-step experiment plan.
The material frames the Gwen 3.5-4B success-rate gain as an early proof of concept; for some open-weight models, later responses become less accurate and less relevant to the ongoing activity and offer weaker actionable guidance, and better task success does not necessarily mean better guidance interactions.