One character changed, the decision changed: 30 text models and 17 vision models take the same attack candidates

A decision model that answers a clean input correctly does not necessarily pick the right action on a modified one. Earlier evaluations mostly reported clean-input accuracy, or tested only fixed single-query text edits. Across 1332 text samples, raw attack success for six models ranges from 38.9% to 69.7%, and drops to 12.5% to 36.6% after label-consistency checks.

Attacks are built on construction models and then handed unchanged to other models, with no further optimization against the target. Candidate search uses Jev (jev-1.13.0), and every evaluated model receives the identical candidate set. 10093 perturbation candidates span 16 source groups, and 9716 injection candidates come from 1232 samples; 24 of the 30 evaluated text models were excluded from candidate selection.

Image perturbations transfer unevenly across models even when optimized against three surrogates at once. Optimization minimizes the summed log-probabilities of clean predictions from Imajev-4B, Visual-Jev-4B, and Jev-Omni. Across 160 image pairs and ten transfer targets, OneJev-4B reaches 32.4% checked success while seven targets stay below 10%.

The evaluation covers content filtering, tool and skill routing, workflow assessment, retrieval relevance, GUI targeting, physical state and motion, market forecasting, visual decisions, and general language diagnostics, keeping each task's native Choice, Noul, and Score interfaces. Thirty models are evaluated on text and 17 on images, with 24 and 12 respectively excluded from candidate selection. The benchmark is available at yifan-lan.github.io/DecisionRobust/, and the authors state that a full version will also be released containing all candidates retained after the applicable label-consistency checks, before transfer-based selection and subsequent semantic or readability review.

The authors note that the benchmark covers individual decisions rather than their downstream effects in complete workflows; image perturbations transfer weakly to some held-out models, and optimizing against multiple surrogates does not ensure broad transfer; computational constraints limit the attack methods and configurations evaluated and their coverage across tasks. They also state that the gap between raw and label-consistent attack success is large, so prediction changes alone can overstate attack effectiveness.