OSWorld-Science tests 12 VLMs on 146 scientific-software tasks: the strongest agent averages 73.7% and only 40% on the hardest set
The work introduces OSWorld-Science, a benchmark and evaluation environment combining expert-defined scientific tasks, artifact-based execution graders, and a shared agent harness, and evaluates 12 vision-language models across 7 scientific domains, 17 software packages, and 3 languages, finding that the strongest model, Claude Fable 5.1, averages 73.7% and about 40% on the hardest task set, with failures concentrated in the interaction layer and result verification rather than in scientific reasoning itself.