Answering belief questions is not collaborating: Claude recovers every agent belief but returns optimal plans in only 67.9% of matched cases
Synopsis
Theory-of-mind benchmarks usually ask story questions about what others believe, and a correct answer does not show the model can use that judgment in a task. This evaluation extends household collaboration and calendar scheduling with asymmetric information and separates world understanding, information access, beliefs, decisions and execution into independent interfaces. On matched scheduling cases Claude recovers all agent beliefs yet produces subjectively optimal plans in only 67.9% of cases, while supplying beliefs raises Qwen-27B household task success from 55.0% to 75.0%.
Interpretation
Theory-of-mind evaluation moves onto executable tasks with native validators, so whether a model turns what someone else knows into a workable decision can be checked separately on the same tasks. Earlier benchmarks mostly probe mental-state inference through story questions, and a correct answer does not establish that the model uses those judgments in a later task. On 91 stories complete for both narrative beliefs and direct planning, Claude recovers 455/455 agent beliefs but produces only 309/455 optimal proposals, or 67.9%.
The construction adds asymmetric personal observations and communication histories to household collaboration and calendar scheduling while keeping the original goals, constraints and validators. Five interfaces request world state, personal evidence, beliefs, decisions and execution, and controls supply personal evidence or correct beliefs to separate state recovery from state use. Household collaboration covers 43 configurations across seven houses and 218 event-selected nodes, 1962 fixed requests per model; the calendar set is 45 uniform-cost and 45 tiered-cost tasks with 1260 requests per model, across six model configurations.
Supplying correct beliefs does not reliably remove downstream difficulty, and the gap between fixed decisions and long-horizon coordination points in different directions for different models. In native household continuation, reference beliefs raise Qwen-27B success from 22/40 to 30/40, while Qwen-4B moves only from 5/40 to 8/40 and Qwen-9B stays at 2/40 and 1/40. Qwen-9B issues no assignments in 35/40 episodes under each continuation condition, instead running 1658 and 1712 exploration commands, even though its fixed supplied-belief decisions reach 216/218.
Perspective
The evaluation covers six service configurations, 43 household configurations across seven houses and 90 calendar tasks, all on frozen inputs and references, so the findings are limited to these simulated household and controlled calendar settings.
Each setting uses one fixed seed without independent repeated runs, and reasoning budgets, sampling controls and serving hardware differ, so the results establish neither pure parameter-scaling effects nor statistical significance across independent runs.