Social dilemmas lose their fixed rules: OpenSocialJax samples hidden mechanics from tens of millions of configurations, and eight LLMs still cannot hold a shared resource
Synopsis
OpenSocialJax extends Clean Up and Commons Harvest into an open-ended distribution: each episode samples one of tens of millions of hidden configurations and holds it fixed across five trials, so agents must infer tool recipes, payoffs and regrowth rules through interaction. Eight large language models were evaluated in homogeneous teams and a mixed arena; rule discovery often improved performance but did not reliably sustain cooperation, with the cleaning burden concentrating among fewer agents in OpenCleanup and every team exhausting the shared resource in OpenHarvest. In the mixed arena, one cooperative agent sustained the public good in OpenCleanup at a cost to its own return, but could not prevent resource depletion in OpenHarvest.
Interpretation
Social dilemma evaluation moves from fixed rules to an open-ended distribution of hidden mechanics, so agents must infer environmental rules and other agents' behavior at the same time. Earlier social dilemma benchmarks offer diverse scenarios but do not systematically vary the hidden environmental dynamics behind a dilemma, nor test how agents discover and adapt to them over repeated interaction. Each episode samples one configuration from tens of millions of possibilities, varying spatial layouts and hidden rules governing tool crafting, payoffs and resource dynamics, and holds it fixed across five trials.
Agents accumulate rule evidence from their own experience and carry discoveries into the next trial through notes they write themselves. Agents never see the hidden rule, other agents' prompts or scores, and cannot communicate with them, relying only on the harness record and their own free-form notes. At the end of each trial the model rewrites its notes into a short list of lessons and plans, and only the rule evidence, trial summaries and that list carry over to the next trial.
Across eight large language models in homogeneous teams, rule discovery often brought improvement, but cooperation was not reliably sustained. In OpenCleanup almost every model improved from the first to the second trial, but for several the gain did not last and their return drifted back down after peaking in the second trial. The effective number of cleaners in OpenCleanup falls across trials as cleaning concentrates among fewer agents, and in OpenHarvest no model keeps the orchard alive for a full trial, with open-source and GPT models stripping it early every trial.
In the mixed arena, the effect of a single cooperative agent depends on the type of dilemma. A homogeneous Claude Opus 5.5 team kept the OpenHarvest orchard productive much longer, while the mixed team depleted it as quickly as homogeneous teams of the other models. In the mixed arena Claude Opus 5.5 kept cleaning in OpenCleanup and bore most of the cost, with a return far below its homogeneous team while the least-cleaning models gained the most; in OpenHarvest its advantage disappeared and its return fell to the level of the other models.
Perspective
The evaluation covers eight large language models, three closed source and five open source, in homogeneous teams of five and in a mixed arena where five models each control one agent, with three seeds per environment, five trials per episode and at most 400 steps per trial. The environments are the two grid worlds OpenCleanup and OpenHarvest, and the hidden configuration stays fixed across trials. The authors propose extending the multi-trial protocol with explicit communication and more sophisticated theory of mind mechanisms.
Rule discovery often improves performance but does not necessarily lead to sustained cooperation, as the cleaning burden concentrates in OpenCleanup and the shared resource is exhausted within each trial in OpenHarvest. Claude Opus 5.5's orchard lifetime in OpenHarvest peaks in the second trial and then shortens, and its notes show that from the third trial on most agents conclude that holding back does not pay. The authors note that training learning-based agents that adapt to the latent configuration remains an open challenge.