Open multi-agent RL gets an axiomatic footing: Shapley's three axioms now shape each agent's value function
Synopsis
In open multi-agent systems where only some agents are controllable, individual value functions previously had no structured basis for learning. Researchers treat the Shapley axioms of efficiency, symmetry and linearity as structural constraints during learning rather than constructing values from the explicit formula, yielding two generic templates, Shapley Machine and Banzhaf Machine. On MPE-PP and SMAC tasks adapted to the n-agent ad hoc teamwork (NAHT) setting, instantiating them on IPPO and POAM generally improved learning efficiency and generalisation to unseen agent types and conventions, though the efficiency axiom weakened generalisation to unseen agent types in some cases.
Interpretation
Reinforcement learning in open multi-agent systems can now derive individual value functions from cooperative-game axioms instead of heuristic design. Prior cooperative multi-agent reinforcement learning assumed all agents were controllable and team composition fixed, and the NAHT objective itself did not specify how controlled and uncontrolled agents form cooperation. Splitting team value into internal cooperation among controlled agents and external teamwork involving uncontrolled agents makes the external term vanish when all agents are controlled, recovering the fully controllable cooperative multi-agent setting.
The implementation translates the three axioms into algorithm components: efficiency into reward shaping, linearity into truncated TD(λ) updates, and symmetry into the structure of individual value networks. The number of n-step return components in truncated TD(λ) is set by the number of non-empty coalition basis games, with λ giving each basis game its weight through a geometric distribution. On MPE-PP and SMAC tasks adapted to NAHT, instantiating on IPPO and POAM gives SM-IPPO, SM-POAM, BM-IPPO and BM-POAM; each metric is computed over 128 episodes, averaged across 5 random seeds, with 95% confidence intervals reported.
Dropping the efficiency axiom yields Banzhaf Machine, which corresponds to the Banzhaf index rather than the Shapley value. On MPE-PP and 3sv5z with IPPO as the base algorithm, Banzhaf Machine performs on par with Shapley Machine; Shapley Machine is better in most other scenarios, sometimes by a narrow margin. In evaluation on unseen agent types, BM-POAM is comparable to or better than SM-POAM in most cases, indicating the efficiency axiom can weaken generalisation to unseen agent types.
Perspective
The demonstrated scope is MPE-PP and SMAC tasks adapted to the NAHT setting, with uncontrolled agents drawn from five pre-trained policies: IPPO, MAPPO, VDN, QMIX and IQL. In 8v9 and 10v11 the number of non-empty coalition basis games (127 and 511) exceeds the episode length (120 and 150), so the implementation approximates it by episode length, making Banzhaf Machine equivalent to the baselines in those two scenarios.
How to handle the case where the predefined horizon is far smaller than the number of non-empty coalition basis games remains unclear; the authors' tentative strategy uses the horizon under the minimal horizon schedule as the number of uncontrolled random basis games. Shapley Machine learns less stably in large-scale scenarios than in small ones, the authors identify fluctuation of the shaped reward as the key cause, and they add a value loss clip only for 8v9 and 10v11 while noting that this strategy slows or even hinders the other three scenarios.