Opponent shaping without peeking at opponents: a meta-learned update rule tops a five-game tournament
Synopsis
Methods that shape how an opponent learns in general-sum games have needed access to that opponent's policy, parameters or learning rule. A study carries the single-agent practice of meta-learning an update rule into multi-agent play, meta-training a rule that reads only the agent's own experience and payoff matrix, so deployment needs no privileged opponent information. Across round-robin tournaments in Prisoner's Dilemma, Matching Pennies, Chicken, Stag Hunt and Coin Game, the rule ranks first in normalised mean score over all five games and first in self-play.
Interpretation
An agent can learn, from its own match records and payoff matrix alone, an update rule that changes how its opponent behaves, without reading the opponent's policy, parameters, rewards or learning rule at deployment. Earlier opponent-shaping methods either differentiate through the opponent's update and need its parameters and learning rule, as LOLA does, or hold the learning algorithm inside a black-box network retrained for each game, as M-FOS does. In round-robin tournaments over Prisoner's Dilemma, Matching Pennies, Chicken, Stag Hunt and Coin Game, the rule ranks first in normalised mean score across all five games and first in normalised mean self-play score; in Prisoner's Dilemma the naive learner's cooperation rises from about half the rounds to 92.9% in the final rounds while the rule itself cooperates only 16.4%.
The rule is a small network that, after each match, reads its own states, both players' actions, its own rewards and the payoff matrix, and outputs how far to shift the policy's log-odds. The training objective is a weighted average of lifetime return against a mixture of opponents, taken over four horizons of 10, 25, 50 and 100 updates so the rule adapts quickly without falling off later. To avoid meta-gradients that explode through co-learning, training uses a likelihood-ratio gradient estimator with Gaussian noise added to the log-odds changes; a PSRO-inspired stage then trains exploiters against the current rule and mixes them into the opponent population, lowering exploitability in Matching Pennies from 0.376 to 0.144.
Perspective
The demonstrated setting is round-robin tournaments over four iterated matrix games plus the two-coin Coin Game, with a rule meta-trained separately per game and, at deployment, a frozen meta-network that only updates the inner policy network. A reader interested in larger or continuous-action multi-agent environments would need to see whether the rule form extends to them.
Rules are trained per game, and in Chicken a rule trained on Chicken scores higher than the IPD rule played zero-shot; in Coin Game the learned rule only rescales the REINFORCE step by a multiplier of about 0.99 to 1.00, which the authors attribute to the rule form; outside the tournaments, M-FOS's meta-self-play variant scores higher than the self-play variant, which the authors suspect stems from different training mixtures.