In continuous games, agents stop chasing payoff and chase regret: DRAC brings exploitability to the 10⁻⁴ range in two sequential games
Synopsis
Multi-agent learning in continuous general-sum games has long struggled with non-stationarity, because agents maximizing their own payoffs make the learning target move. DRAC shifts the objective from payoff maximization to regret minimization, fixing opponents' actions and sampling unilateral deviations to build a stationary regret target, with a dual-head critic predicting best-response utility and regret separately. In a three-agent, five-round sequential Cournot oligopoly and a two-agent, five-stage proportional prize contest, DRAC's exploitability converged to 1.37×10⁻⁴ and 7.45×10⁻⁴, with mean relative absolute deviations of 0.011 and 0.030, both below MADDPG at the same training budget.
Interpretation
In continuous-action, general-sum multi-agent games, the learning objective becomes regret minimization rather than each agent separately maximizing its own payoff. Independent payoff optimization makes every agent update at once, so the learning environment becomes a moving target and agents can cycle through policy adaptations instead of settling at a stable equilibrium. After 200,000 training epochs in a three-agent, five-round sequential Cournot oligopoly, DRAC's exploitability converged to 1.37×10⁻⁴ with a mean relative absolute deviation of 0.011; MADDPG reached an exploitability of 4.27×10⁻⁴ and a mean relative absolute deviation of 0.019 in the same setting.
The algorithm builds its training targets from sampled unilateral deviations, so the targets do not drift with opponents' simultaneous policy updates. Methods such as Deep Counterfactual Regret Minimization must explicitly evaluate every available action at a decision node, which confines them to finite discrete action spaces; standard actor-critic handles continuous actions but couples value estimates, so one agent's flawed policy update propagates into the action-value functions of the others. For each agent, the other players' joint action is held fixed and 256 unilateral deviations are sampled from a Gaussian centered on the executed action; the maximum sampled utility becomes the best-response utility target, which is differenced against a target-network action value and rectified at zero to give the regret target.
A dual-head critic predicts best-response utility and regret separately, and the actor minimizes the sum of predicted regrets. Centralized joint optimization on a shared reward tends to induce collusion in non-cooperative settings, whereas individual regret is bounded below by zero, so minimizing its sum pushes each agent's exploitability toward zero. In a two-agent, five-stage proportional prize contest with resource depletion factor 0.8, DRAC's exploitability converged to 7.45×10⁻⁴ with a mean relative absolute deviation of 0.030; MADDPG reached an exploitability of 2.48×10⁻³ and a mean relative absolute deviation of 0.063.
Perspective
The demonstrated scope is continuous-action games with perfect information and general-sum payoff structure: a three-agent, five-round sequential Cournot oligopoly and a two-agent, five-stage proportional prize contest, both compared against analytical equilibria. The author proposes extending the framework to imperfect-information domains and having the actor output categorical probability distributions to compute mixed-strategy equilibria in discrete games.
Sampled best-response approximation is more prone to overlooking beneficial global deviations in highly non-convex payoff landscapes; scaling to higher-dimensional action spaces requires more Monte Carlo samples, although computational cost stays linear rather than exponential in the number of players; and computing regret requires simulating alternative actions to evaluate hypothetical rewards, which is trivial in a simulator but would need an environment model in black-box settings.