# Multi-Agent Reinforcement Learning

In networked average-reward multi-agent reinforcement learning, each agent updates its policy using only its k-hop neighborhood, and prior results in this setting were tabular convergence to stationary points. The work first gives two counterexamples: even in a fully cooperative, fully observable, ergodic setting, product local policies can stall at a suboptimal local optimum, and when local policies cannot see the global state a vanishing gradient need not certify unilateral optimality. It then proves a finite-sample stationarity bound for a decentralized actor-critic with linear critics, and shows that strong fixed-state entropy regularization upgrades the guarantee to global optimality.

In networked average-reward multi-agent reinforcement learning, each agent updates its policy using only its k-hop neighborhood, and prior results in this setting were tabular convergence to stationary points. The work first gives two counterexamples: even in a fully cooperative, fully observable, ergodic setting, product local policies can stall at a suboptimal local optimum, and when local policies cannot see the global state a vanishing gradient need not certify unilateral optimality. It then proves a finite-sample stationarity bound for a decentralized actor-critic with linear critics, and shows that strong fixed-state entropy regularization upgrades the guarantee to global optimality.

Opponent modeling in multi-agent reinforcement learning usually concatenates an inferred peer representation with the current observation and encodes them jointly, leaving their relationship for policy learning to discover implicitly. PCOR instead imposes an asymmetric structure: observation features form the base representation of the current interaction state, and the peer representation only modulates how those features are interpreted. On two partially observable benchmarks, PCOR reaches higher final evaluation performance than the reported baselines on unseen peers; on POO, against a concatenation variant sharing the same encoders and differing only in fusion, evaluation success rises from 30.2% to 51.1%.

Opponent modeling in multi-agent reinforcement learning usually concatenates an inferred peer representation with the current observation and encodes them jointly, leaving their relationship for policy learning to discover implicitly. PCOR instead imposes an asymmetric structure: observation features form the base representation of the current interaction state, and the peer representation only modulates how those features are interpreted. On two partially observable benchmarks, PCOR reaches higher final evaluation performance than the reported baselines on unseen peers; on POO, against a concatenation variant sharing the same encoders and differing only in fusion, evaluation success rises from 30.2% to 51.1%.

Multi-agent learning in continuous general-sum games has long struggled with non-stationarity, because agents maximizing their own payoffs make the learning target move. DRAC shifts the objective from payoff maximization to regret minimization, fixing opponents' actions and sampling unilateral deviations to build a stationary regret target, with a dual-head critic predicting best-response utility and regret separately. In a three-agent, five-round sequential Cournot oligopoly and a two-agent, five-stage proportional prize contest, DRAC's exploitability converged to 1.37×10⁻⁴ and 7.45×10⁻⁴, with mean relative absolute deviations of 0.011 and 0.030, both below MADDPG at the same training budget.

Multi-agent learning in continuous general-sum games has long struggled with non-stationarity, because agents maximizing their own payoffs make the learning target move. DRAC shifts the objective from payoff maximization to regret minimization, fixing opponents' actions and sampling unilateral deviations to build a stationary regret target, with a dual-head critic predicting best-response utility and regret separately. In a three-agent, five-round sequential Cournot oligopoly and a two-agent, five-stage proportional prize contest, DRAC's exploitability converged to 1.37×10⁻⁴ and 7.45×10⁻⁴, with mean relative absolute deviations of 0.011 and 0.030, both below MADDPG at the same training budget.

In downlink NOMA, power allocation has relied on fixed, equal, or centralized learned rules that struggle to serve weak-channel users and throughput at once. A Kampala International University team models each user as an independent learning agent, coordinates them with a shared reward over local observations, and builds imperfect-SIC residual interference into the learning environment. In a three-user simulation the scheme reached about 18 bps/Hz and a fairness index of 0.96 at 30 dB SNR, with outage below 10% at 10 dB.

In downlink NOMA, power allocation has relied on fixed, equal, or centralized learned rules that struggle to serve weak-channel users and throughput at once. A Kampala International University team models each user as an independent learning agent, coordinates them with a shared reward over local observations, and builds imperfect-SIC residual interference into the learning environment. In a three-user simulation the scheme reached about 18 bps/Hz and a fairness index of 0.96 at 30 dB SNR, with outage below 10% at 10 dB.

Network signal control no longer compresses the whole road network into one shared latent vector. Policies trained under nominal flow often cannot tell a brief fluctuation from a sustained capacity loss when blockages, corridor failures or rain arrive. Here each intersection re-weights its neighbours from their joint state, then reads a history window to judge whether a change persists, cutting waiting-time recovery from 329 to 245 seconds under random blockages and by 82 seconds under a central closure.

Network signal control no longer compresses the whole road network into one shared latent vector. Policies trained under nominal flow often cannot tell a brief fluctuation from a sustained capacity loss when blockages, corridor failures or rain arrive. Here each intersection re-weights its neighbours from their joint state, then reads a history window to judge whether a change persists, cutting waiting-time recovery from 329 to 245 seconds under random blockages and by 82 seconds under a central closure.

On real road networks, autonomous vehicles that minimize only their own travel time raise human-driven vehicles' average travel time, because road capacity is shared and the routes that avoid congestion shift cost onto others. The study adds a marginal cost term to the autonomous vehicle reward: a counterfactual simulation removes that vehicle from the network and compares the remaining vehicles' travel times, quantifying its system impact. On the real networks of Ingolstadt and Provins, the social term cuts the increase in human travel time from 1.7% to 1.1% while autonomous vehicles also get faster; computing marginal cost for only about 40% of the most impactful vehicles reduces extra simulations from roughly 4.14 million to about 2.46 million with close performance.

On real road networks, autonomous vehicles that minimize only their own travel time raise human-driven vehicles' average travel time, because road capacity is shared and the routes that avoid congestion shift cost onto others. The study adds a marginal cost term to the autonomous vehicle reward: a counterfactual simulation removes that vehicle from the network and compares the remaining vehicles' travel times, quantifying its system impact. On the real networks of Ingolstadt and Provins, the social term cuts the increase in human travel time from 1.7% to 1.1% while autonomous vehicles also get faster; computing marginal cost for only about 40% of the most impactful vehicles reduces extra simulations from roughly 4.14 million to about 2.46 million with close performance.

Multi-agent reinforcement learning no longer needs peers' private rewards: each agent evaluates others' transitions with its own learned reward model, learns cooperation in Escape Room, Clean Up, and Commons Harvest, and often divides returns more evenly than agents that see the true rewards.

Multi-agent reinforcement learning no longer needs peers' private rewards: each agent evaluates others' transitions with its own learned reward model, learns cooperation in Escape Room, Clean Up, and Commons Harvest, and often divides returns more evenly than agents that see the true rewards.

The work introduces GREQ, a two-phase hybrid model-learning framework: phase one uses a channel-statistics-based greedy algorithm to deterministically satisfy user QoS with minimal subchannels, and phase two exploits the OFDM property that inter-cell interference on a subchannel comes only from the same subchannel to decompose the large-scale MARL problem into independent per-subchannel QMIX subproblems, avoiding exponential action-space growth; in simulations with 50MHz bandwidth and different numerologies, GREQ improves throughput by up to 40% over optimization and learning baselines, achieves 100% constraint satisfaction, and activates only 20–80% of available subchannels.

The work introduces GREQ, a two-phase hybrid model-learning framework: phase one uses a channel-statistics-based greedy algorithm to deterministically satisfy user QoS with minimal subchannels, and phase two exploits the OFDM property that inter-cell interference on a subchannel comes only from the same subchannel to decompose the large-scale MARL problem into independent per-subchannel QMIX subproblems, avoiding exponential action-space growth; in simulations with 50MHz bandwidth and different numerologies, GREQ improves throughput by up to 40% over optimization and learning baselines, achieves 100% constraint satisfaction, and activates only 20–80% of available subchannels.

SynCo formulates agentic data synthesis and task solving as a coupled multi-agent reinforcement learning problem, updating a Synthesizer and a Reasoner from the same online interaction batch: the Reasoner receives rollout-level correctness rewards while the Synthesizer receives a task-level reward gated by task quality, answer reliability, and outcome-grounded teachability, with both updates feeding the next synthesis round. Across eight mathematical reasoning benchmarks, SynCo reaches 78.72% average accuracy versus 75.85% for a frozen-Synthesizer variant and 74.96% for the strongest external baseline, ScaleQuest-Math, with gains arising mainly from problems the initial model failed to solve.

SynCo formulates agentic data synthesis and task solving as a coupled multi-agent reinforcement learning problem, updating a Synthesizer and a Reasoner from the same online interaction batch: the Reasoner receives rollout-level correctness rewards while the Synthesizer receives a task-level reward gated by task quality, answer reliability, and outcome-grounded teachability, with both updates feeding the next synthesis round. Across eight mathematical reasoning benchmarks, SynCo reaches 78.72% average accuracy versus 75.85% for a frozen-Synthesizer variant and 74.96% for the strongest external baseline, ScaleQuest-Math, with gains arising mainly from problems the initial model failed to solve.