# Fixed-State Entropy Regularization

In networked average-reward multi-agent reinforcement learning, each agent updates its policy using only its k-hop neighborhood, and prior results in this setting were tabular convergence to stationary points. The work first gives two counterexamples: even in a fully cooperative, fully observable, ergodic setting, product local policies can stall at a suboptimal local optimum, and when local policies cannot see the global state a vanishing gradient need not certify unilateral optimality. It then proves a finite-sample stationarity bound for a decentralized actor-critic with linear critics, and shows that strong fixed-state entropy regularization upgrades the guarantee to global optimality.

In networked average-reward multi-agent reinforcement learning, each agent updates its policy using only its k-hop neighborhood, and prior results in this setting were tabular convergence to stationary points. The work first gives two counterexamples: even in a fully cooperative, fully observable, ergodic setting, product local policies can stall at a suboptimal local optimum, and when local policies cannot see the global state a vanishing gradient need not certify unilateral optimality. It then proves a finite-sample stationarity bound for a decentralized actor-critic with linear critics, and shows that strong fixed-state entropy regularization upgrades the guarantee to global optimality.