A foundation model as critic backbone speeds multi-agent RL convergence for random access networks by at least 55%

It introduces a consensus-driven, fully decentralized MARL framework that replaces a global controller with local peer-to-peer communication and unifies three random access optimization objectives: fair AoI, max sum-rate, and fair rate. Unlike prior centralized-training-decentralized-execution methods, it needs no central training entity, and relative to the authors' earlier work it covers three objectives with shared observation and action spaces and locally computable reward functions. The paper formalizes the three objectives as MDPs with local rewards and validates them through numerical learning curves, collision counts, and normalized reward sums.

It uses a self-supervised pretrained foundation model as a reward-agnostic critic backbone with only a task-specific head attached, so online learning benefits from latent representations of environment dynamics. Unlike conventional RL models supervised by reward signals, this foundation model is pretrained on observation-action sequences for forward-dynamics prediction without reward labels, making it transferable across tasks; relative to the purely empirical SMART work, it adds theoretical analysis. The paper describes encoders, transformer blocks, and a prediction head, states that pretraining uses more than a given number of reward-independent random access network samples, and keeps the backbone fixed during the RL phase.

It provides a finite-time convergence analysis of the foundation-model-aided decentralized actor-critic algorithm covering local reward consensus and nonlinear (deep neural network) value function approximation. Relative to the authors' earlier linear-approximation result and analyses that assume consensus over the entire critic model, this work restricts consensus to scalar rewards and characterizes the impact of pretrained foundation model estimation and approximation errors. Under six assumptions, Theorem 1 decomposes the suboptimality gap into value function approximation error, actor update error from imperfect reward consensus, TD-error/advantage mismatch, and intrinsic representation error, with the full proof deferred to an online technical report.

Numerical results show the foundation-model-aided method converges faster on all three random access tasks while reaching final rate and AoI performance comparable to end-to-end training. Compared with end-to-end training and conventional decentralized MARL that exchanges full critic parameters, it substantially shortens the time to reach the optimal performance level while keeping comparable final performance. The paper reports at least about 55% improvement in convergence speed, and its tables give mean, min/max, N-Gap, and 95% time metrics across device counts, with results averaged over at least several independent runs.

The framework targets random access networks without a global controller where devices can exchange information over local links, under saturated traffic, slotted time, and listen-before-talk MAC operation; its three downstream objectives are fair AoI, max sum-rate with minimum rate guarantees, and fair rate. The results are meant to support settings where centralized control is infeasible or undesirable, such as IoT, machine-type communications, and smart-grid-like deployments, and to serve as a starting point for extension to larger action spaces such as transmit power control.

The full proof of Theorem 1 is placed in an online technical report, while the main text gives only a proof sketch and assumptions, so readers wanting to verify constants and orders need that report. The convergence bound relies on assumptions of bounded rewards, mixing time, Lipschitz continuity, a doubly stochastic consensus matrix, a stationary distribution, and universal approximation, and how well these hold in real networks remains an open question. The numerical evaluation is simulation-based with specific parameter settings and limited device counts and topologies, so whether the performance and speedup persist at larger scale, under non-saturated traffic, or with power control still needs further verification.