# Markov Decision Processes

Convergence analyses of synchronous Q-learning usually keep only the Gaussian limit, and central-limit normalization removes the smaller-scale displacement produced by the initial nonlinear evolution. This analysis splits the last-iterate error into an optimal-policy temporal-difference component and a nonnegative displacement caused by maximization, proves the latter converges almost surely to a random limit at the n^-1/4 scale, and uses an Edgeworth expansion to give corrections to the Gaussian approximation. In a two-state example, the first-order corrected density tracks the empirical distribution better than the central-limit Gaussian.

Convergence analyses of synchronous Q-learning usually keep only the Gaussian limit, and central-limit normalization removes the smaller-scale displacement produced by the initial nonlinear evolution. This analysis splits the last-iterate error into an optimal-policy temporal-difference component and a nonnegative displacement caused by maximization, proves the latter converges almost surely to a random limit at the n^-1/4 scale, and uses an Edgeworth expansion to give corrections to the Gaussian approximation. In a two-state example, the first-order corrected density tracks the empirical distribution better than the central-limit Gaussian.

This work extends social laws from deterministic, goal-based settings to stochastic, reward-based multi-agent environments by introducing α-robustness, a measure of the fraction of guaranteed utility each agent retains while following its optimal single-agent policy under the assumption that all agents obey the social law, and by presenting a verification approach that reduces to solving a series of Markov decision processes; experiments on grid toy environments show that with action success probability 1, parallel lanes, opposite lanes, and switch corners with a clockwise social law reach α=1 robustness, while a slow-speed social law does not always improve guaranteed utility.

This work extends social laws from deterministic, goal-based settings to stochastic, reward-based multi-agent environments by introducing α-robustness, a measure of the fraction of guaranteed utility each agent retains while following its optimal single-agent policy under the assumption that all agents obey the social law, and by presenting a verification approach that reduces to solving a series of Markov decision processes; experiments on grid toy environments show that with action success probability 1, parallel lanes, opposite lanes, and switch corners with a clockwise social law reach α=1 robustness, while a slow-speed social law does not always improve guaranteed utility.