# Edgeworth expansion for stochastic approximation

Convergence analyses of synchronous Q-learning usually keep only the Gaussian limit, and central-limit normalization removes the smaller-scale displacement produced by the initial nonlinear evolution. This analysis splits the last-iterate error into an optimal-policy temporal-difference component and a nonnegative displacement caused by maximization, proves the latter converges almost surely to a random limit at the n^-1/4 scale, and uses an Edgeworth expansion to give corrections to the Gaussian approximation. In a two-state example, the first-order corrected density tracks the empirical distribution better than the central-limit Gaussian.

Convergence analyses of synchronous Q-learning usually keep only the Gaussian limit, and central-limit normalization removes the smaller-scale displacement produced by the initial nonlinear evolution. This analysis splits the last-iterate error into an optimal-policy temporal-difference component and a nonnegative displacement caused by maximization, proves the latter converges almost surely to a random limit at the n^-1/4 scale, and uses an Edgeworth expansion to give corrections to the Gaussian approximation. In a two-state example, the first-order corrected density tracks the empirical distribution better than the central-limit Gaussian.