# Q-learning

Convergence analyses of synchronous Q-learning usually keep only the Gaussian limit, and central-limit normalization removes the smaller-scale displacement produced by the initial nonlinear evolution. This analysis splits the last-iterate error into an optimal-policy temporal-difference component and a nonnegative displacement caused by maximization, proves the latter converges almost surely to a random limit at the n^-1/4 scale, and uses an Edgeworth expansion to give corrections to the Gaussian approximation. In a two-state example, the first-order corrected density tracks the empirical distribution better than the central-limit Gaussian.

Convergence analyses of synchronous Q-learning usually keep only the Gaussian limit, and central-limit normalization removes the smaller-scale displacement produced by the initial nonlinear evolution. This analysis splits the last-iterate error into an optimal-policy temporal-difference component and a nonnegative displacement caused by maximization, proves the latter converges almost surely to a random limit at the n^-1/4 scale, and uses an Edgeworth expansion to give corrections to the Gaussian approximation. In a two-state example, the first-order corrected density tracks the empirical distribution better than the central-limit Gaussian.

Simulation of stochastic hybrid systems previously handled only discrete nondeterministic choices, while an action allowed to fire anywhere in a time interval gives a single state infinitely many options. The new method discretizes each action's time interval separately, learns a scheduler with Q-learning over the finite abstraction, and uses precomputed symbolic reachable states to steer simulation and prune unreachable regions in a correct-by-construction way. On the water tank benchmark the simulation reproduces the maximum reachability probabilities reported by an existing method; on a water tank variant with a low-probability underflow objective, guidance makes learning clearly faster than unguided simulation and yields a reachability probability of at least 45%.

Simulation of stochastic hybrid systems previously handled only discrete nondeterministic choices, while an action allowed to fire anywhere in a time interval gives a single state infinitely many options. The new method discretizes each action's time interval separately, learns a scheduler with Q-learning over the finite abstraction, and uses precomputed symbolic reachable states to steer simulation and prune unreachable regions in a correct-by-construction way. On the water tank benchmark the simulation reproduces the maximum reachability probabilities reported by an existing method; on a water tank variant with a low-probability underflow objective, guidance makes learning clearly faster than unguided simulation and yields a reachability probability of at least 45%.

The work presents CleanSurvival, a reinforcement-learning (Q-learning) based automated data-preprocessing framework tailored to time-to-event (survival analysis) models for censored data: it selects among combinations of data imputation, outlier detection, and feature extraction techniques to optimize performance for a Cox, random forest, neural network, or user-supplied time-to-event model; on real-world dataset benchmarks, this Q-learning-based preprocessing improved predictive performance relative to simple baselines, runtime behavior was condition-dependent and most clearly interpretable in the best-covered benchmark cells, and a simulation study showed effectiveness across different types and levels of missingness and noise.

The work presents CleanSurvival, a reinforcement-learning (Q-learning) based automated data-preprocessing framework tailored to time-to-event (survival analysis) models for censored data: it selects among combinations of data imputation, outlier detection, and feature extraction techniques to optimize performance for a Cox, random forest, neural network, or user-supplied time-to-event model; on real-world dataset benchmarks, this Q-learning-based preprocessing improved predictive performance relative to simple baselines, runtime behavior was condition-dependent and most clearly interpretable in the best-covered benchmark cells, and a simulation study showed effectiveness across different types and levels of missingness and noise.