Robust Q-learning drops the generative model: one Markovian trajectory suffices, with near-minimax sample complexity
For robust Markov decision processes with total-variation uncertainty sets, a model-free asynchronous Q-learning algorithm relies only on a single Markovian trajectory from a behavior policy, without a generative model or queries to arbitrary state-action pairs. It estimates robust Bellman updates through a resampled surrogate next state, uses $|S||A|$ space, and reaches entrywise accuracy with a sample complexity of order up to logarithmic factors, carrying the extra factor inherent to Q-learning.