# Sample Complexity

For robust Markov decision processes with total-variation uncertainty sets, a model-free asynchronous Q-learning algorithm relies only on a single Markovian trajectory from a behavior policy, without a generative model or queries to arbitrary state-action pairs. It estimates robust Bellman updates through a resampled surrogate next state, uses $|S||A|$ space, and reaches entrywise $\epsilon$ accuracy with a sample complexity of order $\frac{|S||A| t_{\mathrm{mix}}}{\epsilon^2 \mu_{\min}}$ up to logarithmic factors, carrying the extra factor inherent to Q-learning.

For robust Markov decision processes with total-variation uncertainty sets, a model-free asynchronous Q-learning algorithm relies only on a single Markovian trajectory from a behavior policy, without a generative model or queries to arbitrary state-action pairs. It estimates robust Bellman updates through a resampled surrogate next state, uses $|S||A|$ space, and reaches entrywise $\epsilon$ accuracy with a sample complexity of order $\frac{|S||A| t_{\mathrm{mix}}}{\epsilon^2 \mu_{\min}}$ up to logarithmic factors, carrying the extra factor inherent to Q-learning.

When a regulator evaluates a learned policy, the inputs and outputs are usually all that is visible, so it is hard to tell whether value comes from acting on the state or from average behavior. This work shows that, in a finite-horizon Markov decision process, the expected-cost difference between a policy and its open-loop marginal equals exactly the covariance between state and action, and it bounds adaptive value by the mutual information the policy uses. Auditing that quantity under bandit feedback needs a sample size inversely proportional to an average KL divergence, while a deterministic non-constant policy is not identifiable from its own logs.

When a regulator evaluates a learned policy, the inputs and outputs are usually all that is visible, so it is hard to tell whether value comes from acting on the state or from average behavior. This work shows that, in a finite-horizon Markov decision process, the expected-cost difference between a policy and its open-loop marginal equals exactly the covariance between state and action, and it bounds adaptive value by the mutual information the policy uses. Auditing that quantity under bandit feedback needs a sample size inversely proportional to an average KL divergence, while a deterministic non-constant policy is not identifiable from its own logs.

Large-model training corpora often cover public benchmarks, so evaluators can no longer assume a fresh test set. This work formalizes evaluability without assuming the test set was held out and without constraining the model or its training algorithm, showing that accurate evaluation from model outputs on the test set alone is feasible if and only if the distribution is close to small support, with sample-complexity bounds tight up to logarithmic factors; allowing extra queries to the model, via self-correctors from program testing, widens the evaluable distributions.

Large-model training corpora often cover public benchmarks, so evaluators can no longer assume a fresh test set. This work formalizes evaluability without assuming the test set was held out and without constraining the model or its training algorithm, showing that accurate evaluation from model outputs on the test set alone is feasible if and only if the distribution is close to small support, with sample-complexity bounds tight up to logarithmic factors; allowing extra queries to the model, via self-correctors from program testing, widens the evaluable distributions.

Transformer-based masked diffusion models now have an end-to-end statistical guarantee: when the data distribution is well approximated by a mixture of hidden Markov trees, the training samples needed for a target KL accuracy are governed by latent structural complexity rather than growing exponentially with the sequence space. The analysis covers both the parallelization error of parallel unmasking and the finite-sample estimation error of the mask predictor, and the transformer architecture and sampling procedure need no knowledge of tree topologies or component parameters, adapting to unknown latent structure.

Transformer-based masked diffusion models now have an end-to-end statistical guarantee: when the data distribution is well approximated by a mixture of hidden Markov trees, the training samples needed for a target KL accuracy are governed by latent structural complexity rather than growing exponentially with the sequence space. The analysis covers both the parallelization error of parallel unmasking and the finite-sample estimation error of the mask predictor, and the transformer architecture and sampling procedure need no knowledge of tree topologies or component parameters, adapting to unknown latent structure.