Training logs know what deployment cannot: runtime-confounded RL gets doubly robust evaluation and a testable price
Synopsis
When deployment lacks key variables that training logs record in full, runtime confounding can push a reinforcement-learning policy off course over time. The work places the runtime restriction only in the numerator of an importance ratio and estimates nuisance functions from the full data, yielding an asymptotically normal off-policy estimator that attains the semiparametric efficiency bound, a policy-learning regret bound that differs from the full-information optimum by an irreducible runtime gap, and a sample-splitting test for that gap.
Interpretation
When deployment exposes only part of the state, reinforcement learning can still learn and evaluate within a policy class that depends only on runtime-visible variables, instead of discarding the variables missing at deployment. Prior reinforcement-learning work handled settings where both logging and deployment lack confounders, using causal bounds, sensitivity analysis or partially observable formulations; single-timepoint runtime confounding had been studied for counterfactual prediction, but sequential runtime confounding had not been specifically studied. Under sequential ignorability, consistency and overlap, the value of a target policy is identified from observed data, and the identifying functional keeps the runtime confounders in the history rather than dropping them as common practice would.
The runtime restriction enters only through the numerator of an importance ratio, so reweighting on the full data recovers the value of any runtime policy. Standard off-policy importance weights put the full-information policy in the numerator, whereas here the numerator is the runtime policy's action density and the denominator remains the full-information logging policy. The resulting estimator uses cross-fitted nuisance functions and, under the estimation assumptions, is asymptotically normal and attains the semiparametric efficiency bound, giving valid confidence intervals and tests.
The regret of the learned runtime policy against the full-information optimum splits into an irreducible runtime gap plus a convergent estimation error. The full-information optimum is infeasible at deployment, so this gap is not estimation error but the value lost by withholding the runtime confounders. Under product-rate, consistency and entropy conditions on the runtime policy class, the regret bound keeps the same rates as the non-runtime case plus an irreducible gap.
Because runtime confounders are logged, that irreducible gap can be estimated and tested, giving a decision-relevant quantity for accepting or forgoing those variables. The gap is a difference of two maxima, so direct inference can be non-regular; a sample-splitting scheme learns the full-information and runtime policies on one part and estimates the gap on the other. Under the asymptotic normality of Theorem 2, an asymptotically valid confidence interval for the gap can be built, and the gap can be tested for zero when the two policies' scores differ.
Perspective
The scope is sequential decision settings where logged data are fully observable, deployment lacks some variables, and those variables are recorded in the logs; the conclusions rest on sequential ignorability, consistency and overlap, plus several estimation assumptions, and the material reports no numerical or real-data validation.
The test for the runtime gap fails when the two policies' scores coincide, so an additional score-inequality condition is imposed; online settings and decaying overlap are explicitly left for future work, and no simulation or empirical results are reported.