# Off-policy evaluation

When a regulator evaluates a learned policy, the inputs and outputs are usually all that is visible, so it is hard to tell whether value comes from acting on the state or from average behavior. This work shows that, in a finite-horizon Markov decision process, the expected-cost difference between a policy and its open-loop marginal equals exactly the covariance between state and action, and it bounds adaptive value by the mutual information the policy uses. Auditing that quantity under bandit feedback needs a sample size inversely proportional to an average KL divergence, while a deterministic non-constant policy is not identifiable from its own logs.

When a regulator evaluates a learned policy, the inputs and outputs are usually all that is visible, so it is hard to tell whether value comes from acting on the state or from average behavior. This work shows that, in a finite-horizon Markov decision process, the expected-cost difference between a policy and its open-loop marginal equals exactly the covariance between state and action, and it bounds adaptive value by the mutual information the policy uses. Auditing that quantity under bandit feedback needs a sample size inversely proportional to an average KL divergence, while a deterministic non-constant policy is not identifiable from its own logs.

Agent evaluation has relied on fixed benchmarks that report task success or failure but cannot say whether reviewing or steering an agent mid-trajectory is worth it at deployment. Researchers bring micro-randomized trials into agent trajectories, assigning review or direct execution with equal probability at each eligible decision, then estimate causal effects with augmented inverse probability weighting and structural nested mean models. On τ-bench Retail, ToolSandbox and ImpossibleBench, review effects vary substantially across tool-call types and agent states, and the same data yield decision-group risk rankings.

Agent evaluation has relied on fixed benchmarks that report task success or failure but cannot say whether reviewing or steering an agent mid-trajectory is worth it at deployment. Researchers bring micro-randomized trials into agent trajectories, assigning review or direct execution with equal probability at each eligible decision, then estimate causal effects with augmented inverse probability weighting and structural nested mean models. On τ-bench Retail, ToolSandbox and ImpossibleBench, review effects vary substantially across tool-call types and agent states, and the same data yield decision-group risk rankings.

When deployment lacks key variables that training logs record in full, runtime confounding can push a reinforcement-learning policy off course over time. The work places the runtime restriction only in the numerator of an importance ratio and estimates nuisance functions from the full data, yielding an asymptotically normal off-policy estimator that attains the semiparametric efficiency bound, a policy-learning regret bound that differs from the full-information optimum by an irreducible runtime gap, and a sample-splitting test for that gap.

When deployment lacks key variables that training logs record in full, runtime confounding can push a reinforcement-learning policy off course over time. The work places the runtime restriction only in the numerator of an importance ratio and estimates nuisance functions from the full data, yielding an asymptotically normal off-policy estimator that attains the semiparametric efficiency bound, a policy-learning regret bound that differs from the full-information optimum by an irreducible runtime gap, and a sample-splitting test for that gap.