# Contextual Bandits

When a regulator evaluates a learned policy, the inputs and outputs are usually all that is visible, so it is hard to tell whether value comes from acting on the state or from average behavior. This work shows that, in a finite-horizon Markov decision process, the expected-cost difference between a policy and its open-loop marginal equals exactly the covariance between state and action, and it bounds adaptive value by the mutual information the policy uses. Auditing that quantity under bandit feedback needs a sample size inversely proportional to an average KL divergence, while a deterministic non-constant policy is not identifiable from its own logs.

When a regulator evaluates a learned policy, the inputs and outputs are usually all that is visible, so it is hard to tell whether value comes from acting on the state or from average behavior. This work shows that, in a finite-horizon Markov decision process, the expected-cost difference between a policy and its open-loop marginal equals exactly the covariance between state and action, and it bounds adaptive value by the mutual information the policy uses. Auditing that quantity under bandit feedback needs a sample size inversely proportional to an average KL divergence, while a deterministic non-constant policy is not identifiable from its own logs.

For stochastic contextual bandits whose contexts fall into unknown blocks with identical per-arm mean rewards, online expected regret now matches the known lower bound up to logarithmic factors under arbitrary context distributions and without separation between blocks, achieved without recovering the unknown partition.

For stochastic contextual bandits whose contexts fall into unknown blocks with identical per-arm mean rewards, online expected regret now matches the known lower bound up to logarithmic factors under arbitrary context distributions and without separation between blocks, achieved without recovering the unknown partition.