How much of a policy's value comes from reacting to state: a covariance identity isolates adaptivity, but deterministic policies cannot be audited from their own logs

The value a Markov policy adds or destroys relative to its open-loop marginal can be written exactly as a covariance: how far the joint law of state and action departs from the product of its marginals. The open-loop marginal ignores the state at each step and draws actions from the policy's average action frequencies, so it carries no state information. Earlier work observed that expected regret equals the covariance between costs and decisions for stochastic linear programs and portfolio policies, but only under a mean-unbiasedness condition; the identity here extends to any finite-horizon Markov decision process and any Markov policy. The identity holds under endogenous dynamics, where actions affect future states, by averaging under the policy's own state distribution and pairing actions with the policy's own action-value function; a myopic audit using immediate costs instead can reverse sign, with unbounded error, in a two-step, two-state, two-action construction.

The size of adaptive value is limited by the information the policy uses: the more mutual information between state and action, the more a policy can improve cost over its open-loop marginal. This yields that an MDP's adaptivity gap is at most the covariance magnitude of any optimal policy, and gives a rate-value tradeoff when per-step mutual information is capped. The bound states that covariance magnitude is at most a constant times the action span times the square root of mutual information, where action span is the largest range of costs across actions within a single state; the constant cannot be improved, as shown by a binary-symmetric-channel family whose ratio tends to that constant.

Estimating this adaptive value from a policy's own logs is possible under bandit feedback, with sample size governed by an average KL divergence. The audit reduces to off-policy evaluation of the open-loop marginal, since the marginal is directly observable in the logs and all statistical difficulty lies on the off-policy side. A median-of-means inverse-propensity-weighted auditor reaches a given accuracy with high probability within a stated number of rounds; a lower bound shows that for every value there are instances requiring a matching number of rounds, so the two agree up to constants. If the policy never plays an action its marginal would play on a set of contexts of positive measure, the quantity is not identifiable, so a deterministic non-constant policy cannot be audited from its own logs.

The results cover finite-horizon Markov decision processes and Markov policies; the audit setting requires bandit feedback or episodic logs with logged action propensities and a known open-loop marginal. The numerical demonstration is limited to three small examples, including a 20-step lost-sales inventory problem with capacity 12 and orders from 0 to 6.

The information bound is a worst-case bound and leaves slack when the action span is large; the episodic product bound grows exponentially in horizon unless per-step divergences are small, and general off-policy evaluation lower bounds do not apply directly; a deterministic non-constant policy is not identifiable from its own logs.