Researchers propose a counterfactual-information criterion and build a single simple environment showing that optimal policies for count-based, prediction-error, empowerment, and information-gain intrinsic rewards are Pareto-suboptimal under that criterion

The paper proposes a formal criterion for exploration that compares policies by the counterfactual information they acquire, that is, how well their histories can substitute for experience under alternative policies. Relative to the common practice of judging exploration by the magnitude of intrinsic rewards such as prediction error or learning progress, the criterion shifts evaluation toward how well a history substitutes for experience under alternative policies. The evidence is the formal criterion stated in the abstract, a conceptual and formal contribution; the abstract provides no theorem numbers or proof details.

The paper constructs a single, simple environment in which the maximizing policies of specified count-based, prediction-error, empowerment, and information-gain objectives are Pareto-suboptimal at acquiring counterfactual information. This shows that maximizing these intrinsic rewards need not produce the most informative experience available, placing the intuition that intrinsic rewards guide exploration under a testable counterexample. The evidence is one constructed environment and four specified objective families as described in the abstract; the abstract reports no environment size, state counts, or quantitative metrics.

The paper explains these failures and establishes conditions under which existing intrinsic rewards successfully encourage optimal exploration. Beyond offering counterexamples, the work characterizes when intrinsic rewards are effective, extending the result from a negative example to a conditional account. The evidence is the abstract's statement about explaining failures and establishing conditions; the specific form of the conditions and the derivations are not expanded in the abstract.

The paper constructs an objective that assigns a higher value whenever exploration strictly improves under the proposed criterion. This provides an objective construction aligned with the counterfactual-information criterion rather than only diagnosing shortcomings of existing intrinsic rewards. The evidence is the abstract's statement of this objective construction; the abstract gives no optimization properties or experimental validation.

The work addresses the design and evaluation of exploration objectives in reinforcement learning: for researchers and engineers using intrinsic rewards such as count-based, prediction-error, empowerment, or information-gain objectives, it offers a comparison framework based on counterfactual information and a constructed environment for testing whether an objective aligns with information acquisition. The constructed objective points toward assigning higher value when exploration strictly improves under the criterion, suited to settings that separate exploration evaluation from reward maximization. Because the currently visible text is the abstract, the results should be understood as formal conclusions within the scope of that single simple environment and the specified objectives.

Readers would still want to know how the criterion is operationalized across policy classes and information measures; whether the Pareto-suboptimality result in the single simple environment extends to more complex settings; how strongly the conditions for effective intrinsic rewards constrain concrete algorithms; and how tractable and empirically effective the newly constructed objective is. Because the currently visible text is the abstract and contains no figures, theorems, or experimental details, these questions cannot be settled at the abstract level and remain open questions for the full text.