Researchers propose a counterfactual-information criterion and build a single simple environment showing that optimal policies for count-based, prediction-error, empowerment, and information-gain intrinsic rewards are Pareto-suboptimal under that criterion
The work proposes a formal exploration criterion that compares policies by the counterfactual information they acquire, meaning how well their histories can substitute for experience under alternative policies, and constructs a single simple environment in which the maximizing policies of specified count-based, prediction-error, empowerment, and information-gain objectives are Pareto-suboptimal at acquiring counterfactual information; it then explains these failures, establishes conditions under which existing intrinsic rewards successfully encourage optimal exploration, and constructs an objective that assigns higher value whenever exploration strictly improves under the criterion.