# Intrinsic Rewards

A user simulator now writes a first-person evaluation of the assistant before producing its next line, including a 1-to-5 rating and a critique-or-praise judgment; on real user-ChatGPT logs it tracks human users better than prompted simulators of the same size, and it lets multi-turn RL train assistants to comparable quality with less than half the compute.

A user simulator now writes a first-person evaluation of the assistant before producing its next line, including a 1-to-5 rating and a critique-or-praise judgment; on real user-ChatGPT logs it tracks human users better than prompted simulators of the same size, and it lets multi-turn RL train assistants to comparable quality with less than half the compute.

The work proposes a formal exploration criterion that compares policies by the counterfactual information they acquire, meaning how well their histories can substitute for experience under alternative policies, and constructs a single simple environment in which the maximizing policies of specified count-based, prediction-error, empowerment, and information-gain objectives are Pareto-suboptimal at acquiring counterfactual information; it then explains these failures, establishes conditions under which existing intrinsic rewards successfully encourage optimal exploration, and constructs an objective that assigns higher value whenever exploration strictly improves under the criterion.

The work proposes a formal exploration criterion that compares policies by the counterfactual information they acquire, meaning how well their histories can substitute for experience under alternative policies, and constructs a single simple environment in which the maximizing policies of specified count-based, prediction-error, empowerment, and information-gain objectives are Pareto-suboptimal at acquiring counterfactual information; it then explains these failures, establishes conditions under which existing intrinsic rewards successfully encourage optimal exploration, and constructs an objective that assigns higher value whenever exploration strictly improves under the criterion.