Exploration in physical control stops looking one step ahead: HTC2 scores multi-step futures and tops mean return in eleven of twelve settings
Synopsis
Exploration in physical control often relies on immediate prediction error or single-step physical change, which can miss information that only emerges over extended interaction. HTC2 trains a horizon-conditioned contrastive critic in a learned Hamiltonian phase space, uses the representation distance between a state-action and its same-episode multi-step futures as the primary curiosity signal, and modulates it with bounded multi-step Hamiltonian variation. Combined with SAC and PPO on six DeepMind Control Suite tasks, HTC2 achieves the highest mean final return in eleven of twelve settings.
Interpretation
The exploration signal in physical control is no longer set by immediate prediction error or single-step physical change, but by the representation mismatch between a state-action and its multi-step futures. Earlier intrinsic-motivation methods rely on immediate prediction error, state novelty or local physical change, and single-step Hamiltonian-change curiosity stays local in time, since a large instantaneous variation need not mean an action reaches an informative future over longer horizons. On six DeepMind Control Suite tasks, HTC2 with SAC and PPO achieves the highest mean final return in eleven of twelve settings, ranking first on five of six tasks under SAC and on all six under PPO.
The curiosity signal comes from a horizon-conditioned contrastive critic that learns relations between a current state-action and future states within the same episode. The critic is trained with an InfoNCE objective, with the matched future as the positive and other minibatch futures as negatives, and the positive-pair representation distance is taken as the intrinsic reward. Horizons are sampled from a truncated geometric distribution over 1 to 16, only same-episode anchor-future pairs are kept, and the reward is multiplied by a gate from bounded multi-step Hamiltonian variation, with a latent RND warm start early in training.
Physical structure enters the reward as a modulation rather than a replacement, with Hamiltonian variation only reweighting the temporal contrastive signal within bounds. Directly rewarding change in the learned Hamiltonian can favor local perturbations rather than temporally extended discovery. In ablations on walker_walk, hopper_hop and cheetah_run, the full configuration has the highest mean final return, and replacing the temporal contrastive reward with one-step Hamiltonian change produces the largest drop on hopper_hop.
Perspective
The reported scope is six DeepMind Control Suite tasks (walker_walk, walker_run, walker_stand, hopper_hop, hopper_stand, cheetah_run), with HTC2 evaluated alongside both SAC and PPO and compared against ICM, RIDE, E3B, Plan2Explore and a single-step Hamiltonian-change baseline; physical-state coverage is measured on cartpole_swingup, ball_in_cup_catch and finger_spin, and parametric out-of-distribution retention on walker_walk, hopper_stand and cheetah_run under gravity, friction, mass and damping shifts.
Coverage results differ by environment: HTC2 has the highest final coverage on cartpole_swingup and finger_spin, while RIDE has the highest final coverage on ball_in_cup_catch and HTC2 ranks second in coverage AUC. Under parametric out-of-distribution shifts HTC2 has the best retention in seven of twelve settings, and friction and damping shifts are more task-dependent. In ablations, the one-step Hamiltonian-change variant fails to learn effective hopping on hopper_hop.