Why curriculum learning pays off in reinforcement learning: sparse rewards can stall learning without one, while dense rewards gain little
Curriculum learning often helps more in reinforcement learning than in supervised learning, and a tractable target-interception model now explains why. With dense rewards, curricula bring only marginal gains; with sparse rewards, a curriculum can be required for learning to occur at all. When the agent must both minimize effort and accumulate reward, competing optima appear and can trap it in a quitting solution. Useful curricula do two things: they make rewards less sparse to speed learning, and they shape the reward landscape to steer agents away from bad local optima.