Why curriculum learning pays off in reinforcement learning: sparse rewards can stall learning without one, while dense rewards gain little
Synopsis
Curriculum learning often helps more in reinforcement learning than in supervised learning, and a tractable target-interception model now explains why. With dense rewards, curricula bring only marginal gains; with sparse rewards, a curriculum can be required for learning to occur at all. When the agent must both minimize effort and accumulate reward, competing optima appear and can trap it in a quitting solution. Useful curricula do two things: they make rewards less sparse to speed learning, and they shape the reward landscape to steer agents away from bad local optima.
Interpretation
In a tractable target-interception task, whether a curriculum is necessary depends on reward sparsity: with dense rewards curricula bring only marginal benefits, while with sparse rewards learning may not occur at all without one. Curriculum learning often yields little improvement in supervised settings but can be highly effective in reinforcement learning, and existing analytical studies predict at best modest effects. The average REINFORCE learning dynamics are analyzed mathematically and numerically under two return settings, one with dense rewards and one with sparse rewards.
Learning is written as gradient ascent on a set of policy parameters, and the sparser the reward, the less often the agent receives informative feedback and the slower convergence becomes. In the pure-bias case without an effort term, convergence time is set by the ratio of initial bias distance to the effective target width, and grows exponentially once that ratio is somewhat larger than one. Convergence time is expressed exactly through the exponential integral and expanded asymptotically for large ratios, giving exponential rather than logarithmic convergence time.
When the agent must both minimize effort and accumulate high reward, the effective return has two local maxima, a quitting solution near zero and one near the target action, and agents initialized in the quitting basin get stuck. The phase diagram in the pure-bias case is characterized analytically, with saddle-node bifurcations reduced to the roots of a cubic polynomial. A critical temperature parameter separates a regime with a single optimum regardless of the effort coefficient from one with three phases: one stable fixed point, two stable fixed points, and a trivial regime where seeking the target is not optimal.
Perspective
The conclusions rest on an open-loop target-interception model with a single action per episode, a linear architecture and little meaningful representation learning, analyzed through average learning dynamics in the limit of vanishing learning rate. The authors explicitly leave out closed-loop environments, representation learning in recurrent networks, composable learning primitives and value learning.
The authors offer as speculation that curriculum learning's limited effect in supervised learning reflects fairly informative gradients, making those dynamics more like the dense case than the sparse case. The model also assumes an open-loop, single-action, linear architecture, and the authors expect each of these aspects to make theory substantially harder and to require new ideas.