# Vanishing Gradient Problem

The long-range memory bottleneck in gated recurrent networks is traced to the gates themselves having to be learned by the same recurrent computation, and adding short-dependency samples to training brings GRU and LSTM long-dependency test accuracy from near chance back to roughly ninety percent or higher, matching or exceeding Mamba and xLSTM.

The long-range memory bottleneck in gated recurrent networks is traced to the gates themselves having to be learned by the same recurrent computation, and adding short-dependency samples to training brings GRU and LSTM long-dependency test accuracy from near chance back to roughly ninety percent or higher, matching or exceeding Mamba and xLSTM.

The authors introduce SpatialRNN, in which recurrent neuron-to-neuron communication is replaced by a spatially evolving medium governed by discretized PDEs, and prove that the medium state equals a convolution over the entire history of hidden states, i.e. a structured infinite-order RNN with a fixed parameter count; constructive conditions confine gradient modes to circles of prescribed radii, yielding better long-horizon performance than other recurrent models with fewer parameters.

The authors introduce SpatialRNN, in which recurrent neuron-to-neuron communication is replaced by a spatially evolving medium governed by discretized PDEs, and prove that the medium state equals a convolution over the entire history of hidden states, i.e. a structured infinite-order RNN with a fixed parameter count; constructive conditions confine gradient modes to circles of prescribed radii, yielding better long-horizon performance than other recurrent models with fewer parameters.