Gated RNNs stall on who learns the gates: mixing short-dependency samples into training lets GRU and LSTM catch Mamba and xLSTM

Long-range memory in gated recurrent networks depends not only on the gate structure but on whether useful gate values can be learned at all; when every training sample demands long-range memory, the learning signal for the gates decays with sequence length just as it does elsewhere. The standard account of GRU and LSTM holds that additive paths in the gated update give information and gradients a near-identity route, so a forget gate near one or an update gate that keeps the previous state preserves information across long intervals. On a controlled multi-class temporal XOR task, training on long-dependency samples alone makes the vanilla RNN degrade first as sequences lengthen, and GRU and LSTM also fall to near chance, showing that gating does not by itself solve long-range learning.

Exposing the same label rule over shorter temporal gaps lets the shared gates first learn when to write and when to preserve, and that behavior then carries over to long intervals. Gate parameters are shared across sequence positions, so an update learned at one position changes the gates used throughout the sequence; in short-dependency samples the task-relevant inputs sit closer to the final loss, so the learning signal passes through fewer recurrent transformations. In mixed training, where part of the samples in each informative-token group use short-dependency placement, GRU and LSTM hold mean test accuracy of roughly ninety percent or higher across the tested lengths, while the vanilla RNN shows no comparable gain even after its hidden width is raised from the smaller to the larger setting.

The benefit of short-dependency training transfers across classes: giving short-dependency samples to only one informative-token group also markedly improves the other group, which appears exclusively in long-dependency training samples. It was unclear whether gate learning helps only the tokens and labels that receive the support, or whether it acts on a shared memory behavior. With support restricted to one group, the supported group stays above roughly ninety percent mean accuracy across the tested lengths, and the unsupported group rises from the hard-only baseline to roughly seventy to eighty percent, still below the supported group and below the condition where both groups receive support.

The same gate-learning problem appears in minGRU and minLSTM, whose gates depend only on the current input, and with short-dependency samples they reach test accuracy matching or exceeding Mamba and xLSTM. minGRU and minLSTM compute both gates and candidate states from the current input alone, removing dependence on the previous hidden state, but that removal does not by itself guarantee strong learning signals for gates at early positions. On binary temporal XOR, hard-only training drives LSTM, minGRU, and minLSTM toward chance as length grows while Mamba and xLSTM stay above high mean accuracy; adding short-dependency samples brings minGRU and minLSTM to perfect accuracy in every run at every tested length and lifts LSTM markedly.

The report covers controlled synthetic sequence classification: multi-class and binary temporal XOR, where two informative tokens determine the label amid non-informative tokens, and validation and test samples always use long-dependency placement. The findings apply to GRU, LSTM, minGRU, and minLSTM, with Mamba and xLSTM as comparisons, measured in small configurations across repeated multi-seed runs. The authors propose as a next step identifying or introducing comparable short-dependency support for large-scale models trained on real-world data such as language and images.

The diagnostic probes show changes in gradient propagation and selective updates, but their causal role in learning long-range solutions is not isolated; the authors suggest intervening on learned gate functions to clarify which behaviors account for success and partial transfer. It also remains unclear why Mamba and xLSTM learn more reliably from hard samples under the tested configurations. The short-dependency window size shows an inverse-U effect: very small windows lack non-informative context, while very large windows lengthen the dependency again.