Gated RNNs stall on who learns the gates: mixing short-dependency samples into training lets GRU and LSTM catch Mamba and xLSTM
The long-range memory bottleneck in gated recurrent networks is traced to the gates themselves having to be learned by the same recurrent computation, and adding short-dependency samples to training brings GRU and LSTM long-dependency test accuracy from near chance back to roughly ninety percent or higher, matching or exceeding Mamba and xLSTM.