Looped language models stop paying the one-loop tax: one checkpoint matches its non-recurrent baseline at one loop and adds 7.25 points at four
Looped language models used to trail a non-recurrent model of the same size at one loop; a single checkpoint now matches a non-recurrent baseline trained on the same data at one loop and gains 7.25 points on a ten-task average at four loops.