Weakening the skip connection causes a depth-induced rank collapse that no loss term repairs, while restoring the skip reopens the gradient path
The authors collapsed small transformers into depth-induced rank collapse by weakening the skip connection and then treated copies with two rank-based loss terms differing only in whether their gradient vanishes; neither repaired the collapse even though the bounded force reached about a tenth of the task gradient, because the task gradient no longer reached the query and key weights that decide where attention looks, whereas restoring the skip connection, a runtime action changing no weight, reopened that path at once, after which the rank recovered but only far above the scale of collapse and with the loss still above a healthy network.