Zehao Jin

The work measures how far pretrained transformers follow in-context reference chains: thirteen base models reliably follow only 1.4–3.6 lines (median 2.2) and extra pretrained loops add little, yet with all model weights frozen a task-trained rank-8 LoRA at one early layer extends this computation, raising Qwen3-8B from 15.5% to 99% exact accuracy on 24-line chains, reaching 50 lines with a longer-trained LoRA and, in Ouro-1.4B, 60 lines after four loops and at least 160 after eight, via a relay in the middle layers where frozen heads read progressively further up the chain and removing parent-line attention stops it.

The work measures how far pretrained transformers follow in-context reference chains: thirteen base models reliably follow only 1.4–3.6 lines (median 2.2) and extra pretrained loops add little, yet with all model weights frozen a task-trained rank-8 LoRA at one early layer extends this computation, raising Qwen3-8B from 15.5% to 99% exact accuracy on 24-line chains, reaching 50 lines with a longer-trained LoRA and, in Ouro-1.4B, 60 lines after four loops and at least 160 after eight, via a relay in the middle layers where frozen heads read progressively further up the chain and removing parent-line attention stops it.