# Kimi Delta Attention

LOCI introduces a hybrid spatial-memory architecture in which 15 of 30 transformer blocks keep a key-value cache of past observations while the other 15 are restricted to the current chunk and complemented by a recurrent linear-attention memory whose reads and writes are conditioned on projective camera geometry, with recurrent readouts flowing into subsequent cache-backed blocks to supply their queries with accumulated scene context; on the public MIND memory benchmark and on held-out recorded trajectories it reproduces revisited content more faithfully than representative world models and a same-recipe full-softmax model, lowers peak memory at equal length by about 30% with full history, and streams long videos at constant memory with a bounded bank of retained observations.

LOCI introduces a hybrid spatial-memory architecture in which 15 of 30 transformer blocks keep a key-value cache of past observations while the other 15 are restricted to the current chunk and complemented by a recurrent linear-attention memory whose reads and writes are conditioned on projective camera geometry, with recurrent readouts flowing into subsequent cache-backed blocks to supply their queries with accumulated scene context; on the public MIND memory benchmark and on held-out recorded trajectories it reproduces revisited content more faithfully than representative world models and a same-recipe full-softmax model, lowers peak memory at equal length by about 30% with full history, and streams long videos at constant memory with a bounded bank of retained observations.

The work introduces Complex KDA (CKDA): extending Kimi Delta Attention by allowing signed gate entries and an extended delta-rule coefficient range so that a single diagonal-plus-rank-one transition can realize a 2D rotation; the authors prove every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition, show one CKDA layer tracks every finite group isomorphic to a subgroup of O(2), save one layer relative to comparable diagonal-plus-rank-one linear RNNs, and report experiments on group-word problems, periodic audio continuation, and language modeling.

The work introduces Complex KDA (CKDA): extending Kimi Delta Attention by allowing signed gate entries and an extended delta-rule coefficient range so that a single diagonal-plus-rank-one transition can realize a 2D rotation; the authors prove every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition, show one CKDA layer tracks every finite group isomorphic to a subgroup of O(2), save one layer relative to comparable diagonal-plus-rank-one linear RNNs, and report experiments on group-word problems, periodic audio continuation, and language modeling.