# Rotary Position Embedding

Multimodal Rotary Position Embedding (MRoPE), widely used in vision-language models, assigns height and width coordinates to text tokens that have no inherent spatial structure, adding positional preferences to text-vision attention. The study separates intra-visual spatial encoding from cross-modal spatial effects, probing official pretrained checkpoints without training and running controlled training at 2B and 4B from identical initial parameters. Cross-modal spatial effects provide little extra benefit as spatial references and can even impair performance, while intra-visual spatial encoding remains useful for fine-grained grounding.

Multimodal Rotary Position Embedding (MRoPE), widely used in vision-language models, assigns height and width coordinates to text tokens that have no inherent spatial structure, adding positional preferences to text-vision attention. The study separates intra-visual spatial encoding from cross-modal spatial effects, probing official pretrained checkpoints without training and running controlled training at 2B and 4B from identical initial parameters. Cross-modal spatial effects provide little extra benefit as spatial references and can even impair performance, while intra-visual spatial encoding remains useful for fine-grained grounding.

Detecting many-body entanglement no longer requires reconstructing the full density matrix or relying on a predefined witness. EntNet turns two- to four-body Pauli correlations on a one-dimensional spin chain into a representation invariant under local basis rotations, sparsely selects 16 feature channels, and learns their spatial arrangement with a Transformer using rotary positional embeddings. On a finite-temperature XY benchmark it reaches 100% AUROC from expectation values of these low-order observables alone and holds the false-positive rate to 0.003 on 2,000 constructively separable states; the same model transfers without retraining to XXZ, frustrated J1-J2 and random local Hamiltonians and to longer chains.

Detecting many-body entanglement no longer requires reconstructing the full density matrix or relying on a predefined witness. EntNet turns two- to four-body Pauli correlations on a one-dimensional spin chain into a representation invariant under local basis rotations, sparsely selects 16 feature channels, and learns their spatial arrangement with a Transformer using rotary positional embeddings. On a finite-temperature XY benchmark it reaches 100% AUROC from expectation values of these low-order observables alone and holds the false-positive rate to 0.003 on 2,000 constructively separable states; the same model transfers without retraining to XXZ, frustrated J1-J2 and random local Hamiltonians and to longer chains.

The authors propose WavePrune, a training-free method that restricts each RoPE channel to its first rotation period, removing position-aliasing "ghost" sub-diagonals; it raises HELMET scores on four of five long-context models (35.7 to 40.0 on Qwen3-8B), lowers validation loss at extrapolated lengths when pretraining from scratch, and, via FlashWavePrune, delivers 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context.

The authors propose WavePrune, a training-free method that restricts each RoPE channel to its first rotation period, removing position-aliasing "ghost" sub-diagonals; it raises HELMET scores on four of five long-context models (35.7 to 40.0 on Qwen3-8B), lowers validation loss at extrapolated lengths when pretraining from scratch, and, via FlashWavePrune, delivers 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context.

The authors introduce a theoretical framework for contextual influence through inference dynamics, showing that under a controlled single-round setting repeated contextual assertions do not cause unbounded drift but instead drive internal inference trajectories to stable, query-dependent limits, so repetition does not act as accumulating evidence and may fail to change a prediction or make a flip inevitable; they validate representation-level dynamics against output-level answer shifts across six models and several QA benchmarks.

The authors introduce a theoretical framework for contextual influence through inference dynamics, showing that under a controlled single-round setting repeated contextual assertions do not cause unbounded drift but instead drive internal inference trajectories to stable, query-dependent limits, so repetition does not act as accumulating evidence and may fail to change a prediction or make a flip inevitable; they validate representation-level dynamics against output-level answer shifts across six models and several QA benchmarks.

The authors propose MAGEFormer, which embeds physical spacing constraints into representation learning: MASE calibrates positional frequencies by voxel spacing, GCA suppresses physically implausible cross-scale feature correlations, and GVV performs deterministic sub-voxel view aggregation at inference; under a unified 5-fold protocol on BTCV and FLARE 22 it attains the strongest boundary accuracy among compared methods (10.58 mm and 3.40 mm HD95) with consistent Dice gains, and its advantage grows with anisotropy severity.

The authors propose MAGEFormer, which embeds physical spacing constraints into representation learning: MASE calibrates positional frequencies by voxel spacing, GCA suppresses physically implausible cross-scale feature correlations, and GVV performs deterministic sub-voxel view aggregation at inference; under a unified 5-fold protocol on BTCV and FLARE 22 it attains the strongest boundary accuracy among compared methods (10.58 mm and 3.40 mm HD95) with consistent Dice gains, and its advantage grows with anisotropy severity.

The work identifies context reliance in unstructured knowledge editing, where the standard passage-level objective gives later facts richer ground-truth context and thus lower initial losses so they appear easier to learn; it proposes FOVEATED, a plug-and-play framework that constructs focused views of each sentence by randomly shifting the RoPE positions assigned to the keys of its preceding context during editing and removing the perturbation afterward so native positional encoding is unchanged at inference, instantiated for both direct-optimization and locate-then-edit editors, with a theoretical analysis of how it counteracts context-induced difficulty underestimation and empirical improvements across five KE editors, two LLM backbones, and three benchmarks.

The work identifies context reliance in unstructured knowledge editing, where the standard passage-level objective gives later facts richer ground-truth context and thus lower initial losses so they appear easier to learn; it proposes FOVEATED, a plug-and-play framework that constructs focused views of each sentence by randomly shifting the RoPE positions assigned to the keys of its preceding context during editing and removing the perturbation afterward so native positional encoding is unchanged at inference, instantiated for both direct-optimization and locate-then-edit editors, with a theoretical analysis of how it counteracts context-induced difficulty underestimation and empirical improvements across five KE editors, two LLM backbones, and three benchmarks.