VisionHOPE turns visual backbones into self-modifying learning systems, reaching competitive results on ImageNet-1K, COCO, and ADE20K

It introduces VisionHOPE, formulating a visual backbone as a self-modifying learning system in which five memories (content, key, value, learning rate, retention) co-evolve along visual scans, so the model modifies not only what it remembers but also how it learns. The paper distinguishes this from TTT-based visual backbones, which adapt an inner learner while their representation maps and update rule remain largely outer-parameterized; VisionHOPE follows the self-referential construction of Nested Learning so stored content and the learning rule co-evolve. The claim is supported by the method construction and ablations: on P-VisionHOPE-S, removing adaptive K/V and adaptive dynamics gives 81.9 and 82.0, removing DGD gives 81.9, and the full model gives 82.3.

It identifies the unconstrained self-referential update as a stability barrier in vision and derives a stability-matched step-size control combining a soft cap on self-referential injection with a spectral clamp on the retained memory transition, proving non-expansive memory dynamics along each scan. The paper provides proposition- and corollary-level proofs: complementary operator bounds limit the spectral norm of the retained transition and scale the admissible self-referential injection with the contraction margin, yielding non-expansion for both token-wise and chunk-wise recurrences. Ablations show that removing the soft injection cap causes the forward pass to diverge numerically early in training (NaN), while a hard injection clamp and a soft spectral cap remain stable but score lower than the full scheme (81.8 and 81.9 versus 82.3); the spectral clamp is active on only about 1% of token-wise updates in the full model.

It adapts Nested Learning's chunk formulation to four directional scans by aligning chunks with image rows and columns, processing two-dimensional feature maps with computation linear in token count. Forward and reverse row-major scans use row-aligned chunks, forward and reverse column-major scans use column-aligned chunks, each direction keeps independent states, and directional outputs are fused channel-wise; the chunk formulation allows parallel computation of token-dependent quantities within a chunk. The efficiency analysis gives quadratic interaction and memory terms for DeiT while Vim and VisionHOPE remain linear; measurements show P-VisionHOPE-B keeps linear scaling with resolution and achieves higher throughput and lower memory than Vim-B at similar FLOPs. Ablations show gains from directional coverage (1-way 81.6, 2-way 81.9, 4-way fixed sum 82.1, full 82.3).

On ImageNet-1K classification, COCO detection and instance segmentation, and ADE20K semantic segmentation, VisionHOPE achieves best or tied-best results across the evaluated scales. The paper compares against CNN, ViT, SSM, and TTT baselines in both hierarchical and plain backbone layouts across three tasks, indicating self-modifying learning as a practical foundation for general-purpose visual backbones. On ImageNet-1K, VisionHOPE-T/S/B reach 84.1, 85.2, and 85.6, and plain P-VisionHOPE-T/S/B reach 78.4, 82.3, and 83.4; on COCO, VisionHOPE-T/S/B reach box AP of 47.9, 49.5, and 50.5; on ADE20K, mIoU is 49.4, 50.3, and 51.8.

The result targets general-purpose visual recognition: classification, detection and instance segmentation, and semantic segmentation, in both hierarchical and plain backbone layouts, and it keeps linear computation and memory growth as resolution increases. The paper states that its current evaluation focuses on general-purpose visual recognition, that VisionHOPE has not yet been evaluated as a visual encoder in Multimodal Large Language Models, and that its effectiveness on specialized downstream tasks in medical imaging and remote sensing remains to be empirically validated, with the authors planning to extend to these settings. For a reader, this means the method is currently best suited as an alternative general-purpose visual backbone rather than something to assume works equally well in specialized domains or multimodal understanding.

Open questions a careful reader might watch: the spectral clamp is active on only about 1% of token-wise updates in the full model and removing it leaves accuracy unchanged in this setting, so its role at different data scales or higher resolutions needs more observation; removing the soft injection cap leads to numerical divergence, suggesting the coupling between stability control and training configuration deserves further characterization; the paper reports no empirical results for multimodal large language model encoders, medical imaging, or remote sensing, so effectiveness there remains open; and ablations are mainly run on P-VisionHOPE-S, leaving the sensitivity of design choices at other scales and backbone layouts to be filled in.