ELLIS Institute Tübingen

The work introduces DSReg (Dependency-Sparsity Regularization), which, on top of the linear-identifiability premise provided by LeJEPA, proves that under Structural Diversity—distinct dependency footprints of latents on observations—each world latent can be recovered up to signed permutation with no reconstruction, no decoder, and no labels; the method applies post hoc to any linearly identified representation, reuses trained checkpoints, and across synthetic regimes, world-model probes, pixel-trained encoders, and external renderers preserves dense prediction while improving individual-latent recovery and downstream sparse control, editing, and few-shot readout.

The work introduces DSReg (Dependency-Sparsity Regularization), which, on top of the linear-identifiability premise provided by LeJEPA, proves that under Structural Diversity—distinct dependency footprints of latents on observations—each world latent can be recovered up to signed permutation with no reconstruction, no decoder, and no labels; the method applies post hoc to any linearly identified representation, reuses trained checkpoints, and across synthetic regimes, world-model probes, pixel-trained encoders, and external renderers preserves dense prediction while improving individual-latent recovery and downstream sparse control, editing, and few-shot readout.

The work proposes a more realistic threat model for distillation attacks in which attackers continue training with reinforcement learning after distillation, and shows experimentally that distillation followed by RL outperforms either alone, that a well-known defense (antidistillation sampling) can appear effective after distillation but be broken after RL, and that distilling on approximate reasoning traces reconstructed from closed-source API summaries and final answers, followed by RL, yields reasoning gains similar to using full hidden traces.

The work proposes a more realistic threat model for distillation attacks in which attackers continue training with reinforcement learning after distillation, and shows experimentally that distillation followed by RL outperforms either alone, that a well-known defense (antidistillation sampling) can appear effective after distillation but be broken after RL, and that distilling on approximate reasoning traces reconstructed from closed-source API summaries and final answers, followed by RL, yields reasoning gains similar to using full hidden traces.

The work introduces Complex KDA (CKDA): extending Kimi Delta Attention by allowing signed gate entries and an extended delta-rule coefficient range so that a single diagonal-plus-rank-one transition can realize a 2D rotation; the authors prove every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition, show one CKDA layer tracks every finite group isomorphic to a subgroup of O(2), save one layer relative to comparable diagonal-plus-rank-one linear RNNs, and report experiments on group-word problems, periodic audio continuation, and language modeling.

The work introduces Complex KDA (CKDA): extending Kimi Delta Attention by allowing signed gate entries and an extended delta-rule coefficient range so that a single diagonal-plus-rank-one transition can realize a 2D rotation; the authors prove every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition, show one CKDA layer tracks every finite group isomorphic to a subgroup of O(2), save one layer relative to comparable diagonal-plus-rank-one linear RNNs, and report experiments on group-word problems, periodic audio continuation, and language modeling.

The authors introduce PlaneCycle, a parameter-free, training-free, adapter-free 2D-to-3D lifting operator that lets a pretrained 2D backbone acquire 3D fusion by cyclically distributing spatial aggregation across the orthogonal HW, DW, and DH planes throughout network depth without modifying any pretrained parameter; using DINOv3 ViT-S/16, ViT-B/16, and ViT-L/16 on six 3D classification and three 3D segmentation benchmarks, PCg reaches 87.8 average AUC and 82.0 average ACC on ViT-B/16 under linear probing, surpassing slice-wise 2D and 3D-flattening baselines with paired t-test significance (p<0.05) on 5/6 datasets, and after full fine-tuning it matches standard 3D architectures, exceeding 3D flattening by up to 2.6 Dice points on segmentation while retaining 2D-level attention complexity.

The authors introduce PlaneCycle, a parameter-free, training-free, adapter-free 2D-to-3D lifting operator that lets a pretrained 2D backbone acquire 3D fusion by cyclically distributing spatial aggregation across the orthogonal HW, DW, and DH planes throughout network depth without modifying any pretrained parameter; using DINOv3 ViT-S/16, ViT-B/16, and ViT-L/16 on six 3D classification and three 3D segmentation benchmarks, PCg reaches 87.8 average AUC and 82.0 average ACC on ViT-B/16 under linear probing, surpassing slice-wise 2D and 3D-flattening baselines with paired t-test significance (p<0.05) on 5/6 datasets, and after full fine-tuning it matches standard 3D architectures, exceeding 3D flattening by up to 2.6 Dice points on segmentation while retaining 2D-level attention complexity.