# Vision Transformer

10 related articles

Subscribe to updates

arXiv Nov 12, 2026
MRI reconstruction stops learning from clinical raw data: SIM trains on physics simulation and transfers zero-shot across 14 datasets
SIM replaces clinical raw k-space with training samples generated from multi-coil MRI physics, then scales data and model capacity together in a generic Transformer. Across 14 public and in-house datasets spanning seven centers, five vendors, 0.3 to 7 T and more than ten protocols, its zero-shot reconstruction beats the second-best method by 1.06 to 3.95 dB PSNR; in a 150-scan blinded reader study, SIM reached 100% diagnostic usability at an acceleration factor the source does not state.
SIM replaces clinical raw k-space with training samples generated from multi-coil MRI physics, then scales data and model capacity together in a generic Transformer. Across 14 public and in-house datasets spanning seven centers, five vendors, 0.3 to 7 T and more than ten protocols, its zero-shot reconstruction beats the second-best method by 1.06 to 3.95 dB PSNR; in a 150-scan blinded reader study, SIM reached 100% diagnostic usability at an acceleration factor the source does not state.
Peking University Shenzhen Hospital Shenzhen Institutes of Advanced Technology
Hong Shang（First author）
Scalable Inverse Model (SIM)MRI Reconstruction Physics-based Simulation Neural Scaling Laws Vision Transformer
arXiv Nov 5, 2026
Galaxy morphology as a fossil record: Katachi-PRIME infers mass, age and metallicity from three-band images
Katachi-PRIME (形') feeds three-band gri galaxy images together with photometry into implicit likelihood inference to return full posteriors for stellar mass, specific star formation rate, redshift, age, dust attenuation and gas-phase metallicity. On roughly 232,000 GSWLC galaxies, image-only posteriors are tighter than those from 5-band photometry and combining both is best; the same image embeddings show that main-sequence and quiescent morphologies retain star formation history continuously for several Gyr, while starbursts and green-valley galaxies keep only the most recent activity.
Katachi-PRIME (形') feeds three-band gri galaxy images together with photometry into implicit likelihood inference to return full posteriors for stellar mass, specific star formation rate, redshift, age, dust attenuation and gas-phase metallicity. On roughly 232,000 GSWLC galaxies, image-only posteriors are tighter than those from 5-band photometry and combining both is best; the same image embeddings show that main-sequence and quiescent morphologies retain star formation history continuously for several Gyr, while starbursts and green-valley galaxies keep only the most recent activity.
Johns Hopkins University University of Cambridge Tohoku University
Juan Pablo Alfonzo（First author）
Katachi-PRIME Vision Transformer Neural Posterior Estimation Star Formation Histories
bioRxiv Oct 14, 2026
SVC treats each gene's intracellular spread as an image patch: other genes in the same cell supply the context for predicting where unmeasured transcripts sit
Subcellular-resolution spatial transcriptomics can locate transcripts inside cells, but earlier methods targeted single analysis tasks and lacked a unified framework for characterizing and predicting gene localization in a cell-specific, context-aware way. SVC represents each gene's within-cell spatial expression as an image patch and uses other genes in the same cell as context, learning cross-gene spatial dependencies through self-supervised masking. On seqFISH+ mouse embryonic fibroblast data, SVC beat four image-based baselines on all six metrics for predicting subcellular patterns of unmeasured genes, with median gains of 5.2%–21.0% over the more challenging baseline pair.
Subcellular-resolution spatial transcriptomics can locate transcripts inside cells, but earlier methods targeted single analysis tasks and lacked a unified framework for characterizing and predicting gene localization in a cell-specific, context-aware way. SVC represents each gene's within-cell spatial expression as an image patch and uses other genes in the same cell as context, learning cross-gene spatial dependencies through self-supervised masking. On seqFISH+ mouse embryonic fibroblast data, SVC beat four image-based baselines on all six metrics for predicting subcellular patterns of unmeasured genes, with median gains of 5.2%–21.0% over the more challenging baseline pair.
Texas A&M University Yale University
Hui Song Wan（First author）Xiang Zhou（Corresponding author）
SVC Spatial transcriptomics Vision Transformer Masked Image Modeling
Hugging Face - Blog Oct 23, 2026
OlmoEarth v1.3 turns a year of satellite data into one vector per pixel, leading 20 of 27 evaluations
Ai2 released OlmoEarth v1.3, a satellite embedding model that compresses a year of radar and multispectral time series into one 128-dimensional vector per 10-meter pixel, with precomputed annual embeddings for the whole planet shared openly. Until now, using OlmoEarth mainly meant fine-tuning a model per task in OlmoEarth Studio, which requires GPU access and storing the imagery. Against AlphaEarth Foundations and Tessera v2 across 27 evaluations on six continents, the 128-dimensional embeddings scored highest on 20 and the 64-dimensional version led on 18.
Ai2 released OlmoEarth v1.3, a satellite embedding model that compresses a year of radar and multispectral time series into one 128-dimensional vector per 10-meter pixel, with precomputed annual embeddings for the whole planet shared openly. Until now, using OlmoEarth mainly meant fine-tuning a model per task in OlmoEarth Studio, which requires GPU access and storing the imagery. Against AlphaEarth Foundations and Tessera v2 across 27 evaluations on six continents, the 128-dimensional embeddings scored highest on 20 and the 64-dimensional version led on 18.
OlmoEarth v1.3 Satellite embedding model OlmoEarth Embeddings Explorer PLANTEUR Vision Transformer
arXiv Oct 12, 2026
Event cameras stop reading one frame at a time: splitting long streams into multiple frames lifts THU-EACT-50 CHL accuracy from 30.5% to 78.7%
Sweden's defence research agency splits event streams into multiple frames before pre-training, reaching 78.7% on THU-EACT-50 CHL and 66.9% on DailyDVS-200 for long, complex actions using event data alone.
Swedish Defence Research Agency
Emma Norrby（First author）
Event-based Action Recognition Masked Autoencoder Pre-training Multi-Frame Event Representation Sparsity-aware Masking Activity-based Token Selection
arXiv Oct 7, 2026
REViT-v2 cuts roto-reflection equivariant attention cost from quadratic to linear via windowed group-convolutional self-attention, beating rotation/flip-augmented ViT-S and RE-ResNet on ImageNet-1K
The authors propose REViT-v2, which restricts group-convolutional self-attention to local windows (wG-CSA) and builds a multi-stage equivariant feature pyramid from group-convolutional tokenization and downsampling, reducing the spatial complexity of roto-reflection equivariant attention from quadratic to linear in image area; on ImageNet-1K the equivariant REViT-v2-S reaches 79.27% and 80.9% Top-1, above rotation/flip-augmented ViT-S (72.08%) and RE-ResNet (77.37%), and on Rotated MNIST it keeps about 98.26% accuracy with smaller model size, FLOPs, latency, and peak memory.
The authors propose REViT-v2, which restricts group-convolutional self-attention to local windows (wG-CSA) and builds a multi-stage equivariant feature pyramid from group-convolutional tokenization and downsampling, reducing the spatial complexity of roto-reflection equivariant attention from quadratic to linear in image area; on ImageNet-1K the equivariant REViT-v2-S reaches 79.27% and 80.9% Top-1, above rotation/flip-augmented ViT-S (72.08%) and RE-ResNet (77.37%), and on Rotated MNIST it keeps about 98.26% accuracy with smaller model size, FLOPs, latency, and peak memory.
Sheir Zaheer（First author）
REViT-v2 Windowed Group Convolutional Self-Attention Group-Equivariant Vision Transformer Vision Transformer
bioRxiv Sep 29, 2026
SpatialTRACE extends sparse annotations into tissue-wide anatomical axis and region maps and predicts the same coordinates from DAPI images alone
The authors developed SpatialTRACE, comprising graph- and image-based models: SpatialTRACE-Graph combines gene-expression profiles with spatial neighborhoods to predict crypt-villus and epithelial-distance axis coordinates and to identify Peyer's patches in mouse small-intestine sections using as few as 10 annotated training villi, while SpatialTRACE-Image, a multiscale vision transformer that learns from the coordinate and region predictions generated by SpatialTRACE-Graph, predicts the same anatomical axis coordinates and regions across entire tissue images from DAPI alone and was applied to immunofluorescence images to map antigen-specific P14 CD8 T cells responding to acute systemic LCMV Armstrong infection in the small intestine, showing that a retinoic acid receptor inhibitor-treated
The authors developed SpatialTRACE, comprising graph- and image-based models: SpatialTRACE-Graph combines gene-expression profiles with spatial neighborhoods to predict crypt-villus and epithelial-distance axis coordinates and to identify Peyer's patches in mouse small-intestine sections using as few as 10 annotated training villi, while SpatialTRACE-Image, a multiscale vision transformer that learns from the coordinate and region predictions generated by SpatialTRACE-Graph, predicts the same anatomical axis coordinates and regions across entire tissue images from DAPI alone and was applied to immunofluorescence images to map antigen-specific P14 CD8 T cells responding to acute systemic LCMV Armstrong infection in the small intestine, showing that a retinoic acid receptor inhibitor-treated
University of California San Diego Allen Institute
Alexander Monell（First author）Maximilian Heeg（Corresponding author）
SpatialTRACE Spatial transcriptomics Vision Transformer Anatomical axis and region annotation
Frontiers in Oncology Sep 28, 2026
Bibliometric analysis of 608 AI breast cancer imaging studies finds diagnosis at 71.22% and explainable AI with a 1.00 burst ratio as the hottest frontier
Using Scopus and Web of Science and a PRISMA workflow that narrowed 4,831 records to 608 peer-reviewed journal articles and reviews from 2020 to 2026, this study applied Bibliometrix and VOSviewer for a task-aware, methodology-centric bibliometric and thematic analysis, finding that diagnosis accounts for 71.22% of studies, mammography for 42.11%, explainable AI shows the strongest burst ratio at 1.00, while treatment-response prediction covers only 2.30% and only about 15-25% of studies explicitly report hyperparameter tuning strategies.
Using Scopus and Web of Science and a PRISMA workflow that narrowed 4,831 records to 608 peer-reviewed journal articles and reviews from 2020 to 2026, this study applied Bibliometrix and VOSviewer for a task-aware, methodology-centric bibliometric and thematic analysis, finding that diagnosis accounts for 71.22% of studies, mammography for 42.11%, explainable AI shows the strongest burst ratio at 1.00, while treatment-response prediction covers only 2.30% and only about 15-25% of studies explicitly report hyperparameter tuning strategies.
University of Louisville Assiut University Mansoura University Taibah University Midway University
Yathreb Bayan Mohamed（First author）Mohamed Shehata（Corresponding author）Mostafa A. Elhosseini（Corresponding author）
Bibliometric Analysis Systematic Review AI-based Breast Cancer Detection Explainable AI Hyperparameter Optimization
arXiv Sep 25, 2026
ICHOR self-supervised pretraining on 11,405 ASL CBF scans outperforms structural-MRI pretrained baselines across four downstream tasks
The study introduces ICHOR, a self-supervised pretraining framework for ASL cerebral blood flow (CBF) maps based on 3D masked autoencoders (a ViT-Base encoder with a light decoder), pretrained on 11,405 ASL CBF scans from 14 studies spanning multiple sites and protocols, and evaluated on three diagnostic classification tasks plus one ASL CBF map quality prediction regression task, where it outperformed structural-MRI pretrained baselines BrainIAC, BrainSegFounder, and MedicalNet overall across all four tasks.
The study introduces ICHOR, a self-supervised pretraining framework for ASL cerebral blood flow (CBF) maps based on 3D masked autoencoders (a ViT-Base encoder with a light decoder), pretrained on 11,405 ASL CBF scans from 14 studies spanning multiple sites and protocols, and evaluated on three diagnostic classification tasks plus one ASL CBF map quality prediction regression task, where it outperformed structural-MRI pretrained baselines BrainIAC, BrainSegFounder, and MedicalNet overall across all four tasks.
University of Pennsylvania Philadelphia University University of Maryland, Baltimore
Xavier Beltran-Urbano（First author）Sudipto Dolui（Corresponding author）
ICHOR Masked Image Modeling ASL CBF Maps Vision Transformer LoRA
arXiv Sep 25, 2026
TranSamba uses cross-plane Mamba to push weakly supervised volumetric segmentation to 64.1% DSC, topping BraTS, KiTS and LASC
The work proposes TranSamba, which places a Cross-Plane Mamba (CPM) block before the Transformer block in every layer of a DeiT-S Vision Transformer encoder so that patch tokens in neighboring planes exchange information in linear time, improving in-plane self-attention and the class-to-patch attention maps; trained with only plane-level labels, 16 contiguous planes per volume and 16 volumes per batch for 100 epochs, it reaches 64.1%, 26.8% and 41.0% DSC and 46.8%, 25.8% and 25.6% IoU on the BraTS, KiTS and LASC test sets, the highest among the compared methods, with time complexity linear in the number of planes and batch space complexity independent of it.
The work proposes TranSamba, which places a Cross-Plane Mamba (CPM) block before the Transformer block in every layer of a DeiT-S Vision Transformer encoder so that patch tokens in neighboring planes exchange information in linear time, improving in-plane self-attention and the class-to-patch attention maps; trained with only plane-level labels, 16 contiguous planes per volume and 16 volumes per batch for 100 epochs, it reaches 64.1%, 26.8% and 41.0% DSC and 46.8%, 25.8% and 25.6% IoU on the BraTS, KiTS and LASC test sets, the highest among the compared methods, with time complexity linear in the number of planes and batch space complexity independent of it.
Harry Perkins Institute of Medical Research Victor Chang Cardiac Research Institute The University of Western Australia
Yiheng Lyu（First author, Corresponding author）
TranSamba Cross-Plane Mamba Weakly Supervised Volumetric Medical Segmentation Vision Transformer Mamba

You’re all caught up