Genes no longer queue by index in attention: SPEAR folds co-expression geometry and perturbation-time repositioning into rotation angles
Synopsis
Transformer models for single-cell perturbation prediction need a position for each gene, yet genes have no natural order, and static co-expression masks cannot reflect the regulatory rewiring a perturbation induces. SPEAR replaces that position with continuous coordinates from the co-expression graph Laplacian and, at every layer, re-derives a scalar position from the perturbation-conditioned hidden state, summing both into one rotation angle. On genetic, pharmacological and cytokine perturbation benchmarks it beats baselines including scDFM on differential-expression direction and magnitude, with the largest gap on held-out combinations.
Interpretation
A gene's position in attention is no longer set by chromosomal index or a fixed co-expression switch, but by continuous geometric coordinates of the co-expression network plus a per-layer, perturbation-dependent displacement. Earlier perturbation models either ordered genes by expression rank or integer index, or restricted attention with a static binary co-expression mask built from unperturbed cells, identical across all cells and conditions. On the ComboSciPlex drug-perturbation benchmark, SPEAR has the lowest reconstruction error across all pointwise metrics and the highest biological fidelity scores, improving on scDFM, which shares the differential transformer backbone and flow-matching objective, on every metric, with the largest relative gains on Spearman-LFC and Pearson-Δ.
The mechanism turns both position signals into rotation angles applied to attention queries and keys, so the influence genes exert on one another rearranges with the perturbation context. Spectral encoding takes a normalized Laplacian eigendecomposition of the co-expression graph, keeps the first K coordinates beyond the zero eigenvector as each gene's continuous position, and maps them to rotation angles through per-head learnable frequency projections. Adaptive repositioning projects each gene's hidden state at every layer through a SwiGLU gate into one scalar position per head and converts it to rotation angles at geometric frequencies; on ComboSciPlex the spectral-only variant gives the largest early gain over the no-position baseline, adaptive repositioning adds gains later in training, and the combined SPEAR reaches the highest values on three metrics at convergence.
Positional distance itself carries readable regulatory information, and a knocked-out hub gene's neighborhood reorganizes. A static co-expression mask counts indirect co-regulation and technical covariation as edges, yielding far more edges than neighbors selected by positional distance. On the Norman additive split, scalar positions from the first cross-attention query module serve as unsupervised edge scores; taking the top 30 neighbors per transcription factor gives roughly double the TRRUST precision of the static baseline with orders of magnitude fewer edges.
Perspective
The demonstrated setting is Norman genetic perturbations in K562 cells, ComboSciPlex drug perturbations in A549 cells and Parse-PBMC cytokine perturbations in peripheral blood mononuclear cells, with evaluation restricted to the top 1,000 highly variable genes and separated into additive and holdout extrapolation regimes.
Spectral coordinates are computed once from the training co-expression graph and frozen, so they cannot cover genes outside the training vocabulary; adaptive positions are computed in a single forward pass, so upstream rotations cannot use feedback from downstream layers; whether the advantage holds under noisier co-expression estimates at full transcriptome scale is left open.