# Transformer

Finite-element surrogates are usually trained and compared by displacement error, although engineering decisions rest on stresses. This study replaces the norm of the supervised loss with the energy norm of the assembled stiffness matrix, leaving everything else fixed; in pre-registered comparisons a three-dimensional transformer's median von Mises error fell 6.1-fold and the two-dimensional mean fell by more than half, also on held-out instances, and training on the energy alone without solutions came close to that accuracy.

Finite-element surrogates are usually trained and compared by displacement error, although engineering decisions rest on stresses. This study replaces the norm of the supervised loss with the energy norm of the assembled stiffness matrix, leaving everything else fixed; in pre-registered comparisons a three-dimensional transformer's median von Mises error fell 6.1-fold and the two-dimensional mean fell by more than half, also on held-out instances, and training on the energy alone without solutions came close to that accuracy.

Beamforming for near-field extreme massive MIMO integrated sensing and communication usually requires repeatedly solving nonconvex constrained optimization, and online latency grows with array size. PATBO fixes the communication and sensing beam directions geometrically and lets a Transformer predict only the normalized stream powers, trained with Cramér–Rao bound (CRB) sensitivity weighting. On a retained test set with 1024 antennas, 4 users and 2 targets, direct inference reaches 19.54% joint feasibility, physics-constrained power repair raises it to 87.79%, and optimizer fallback for the remainder brings it to 100%, at 1.11 ms per sample.

Beamforming for near-field extreme massive MIMO integrated sensing and communication usually requires repeatedly solving nonconvex constrained optimization, and online latency grows with array size. PATBO fixes the communication and sensing beam directions geometrically and lets a Transformer predict only the normalized stream powers, trained with Cramér–Rao bound (CRB) sensitivity weighting. On a retained test set with 1024 antennas, 4 users and 2 targets, direct inference reaches 19.54% joint feasibility, physics-constrained power repair raises it to 87.79%, and optimizer fallback for the remainder brings it to 100%, at 1.11 ms per sample.

A small transformer with a single attention layer was trained to predict the left and right descent sets of elements of the infinite Coxeter group Ã₂, meaning the possible first and last letters of minimal-length words. Test accuracy improved as training data moved from random words to reduced words to ShortLex normal-form words: the random-word and reduced-word datasets each began deteriorating at some length, while the set of all 1,998 ShortLex words of length at most 36 never deteriorated. From the trained model's attention patterns, the researchers formulated and independently proved two word-theory propositions about right and left descent sets.

A small transformer with a single attention layer was trained to predict the left and right descent sets of elements of the infinite Coxeter group Ã₂, meaning the possible first and last letters of minimal-length words. Test accuracy improved as training data moved from random words to reduced words to ShortLex normal-form words: the random-word and reduced-word datasets each began deteriorating at some length, while the set of all 1,998 ShortLex words of length at most 36 never deteriorated. From the trained model's attention patterns, the researchers formulated and independently proved two word-theory propositions about right and left descent sets.

The outcome of a temporary team is often foreshadowed before play begins, yet a single player rating says neither whether that player fits the current role nor whether the five assignments complement one another. This work puts both questions into one deep-learning model: it first reads each player's earlier matches against the current champion and role, then compares the two ten-player lineups as a whole. On a chronologically split validation set, the model predicts the winner before play at 61.33% accuracy, above the TrueSkill, CatBoost, GRU and DraftRec comparisons.

The outcome of a temporary team is often foreshadowed before play begins, yet a single player rating says neither whether that player fits the current role nor whether the five assignments complement one another. This work puts both questions into one deep-learning model: it first reads each player's earlier matches against the current champion and role, then compares the two ten-player lineups as a whole. On a chronologically split validation set, the model predicts the winner before play at 61.33% accuracy, above the TrueSkill, CatBoost, GRU and DraftRec comparisons.

Whether colorectal cancer recurs after resection has been judged mainly from routine pathology such as lymphovascular invasion, perineural invasion, tumor budding and Ki-67. AlphaFISH pairs a sequencing-free nanoprobe biochip with a transformer model to read 41 host transcripts and 51 bacterial genera across 149 colorectal biopsies, and in 28 patients followed for one year after surgery it predicted recurrence or metastasis with an AUC of 0.81, against 0.45 for the combined routine clinical markers. The result comes from a preprint that has not been peer reviewed, on a limited sample.

Whether colorectal cancer recurs after resection has been judged mainly from routine pathology such as lymphovascular invasion, perineural invasion, tumor budding and Ki-67. AlphaFISH pairs a sequencing-free nanoprobe biochip with a transformer model to read 41 host transcripts and 51 bacterial genera across 149 colorectal biopsies, and in 28 patients followed for one year after surgery it predicted recurrence or metastasis with an AUC of 0.81, against 0.45 for the combined routine clinical markers. The result comes from a preprint that has not been peer reviewed, on a limited sample.

The electron density is the central variable of density functional theory, and obtaining it normally means iterating to self-consistency. Two Transformer-inspired networks, SwinUNETR-V2 and its attention-free counterpart NeXtUNETR, predict the density directly, reaching normalized mean absolute density errors of 0.075% and 0.046% on QM9. For NeXtUNETR, a single Kohn-Sham step on the predicted density reproduces the self-consistent electronic structure, with median errors of 0.036 meV in total energy, 0.65 meV/angstrom in forces and 1.8 meV in the HOMO-LUMO gap.

The electron density is the central variable of density functional theory, and obtaining it normally means iterating to self-consistency. Two Transformer-inspired networks, SwinUNETR-V2 and its attention-free counterpart NeXtUNETR, predict the density directly, reaching normalized mean absolute density errors of 0.075% and 0.046% on QM9. For NeXtUNETR, a single Kohn-Sham step on the predicted density reproduces the self-consistent electronic structure, with median errors of 0.036 meV in total energy, 0.65 meV/angstrom in forces and 1.8 meV in the HOMO-LUMO gap.

A two-layer, 427k-parameter transformer shows grokking on the finite-field discrete logarithm map: after a long memorization phase it reaches 99.8% held-out accuracy at p = 113. A 12-model capacity sweep from 90k to 14.4M parameters finds grokking recurring at multiple scales, with a non-monotonic region whose outcome changes when training data increase; hidden states grow aligned with multiplicative-character structure, and prime-order and disjoint-generator controls also generalize.

A two-layer, 427k-parameter transformer shows grokking on the finite-field discrete logarithm map: after a long memorization phase it reaches 99.8% held-out accuracy at p = 113. A 12-model capacity sweep from 90k to 14.4M parameters finds grokking recurring at multiple scales, with a non-monotonic region whose outcome changes when training data increase; hidden states grow aligned with multiplicative-character structure, and prime-order and disjoint-generator controls also generalize.