Swapping matrix multiplication for an associative-algebra product: a 110M-parameter model gains 6.2–7.8% generation throughput while GSM8K, MBPP and IFEval all drop

The authors treat the multiplication table itself as an architectural choice: instead of fixing ordinary matrix multiplication and searching for a cheaper evaluation algorithm, they keep the same learned weight blocks and use a sparser interaction table, so the q=2 case needs six block GEMMs rather than eight. Prior fast-matrix-multiplication work, including Strassen and later automated searches, keeps the target product fixed and changes only the algorithm; here the product itself changes while all weight blocks remain independently learned parameters. The paper proves associativity (Proposition 3.1) and argues via the Alder–Strassen bound that the table's bilinear rank is 6, optimal for that table; for q=2 this is compared with rank 7 for Strassen's 2×2 algorithm.

The construction generalizes to a directed-graph family in which vertices are diagonal slots and edges are off-diagonal interaction slots with edge–edge products vanishing; with physical block size fixed and the number of groups growing, square-operand arithmetic becomes quadratic in the matrix dimension. It supplies an algebra family that can grow with layer width and derives finite-shape constraints for GPU execution (for example, the admissible group counts when all three axes must be at least 128), separating algebra size from physical block size as two independent choices. Proposition 5.1 gives the MAC count and the quadratic order and notes that this order requires no scalar-size leaves; Appendix F measures a log–log slope of 2.01 for Q latency across an eightfold increase in size on Qwen3-32B shapes, which the authors describe as a description of those four measured sizes, separate from the exact quadratic arithmetic bound of Proposition 5.1.

The row-typed projections can be realized as rectangular blocks compatible with causal masking and KV-cached decoding, evaluating all row types in parallel during training and only the new position's row during decoding. It carries the algebra down to the Transformer projection level, giving parameter counts, rectangular cost, gradient support and a causality proposition, and shows the weight bank is exposed jointly across row types and acts injectively overall. Proposition 4.1 gives the parameter and cost formulas and Proposition 6.1 proves causality; Appendix B gives vector–Jacobian formulas matching the forward block support, showing forward plus backward equals three forward evaluations in matrix-product MACs.

In a controlled comparison of two approximately 110M-parameter decoder-only LMs over 12.3B tokens differing only in the FFN multiplication law, the algebraic model has higher throughput in all four prompt domains (6.2–7.8%) but lower scores on GSM8K, IFEval and MBPP. This is a small-scale feasibility and trainability check of the construction; the authors explicitly frame it as such rather than as evidence of an advantage at equal quality or training time. One training run per model with identical architecture, data and optimization recipe, differing only in the FFN multiplication law; throughput uses 100 unique prompts per domain with four measurement runs and three retained after warm-up, reported with 95% confidence intervals; downstream evaluation uses lm-eval-harness with five-shot GSM8K and zero-shot IFEval and MBPP, and the authors note the standard errors quantify task-example uncertainty, not training-run variation.

The result is aimed at architecture and systems researchers who want to reduce projection arithmetic while keeping the weight bank and parameter count, and it applies to square-operand projection settings with fixed physical block size and a growable number of groups, and to row-typed rectangular projections that support causal masking and KV-cached decoding. Kernel measurements cover public shapes including Qwen3-1.7B/4B/8B, a Qwen3-30B-A3B expert, a DeepSeek-V3 expert and Qwen3-32B, using synthetic BF16 tensors with FP32 accumulation; the pretraining check is limited to approximately 110M parameters, 12.3B tokens and one recursive law replacing only the FFN projections. The authors state that larger algebras, algebraic attention, larger models and repeated seeds remain untested, and Appendix G develops the separate extension to algebraic attention scores while the experiment uses dense attention.

Pretraining uses one run per model at one scale, and the authors state this does not establish an advantage at equal quality or training time; the throughput gain is smaller than the FFN arithmetic reduction because attention, the LM head, sampling, tokenization and memory traffic also contribute, and generation lengths differ, so fixed-length repeated full-model timings are needed to isolate the layer effect. Kernel benchmarks use synthetic tensors and report projection/FFN-level rather than end-to-end Transformer speedups, MoE measurements exclude dispatch and communication, and DeepSeek dimensions are evaluated in BF16 with its production FP8 implementation outside the comparison; performance counters were unavailable, so no counter-based bottleneck attribution is made. Growing the algebra also restricts each row's feature interactions and changes how often off-diagonal parameters are used, so computational scaling alone does not establish a quality-preserving scaling law. This evidence bundle is full text, but some table values appear as fractions, so precise comparisons should still return to the original tables.