Each Transformer layer read as one EM iteration: attention does belief propagation, the layer-to-layer jump updates parameters
Transformer attention is interpreted as max-product belief propagation on a Markov random field that varies from instance to instance, while the parameter change between layers acts as the parameter-update step of EM, a view that groups self-attention, mixture-of-experts, gated units and selective state-space models under one meta-model principle in which the input generates the parameters.