# Tensor parallelism

Muse Spark Open decoding rises from a 517 tok/s baseline to 1,109 tok/s by fusing each layer's attention and MoE into a single kernel that stays resident on the GPU, overlapping weight loads, tensor-core work and cross-GPU communication. Previously SGLang launched about 1,450 kernels per decoding step, and the GPU drained and refilled between them, leaving some SMs and threads idle. The figure was measured on Blackwell GPUs at batch size one with DFlash V2 speculative decoding and has not yet been extended to larger batches.

Muse Spark Open decoding rises from a 517 tok/s baseline to 1,109 tok/s by fusing each layer's attention and MoE into a single kernel that stays resident on the GPU, overlapping weight loads, tensor-core work and cross-GPU communication. Previously SGLang launched about 1,450 kernels per decoding step, and the GPU drained and refilled between them, leaving some SMs and threads idle. The figure was measured on Blackwell GPUs at batch size one with DFlash V2 speculative decoding and has not yet been extended to larger batches.

Running Hugging Face models on AWS Trainium now rests on a different foundation: TorchNeuron, a native PyTorch backend, exposes Neuron chips as an ordinary device with eager execution, torch.compile and native distributed APIs, replacing the PyTorch XLA-based optimum-neuron. By Hugging Face's comparison, out-of-the-box support among the 100 most-used text-generation architectures rises from 46% with Optimum Neuron v0.4.5 to 89%. The backend enters public beta on October 20, 2026.

Running Hugging Face models on AWS Trainium now rests on a different foundation: TorchNeuron, a native PyTorch backend, exposes Neuron chips as an ordinary device with eager execution, torch.compile and native distributed APIs, replacing the PyTorch XLA-based optimum-neuron. By Hugging Face's comparison, out-of-the-box support among the 100 most-used text-generation architectures rises from 46% with Optimum Neuron v0.4.5 to 89%. The backend enters public beta on October 20, 2026.

On the public AgentX agent workload, an eight-GPU AMD Instinct MI355X system serving Kimi K3 with vLLM reaches about 7,567 total tokens/s/GPU at 50 tokens/s/user P90 interactivity, 24.4% above NVIDIA HGX B300's 6,083; under SemiAnalysis's modeled $1.50 versus $2.26 per GPU-hour, that is 18.16 million versus 9.69 million tokens per dollar of TCO, 87.4% more. The result comes from sustained AMD and vLLM community engineering across attention, recurrent state, MoE and caching rather than a single change.

On the public AgentX agent workload, an eight-GPU AMD Instinct MI355X system serving Kimi K3 with vLLM reaches about 7,567 total tokens/s/GPU at 50 tokens/s/user P90 interactivity, 24.4% above NVIDIA HGX B300's 6,083; under SemiAnalysis's modeled $1.50 versus $2.26 per GPU-hour, that is 18.16 million versus 9.69 million tokens per dollar of TCO, 87.4% more. The result comes from sustained AMD and vLLM community engineering across attention, recurrent state, MoE and caching rather than a single change.