Tensor-parallel traffic drops to 4 bits: Echo feeds each rounding error into later calls, beating NCCL by 2.09x on 1 GB messages

The tensor-parallel all-reduce sits between two dependent matrix multiplications, so its cost cannot be hidden, and the values it moves have stayed at 16 bits. Echo replaces that communication with 4-bit integer transport and a high-precision sum, using one mechanism for both training and inference. Earlier work either hides or accelerates the all-reduce without reducing the bytes it moves, or reduces the error of a single quantization through finer groups, preserved outlier channels, or rotations, without compensating the errors that earlier calls already injected. On eight A100 GPUs over NVLink, the uncompensated 4-bit all-reduce is 2.65x faster than the 16-bit NCCL all-reduce on 1 GB messages and takes 37.8% of its time; with compensation it is 2.09x faster, end-to-end time to first token improves by up to 1.16x and the training step by up to 1.09x, rising to 1.44x and 1.38x as background traffic grows.

The error can be controlled because the all-reduce reveals the rounding error it has just injected when it forms the sum, and later calls can subtract that error back out. Error feedback was previously used for gradient compression, where the gradient of the same parameters reappears at the next step; a hidden state is produced once and consumed by the next layer, and at inference there is no training loop at all, so applying it directly gives no gain. Echo builds the compensator from two moving averages with different decay rates, adds a correction to the input before each all-reduce, and refreshes the averages from the output residual; fused into the entry pass with the update running beside the next matrix multiplication, it adds 0.90 ms to the critical path on 1 GB messages, against 14.09 ms for separate PyTorch operations.

The reported setting is a single server with eight A100 GPUs connected by NVLink, serving Llama-3-8B and Qwen3-8B at tensor-parallel degree 8, and pretraining Qwen3-0.6B from scratch at the same degree without pipeline, context, data, or sequence parallelism. Speed numbers come from an idle interconnect and from background peer-to-peer traffic occupying 50%, 75%, and 100% of the time, and accuracy is measured in one evaluation harness with each task's standard shot count. The authors propose the reduce-scatter of sequence parallelism and the combine step of expert parallelism, which share the same two structural properties, as natural next targets.

The depth-independent propagation bound rests on one assumption about the trained network, namely that it does not keep amplifying a perturbation along one direction from site to site; the authors also construct a network that satisfies the noise model yet makes the deviation grow exponentially with depth, so an assumption of this kind cannot be dropped. The backward source bound holds in mean square under a smoothness condition on the normalization Jacobians, with constants degraded because an error and its correction pass through different operators. On Llama-3-8B, 4-bit inference loses 1.66 points on average over seven benchmarks, and GSM8K strict-match falls from 49.81 to 46.17; the authors recover most of this by promoting the first half of the layers to 8 bits, a policy used only in the serving experiments, while the training experiments use uniform 4-bit communication. The code will be released upon acceptance.