AWT uses activation-derived diagonal preconditioning to close 12–60% of the tensor-network compression perplexity gap on Llama, Ministral, and Qwen
The authors propose Activation-aware Weight Tensorization (AWT), a training-free, solver-agnostic calibration-time preconditioner that applies an activation-derived diagonal equivalent reparameterization to each weight matrix before an unchanged TT/TTN solver and deploys it with only an input-side elementwise rescaling. Across Llama 3.1 8B, Ministral 8B, and Qwen2.5 7B at 2–6 compression, AWT consistently improves vanilla tensorization: it closes 12–35% of the WikiText perplexity gap under single-operator replacement and 27–60% under Llama multi-operator suffix replacement, with gains transferring to HellaSwag and ARC-Challenge.