Jichao Jiang

The authors propose TACO, an optimizer that computes the exact steepest-descent direction under a dimension-normalized 1-to-1 operator norm by taking the sign of the largest-magnitude entry in each column of two-dimensional weight matrices, keeping only a small set of low-precision gradient components per column; on OPT-13B it reduces persistent optimizer state 174x versus AdamW8bit (27.7 GB to 0.16 GB) and peak training memory 2.9x (80.6 GB to 27.5 GB) with comparable accuracy and runtime, and it enables full-parameter fine-tuning of 30-32B-parameter models on a single 80 GB H100.

The authors propose TACO, an optimizer that computes the exact steepest-descent direction under a dimension-normalized 1-to-1 operator norm by taking the sign of the largest-magnitude entry in each column of two-dimensional weight matrices, keeping only a small set of low-precision gradient components per column; on OPT-13B it reduces persistent optimizer state 174x versus AdamW8bit (27.7 GB to 0.16 GB) and peak training memory 2.9x (80.6 GB to 27.5 GB) with comparable accuracy and runtime, and it enables full-parameter fine-tuning of 30-32B-parameter models on a single 80 GB H100.