Joey David

The information a fine-tune writes into its adapter is now measured not by parameter count or weight reconstruction error but as the smallest serialized size of a compressed adapter that, reloaded into the frozen model, keeps 90% of the held-out fine-tuning gain. The same dataset demands very different rates across frozen models: keeping 90% of the math gain takes 1.03 bits per value on Qwen2.5-Math-7B but 0.41 on Qwen2.5-7B, and on an adapter trained with deliberately mismatched reasoning supervision, compression lifts GSM8K accuracy from 23.7% to 82.9%, above the frozen Qwen2.5-7B's 77.6%.

The information a fine-tune writes into its adapter is now measured not by parameter count or weight reconstruction error but as the smallest serialized size of a compressed adapter that, reloaded into the frozen model, keeps 90% of the held-out fine-tuning gain. The same dataset demands very different rates across frozen models: keeping 90% of the math gain takes 1.03 bits per value on Qwen2.5-Math-7B but 0.41 on Qwen2.5-7B, and on an adapter trained with deliberately mismatched reasoning supervision, compression lifts GSM8K accuracy from 23.7% to 82.9%, above the frozen Qwen2.5-7B's 77.6%.