How many bits a fine-tune must keep: the same dataset needs 1.03 bits per value on one frozen model and 0.41 on another

How many bits a fine-tuned adapter needs now has a behavioral definition: the smallest serialized size that, once reloaded into the frozen model, retains 90% of the held-out fine-tuning gain. Adapter rank, parameter count and weight reconstruction error were the usual ways to describe how much information a fine-tune writes, and they measure storage rather than the bits needed to preserve behavior. Across frozen models from 0.5B to 32B, an adaptive codec that minimizes LoRA weight reconstruction error reaches a relative weight RMSE of 0.54 at about one bit per value, below uniform quantization's 0.56, yet retains only 3% of the held-out gain against 86% for uniform quantization.

The same dataset requires different bit counts on different frozen models, and the ordering can even reverse, so compression difficulty belongs to the dataset–receiver pair rather than to the dataset alone. If rate were set by the data, one corpus would keep the same ordering across models; instead XBRL tagging is the lowest-rate corpus on six receivers but not on Qwen2.5-7B, where math is cheaper. Keeping 90% of the math gain takes 1.03 bits per value on Qwen2.5-Math-7B and 0.41 bits per value on Qwen2.5-7B.

Compression is not only loss: it can attenuate the part of an adapter that supervision wrote badly, and a corrupted adapter's accuracy rises sharply after compression. The uncompressed corrupted adapter falls far below the frozen model, while after compression it scores above it. On Qwen2.5-7B, GSM8K accuracy drops from 77.6% for the frozen model to 23.7% for the corrupted adapter and returns to 82.9% after compression, with the effect also appearing on Mistral-7B-Instruct and Llama-3.1-8B.

Harmful and useful effects are unevenly distributed across the adapter's singular directions, with leading directions most detrimental, and compression preferentially attenuates them. Decomposing a corrupted adapter into singular directions and evaluating contiguous windows shows leading directions strongly harmful, an intermediate range contributing positively, and trailing directions having little effect. On Mistral-7B-Instruct, every window containing the leading direction averages 24.0%, at or below the frozen baseline's 25.0%, while windows starting between directions 2 and 9 average 28.7% to 31.8%.

The conclusions rest on post-training compression of LoRA adapters, with frozen models spanning Qwen2.5 from 0.5B to 32B, Qwen3-8B-Base, Mistral-7B, Llama-3.1-8B and Gemma-2-9B, and with evaluation mainly on GSM8K-style math and structured tasks; the recovery experiments use deliberately constructed mismatched supervision rather than naturally occurring fine-tuning errors.

The behavioral rate is an operational, codec-dependent achievable rate rather than an information-theoretic optimum; the correction-spectrum statistic ranks rates well within a fixed receiver but does not transfer reliably across receivers; the shrinkability rule reaches an RMSE of 0.068 bits per value on receivers not used to fix it, yet the rates had already been measured when the rule was formulated, so the validation remains retrospective; whether naturally occurring fine-tuning errors share the same directional structure is left open.