Few-shot CLIP tuning stops trusting gradient size: sign-only updates with a refreshed support reach 72.7% on ViT-B/16 at one shot

In few-shot vision-language adaptation, gradient magnitudes are unreliable while signs remain usable, and this separation becomes two update rules that keep no optimizer state. Earlier parameter-efficient fine-tuning and sparse optimization methods mostly kept the Adam or SGD update recipe, assuming few-shot gradients were reliable enough to drive learning and leaving noise to hyperparameter tuning. On CLIP (ViT-B/16), comparing the 1-shot gradient with a 16-shot reference gradient, the coordinate-wise magnitude ratio spans orders of magnitude, while a magnitude-weighted sign reliability stays positive over iterations and rises when restricted to the top-magnitude tail, from about 0.58 to about 0.71 on the top 0.1% of coordinates on EuroSAT.

Gradient magnitude decides where to update and the sign decides how far, with each selected nonzero coordinate moving by a fixed step. SignSparseSGD fixes its support after the first minibatch, DynSignSparseSGD recomputes the support at every step, and neither keeps first- or second-moment buffers. Across 11 datasets, the ViT-B/32, ViT-B/16 and ViT-L/14 backbones and 1, 2, 4, 8 and 16 shots, DynSignSparseSGD has the highest mean accuracy in all 15 backbone–shot settings, reaching 72.7% on ViT-B/16 at one shot versus 71.4% for SO and 70.5% for LoRA.

Under a 300-epoch training budget, both sign-sparse variants show far smaller average peak-to-final drops than the compared methods, and the fixed support drops less. At the shared learning rate of 10⁻⁴, the average peak-to-final drops over 11 datasets are 0.93 and 1.85 points for SignSparseSGD and DynSignSparseSGD, versus 5.26 for SO, 6.06 for SAFT, 7.97 for LoRA and 9.58 for GaLoRE. The dynamic support is the stronger default under a fixed short budget, while the static support shows smaller late-epoch degradation when training is extended to 300 epochs; SignSparseSGD stays within 5 points of its peak on all 11 datasets and DynSignSparseSGD on 10 of 11.

The conclusions rest on few-shot adaptation for CLIP-style image classification, evaluated on 11 datasets, the ViT-B/32, ViT-B/16 and ViT-L/14 backbones, 1 to 16 labeled examples per class and three fixed random seeds, with all methods sharing one configuration. Both optimizers still require dense backpropagation, so the savings occur at the optimizer stage rather than in backpropagation.

The study is explicitly limited to CLIP-style image classification, and other architectures or modalities may require recalibrating the sign update scale. The static support can trail at larger shot counts because the initial support becomes less representative as optimization proceeds. The dynamic support is the stronger default under a fixed short budget, while the static support shows smaller late-epoch accuracy degradation under very long training.