SAM 3 at 4-bit weights stops grabbing the wrong object: SPQ lifts RefCOCO hit rate from 42.1 to 72.5
Quantizing SAM 3's image-encoder weights to 4 bits while keeping activations at 8 bits produces prompt-alignment drift: masks look plausible but land on the wrong object or lose the target across video frames. SPQ gives more calibration weight to parameters that affect text-concept scores, constrains feature error along low-variance residual directions, and folds a mean-error correction estimated from paired video rollouts into existing biases. Under W4A8, RefCOCO Pr@0.5 rises from 42.1 for PTQ4SAM to 72.5 while retaining 54.8 COCO mask AP, with no added inference operators.