SAM 3 at 4-bit weights stops grabbing the wrong object: SPQ lifts RefCOCO hit rate from 42.1 to 72.5
Synopsis
Quantizing SAM 3's image-encoder weights to 4 bits while keeping activations at 8 bits produces prompt-alignment drift: masks look plausible but land on the wrong object or lose the target across video frames. SPQ gives more calibration weight to parameters that affect text-concept scores, constrains feature error along low-variance residual directions, and folds a mean-error correction estimated from paired video rollouts into existing biases. Under W4A8, RefCOCO Pr@0.5 rises from 42.1 for PTQ4SAM to 72.5 while retaining 54.8 COCO mask AP, with no added inference operators.
Interpretation
The problem with low-bit SAM 3 is not only weaker masks: the link between a text prompt and its intended object breaks, so the quantized model can outline a plausible but unintended object or switch identities as a video unfolds. Earlier quantization methods mainly minimize feature reconstruction error, which preserves most accuracy on structural metrics such as COCO and LVIS but degrades markedly on referring expressions and long video sequences. With a W4A8 image encoder, PTQ4SAM reaches 54.1 COCO mask AP but only 42.1 RefCOCO Pr@0.5, a visible gap between structural and prompt-grounding scores.
SPQ shifts calibration from minimizing total feature error toward preserving the feature directions that decide which object a prompt selects. Standard post-training quantization minimizes reconstruction error equally across feature directions, whereas controlled perturbations show that perturbing low-variance residual directions affects target selection far more than mask AP. Under W4A8, SPQ reaches 54.8 COCO mask AP and 72.5 RefCOCO Pr@0.5, against 54.1 and 42.1 for PTQ4SAM, with consistent gains also reported on LVIS, the RefCOCO variants and SA-V.
Three calibration components each address one kind of error, and all run offline, so the exported inference graph matches an ordinary quantized model. Semantic-anchor calibration upweights parameters that affect text-concept scores, orthogonal subspace rectification constrains feature error along residual directions, and temporal distribution alignment estimates the mean shift at video memory interfaces and folds it into existing biases. Under one W4A8 protocol, semantic-anchor calibration alone raises COCO AP from 45.2 to 52.5, orthogonal subspace rectification alone raises RefCOCO Pr@0.5 from 31.1 to 65.9, temporal alignment alone raises SA-V J&F from 56.5 to 75.6, and all three together reach 54.8/72.5/81.2.
Perspective
The demonstrated scope is SAM 3 under a W4A8 mixed-precision setting: 4-bit image-encoder weights and 8-bit activations, with the multimodal decoder, detector queries and video memory kept at W8A8 and the text encoder and prediction heads at FP16; evaluation covers COCO 2017, LVIS, RefCOCO/+/g and SA-V, plus one CLIP transfer check. The authors list extension to more architectures and lower activation bit widths as future work.
The authors note that SPQ depends on representative calibration images and text prompts, and substantial shifts in deployment prompts may require recalibration; the functional controls measure selective feature sensitivity, while the temporal diagnostics track correlations between feature changes and accuracy, and establishing their causal relationship requires further study. More aggressive W6A6 or W4A4 decoder and memory settings also remain to be evaluated.