Visual grounding drops coordinate-by-coordinate decoding: a 0.8B model returns the box in one pass at 65.97 ms

A visual grounding model can return a bounding box as four continuous values regressed from an existing multimodal input state instead of writing coordinates as text. Earlier multimodal models express the box as a coordinate sequence, which requires sequential decoding and parsing on top of the image-text processing needed for localization. On the official test splits of RefCOCO, RefCOCO+, and RefCOCOg, Jag improves over the base model without grounding fine-tuning on every split, by 9.63 to 17.51 accuracy points.

The regression head reads the hidden state of the last valid input token, a position that follows all visual and textual input, so box prediction reuses the same multimodal computation. Existing continuous box regression reads from a generated trigger token state, whereas Jag reads an existing input state and generates no trigger. The head applies layer normalization, a hidden projection with GELU, and a four-dimensional output projection, predicting all four coordinates together; training supervises normalized centers and dimensions with an L1 loss and box overlap and enclosure with a GIoU loss.

Geometric supervision adapts not only the regression head but also the language backbone and visual merger, so localization losses shape both the multimodal representations and the coordinate mapping. Fitting a regression head to frozen features confines adaptation to the coordinate mapping, and component ablations show this falls far short of joint adaptation. On a shared training subset, jointly updating the language backbone, visual merger, and regression head improves Acc@0.5 by more than 60 points on each dataset relative to head-only training.

Direct prediction keeps accuracy close to an autoregressive model at the standard overlap threshold while cutting single-request inference time substantially. An autoregressive model generates coordinate tokens sequentially, while Jag returns four coordinates after one complete multimodal forward pass without coordinate generation or text parsing. Mean end-to-end single-request latency is 65.97 ms, 19.60 times faster than the base model, 1.82 times faster than NExT-Chat, 3.62 times faster than LocateAnything, and 9.35 times faster than Hi-Token, with 2.52 GiB peak process memory.

The reported scope is single-object localization in static images, evaluated on the official training and test splits of RefCOCO, RefCOCO+, and RefCOCOg, with 30,969 test expressions. Latency and memory numbers come from end-to-end measurement on one NVIDIA A100-PCIE-40GB, covering image opening, preprocessing, host-device transfers, model execution, and output parsing. Inference efficiency depends on input resolution, expression length, hardware, and deployment settings, so applications with different input distributions or serving configurations should measure accuracy and latency under their intended operating conditions.

At stricter overlap thresholds the autoregressive model is more accurate on all three datasets, indicating a larger difference in precise box placement than in Acc@0.5. In the multi-seed comparison the autoregressive model also has higher mean accuracy, and the paired interval on RefCOCOg favors it. Extending the setting to multiple referents, video, or segmentation calls for task-specific design and evaluation, and specialized-domain applications should assess performance on representative data.