Decision models skip the explanation: SGLang's Score API reuses a shared query across candidates, so latency barely grows with option count
Synopsis
For decision tasks such as checking an order's status, an application often needs a category or a score rather than prose. SGLang's Score API lets callers declare label tokens explicitly and read their scores directly, while multi-item scoring (MIS) reuses the repeated query computation inside one request and keeps candidates isolated. On Open-Jev choice tasks, MIS latency stays nearly flat as options grow from 2 to 16: 18.7 ms on Qwen3-0.6B versus 39.6 ms for Generate and 24.3 ms for SIS; the advantage is largest as load rises and is not consistently faster at low load.
Interpretation
Decision applications can now ask the model for scores on specific labels instead of hoping those labels appear in a generation response's top-k logprobs. One-token generation with logprobs was a workable baseline, but its top-k response may omit a label the application needs. SGLang's /v1/score declares labels through label_token_ids and returns one score row per item in the requested label order, exercised on Open-Jev's independent candidate prompts.
Multi-item scoring reuses the query computation repeated within a request while restricting each candidate to that query and its own tokens. Single-item scoring treats each query-plus-item pair as an independent logical sequence, even when one request contains several items; continuous batching and prefix caching help, but batching alone does not guarantee shared computation. On SGLang with CUDA 13.0 on one NVIDIA H200, using Qwen3-0.6B, Qwen3-8B and Qwen3.5-4B on Open-Jev choice tasks, MIS latency stays nearly flat as options grow from 2 to 16; at 16 options Qwen3-0.6B takes 18.7 ms versus 39.6 ms for Generate and 24.3 ms for SIS.
MIS's latency advantage opens up mainly as load rises, and it delays saturation rather than eliminating it. At low load MIS is not consistently faster; on Qwen3.5-4B, Generate is faster at low load. Qwen3-0.6B's p95 stays below roughly 100 ms across the plotted range while Generate and SIS climb into seconds; Qwen3-8B's MIS holds near 130 ms through 90 questions/s then rises sharply at the next load point; at the highest plotted load Qwen3.5-4B's MIS p95 is roughly one-third that of the other two paths.
Perspective
These measurements evaluate serving performance only, covering pointwise candidate evaluation on Open-Jev choice tasks and a Fused-Choice versus anchor-based Setwise scoring comparison under one shared server configuration; they do not establish equivalent decisions across prompt formulations or execution modes.
MIS requires a supported model, backend and server configuration, and selecting /v1/score alone does not enable it; candidate-count groups contain different questions with different token lengths, so this is an observed workload trend rather than an experiment isolating candidate count; high-load points also include substantial client queueing, and each load was swept once.