Service-call code gets an execution test: SCCGBench finds top models hit the right endpoint only 3.76% of the time

The yardstick for service-call code shifts from whether the code resembles a reference implementation to whether the request actually reaches the intended service interface. Earlier code benchmarks rely on CodeBLEU, edit similarity, or unit-test pass rates, none of which can tell whether a generated HTTP request hits the correct service endpoint. On 612 test samples, the BM25 retrieval baseline posts the highest CodeBLEU at 28.23, yet its Mock hit rate is 0, showing that static similarity and executability diverge.

Evaluation intercepts the request a generated program tries to issue inside an isolated virtual environment and compares host, endpoint, HTTP method, and parameters without real network access or credentials. Mock requires host, endpoint, and HTTP method to match simultaneously; Robust additionally requires status checking and exception handling, while ParamF1 measures parameter alignment separately. Across six code models in the intent-only setting, host accuracy is only 3% to 8% and endpoint accuracy 12% to 19%, even though request constructability reaches 80% to 92%.

Failures concentrate at the step of choosing the right service, not at writing a request. Attributing 3,672 generations from six neural models to the earliest failed stage, host errors account for 69.8%, while syntax errors and request-construction failures together account for about a quarter. With gold documentation, most open-source models reach about 89% to 90% Mock and DeepSeek-V4-Pro reaches 76.47% to 81.86%, whereas BM25 retrieval reaches only 33.99% Endpoint@10.

The evaluation targets the service-computing setting where a natural-language intent becomes an executable Web service call, and its readers are researchers and engineers working on code generation, API recommendation, and retrieval-augmented generation. SCCGBench contains 3,135 RapidAPI call functions from real GitHub repositories, covering 850 APIs and 2,082 repositories, with an API-level split that keeps the same API out of training, validation, and test sets. The virtual execution environment replaces network libraries to intercept requests, consuming no API quota and incurring no real billing. The authors propose future directions including service-interface-specific retrieval and reranking models, joint optimization of retrieve-select-generate pipelines, and extension to multiple programming languages and API ecosystems.

Mock and Robust rest on virtual execution and are not equivalent to success over a real network. In the bounded real-call check, 21 of 75 screened services returned 2xx for gold requests, and Mock-positive requests returned 2xx in 91.3% of effective attempts, but five of nine eligible Mock-negative requests also returned 2xx because they target different but accepting endpoints on the same host, showing that Mock is stricter than a real 2xx response. The data covers only Python and the RapidAPI ecosystem. Comment-contamination, temporal-slicing, and human-agreement analyses find no execution advantage from redacted intents or older samples, but these analyses cannot prove contamination is entirely absent.