GateScope black-box audits 10 commercial LLM API gateways: some identify gpt-5 as the claimed model only 13.09% of the time, and o*ey bills 62.8% above expected
The authors introduce GateScope, a lightweight black-box auditing framework that uses only public APIs to evaluate LLM API gateways along response content, multi-turn conversation consistency, billing accuracy, and latency characteristics; controlled validation on official endpoints yields an average F1 of 0.968±0.085 across 24 models, and auditing 10 commercial gateways reveals identification rates as low as 13.09% for gpt-5, degraded multi-turn memory checkpoints, a 62.8% billing gap for o*ey on gpt-4o, and markedly higher latency variation for b*ie.