AI-generated editorial illustration; not a documentary photograph.
NVIDIA’s SWE-Serve benchmark measures whether AI coding agents can make repository-scale changes that continue to work when an inference server loads a real model and handles requests. In the benchmark’s 19 live-serving tasks, patches passed 45.9% of the time with the complete verifier, compared with 69.4% when live-serving checks were excluded.
Testing the serving path
Developed with input from the SGLang team, SWE-Serve converts 83 merged SGLang pull requests into 53 executable tasks. The tasks cover model enablement, decoding, caching, scheduling, serving APIs and runtime performance, extending beyond benchmarks focused on isolated kernels or optimization work.
Twelve tasks run on CPUs and 41 use a single NVIDIA H100. The benchmark does not evaluate other inference engines, multi-GPU execution or multi-node serving in this first release.
Where local checks fall short
The live-serving portion contains 276 tests, including checks for model loading, request handling, batched generation and output behavior. Across the 19 tasks, 147 of 627 patches changed from failing to passing when live-serving tests were removed, showing how local or partial verification can conceal runtime failures.
One example requires support for both dense and mixture-of-experts Qwen3.5 models on one H100. Its verifier checks registration, configuration, weight loading, multimodal inputs, OpenAI-compatible requests, native batching, log probabilities and routed-expert execution.
Cross-domain work is harder
SWE-Serve reports a 69.0% pass rate for 26 tasks limited to one runtime domain. The rate falls to 47.7% for 27 tasks spanning multiple domains, including request I/O, scheduling, model execution and KV-cache or resource management.
Across the best tested configuration for each of 11 models, mean pass@1 ranges from 34.6% to 75.5%. Four models reached 64%, but their mean cost ranged from $0.95 to $7.24 per task and their mean runtimes from 25.5 to 99.9 minutes.
A narrow benchmark result
The evaluations used 31 model-effort configurations, with each configuration running the 53-task benchmark three times. Sessions were limited to 210 minutes and 350 steps, and a task counted as solved only when the complete verifier passed on the declared hardware.
NVIDIA stresses that a SWE-Serve pass means only that a patch satisfied the benchmark verifier. It does not show that an agent-generated patch or reference solution is deployable, ready to merge or endorsed by SGLang maintainers. The task environments, verifiers and baseline configurations are available through the project’s GitHub release.
Source: How SWE-Serve Exposes the Gap Between Local Tests and Live Serving
SIGNAL TITANS
Leave a Reply