PyTorch developers have integrated Helion, a domain-specific language (DSL) for hardware-agnostic kernel writing, into vLLM's linear backend. The implementation uses ahead-of-time (AOT) autotuning to optimize large language model (LLM) inference performance on NVIDIA Hopper GPUs, outperforming vLLM's default CUTLASS, FlashInfer, and DeepGEMM implementations across tested workloads.
Unified GEMM Architecture
Rather than maintaining separate custom kernels and manual dispatch heuristics for algorithmic variants like Split-K and Swap-AB, the Helion linear backend unifies general matrix multiplication (GEMM) into a single implementation. Algorithmic variations are exposed as tunable parameters, allowing an automated search process to select the optimal variant alongside low-level scheduling parameters for each tensor shape. The evaluation targeted 8-bit quantized formats on Hopper GPUs, including FP8 Dynamic, W8A8 INT8, and Block FP8.
Hybrid Dispatch and Autotuning
To manage latency, the integration uses a hybrid dispatch strategy based on token sequence length and CUDA Graph execution. Helion kernels handle small tensor shapes up to 32 tokens during CUDA Graph replay, while larger shapes fall back to default libraries. This structure avoids runtime CPU launch overhead and limits fine-grained autotuning to the small token counts typical of LLM decoding phases. The kernel configurations were tuned per input shape using an LLM-guided search algorithm, LLMSeededLFBOTreeSearch, seeded by Claude Opus 4.8.
Kernel Benchmarks and Serving Limitations
Isolated kernel benchmarks on an NVIDIA H100 GPU demonstrated geometric mean speedups over default libraries: 1.110× for FP8 Dynamic, 1.178× for W8A8 INT8, and up to 1.177× for Block FP8 over DeepGEMM. End-to-end serving benchmarks across Qwen3 model variants showed throughput improvements exceeding 10% for specific batch sizes. However, offline autotuning can require hours to process target shapes, and running Helion outside CUDA Graphs introduces CPU launch overhead. The code and tuning scripts are accessible via a dedicated vLLM repository fork.
AI-generated editorial illustration; not a documentary photograph.
Source: Building a High-Performance and Portable vLLM Linear Backend with Helion
SIGNAL TITANS
Leave a Reply