Summary: vLLM is a fast open-source engine for serving LLMs behind an OpenAI-compatible API, but first runs often stop at the same few walls: CUDA out-of-memory at startup, a driver/toolchain mismatch, a download that seems frozen, or chat requests that fail because the model has no chat template. This guide groups those problems by symptom and gives the fix the official vLLM docs recommend, checked against the stable docs on 2026-11-09. Official documentation screenshots are at the end of this post.
Before you debug: confirm your setup is supported
A surprising share of "vLLM is broken" reports are really "this machine is outside the supported matrix". Spend two minutes on these checks before you touch any flags.
- Operating system: the GPU installation page lists Linux as the OS requirement and says vLLM does not support Windows natively. On Windows, the docs point you to WSL (Windows Subsystem for Linux) with a compatible distribution, or to community-maintained forks.
- Python: the requirement is Python 3.11 to 3.14. An older 3.10 environment is the first thing to rule out when
piprefuses to resolve. - NVIDIA GPU: compute capability 7.5 or higher. The docs give T4, RTX 20xx, A100, L4, H100 and B200 as examples. Older cards such as Pascal-generation GPUs fall below that line.
- CUDA build: the pre-built wheels contain binaries compiled with CUDA 12.9, and the project also publishes builds for CUDA 12.8 and 13.0. Blackwell GPUs (B200, GB200) need at least CUDA 12.8.
The installation page also explains why a fresh environment matters: vLLM compiles many CUDA kernels, and that creates binary incompatibility with other CUDA and PyTorch builds, even the same PyTorch version built differently. The recommended route is a new uv environment:
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels
--torch-backend=auto lets uv pick the PyTorch index that matches your installed CUDA driver. The docs also warn that PyTorch installed via conda statically links NCCL, which can cause problems when vLLM tries to use NCCL, so mixing a conda PyTorch with a pip vLLM is a common source of odd multi-GPU errors.
Symptom 1: startup is slow or seems to hang
Stuck while downloading the model
If the model is not on disk yet, vLLM downloads it from Hugging Face first, and on a slow link that can look exactly like a hang. The troubleshooting page recommends downloading the model separately with the hf CLI, then passing the local path to vLLM. That isolates the network step from the engine step, so you know which one is actually slow.
Stuck while loading from disk
Large models take a long time to read, and shared or network filesystems on clusters can make it much worse. The docs suggest keeping weights on a local disk and watching CPU memory, because a model that barely fits in RAM can push the system into swapping. To prove whether loading is the bottleneck, start once with --load-format dummy, which skips loading the real weights. If startup suddenly becomes quick, the problem is storage or download, not your GPU.
Several minutes of kernel compilation on new GPUs
The installation guide says that without the precompiled FlashInfer kernels, vLLM downloads and compiles kernels at startup, which can add several minutes on Hopper and newer GPUs, especially Blackwell, and it logs a warning. The fix is simply to run vllm download-kernels in the same Python environment after every install or upgrade, including inside custom container images.
Symptom 2: CUDA out of memory (OOM)
This is the error most people hit first. Two things are going on. First, the weights themselves may be too large for one GPU. Second, vLLM deliberately reserves most of the card: the engine arguments page says --gpu-memory-utilization defaults to 0.92 when unspecified, meaning about 92% of GPU memory is used by that vLLM instance for weights, activations and KV cache. The setting is per instance, so if something else already holds GPU memory (a notebook, a desktop session, a second server), the default can fail even when the model itself would fit.
Work through the options in the official "Conserving Memory" page, cheapest first:
- Shorten the context. Set
--max-model-len(ormax_model_lenin Python). If unspecified, it is derived from the model config, and modern models often advertise very long contexts that need a large KV cache. Lowering it is usually the quickest win. - Lower concurrency. Reduce
--max-num-seqs, the maximum number of sequences processed per iteration. The docs' example usesmax_model_len=2048, max_num_seqs=2for a small vision model. - Share the GPU explicitly. If you run two vLLM instances on one GPU, the docs suggest setting
--gpu-memory-utilization 0.5for each, rather than letting both try for the default. - Drop CUDA graphs. CUDA graphs speed up inference but take extra GPU memory.
--enforce-eagerdisables graph capture (and torch.compile) entirely; expect lower throughput in exchange for memory headroom. - Use a quantized model. Statically quantized checkpoints from the Hugging Face Hub can be used directly and take less memory, at the cost of some precision.
- Split across GPUs.
--tensor-parallel-size 2(or-tp 2) splits the model across two GPUs. Choose which GPUs with theCUDA_VISIBLE_DEVICESenvironment variable rather than calling device functions before vLLM starts; the docs warn that doing so can triggerRuntimeError: Cannot re-initialize CUDA in forked subprocess.
For multimodal models there is one more lever: limit_mm_per_prompt lets you cap how many images, videos or audio clips one prompt may contain, or disable a modality you never use (for example {"video": 0}), so vLLM does not reserve memory for it.
Symptom 3: CUDA, driver and PTX errors
"the provided PTX was compiled with an unsupported toolchain" or "driver is too old"
Per the troubleshooting page, both messages mean the CUDA code in the vLLM wheel was built with a newer toolkit than your driver supports. If you use the official Docker image, add -e VLLM_ENABLE_CUDA_COMPATIBILITY=1 to docker run to turn on the bundled forward-compatibility libraries. Outside Docker, install the cuda-compat package (the Ubuntu example is sudo apt-get install cuda-compat-12-9), then set VLLM_ENABLE_CUDA_COMPATIBILITY=1 and point VLLM_CUDA_COMPATIBILITY_PATH at the compat folder. Note the docs' caveat: CUDA forward compatibility is only supported on select professional and data-center NVIDIA GPUs. On a consumer card, updating the NVIDIA driver is the realistic fix.
Crash near self.graph.replay()
A traceback that ends around CUDA graph replay is a CUDA error inside a captured graph. Restart with --enforce-eager (or enforce_eager=True in Python) so the real failing CUDA operation shows up in the trace instead of being hidden by the graph.
Multi-GPU hangs and NCCL errors
If GPUs cannot talk to each other, the troubleshooting page provides a small test script to run with NCCL_DEBUG=TRACE torchrun --nproc-per-node=<number-of-GPUs> test.py. If that script itself hangs or crashes, the docs say the hardware or drivers are the likely problem; one common workaround to try is export NCCL_P2P_DISABLE=1. On machines with complicated networking, also check that the IP address in the debug log is correct and override it with VLLM_HOST_IP if not.
Symptom 4: Python script errors before inference starts
If your script prints a warning that CUDA was previously initialized and vLLM is switching the multiprocessing start method to spawn, or Python raises a related RuntimeError as worker processes start, the fix in the docs is structural: put your vLLM code behind a main guard.
if __name__ == '__main__':
import vllm
llm = vllm.LLM(...)
Two model-loading errors are also worth recognizing. Model architectures [...] failed to be inspected means vLLM could not import the model file, usually because of missing dependencies or an outdated build, so read the full log above it. Model architectures [...] are not supported for now means the architecture is not in vLLM's registry; check the Supported Models list before assuming a bug. Separately, the --trust-remote-code flag defaults to off, so models that ship custom code from Hugging Face will not run their code unless you opt in, and you should only do that for repositories you trust.
Symptom 5: the server runs but requests fail or look wrong
Chat requests error, plain completions work
The online serving docs are clear on this one: the Chat Completions API only works for models with a chat template in their tokenizer configuration, and without one "all chat requests will error". Some instruction-tuned models ship without it. Supply a template file yourself:
vllm serve <model> --chat-template ./path-to-chat-template.jinja
The project keeps example chat templates for popular models in its examples directory on GitHub.
Wrong model name, wrong port, or 401s
By default vllm serve listens on http://localhost:8000 and hosts one model at a time. The name clients must send in the model field is the --model argument unless you set --served-model-name, so run curl http://localhost:8000/v1/models and copy the name exactly. If you started the server with --api-key (or the VLLM_API_KEY environment variable), every client must send that key in its header; multiple keys are accepted for rotation. Need two models behind one port? The vLLM FAQ says that is not currently supported: run one server per model and put a router in front.
Answers got worse after an upgrade
Since v0.8.0, vLLM takes default sampling parameters from the model creator's generation_config.json when it exists. If quality changed, test with --generation-config vllm (online) or generation_config="vllm" (offline) to compare against vLLM's neutral defaults. And if the same prompt gives slightly different outputs across runs, the FAQ explains that batching and numerical precision make this expected; using float32 or request seeds reduces the variance but does not remove it.
When nothing above matches: turn on logging
The troubleshooting page lists environment variables for a stuck instance: VLLM_LOGGING_LEVEL=DEBUG for more logs, CUDA_LAUNCH_BLOCKING=1 to pinpoint the failing CUDA kernel, and NCCL_DEBUG=TRACE for multi-GPU communication. VLLM_TRACE_FUNCTION=1 records every function call but, per the docs, can slow token generation by over 100x, so use it only as a last resort. When you are done, unset these or open a new shell; the docs note leftover debug variables can make the system slow.
vLLM vs SGLang vs llama.cpp: is vLLM the right tool?
Sometimes the real fix is a different engine. All three are free and open source; the differences are in hardware focus and setup style. Facts below come from each project's official docs or GitHub README on 2026-11-09.
| Item | vLLM | SGLang | llama.cpp |
|---|---|---|---|
| Price / license | Free, Apache-2.0 | Free, Apache-2.0 | Free, MIT |
| Main platform | Linux (no native Windows; WSL suggested); Python 3.11–3.14 | Install guide targets Linux with NVIDIA GPUs; README lists GPUs, TPUs, NPUs, CPUs and Apple Silicon | Plain C/C++; CPU, Apple silicon (Metal), CUDA, HIP, Vulkan, SYCL |
| NVIDIA requirement | Compute capability 7.5+ | Docker images need a CUDA 13-compatible driver | CUDA backend optional; CPU+GPU hybrid for models larger than VRAM |
| Install | uv pip install vllm --torch-backend=auto |
uv pip install --prerelease=allow sglang or Docker lmsysorg/sglang |
Install script, pre-built binaries, Docker or build from source |
| OpenAI-compatible server | vllm serve <model>, default port 8000 |
sglang serve (docs example uses port 30000) |
llama serve -hf <repo>, with built-in web UI |
| Best fit | GPU servers serving many users | Data-center GPU/TPU fleets | Laptops, Macs, small GPUs, quantized GGUF models |
If you are on a Windows laptop or a GPU older than compute capability 7.5, llama.cpp-based tools are usually the less painful path; see our LM Studio tutorial for a desktop option. If you just want hosted open models without managing GPUs, our Hugging Face Inference errors guide covers that route, and our Open WebUI playbook shows how to put a chat interface on top of any OpenAI-compatible server, including vLLM.
Editor's note
- Who it's for: developers with a Linux machine and a recent NVIDIA (or supported AMD/Intel) GPU who want high-throughput, OpenAI-compatible serving for an app or a team.
- Who it's not for: Windows users who don't want WSL, owners of GPUs below compute capability 7.5, and anyone who just wants to chat with a model on a laptop.
- Gotcha 1: the 0.92 default for
--gpu-memory-utilizationassumes the GPU is otherwise free. Close notebooks and other servers, or lower the value. - Gotcha 2: a long default context is the silent OOM cause. Set
--max-model-lento what your app actually needs. - Gotcha 3: chat requests need a chat template; base models usually don't have one.
- Common error → fix: PTX/"driver too old" → update the driver, or use
VLLM_ENABLE_CUDA_COMPATIBILITY=1on supported data-center GPUs;Cannot re-initialize CUDA in forked subprocess→ select GPUs withCUDA_VISIBLE_DEVICESand add a main guard; crash atgraph.replay()→ rerun with--enforce-eager.
FAQ
Can I run vLLM on Windows?
Not natively. The official installation page says to use WSL with a compatible Linux distribution, or community-maintained forks, which the vLLM project does not maintain.
What value should I use for --gpu-memory-utilization?
The default is 0.92 for a single instance with the GPU to itself. Lower it when other processes use the same GPU; the docs' example for two vLLM instances on one GPU is 0.5 each. If you still hit OOM, reduce --max-model-len and --max-num-seqs before buying hardware.
Can one vLLM server host several models?
According to the vLLM FAQ, serving multiple models on one port through the OpenAI-compatible server is not currently supported. Run one instance per model and route requests to the right one with a proxy layer.
Check the supported matrix first, then shrink context and memory before chasing exotic flags; that order resolves most first-run vLLM failures.
Last verified: 2026-11-09
Sources: vLLM Troubleshooting, vLLM GPU installation, vLLM Quickstart, vLLM Conserving Memory, vLLM Engine Arguments, vLLM Online Serving, vLLM FAQ, vLLM on GitHub, SGLang on GitHub, SGLang installation, llama.cpp on GitHub.
Image credit: screenshots of the public vLLM documentation (Troubleshooting, GPU installation requirements, Engine Arguments, Online Serving chat template section), captured 2026-11-09. See the official screenshots at the end of this post.
댓글
댓글 쓰기