기본 콘텐츠로 건너뛰기

vLLM Errors Fixed: Out of Memory, CUDA and Chat Templates

Summary: vLLM is a fast open-source engine for serving LLMs behind an OpenAI-compatible API, but first runs often stop at the same few walls: CUDA out-of-memory at startup, a driver/toolchain mismatch, a download that seems frozen, or chat requests that fail because the model has no chat template. This guide groups those problems by symptom and gives the fix the official vLLM docs recommend, checked against the stable docs on 2026-11-09. Official documentation screenshots are at the end of this post.

Before you debug: confirm your setup is supported

A surprising share of "vLLM is broken" reports are really "this machine is outside the supported matrix". Spend two minutes on these checks before you touch any flags.

  • Operating system: the GPU installation page lists Linux as the OS requirement and says vLLM does not support Windows natively. On Windows, the docs point you to WSL (Windows Subsystem for Linux) with a compatible distribution, or to community-maintained forks.
  • Python: the requirement is Python 3.11 to 3.14. An older 3.10 environment is the first thing to rule out when pip refuses to resolve.
  • NVIDIA GPU: compute capability 7.5 or higher. The docs give T4, RTX 20xx, A100, L4, H100 and B200 as examples. Older cards such as Pascal-generation GPUs fall below that line.
  • CUDA build: the pre-built wheels contain binaries compiled with CUDA 12.9, and the project also publishes builds for CUDA 12.8 and 13.0. Blackwell GPUs (B200, GB200) need at least CUDA 12.8.

The installation page also explains why a fresh environment matters: vLLM compiles many CUDA kernels, and that creates binary incompatibility with other CUDA and PyTorch builds, even the same PyTorch version built differently. The recommended route is a new uv environment:

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

--torch-backend=auto lets uv pick the PyTorch index that matches your installed CUDA driver. The docs also warn that PyTorch installed via conda statically links NCCL, which can cause problems when vLLM tries to use NCCL, so mixing a conda PyTorch with a pip vLLM is a common source of odd multi-GPU errors.

Symptom 1: startup is slow or seems to hang

Stuck while downloading the model

If the model is not on disk yet, vLLM downloads it from Hugging Face first, and on a slow link that can look exactly like a hang. The troubleshooting page recommends downloading the model separately with the hf CLI, then passing the local path to vLLM. That isolates the network step from the engine step, so you know which one is actually slow.

Stuck while loading from disk

Large models take a long time to read, and shared or network filesystems on clusters can make it much worse. The docs suggest keeping weights on a local disk and watching CPU memory, because a model that barely fits in RAM can push the system into swapping. To prove whether loading is the bottleneck, start once with --load-format dummy, which skips loading the real weights. If startup suddenly becomes quick, the problem is storage or download, not your GPU.

Several minutes of kernel compilation on new GPUs

The installation guide says that without the precompiled FlashInfer kernels, vLLM downloads and compiles kernels at startup, which can add several minutes on Hopper and newer GPUs, especially Blackwell, and it logs a warning. The fix is simply to run vllm download-kernels in the same Python environment after every install or upgrade, including inside custom container images.

Symptom 2: CUDA out of memory (OOM)

This is the error most people hit first. Two things are going on. First, the weights themselves may be too large for one GPU. Second, vLLM deliberately reserves most of the card: the engine arguments page says --gpu-memory-utilization defaults to 0.92 when unspecified, meaning about 92% of GPU memory is used by that vLLM instance for weights, activations and KV cache. The setting is per instance, so if something else already holds GPU memory (a notebook, a desktop session, a second server), the default can fail even when the model itself would fit.

Work through the options in the official "Conserving Memory" page, cheapest first:

  1. Shorten the context. Set --max-model-len (or max_model_len in Python). If unspecified, it is derived from the model config, and modern models often advertise very long contexts that need a large KV cache. Lowering it is usually the quickest win.
  2. Lower concurrency. Reduce --max-num-seqs, the maximum number of sequences processed per iteration. The docs' example uses max_model_len=2048, max_num_seqs=2 for a small vision model.
  3. Share the GPU explicitly. If you run two vLLM instances on one GPU, the docs suggest setting --gpu-memory-utilization 0.5 for each, rather than letting both try for the default.
  4. Drop CUDA graphs. CUDA graphs speed up inference but take extra GPU memory. --enforce-eager disables graph capture (and torch.compile) entirely; expect lower throughput in exchange for memory headroom.
  5. Use a quantized model. Statically quantized checkpoints from the Hugging Face Hub can be used directly and take less memory, at the cost of some precision.
  6. Split across GPUs. --tensor-parallel-size 2 (or -tp 2) splits the model across two GPUs. Choose which GPUs with the CUDA_VISIBLE_DEVICES environment variable rather than calling device functions before vLLM starts; the docs warn that doing so can trigger RuntimeError: Cannot re-initialize CUDA in forked subprocess.

For multimodal models there is one more lever: limit_mm_per_prompt lets you cap how many images, videos or audio clips one prompt may contain, or disable a modality you never use (for example {"video": 0}), so vLLM does not reserve memory for it.

Symptom 3: CUDA, driver and PTX errors

"the provided PTX was compiled with an unsupported toolchain" or "driver is too old"

Per the troubleshooting page, both messages mean the CUDA code in the vLLM wheel was built with a newer toolkit than your driver supports. If you use the official Docker image, add -e VLLM_ENABLE_CUDA_COMPATIBILITY=1 to docker run to turn on the bundled forward-compatibility libraries. Outside Docker, install the cuda-compat package (the Ubuntu example is sudo apt-get install cuda-compat-12-9), then set VLLM_ENABLE_CUDA_COMPATIBILITY=1 and point VLLM_CUDA_COMPATIBILITY_PATH at the compat folder. Note the docs' caveat: CUDA forward compatibility is only supported on select professional and data-center NVIDIA GPUs. On a consumer card, updating the NVIDIA driver is the realistic fix.

Crash near self.graph.replay()

A traceback that ends around CUDA graph replay is a CUDA error inside a captured graph. Restart with --enforce-eager (or enforce_eager=True in Python) so the real failing CUDA operation shows up in the trace instead of being hidden by the graph.

Multi-GPU hangs and NCCL errors

If GPUs cannot talk to each other, the troubleshooting page provides a small test script to run with NCCL_DEBUG=TRACE torchrun --nproc-per-node=<number-of-GPUs> test.py. If that script itself hangs or crashes, the docs say the hardware or drivers are the likely problem; one common workaround to try is export NCCL_P2P_DISABLE=1. On machines with complicated networking, also check that the IP address in the debug log is correct and override it with VLLM_HOST_IP if not.

Symptom 4: Python script errors before inference starts

If your script prints a warning that CUDA was previously initialized and vLLM is switching the multiprocessing start method to spawn, or Python raises a related RuntimeError as worker processes start, the fix in the docs is structural: put your vLLM code behind a main guard.

if __name__ == '__main__':
    import vllm
    llm = vllm.LLM(...)

Two model-loading errors are also worth recognizing. Model architectures [...] failed to be inspected means vLLM could not import the model file, usually because of missing dependencies or an outdated build, so read the full log above it. Model architectures [...] are not supported for now means the architecture is not in vLLM's registry; check the Supported Models list before assuming a bug. Separately, the --trust-remote-code flag defaults to off, so models that ship custom code from Hugging Face will not run their code unless you opt in, and you should only do that for repositories you trust.

Symptom 5: the server runs but requests fail or look wrong

Chat requests error, plain completions work

The online serving docs are clear on this one: the Chat Completions API only works for models with a chat template in their tokenizer configuration, and without one "all chat requests will error". Some instruction-tuned models ship without it. Supply a template file yourself:

vllm serve <model> --chat-template ./path-to-chat-template.jinja

The project keeps example chat templates for popular models in its examples directory on GitHub.

Wrong model name, wrong port, or 401s

By default vllm serve listens on http://localhost:8000 and hosts one model at a time. The name clients must send in the model field is the --model argument unless you set --served-model-name, so run curl http://localhost:8000/v1/models and copy the name exactly. If you started the server with --api-key (or the VLLM_API_KEY environment variable), every client must send that key in its header; multiple keys are accepted for rotation. Need two models behind one port? The vLLM FAQ says that is not currently supported: run one server per model and put a router in front.

Answers got worse after an upgrade

Since v0.8.0, vLLM takes default sampling parameters from the model creator's generation_config.json when it exists. If quality changed, test with --generation-config vllm (online) or generation_config="vllm" (offline) to compare against vLLM's neutral defaults. And if the same prompt gives slightly different outputs across runs, the FAQ explains that batching and numerical precision make this expected; using float32 or request seeds reduces the variance but does not remove it.

When nothing above matches: turn on logging

The troubleshooting page lists environment variables for a stuck instance: VLLM_LOGGING_LEVEL=DEBUG for more logs, CUDA_LAUNCH_BLOCKING=1 to pinpoint the failing CUDA kernel, and NCCL_DEBUG=TRACE for multi-GPU communication. VLLM_TRACE_FUNCTION=1 records every function call but, per the docs, can slow token generation by over 100x, so use it only as a last resort. When you are done, unset these or open a new shell; the docs note leftover debug variables can make the system slow.

vLLM vs SGLang vs llama.cpp: is vLLM the right tool?

Sometimes the real fix is a different engine. All three are free and open source; the differences are in hardware focus and setup style. Facts below come from each project's official docs or GitHub README on 2026-11-09.

Item vLLM SGLang llama.cpp
Price / license Free, Apache-2.0 Free, Apache-2.0 Free, MIT
Main platform Linux (no native Windows; WSL suggested); Python 3.11–3.14 Install guide targets Linux with NVIDIA GPUs; README lists GPUs, TPUs, NPUs, CPUs and Apple Silicon Plain C/C++; CPU, Apple silicon (Metal), CUDA, HIP, Vulkan, SYCL
NVIDIA requirement Compute capability 7.5+ Docker images need a CUDA 13-compatible driver CUDA backend optional; CPU+GPU hybrid for models larger than VRAM
Install uv pip install vllm --torch-backend=auto uv pip install --prerelease=allow sglang or Docker lmsysorg/sglang Install script, pre-built binaries, Docker or build from source
OpenAI-compatible server vllm serve <model>, default port 8000 sglang serve (docs example uses port 30000) llama serve -hf <repo>, with built-in web UI
Best fit GPU servers serving many users Data-center GPU/TPU fleets Laptops, Macs, small GPUs, quantized GGUF models

If you are on a Windows laptop or a GPU older than compute capability 7.5, llama.cpp-based tools are usually the less painful path; see our LM Studio tutorial for a desktop option. If you just want hosted open models without managing GPUs, our Hugging Face Inference errors guide covers that route, and our Open WebUI playbook shows how to put a chat interface on top of any OpenAI-compatible server, including vLLM.

Editor's note

  • Who it's for: developers with a Linux machine and a recent NVIDIA (or supported AMD/Intel) GPU who want high-throughput, OpenAI-compatible serving for an app or a team.
  • Who it's not for: Windows users who don't want WSL, owners of GPUs below compute capability 7.5, and anyone who just wants to chat with a model on a laptop.
  • Gotcha 1: the 0.92 default for --gpu-memory-utilization assumes the GPU is otherwise free. Close notebooks and other servers, or lower the value.
  • Gotcha 2: a long default context is the silent OOM cause. Set --max-model-len to what your app actually needs.
  • Gotcha 3: chat requests need a chat template; base models usually don't have one.
  • Common error → fix: PTX/"driver too old" → update the driver, or use VLLM_ENABLE_CUDA_COMPATIBILITY=1 on supported data-center GPUs; Cannot re-initialize CUDA in forked subprocess → select GPUs with CUDA_VISIBLE_DEVICES and add a main guard; crash at graph.replay() → rerun with --enforce-eager.

FAQ

Can I run vLLM on Windows?

Not natively. The official installation page says to use WSL with a compatible Linux distribution, or community-maintained forks, which the vLLM project does not maintain.

What value should I use for --gpu-memory-utilization?

The default is 0.92 for a single instance with the GPU to itself. Lower it when other processes use the same GPU; the docs' example for two vLLM instances on one GPU is 0.5 each. If you still hit OOM, reduce --max-model-len and --max-num-seqs before buying hardware.

Can one vLLM server host several models?

According to the vLLM FAQ, serving multiple models on one port through the OpenAI-compatible server is not currently supported. Run one instance per model and route requests to the right one with a proxy layer.

Check the supported matrix first, then shrink context and memory before chasing exotic flags; that order resolves most first-run vLLM failures.

Last verified: 2026-11-09
Sources: vLLM Troubleshooting, vLLM GPU installation, vLLM Quickstart, vLLM Conserving Memory, vLLM Engine Arguments, vLLM Online Serving, vLLM FAQ, vLLM on GitHub, SGLang on GitHub, SGLang installation, llama.cpp on GitHub.

Image credit: screenshots of the public vLLM documentation (Troubleshooting, GPU installation requirements, Engine Arguments, Online Serving chat template section), captured 2026-11-09. See the official screenshots at the end of this post.

Uploaded Image Uploaded Image Uploaded Image Uploaded Image

댓글

이 블로그의 인기 게시물

브루(Vrew) 요금제 정리: 무료 200크레딧으로 어디까지 쓸 수 있나

요약: 영상 자막 편집기 브루(Vrew)는 2026년 4월 22일부터 기능별 사용량 대신 '통합 크레딧' 하나로 요금을 매겨요. 무료 플랜은 매달 200크레딧이 들어오는데, 음성 분석(자동 자막)만 쓰면 최대 200분 분량이라 짧은 영상 몇 편은 충분하지만, 영상 초반의 Vrew 마크를 지우려면 유료 플랜이 필요해요. 이 글은 브루 공식 가격 페이지와 고객센터 공지를 2026년 10월 23일에 다시 확인해, 크레딧 계산법과 플랜 고르는 기준을 정리했어요. 목차 브루 요금 구조: 통합 크레딧이란 1크레딧으로 할 수 있는 일 무료 플랜으로 되는 것과 안 되는 것 유료 플랜 4종 비교표 월 구독·연 구독·기간 이용권, 무엇이 이득일까 상황별 추천: 학생·직장인·유튜버 크레딧이 모자랄 때: 사용 기록 지우개 업그레이드·환불 전에 알아둘 규칙 운영자 메모, 자주 묻는 질문 1. 브루 요금 구조: 통합 크레딧이란 예전 브루는 플랜마다 음성 분석 몇 분, AI 목소리 몇 자, 번역 몇 자처럼 기능별 한도를 따로 줬어요. 그런데 보이저엑스(브루 운영사)가 실제 이용 데이터를 보니 한두 가지 기능만 집중해서 쓰는 사람이 훨씬 많았다고 해요. 그래서 2026년 4월 22일 업데이트부터는 플랜마다 정해진 크레딧을 받고, 어떤 기능에 쓸지는 내가 정하는 방식으로 바뀌었어요. 플랜별 매달 지급량은 Free 200 · Light 2,000 · Standard 10,000 · Business 12,000크레딧 이에요. 공식 공지에 따르면 무료 사용자는 매달 1일에, 유료 사용자는 매월 결제일에 충전되고, 남은 크레딧은 다음 달로 넘어가지 않아요. 바뀐 뒤 무엇이 좋아졌나 공지에 나온 예시가 이해하기 쉬워요. Standard 플랜에서 AI 목소리만 쓰면 예전에는 최대 50만 자였지만, 지금은 1만 크레딧을 모두 AI 목소리에 몰아 쓸 수 있어 최대 100만 자가 돼요. 반대로 여러 기능을 골고루 쓰는 사람은 크레딧이 기능끼리 ...

에이닷 전화 vs 익시오: 통화 녹음·AI 요약, 내 통신사·폰에 맞는 앱 고르기

요약: 통화 녹음과 AI 요약을 쓰고 싶다면 먼저 내 통신사와 휴대폰 종류부터 확인해야 해요. LG유플러스 고객은 무료 앱 익시오, SKT 고객은 에이닷 전화가 가장 기능이 많고, 다른 통신사의 아이폰 사용자는 iOS에 들어 있는 기본 통화 녹음·전사 기능이 현실적인 선택이에요. 이 글은 2026-10-30에 SKT·LG유플러스·Apple 공식 페이지와 도움말을 직접 확인해 세 가지를 비교했어요. 왜 '앱 기능'보다 '내 조건'을 먼저 봐야 할까요 통화 요약 앱 비교 글을 보면 기능 목록부터 나열하는 경우가 많아요. 그런데 실제로 써 보려는 사람에게 가장 먼저 걸리는 건 기능이 아니라 이용 조건 이에요. 익시오는 LG유플러스 고객 전용이고, 에이닷 전화는 누구나 설치할 수 있지만 통역콜·baro 같은 기능은 SKT 고객에게만 열려 있어요. 아이폰은 운영체제 구조상 앱이 통화를 마음대로 녹음하기 어려워서, 통신사 앱이 안 되면 iOS 기본 기능에 기대야 해요. 그래서 이 글은 "어떤 앱이 제일 좋은가"보다 "내 통신사·내 폰에서 무엇을 쓸 수 있나" 순서로 정리했어요. 기능 차이는 그다음이에요. 한눈에 비교: 에이닷 전화 vs 익시오 vs 아이폰 기본 통화 녹음 구분 에이닷 전화 (SKT) 익시오 ixi-O (LG유플러스) 아이폰 기본 통화 녹음 (Apple) 누가 쓸 수 있나 통신사 구분 없이 앱 설치 가능. 단, 통역콜·baro·그룹통화 발신 등은 SKT 고객 전용 U+ 모바일 고객. 타 통신사, 만 14세 미만, 선불요금제 고객은 제한 (공식 사이트 FAQ상 알뜰폰 중 KB리브엠은 예외) 지원 국가·언어의 iPhone 사용자. 1명과 통화할 때만 녹음 가능 지원 OS·기기 Android 10 이상, iOS 16 이상 iOS 17 이상 iPhone 12 이상 / Android...

타입캐스트 vs 클로바더빙: 수익 나는 채널에 쓸 때 무료·유료 조건 비교

요약 타입캐스트와 클로바더빙은 둘 다 글만 입력하면 한국어 AI 목소리를 만들어 주는 서비스지만, 무료로 쓸 수 있는 범위가 꽤 다릅니다. 클로바더빙 무료는 매달 15,000자까지 쓸 수 있는 대신 수익이 생기는 채널에는 쓸 수 없고, 타입캐스트 무료는 처음에 주는 약 5분 분량의 다운로드로 끝나는 체험판에 가깝습니다. 광고 수익이 붙은 유튜브나 쇼핑몰 영상에 넣을 목소리라면 처음부터 유료 플랜 조건을 보고 고르는 편이 안전합니다. 왜 이 두 서비스를 같이 비교하나 쇼츠 내레이션, 수업 자료 더빙, 상품 소개 영상처럼 "내 목소리 대신 AI 목소리"가 필요할 때 한국 사용자들이 가장 많이 찾는 곳이 네오사피엔스의 타입캐스트 와 네이버의 클로바더빙 입니다. 기능 소개만 보면 비슷해 보이지만, 실제로 문제가 되는 건 음질보다 "이 목소리를 내 채널에 써도 되나" 입니다. 그래서 이 글은 기능 나열보다 무료 사용 범위, 출처 표기 의무, 유료 플랜 가격과 다운로드 한도를 중심으로 정리했습니다. 모든 수치는 2026년 10월 25일에 두 서비스의 공식 요금 페이지와 고객센터 도움말에서 직접 다시 확인한 내용입니다. 한눈에 보는 비교표 항목 타입캐스트 무료 타입캐스트 베이직 클로바더빙 Free 클로바더빙 Standard 월 요금 0원 9,900원 (연간 결제 시 10% 할인) 0원 19,900원 (네이버 클라우드 플랫폼에서 결제) 무료·포함 사용량 음성 생성·재생 무제한, 다운로드는 최초 3,000크레딧(약 5분) 1회 지급 월 30,000크레딧(약 60분) 월 15,000자, 다운로드 월 20회, 프로젝트 누적 5개 (매월 1일 갱신) 월 30,000자, 다운로드 월 40회, 프로젝트 누적 20개 쓸 수 있...