At a glanceThree ways to run a language model yourself, compared on platforms, model files, concurrency, defaults and security, with a short guide to choosing.
Not testedCompiled from each project's own documentation. None of the three was installed or timed.
vLLM, Ollama and llama.cpp all do the same basic thing: they load an open-weight model and answer requests in the OpenAI API format. People compare them as rivals, but they were built for different situations, and the right choice usually follows from two questions. Who will use it, and on what hardware?
1. The short answer
One person, one computerYou want it working in five minutes
→
Ollama
Unusual hardware, or full controlCPU only, a small device, your own quantization
→
llama.cpp
Many users, GPUs in a serverAn application or a team sends requests at once
→
vLLM
Start from who uses it and what it runs on, not from the tool.
Many setups use two of them: Ollama on each developer's laptop, and vLLM on the shared server. Because all three speak the same API, the application code does not change.
2. What each one is
Ollama
llama.cpp
vLLM
In one line
An app that downloads models and runs them
"LLM inference in C/C++": a command-line tool and a server
A server for running models on GPUs
Install
A desktop app; on Linux, an install script
An install script, a prebuilt binary, Docker, or build from source
A Python package or a Docker image
First command
ollama run <model>
llama serve -hf <repo>
vllm serve <repo>
Comes with
A model library you pull from by name
A chat page served by the server itself
Metrics, batching and multi-GPU support
3. Where each one runs
Ollama
llama.cpp
vLLM
macOS
Yes, macOS 14 or newer; GPU on Apple M-series
Yes; the README calls Apple silicon "a first-class citizen"
Through a separate build, vLLM-Metal
Windows
Yes
Yes
Not natively; the documentation points to WSL
Linux
Yes
Yes
Yes; this is its home
NVIDIA GPUs
Yes, compute capability 5.0 and up, with a recent driver
Yes, with its own CUDA kernels
Yes; the main target
Other accelerators
See its hardware page
Many backends, including Vulkan, SYCL and AMD through HIP
AMD and Intel builds are documented
CPU only
Works, slowly
A core use case
A CPU backend exists, but GPUs are the target
This table alone settles many choices. On a Windows desktop or a Mac, start with Ollama or llama.cpp. vLLM assumes a Linux machine with a serious GPU.
4. Where the model files come from
Ollama
llama.cpp
vLLM
Usual source
Its own model library, by short name
GGUF files, downloaded from Hugging Face with -hf
Hugging Face repositories, by name
File format
Imports GGUF or Safetensors through a Modelfile
GGUF
Safetensors, with a fallback to the older PyTorch format
Quantization
Does not quantize GGUF files on import; prepare them first
Built in, from 1.5-bit to 8-bit; -hf picks Q4_K_M by default
Runs pre-quantized models from the Hub
Quantization is why the same model can need very different memory in each tool. A default GGUF download is a 4-bit file, about a third the size of the 16-bit weights that vLLM loads by default. That makes llama.cpp and Ollama far easier to fit on small hardware, at some cost in precision. A fair comparison between tools uses the same precision in each.
5. What happens with several users
This is the real dividing line.
Ollama
llama.cpp
vLLM
Requests at once, by default
1 per model; OLLAMA_NUM_PARALLEL raises it
A number of slots chosen automatically; -np sets it
Many, batched together at every step
When busy
Queues up to 512 requests, then answers with 503
See its server README
Requests queue; under memory pressure some are paused and redone
Memory cost of parallelism
Grows with parallel requests times context length
The slots draw on one shared pool
A shared cache, sized from free GPU memory
One model across several GPUs
Not covered in the pages read
Not covered in the pages read
Yes, with tensor parallelism
Ollama's default of one request at a time per model is right for a personal assistant and wrong for a team. llama.cpp's server enables continuous batching by default and can serve a small group well. vLLM is designed around this problem: its scheduler, its cache and its metrics all exist to keep many requests moving on expensive GPUs.
6. Three defaults that surprise people
Ollama
llama.cpp
vLLM
Context length
4,096 tokens unless you raise it
Taken from the model
Taken from the model, often tens of thousands
Listens on
127.0.0.1, port 11434
127.0.0.1; check the port it prints
Port 8000
API key
None; clients send one that is ignored
--api-key, off by default
--api-key, off by default, and it covers only some paths
Metrics for monitoring
Not covered in the pages read
--metrics, off by default
On by default at /metrics
The same model can look worse in one toolIf answers seem to forget the start of a long document in Ollama but not in vLLM, the cause is usually the 4,096-token default, not the model. Raise it with OLLAMA_CONTEXT_LENGTH before you compare quality.
None of the three is safe to put on the open internet as installed. Two listen only on the local machine by default, and vLLM's own documentation says not to rely on its API key alone. Put a proxy in front, as in vLLM for a team.
7. One client, three servers
All three accept the OpenAI client. Only the address changes:
from openai import OpenAI
# Pick one base_url:
# Ollama: http://localhost:11434/v1
# llama.cpp: http://localhost:PORT/v1 (the port the server printed)
# vLLM: http://localhost:8000/v1
client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed-locally")
reply = client.chat.completions.create(
model="MODEL_NAME", # the name each server reports at /v1/models
messages=[{"role": "user", "content": "Explain a KV cache in one sentence."}],
max_tokens=100,
)
print(reply.choices[0].message.content)
This is the practical argument for not agonizing over the choice. Write your application against the OpenAI format, start with whichever tool fits your machine today, and move later by changing one line.
8. How to choose
Your situation
Start with
Why
A laptop or desktop, just you
Ollama
One install, models by name, sensible for one user
llama.cpp server, or Ollama with parallelism raised
Enough concurrency without a server project
A team, or an application with real traffic
vLLM
Batching, multi-GPU, metrics
A model too large for one GPU
vLLM
Tensor parallelism
Windows, and you need vLLM
vLLM inside WSL, or a Linux server
No native Windows support
If you are heading for vLLM, the series Run your own LLM server with vLLM starts from the first command. If you want numbers, measure the same model at the same precision on your own hardware; the sizing guide shows how to run a load test and read it.
This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.
Every entry in the tables comes from each project's own documentation or README as read on November 12, 2026. None of the three was installed or timed for this article, so it contains no speed comparison; speed depends on your hardware and model, and the only honest figure is one you measure. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy
Found this useful? Share it, or follow for new guides.
OpenAI broadly launched a new GPT-6-based ChatGPT experience on October 7. Paid plans default to GPT-6 Sol, while Free and Go default to GPT-6 Luna. Check the models used in ChatGPT, Codex, and Work separately.
Point Claude Code at a model you host with vLLM: the six settings, what Claude Code really sends, what stops working, and what Anthropic does not support.
Install the NVIDIA Container Toolkit, prove the GPU works inside a container, start vLLM with docker run, then make it a Compose service that survives reboots.