Seongwoo Choi / Journal

vLLM vs Ollama vs llama.cpp: which one to run

At a glanceThree ways to run a language model yourself, compared on platforms, model files, concurrency, defaults and security, with a short guide to choosing.

Not testedCompiled from each project's own documentation. None of the three was installed or timed.

In this article 8 sections
  1. 1. The short answer
  2. 2. What each one is
  3. 3. Where each one runs
  4. 4. Where the model files come from
  5. 5. What happens with several users
  6. 6. Three defaults that surprise people
  7. 7. One client, three servers
  8. 8. How to choose

vLLM, Ollama and llama.cpp all do the same basic thing: they load an open-weight model and answer requests in the OpenAI API format. People compare them as rivals, but they were built for different situations, and the right choice usually follows from two questions. Who will use it, and on what hardware?

1. The short answer

One person, one computerYou want it working in five minutes
→
Ollama
Unusual hardware, or full controlCPU only, a small device, your own quantization
→
llama.cpp
Many users, GPUs in a serverAn application or a team sends requests at once
→
vLLM
Start from who uses it and what it runs on, not from the tool.

Many setups use two of them: Ollama on each developer's laptop, and vLLM on the shared server. Because all three speak the same API, the application code does not change.

2. What each one is

Ollamallama.cppvLLM
In one lineAn app that downloads models and runs them"LLM inference in C/C++": a command-line tool and a serverA server for running models on GPUs
InstallA desktop app; on Linux, an install scriptAn install script, a prebuilt binary, Docker, or build from sourceA Python package or a Docker image
First commandollama run <model>llama serve -hf <repo>vllm serve <repo>
Comes withA model library you pull from by nameA chat page served by the server itselfMetrics, batching and multi-GPU support

3. Where each one runs

Ollamallama.cppvLLM
macOSYes, macOS 14 or newer; GPU on Apple M-seriesYes; the README calls Apple silicon "a first-class citizen"Through a separate build, vLLM-Metal
WindowsYesYesNot natively; the documentation points to WSL
LinuxYesYesYes; this is its home
NVIDIA GPUsYes, compute capability 5.0 and up, with a recent driverYes, with its own CUDA kernelsYes; the main target
Other acceleratorsSee its hardware pageMany backends, including Vulkan, SYCL and AMD through HIPAMD and Intel builds are documented
CPU onlyWorks, slowlyA core use caseA CPU backend exists, but GPUs are the target

This table alone settles many choices. On a Windows desktop or a Mac, start with Ollama or llama.cpp. vLLM assumes a Linux machine with a serious GPU.

4. Where the model files come from

Ollamallama.cppvLLM
Usual sourceIts own model library, by short nameGGUF files, downloaded from Hugging Face with -hfHugging Face repositories, by name
File formatImports GGUF or Safetensors through a ModelfileGGUFSafetensors, with a fallback to the older PyTorch format
QuantizationDoes not quantize GGUF files on import; prepare them firstBuilt in, from 1.5-bit to 8-bit; -hf picks Q4_K_M by defaultRuns pre-quantized models from the Hub

Quantization is why the same model can need very different memory in each tool. A default GGUF download is a 4-bit file, about a third the size of the 16-bit weights that vLLM loads by default. That makes llama.cpp and Ollama far easier to fit on small hardware, at some cost in precision. A fair comparison between tools uses the same precision in each.

5. What happens with several users

This is the real dividing line.

Ollamallama.cppvLLM
Requests at once, by default1 per model; OLLAMA_NUM_PARALLEL raises itA number of slots chosen automatically; -np sets itMany, batched together at every step
When busyQueues up to 512 requests, then answers with 503See its server READMERequests queue; under memory pressure some are paused and redone
Memory cost of parallelismGrows with parallel requests times context lengthThe slots draw on one shared poolA shared cache, sized from free GPU memory
One model across several GPUsNot covered in the pages readNot covered in the pages readYes, with tensor parallelism

Ollama's default of one request at a time per model is right for a personal assistant and wrong for a team. llama.cpp's server enables continuous batching by default and can serve a small group well. vLLM is designed around this problem: its scheduler, its cache and its metrics all exist to keep many requests moving on expensive GPUs.

6. Three defaults that surprise people

Ollamallama.cppvLLM
Context length4,096 tokens unless you raise itTaken from the modelTaken from the model, often tens of thousands
Listens on127.0.0.1, port 11434127.0.0.1; check the port it printsPort 8000
API keyNone; clients send one that is ignored--api-key, off by default--api-key, off by default, and it covers only some paths
Metrics for monitoringNot covered in the pages read--metrics, off by defaultOn by default at /metrics
The same model can look worse in one toolIf answers seem to forget the start of a long document in Ollama but not in vLLM, the cause is usually the 4,096-token default, not the model. Raise it with OLLAMA_CONTEXT_LENGTH before you compare quality.

None of the three is safe to put on the open internet as installed. Two listen only on the local machine by default, and vLLM's own documentation says not to rely on its API key alone. Put a proxy in front, as in vLLM for a team.

7. One client, three servers

All three accept the OpenAI client. Only the address changes:

from openai import OpenAI

# Pick one base_url:
#   Ollama:     http://localhost:11434/v1
#   llama.cpp:  http://localhost:PORT/v1   (the port the server printed)
#   vLLM:       http://localhost:8000/v1
client = OpenAI(base_url="http://localhost:11434/v1", api_key="not-needed-locally")

reply = client.chat.completions.create(
    model="MODEL_NAME",   # the name each server reports at /v1/models
    messages=[{"role": "user", "content": "Explain a KV cache in one sentence."}],
    max_tokens=100,
)
print(reply.choices[0].message.content)

This is the practical argument for not agonizing over the choice. Write your application against the OpenAI format, start with whichever tool fits your machine today, and move later by changing one line.

8. How to choose

Your situationStart withWhy
A laptop or desktop, just youOllamaOne install, models by name, sensible for one user
A Mac mini as a home assistantOllamaSee the Mac mini guide
No GPU, or an unusual onellama.cppThe widest hardware support, and it runs on CPU
You want to choose the quantization yourselfllama.cppQuantization is built in
A few colleagues on one workstationllama.cpp server, or Ollama with parallelism raisedEnough concurrency without a server project
A team, or an application with real trafficvLLMBatching, multi-GPU, metrics
A model too large for one GPUvLLMTensor parallelism
Windows, and you need vLLMvLLM inside WSL, or a Linux serverNo native Windows support

If you are heading for vLLM, the series Run your own LLM server with vLLM starts from the first command. If you want numbers, measure the same model at the same precision on your own hardware; the sizing guide shows how to run a load test and read it.

Sources

This article reflects public information available when it was written. Versions, defaults and prices can change; check the linked official documentation.

Every entry in the tables comes from each project's own documentation or README as read on November 12, 2026. None of the three was installed or timed for this article, so it contains no speed comparison; speed depends on your hardware and model, and the only honest figure is one you measure. Written for this site with AI assistance, from the sources above. For errors or additional information, please request a correction. Editorial policy