vLLM vs Ollama: which one should serve your model?

What vLLM and Ollama actually are

vLLM is an open-source inference and serving engine for large language models, released under Apache 2.0. Its documentation lists PagedAttention, continuous batching of incoming requests, chunked prefill and prefix caching as core features, alongside an OpenAI-compatible server so existing SDK code can point at it without changes.

Ollama is a single-binary model runner under the MIT license, with installers for macOS, Windows, Linux and a Docker image. You pull a model from Ollama's library and run it. It also exposes an OpenAI-compatible endpoint at http://localhost:11434/v1/, and its docs demonstrate the official OpenAI Python client against it.

The split matters more than the marketing: one is built to keep a GPU busy serving many callers, the other is built to make one model run on the machine in front of you.

The default setting that decides your throughput

Ollama's FAQ is explicit. OLLAMA_NUM_PARALLEL, the number of requests each loaded model handles at once, defaults to 1. Set nothing and the second caller waits. The same section states that required memory scales with OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH, so turning parallelism up raises KV cache memory, not just throughput.

Two more defaults matter. OLLAMA_MAX_LOADED_MODELS defaults to three times the number of GPUs, or three for CPU inference. When a new model does not fit in available memory, requests queue until a slot frees. And the default context window is 4,096 tokens, far below the 256K that Ollama's own qwen3.8 page advertises for the model.

vLLM answers the same problem architecturally. Continuous batching admits new requests into the running batch instead of waiting for the current one to drain, and prefix caching reuses shared prompt prefixes, which is exactly the shape of agent workloads where every call re-sends the same system prompt. That is why the two produce very different numbers as soon as more than one client shows up, and why single-prompt benchmarks of the pair tell you almost nothing.

Where each one breaks

• Ollama breaks on concurrency, not on quality. One request per model at a time means a three-person team shares a queue. Raising OLLAMA_NUM_PARALLEL helps, but the docs warn the memory cost scales with it.
• vLLM breaks on ops. It is a serving stack: you manage a GPU host, a process supervisor, model weight storage and version pinning. On a laptop there is nothing to install it into.
• Both break on context. The 256K window on qwen3.8 is a ceiling, not a budget. KV cache comes out of the same VRAM as the weights, and a long-context session on a 24GB card runs out before the weights do.

What people are actually running in November 2026

Qwen3.8-27B is the practical target for most teams. The Hugging Face repository lists 27.78B parameters in BF16, Apache 2.0, an image-text-to-text pipeline and 7.8M downloads. Ollama's qwen3.8:27b tag is 18GB with a 256K context window, which is only possible because the library build is quantized: the same weights are 55.6GB in BF16.

Kimi K3.1 Pro, uploaded 2026-11-02, sits at the other end. Its repository lists 2.78 trillion parameters and roughly 2.72TB of weights stored as MXFP4, under a custom license rather than an OSI one. Ollama's library has no Kimi K3.1 Pro page: the path returns 404. That is not a gap in Ollama, it is a size class. Nothing about that model is a laptop story, and it does not fit on one GPU at all.

Rent or run: what the numbers look like

RunPod's on-demand rates, checked on 2026-11-06: RTX 3090 24GB at $0.50/hr, L4 24GB at $0.59/hr, RTX 4090 24GB at $0.89/hr, L40S 48GB at $1.09/hr, RTX Pro 6000 96GB at $2.49/hr and H200 141GB at $5.29/hr. Left running for 730 hours, the 24GB RTX 4090 is about $650 a month and the L4 about $431.

Those rates set the break-even. Serving yourself one model with Ollama on hardware you already own costs nothing extra, and paying $650 a month for a pod to answer your own questions is waste. Serving a team or an agent fleet flips it: a single 24GB pod is cheaper than most seat-based AI plans, and vLLM is what makes that pod worth paying for, because the batching is what lets it absorb more than one caller.

Then there is the Kimi case. Capacity math alone: 2.72TB of weights across 141GB H200s is roughly 20 GPUs, and at $5.29/hr each that runs about $106/hr before a single token is served. Self-hosting that model is a procurement decision, not a weekend project.

Which one should you pick?

Pick Ollama if the endpoint serves you, or a handful of people who can tolerate a queue, and the machine is a Mac, a Windows box or a small Linux server. The install is one command, the model library is the fastest path from "I want to try Qwen3.8" to a running chat, and the OpenAI-compatible endpoint means your client code does not care which runner sits behind it.

Pick vLLM if the endpoint is shared, the prompts are long and repetitive, or you are paying for a GPU by the hour and want utilization above idle. Continuous batching and prefix caching exist precisely to stop a rented card from sitting at 5% utilization.

Pick neither if your real question is cost per token on a hosted API. A rented 24GB pod at $650 a month buys a lot of API calls, and self-hosting only wins once you have steady traffic or a data-residency reason. Our breakdown of what self-hosting an LLM really costs covers that trade.

The migration path is short, which is the reason not to over-think it: both expose OpenAI-compatible endpoints, so moving from Ollama to vLLM is mostly a base URL and a model name change, plus someone to keep the GPU host alive.

At a glance

vLLM vs Ollama, from each project's own documentation (checked 2026-11-06)

FAQ

Is vLLM faster than Ollama?

Neither project publishes a head-to-head benchmark, and on one request at a time the gap is not the interesting part. The documented difference is concurrency: Ollama handles one request per model by default, vLLM batches incoming requests continuously. vLLM pulls ahead once several callers share the endpoint.

Can I run vLLM on a Mac?

vLLM's documentation describes GPU execution with CUDA and HIP graphs and GPU quantized kernels. Ollama ships official macOS, Windows, Linux and Docker installers, which is why laptop comparisons almost always mean Ollama.

Does Qwen3.8-27B fit on a 24GB GPU?

Not at full precision: the Hugging Face weights are 55.6GB in BF16. Ollama's qwen3.8:27b tag is 18GB of quantized weights and does fit a 24GB card, but the KV cache comes out of the same memory, so the 256K context window is a ceiling you cannot afford in full.

Related reading

what self-hosting an LLM really costs, LM Studio vs Ollama, cheapest GPU cloud for LLM inference, RunPod alternatives

Sources

• vLLM documentation (features and serving)
• vLLM LICENSE (Apache 2.0)
• Ollama FAQ (concurrency, context, memory)
• Ollama OpenAI compatibility
• Ollama LICENSE (MIT)
• Ollama model library: qwen3.8
• Hugging Face: Qwen3.8-27B model card
• Hugging Face: Kimi-K3.1-Pro model card
• RunPod GPU cloud pricing