What does a self-hosted LLM actually cost in 2026?

What self hosting an LLM means now

A self-hosted LLM is a model whose weights you download and serve on hardware you rent or own, rather than calling a vendor's API. In 2026 that stopped being an experiment for hobbyists and became a line item: the weights are permissively licensed, the GPUs are rentable by the second, and the question is no longer whether it runs but whether the invoice beats the API.

OpenAI's gpt-oss-120b is the current reference point. Hugging Face lists it under the Apache 2.0 licence, ungated, with 2,854,564 downloads and 5,440 likes as of 26 October 2026, and the model card ships it in MXFP4 so the file size stays inside a single high-end card. That combination (permissive licence, ungated download, quantised weights) is what makes a self-hosted deployment a purchase decision rather than a research project.

The other half of the equation changed in the same year. IBM CEO Arvind Krishna told investors on the Q3 call that open-weight models now deliver "anywhere from 25 percent to 10 percent of the token cost of the frontier models" while being "now 90 percent as effective", per The Register's report on 22 October 2026. He was arguing for on-prem purchases. The numbers behind that argument are worth checking against the rental market yourself.

What a GPU-hour costs on the rental market

Runpod publishes its on-demand rates per GPU, per hour, and they are current as of 26 October 2026. The relevant rungs for LLM serving:

• RTX A5000, 24 GB: $0.27/hr. Fine for embeddings and small classification models, not for a 120B mixture-of-experts.
• L40S, 48 GB: $1.09/hr.
• A100 SXM, 80 GB: $1.79/hr.
• H100 PCIe, 80 GB: $2.89/hr. H100 SXM, 80 GB: $3.99/hr.
• H200, 141 GB: $5.29/hr. B200, 180 GB: $7.99/hr.
• B300, 288 GB: $8.99/hr.

Multiply by 730 hours (an average month) and the cheap end stops looking cheap. One H100 SXM runs $2,912.70 a month on-demand. Two A100s, the kind of pair a 78B-class model needs, run $2,613.40. A B200 runs $5,832.70. Those are compute-only numbers: storage, egress and the idle time between requests are billed separately or not billed at all because the pod is stopped.

This is also where the budget-VPS instinct fails. A RackNerd or similar $10-a-month box is a sensible host for a router, a queue or an eval harness. It cannot hold 120B of parameters in any format, and no amount of swap changes that.

The same money, priced in API tokens

Here is the comparison the GPU price lists never put side by side. Take the $2,912.70 that one H100 SXM costs for a month and spend it on tokens instead. At Groq's published rate of $0.15 per million input tokens for gpt-oss-120b, that budget buys 19.42 billion input tokens, or 4.86 billion output tokens at $0.60. Against a frontier model at $10 input and $50 output per million, the same money buys 291 million input tokens or 58 million output tokens.

The gap between those two token counts is the entire argument for open weights, and IBM's claim of 10% to 25% of frontier cost lands in the same range: gpt-oss-120b input is 66.7 times cheaper per token than gpt-6-astra input on the numbers above.

Then check the small-model rung, because it moves the floor. OpenAI's own price page on 26 October 2026 lists gpt-6-luna at $0.10 input and $0.50 output per million, with cached input at $0.01. That is cheaper than Groq charges to host the open-weight 120B model, and it means the "self-host to save money" case now has to be made against a closed model at ten cents, not against a $10 flagship. For a summarisation or classification workload, $2,912.70 buys 29.1 billion luna input tokens.

One caveat that matters for honesty: Groq's $0.15 rate is served on Groq's own silicon at 500 output tokens per second, not on an H100. The token price tells you what a competitor charges for the same weights. It does not tell you what your H100 will produce in an hour.

Utilisation is the number that decides it

Nobody runs a GPU at 100% for a month. Price the card by how long it is actually working, because that is what you are buying:

• 24/7 (730 hours): $2,912.70/mo, equivalent to 19.4 billion gpt-oss-120b input tokens. You need serious, steady traffic to justify it.
• 12 hours a day (365 hours): $1,456.35/mo, or the price of 9.7 billion input tokens.
• 8 hours a day (243 hours): $969.57/mo, or 6.5 billion input tokens.
• 4 hours a day (122 hours): $486.78/mo, or 3.2 billion input tokens.

The rule that falls out of this: a rented GPU has to serve more tokens in the hours you pay for than the API would charge for the same tokens in that month. At $0.10 per million for a small closed model, an 8-hour-a-day H100 has to beat 9.7 billion tokens. Below that volume, a rented card is a fixed cost you cannot shrink and an API is a variable cost that follows demand to zero.

The exceptions are real, though. Batch jobs that can saturate a card overnight, fine-tuning and distillation runs that APIs price poorly or not at all, and workloads where the prompt cannot leave your infrastructure all justify the fixed cost regardless of the token maths.

The costs the hourly rate leaves out

Krishna put the operational problem in plain terms: most businesses can build useful AI infrastructure in two or three racks, but the enterprise struggle is "where are my kernel engineers? Where are the people who worry about the underlying network?" That question does not appear on a GPU price page and it is the most common reason a self-hosted deployment quietly goes back to an API.

• You become the inference team. Serving stack, batching, quantisation, model updates, monitoring, and the pager rotation when throughput drops at 2 a.m.
• Capacity is your problem in both directions. A spike queues behind your own hardware. An idle weekend is money spent on nothing, which the per-second billing makes visible rather than free.
• The quality gap is real at the top. A 90%-as-effective model is fine for summarising a contract and wrong for the task that needed the frontier. The sensible architecture is usually both: open weights for volume, a frontier API for the hard tail.
• Licence homework still applies. gpt-oss-120b is Apache 2.0 and ungated. Not every downloadable model is: check the licence field on the repository before you plan a product around it.

What to actually run on the card

Three open-weight models give you a usable spread across the rental ladder, and Groq's model list publishes the per-token price of each so you can check the self-hosted bill against a hosted one before you rent anything.

• gpt-oss-120b: Apache 2.0, ungated, MXFP4 weights, 2,854,564 downloads on Hugging Face. Hosted at $0.15 input and $0.60 output per million tokens, 500 tokens per second, 131,072-token context. This is the sensible first candidate for an H100 or H200.
• gpt-oss-20b: the smaller sibling, hosted at $0.075 input and $0.30 output, 1,000 tokens per second. It fits comfortably on a 24 to 48 GB card, which is where Runpod's $1.09/hr L40S starts to make sense for steady internal traffic.
• Qwen3.8-27B: hosted at $0.80 input and $4.00 output per million tokens, 450 tokens per second, 131,072-token context. The hosted price is high enough that a self-hosted deployment on a 48 GB card has a real chance of beating it, which is not true at the 20B tier.

That last point is the practical filter. Compute the hosted price of the exact model you plan to run, multiply it by your monthly token volume, and compare it against the hours you would pay for. When the hosted rate is $0.075 per million input, the GPU almost never wins. When it is $0.80, the arithmetic flips much sooner.

When self hosting wins

Rent the GPU when the workload is steady, the model class is open-weight, the data cannot leave your network, or you are doing training-adjacent work that APIs do not sell per token. On the numbers above, that means committing to billions of tokens a month, not millions.

Stay on the API when traffic is spiky, when the task needs flagship quality, or when nobody on the team wants to own a serving stack. At $0.10 per million input tokens for gpt-6-luna and $0.15 for gpt-oss-120b on Groq, the API is not just the low-volume option any more. It is often the cheaper one at volumes that would have justified hardware two years ago.

The decision is a spreadsheet, not a principle: tokens per month, divided by the hours you would actually pay for, compared against the current per-token rate for the quality tier you need. Every input in that calculation is published. Most self-hosting debates skip it and argue about benchmarks instead.

At a glance

GPU rental versus token pricing, verified 26 October 2026 (USD)

FAQ

Is a self-hosted LLM cheaper than an API?

Only at volume. A Runpod H100 SXM costs $3.99 an hour, $2,912.70 for a full month, which equals 19.4 billion gpt-oss-120b input tokens at Groq's $0.15 per million. If your workload is smaller than that, or the GPU sits idle, the API is cheaper.

What hardware do I need to self-host an open-weight model?

It depends on the quantised weight footprint. Runpod's ladder starts at $1.09/hr for a 48 GB L40S, $3.99/hr for an 80 GB H100 SXM and $5.29/hr for a 141 GB H200, so the card's VRAM, not the parameter count on the label, sets the bill.

Do I still need an API if I self-host?

Most teams keep one. IBM's Arvind Krishna puts open-weight models at roughly 90% of frontier effectiveness, so the usual split is open weights for high-volume work and a frontier API for tasks that need the last 10%.

Related reading

cheapest GPU cloud for LLM inference, best local LLM tools, Ollama alternatives, sovereign AI open-weight models

Sources

• The Register: IBM CEO says open weight models are now irresistibly good and cheap on-prem (22 Oct 2026)
• Runpod GPU cloud pricing, per-hour on-demand rates
• GroqCloud models and per-token pricing
• OpenAI API pricing (gpt-6-astra, gpt-6-luna)
• Hugging Face model card: openai/gpt-oss-120b (Apache 2.0, downloads)