How much does LLM inference really cost?

There is no single price for running an LLM.

You can pay for tokens.

You can reserve a managed model endpoint.

Or you can rent the GPUs and run the model yourself.

All three can be described as “LLM inference.” Their pricing units are different because the products underneath them are different.

That creates a common mistake in AI buying: compare one provider's price per million tokens with another provider's GPU-hour, decide which number looks smaller, and assume you found the cheaper option.

You probably haven't.

The real question is:

What does it cost to serve your workload at the quality, latency, throughput, and reliability you actually need?

Once you calculate that, the cheapest path can change considerably.

Start with the three ways to buy inference

Most LLM infrastructure costs fall into three broad models.

1. Pay per token

This is the familiar API model.

You send input tokens, receive output tokens, and pay according to usage.

The basic calculation is:

Monthly cost = input tokens × input rate + output tokens × output rate

That makes per-token APIs easy to start with because the cost tracks consumption.

Use almost nothing and the bill stays small.

Send ten times as much work and the bill grows with it.

For uncertain or intermittent workloads, that flexibility is valuable.

2. Pay for a dedicated endpoint

A dedicated endpoint changes the unit from consumption to capacity.

With Hivenet Inference API, for example, dedicated endpoints use per-replica pricing billed by the second.

The calculation becomes:

Monthly cost = replicas × endpoint rate × runtime

For this example, we use Hivenet's advertised 2 × RTX 5090 endpoint tier, starting at €2.10/hour per replica, checked September 28, 2026. Confirm the model, serving variant, configuration, and current rate in the console before deploying.

Run one continuously for a 730-hour month and the starting infrastructure cost is about €1,533.

There is no token meter on Hivenet's dedicated endpoint path. Your cost follows the capacity you deploy and how long it runs.

That makes the model more attractive as utilization rises.

3. Rent GPUs and run inference yourself

Raw GPU rental moves another layer of responsibility to the buyer.

With GPU/CPU rental from Hivenet, an RTX 5090 currently starts at €0.75 per GPU-hour, billed by the second.

The infrastructure calculation looks simple:

GPU cost = number of GPUs × GPU rate × runtime

One RTX 5090 running continuously at €0.75/hour works out to €547.50 for a 730-hour month.

That number can look dramatically cheaper than a managed endpoint.

It is also buying much less.

You have the GPU.

Your team still has to turn it into a reliable inference service.

‍

‍

The cheapest headline rate is often the least complete number

This is the point where many LLM cost calculators become misleading.

They compare:

€0.75/hour

with:

€2.10/hour

and conclude the first is 64% cheaper.

Mathematically, yes.

Operationally, those numbers describe different units.

Hivenet's raw Compute path gives your team the instance and control of the stack. Hivenet's managed inference path operates the serving layer and exposes an OpenAI-compatible endpoint.

If you rent the GPU, your actual stack may also need:

• a model checkpoint and serving configuration
• vLLM, TGI, SGLang, llama.cpp, or another serving engine
• a gateway
• authentication
• routing
• monitoring and metrics
• replica management
• failure handling
• upgrades
• someone to keep all of it working

None of those things makes self-managed inference a bad deal.

For teams that already have the infrastructure expertise, need deep customization, or want full control over the serving environment, raw GPUs can be exactly the right choice.

But engineering work does not become free because it does not appear on the cloud invoice.

A one-billion-token example shows why the unit matters

Consider a production workload processing 1 billion tokens per month.

Assume:

• 750 million input tokens
• 250 million output tokens
• continuous availability

Now compare three ways to buy the infrastructure.

Per-token API

OVHcloud's Qwen3.6-27B catalog entry lists €0.40 per million input tokens and €2.70 per million output tokens, checked September 28, 2026. These are the rates used in the example below.

The calculation is:

750 × €0.40 = €300

plus:

250 × €2.70 = €675

Total:

€975/month

At this volume, the per-token API is still below the continuous cost of the Hivenet dedicated endpoint used in this example.

Hivenet dedicated endpoint

The advertised 2 × RTX 5090 endpoint tier used here starts at €2.10/hour per replica. Hivenet bills dedicated endpoints per replica and by runtime rather than by token.

For continuous operation:

€2.10 × 730 = €1,533

Total:

from €1,533/month

One replica has to be sufficient for the workload's concurrency and latency requirements. If more capacity is required, the cost rises.

Raw RTX 5090 capacity

One Hivenet RTX 5090 currently starts at €0.75/hour.

For continuous operation:

€0.75 × 730 = €547.50

Raw infrastructure cost:

from €547.50/month

It is the smallest number on the page.

It should not automatically win.

You still need to determine whether the model and serving configuration fit that hardware and whether your team wants to operate the serving stack.

That is the cost most simple calculators leave out.

Use total cost, not just cloud cost

There are four parts of LLM inference cost worth calculating separately.

Infrastructure or API spend

This is the visible bill.

Tokens, replicas, GPUs, storage, networking, or whichever unit the provider charges.

It is the easiest part to compare.

Utilization

A dedicated endpoint or GPU can be cheap per unit of work when it stays busy.

It can be expensive when it sits idle.

Per-token APIs reverse that relationship. Low utilization is their strength because you are not reserving the underlying capacity.

This is why pay-per-token and dedicated inference become different economic choices rather than two interchangeable pricing pages.

Operational cost

If your team manages the serving stack, someone has to build and maintain it.

The relevant question is not whether your engineers can run vLLM.

It is what else they would be doing if they did not have to.

For a company building AI infrastructure as a core capability, the work may be strategic.

For a company whose actual product happens to use an LLM, it may be overhead.

Model efficiency

The same workload can require different infrastructure depending on how the model is served.

This is where quantization and serving optimization turn into commercial factors.

The published Qwen3.6-27B NVFP4 model card documents a two-GPU serving configuration. Its speed comparison uses four GPUs for both NVFP4 and FP16, so that comparison alone does not establish a universal hardware reduction. Our hardware-efficiency discussion explains the cost implications; capacity still needs to be sized for your workload.

Across the reported accuracy evaluations, it retains between 95.5% and 100% of the corresponding FP16 benchmark scores.

A cost calculator that ignores the model configuration underneath the endpoint misses part of the economics.

Per-token pricing is often cheapest before utilization builds

At low volume, a token API has an obvious advantage:

You do not pay for idle capacity.

Suppose your workload falls from 1 billion tokens to 100 million.

Using the same OVHcloud pricing and 3:1 mix, the example bill falls from €975 to about:

€97.50

A continuously running €2.10/hour endpoint would still cost €1,533.

A continuously running €0.75/hour GPU would still cost €547.50 before operating costs.

For a prototype or sporadic application, there is no mystery here.

The token API wins the direct cost comparison by a wide margin.

That is why starting with consumption pricing is often sensible.

The economics change only when usage does.

Dedicated endpoints improve as the capacity gets used

Now move in the other direction.

At two billion tokens per month, the same OVHcloud example reaches approximately:

€1,950

At three billion:

€2,925

The example Hivenet endpoint's starting monthly cost is still €1,533 if one replica can carry that traffic and remains running continuously.

This is the crossover effect we explored in when dedicated inference becomes cheaper than paying per token.

Using the OVHcloud rates and Hivenet starting endpoint rate checked September 28, 2026, this example reaches cost parity at roughly 1.57 billion total tokens per month: €1,533 divided by €0.975 per million tokens at a 3:1 input/output ratio. This assumes one replica runs for 730 hours and can meet the workload's capacity and latency needs. It is a single-provider cost illustration, not a verified market-wide range or a claim of equivalent performance or quality.

Dedicated capacity is not inherently cheap.

Used dedicated capacity can be cheap.

That distinction matters.

Raw GPU rental can win when your team wants the work

There is another point where the calculation can change.

A technical team may already have:

• infrastructure automation
• serving expertise
• monitoring
• deployment pipelines
• model evaluation
• GPU scheduling
• people responsible for production ML systems

For that organization, self-managing a model may add little marginal operational cost.

Raw GPU rental can then become extremely attractive.

Hivenet's current Compute offering explicitly separates that use case from managed inference: use Compute when you want full control over the instance and serving stack; use Inference API when you want Hivenet operating the model endpoint.

That gives buyers a useful choice rather than forcing every workload into the same product.

If your team wants to run vLLM itself, rent the GPU.

If your team wants to call the model through an API, price the managed endpoint.

Availability changes the math too

Most simple monthly calculations assume 24/7 infrastructure.

Your workload may not need that.

Hivenet bills both Compute and dedicated inference by runtime rather than forcing a full monthly commitment. The Inference API uses per-replica, per-second billing, while Compute also uses per-second billing.

That means the 730-hour examples are deliberately conservative.

If an internal batch workflow only needs inference for eight hours a day, the runtime cost can be very different.

If a customer-facing product has to answer requests at 3 a.m., continuous availability may be unavoidable.

Again, the workload decides.

Your cost calculator needs at least seven inputs

A useful LLM inference calculation should collect more than monthly tokens.

At minimum, know:

Input
Why you need it

• Monthly input tokens
  Determines metered input cost
• Monthly output tokens
  Often the more expensive part of metered APIs
• Traffic distribution
  Determines how efficiently dedicated capacity can be used
• Peak concurrency
  Helps size replicas or GPUs
• Prompt/context length
  Affects memory and serving performance
• Latency target
  Can change required capacity
• Availability requirement
  Determines how many hours capacity must stay running

Then add the question that does not fit neatly into a spreadsheet:

Do we want to operate the inference stack ourselves?

That answer determines whether raw GPU rental and managed inference are genuinely substitutes for your team.

The cheapest model can still be the expensive choice

Infrastructure is only useful if the model does the job.

Suppose Model A costs half as much as Model B but fails your workflow often enough that requests need retries, human review, or escalation to Model B.

The cheap call just became more expensive.

The same logic applies to aggressive quantization.

Saving hardware while moving below the application's quality threshold is not efficiency.

This is why Hivenet measures model quality alongside serving efficiency rather than presenting infrastructure cost as an isolated win.

In Hivenet's reported Qwen3.6-27B comparison, LiveCodeBench scored 81.60 with NVFP4 versus 85.42 with FP16, or 95.5% of the FP16 score. The published model card and evaluation details describe the test conditions; this is not a guarantee of equivalent results on every workload.

That gives a buyer something concrete to evaluate against their own acceptance threshold.

Sometimes the biggest saving comes from using a different model

There is one more lever before you optimize the infrastructure.

Ask whether every request needs the model it currently receives.

A company can spend weeks reducing the serving cost of a frontier model while routing simple extraction, classification, RAG, and summarization calls through capability they barely use.

Our analysis of which frontier-model calls a 27B model can handle for less looks at that problem directly.

The most effective cost architecture may combine several approaches:

EASY / REPEATABLE WORK
       ↓
efficient 27B model

HARD / AMBIGUOUS WORK
       ↓
frontier model

STEADY HIGH-VOLUME TRAFFIC
       ↓
dedicated capacity

VARIABLE / LOW-VOLUME TRAFFIC
       ↓
per-token API

Model selection and infrastructure selection are part of the same cost problem.

A better LLM cost calculation starts with the workload

There is no universally cheapest way to run inference.

Per-token APIs are hard to beat when usage is low and unpredictable.

Dedicated endpoints become more interesting when production traffic is steady enough to use the capacity.

Raw GPU rental can deliver the lowest direct infrastructure cost when your team wants the control and is prepared to run the stack.

The mistake is comparing the billing units before comparing the jobs they perform.

So before choosing an inference provider, write down:

How much do we use?

When do we use it?

How hard is the traffic to serve?

What quality do we need?

Who do we want operating the stack?

Then calculate the cost.

That gives you a number worth comparing.

Explore Hivenet Inference API for managed dedicated capacity, compare GPU/CPU rental if you want to operate the stack yourself, or talk to our team with your current model, token volume, traffic pattern, and latency target. We can help you model the three paths against the workload you actually run.