Stackrig

Home Blog

measured measured on Google Cloud, 8 October 2026 published

Self-hosted LLM or API: when does a rented GPU pay off?

We earn commissions when you shop through the links below.

gpt-oss-20b on one rented L4, measured: what a million output tokens cost at 1 to 64 requests at once, and how busy the GPU must be to beat five per-token APIs.

Short answer: only when it is kept busy with many requests at once. We served gpt-oss-20b on one rented NVIDIA L4. With one to four requests at a time it delivered 56 to 152 output tokens per second, which costs $1.29 to $3.48 per million output tokens at Google Cloud's on-demand price: several times what each of five per-token APIs charges for short chats and long answers.

With 64 requests at once it reached 1,008 output tokens per second for short chats, about $0.19 per million. At that load it beats four of the five APIs, but only if it carries that load for 39 to 61 % of the hours you pay for. The cheapest API in our table it does not beat at all at the on-demand price.

An open-weight model can be paid for in two ways. Through an API, often called serverless inference: you send requests, the provider runs the model on its GPUs, and you pay per token, nothing for idle time. Or on a GPU you rent by the hour and run the model on yourself: you pay every hour, busy or not. The question is how busy the rented GPU has to be before its hours cost less than the tokens. We measured one GPU with one model and did the arithmetic against the APIs' list prices. The answer depends on the model: a larger or a denser model moves it.

The setup

  • The machine: Google Cloud g2-standard-4 (one NVIDIA L4 with 24 GB, 4 vCPUs, 16 GiB) in us-central1, run as a spot VM on 8 October 2026. Spot only changes the price, not the speed; we apply both the on-demand and the spot price to the same numbers.
  • The model: OpenAI's gpt-oss-20b (Apache 2.0), a mixture of experts with 21B parameters, 3.6B of them active per token.
  • The server: vLLM 0.31.0 in its official Docker image, defaults except --max-model-len 16384.
  • The load: vllm bench serve with random prompts of fixed length and --ignore-eos, so every answer has its full length; seed 1, a warm-up of 10 requests. Three shapes: short chats (256 tokens in, 256 out), long prompts (2,048 in, 256 out) and long answers (128 in, 1,024 out), each with 1, 4, 16 and 64 requests at once, 20 to 320 requests per step. No request failed.

For one request at a time, others have published about 52 and 62 output tokens per second for this model on an L4 (a GKE setup on dev.to, devforth), close to our 59. We found no published number for what one L4 does with many requests at once.

What one L4 delivered, and what a million tokens cost on it

Measured 2026-10-08. Cost per million output tokens at 100 % use, our arithmetic from Google Cloud's prices of that day ($0.7068 an hour on demand, $0.4028 spot) and RunPod's L4 ($0.49 an hour), another host we did not run on. TTFT p99 = the slowest 1 % of waits for the first token.
ShapeAt onceOutput tokens/sTTFT p99On demandSpotRunPod
Short chats15975 ms$3.32$1.89$2.30
Short chats415283 ms$1.29$0.74$0.89
Short chats16379620 ms$0.52$0.30$0.36
Short chats641,0081,926 ms$0.19$0.11$0.14
Long prompts156361 ms$3.48$1.98$2.41
Long prompts4145107 ms$1.36$0.77$0.94
Long prompts162733,932 ms$0.72$0.41$0.50
Long prompts6445018,043 ms$0.44$0.25$0.30
Long answers16058 ms$3.29$1.88$2.28
Long answers414383 ms$1.37$0.78$0.95
Long answers16364398 ms$0.54$0.31$0.37
Long answers641,0481,102 ms$0.19$0.11$0.13

More requests at once means more tokens per second for the same hourly price, so the price per token falls. Short chats and long answers were still rising from 16 to 64 at once: 64 is our highest step, not the L4's ceiling. Long prompts reached only 450 output tokens per second at 64 at once, and there the slowest 1 % of them waited 18 s for their first token, past what a chat user waits for.

What the APIs charge

Standard tier, US dollars per million tokens, read on each provider's own page on 2026-10-08, in alphabetical order. The last three columns are one million output tokens with the input tokens each shape sends with them (our arithmetic).
APIInputOutputShort chatsLong promptsLong answers
Cloudflare Workers AI$0.20$0.30$0.50$1.90$0.325
DeepInfra$0.03$0.14$0.17$0.38$0.144
DigitalOcean$0.05$0.45$0.50$0.85$0.456
Google Vertex AI$0.07$0.25$0.32$0.81$0.259
Groq$0.075$0.30$0.375$0.90$0.309

At Google Cloud's on-demand price, the L4 at one to four requests at a time costs more than every one of these, with one exception: long prompts at four at once against Cloudflare, whose input price is the highest of the five.

How busy the GPU must be

A rented GPU costs the same whether it works or waits. If it carries its load for a share u of the hours you pay for and idles the rest, its cost per token is the table's number divided by u. It is cheaper than an API when u is above the break-even share:

u* = H × 1,000,000 / (3,600 × T × (p_out + r × p_in))
  • H: the GPU's price per hour, in dollars.
  • T: the output tokens per second it delivers at your load.
  • p_out, p_in: the API's price per million output and input tokens.
  • r: input tokens per output token in your traffic (short chats 1, long prompts 8, long answers 0.125).

One line from our tables: short chats at 64 at once against Groq's API, 0.7068 × 1,000,000 / (3,600 × 1,007.9 × (0.30 + 1 × 0.075)) = 0.52. The L4 is cheaper if it is busy at that load for more than 52 % of the hours you pay for. Put in your own prices and your own T; above 100 % the GPU is never cheaper at that load.

Break-even share u* of the paid hours for the L4 at Google Cloud's on-demand price, our arithmetic. Above 100 % the L4 is never cheaper at that load. The steps left out (one at a time, and four at once for short chats and long answers) are above 100 % against every API.
ShapeAt onceCloudflareDeepInfraDigitalOceanVertex AIGroq
Short chats16104 %305 %104 %162 %138 %
Short chats6439 %115 %39 %61 %52 %
Long prompts471 %357 %159 %167 %151 %
Long prompts1638 %189 %84 %89 %80 %
Long prompts6423 %115 %51 %54 %48 %
Long answers16166 %375 %118 %208 %174 %
Long answers6458 %130 %41 %72 %61 %

At 64 at once the L4 beats Cloudflare, DigitalOcean, Vertex AI and Groq if that load fills 39 to 72 % of the hours for short chats and long answers. DeepInfra's prices it does not beat on demand in any shape (115 to 130 % at 64 at once). Long prompts favour the GPU against the APIs that charge more for input, Cloudflare above all (23 %), but that is the shape whose first token came slowest. On spot the L4 costs 57 % of its on-demand price, so every share drops to 57 % of the table's: Cloudflare's 39 % for short chats becomes 22 %, and DeepInfra's 115 % becomes 65 %. A spot VM can be stopped by Google at any time, as Google's page on Spot VMs says.

In a month the same sum looks like this (our arithmetic): 730 hours of the on-demand L4 are $516. Busy around the clock with 64 short chats at once it would make about 2,649 million output tokens; at DigitalOcean's price, with their input, those cost $1,324. At 39 % of the hours busy the two are even.

Before the first token: 10.5 minutes

A rented GPU has to start before it answers. From "create the VM" to the first answer took 10 min 28 s: the machine and its driver were up after 41 s and Docker with the GPU after 74 s; pulling the vLLM image took 252 s and starting its server 27 s, and from the server's start to its first answer, which includes downloading and loading the model, took 275 s more. A GPU you start for a working day and stop at night pays that every morning, and a GPU that starts only when traffic comes makes its first users wait that long.

What this measurement does not tell you

  • One model, one GPU type, one serving stack in one version, one run on one day. How far the numbers move between runs is not known.
  • Real traffic does not arrive as exactly 16 or 64 requests at a time; the break-even share assumes the GPU carries the step's load for that share of the hours and idles the rest.
  • Standard prices only: no batch, flex or cached-input prices. The hourly price is the machine's; its disk, its network and the people who run it are not in it.
  • We did not compare the APIs' speed, limits, privacy terms, quantisation or answer quality.
  • Other GPUs were not run. DigitalOcean's GPU Droplets, for example, start at 20 GB, not the L4's 24, and are not in the table.

The same "what does one machine carry" question for a web app is How many users can a $12 server handle?

The app is free to try in your browser, without an account. To hear when Pro is ready, join the waitlist.

Sources

  • Our run on Google Cloud, 2026-10-08: vllm/vllm-openai 0.31.0, openai/gpt-oss-20b at commit 6cee5e8.
  • Google Cloud prices, read 2026-10-08: on demand and spot for the g2-standard-4 in Iowa (us-central1), and Spot VMs, which can be preempted at any time.
  • RunPod's prices, L4 on Secure Cloud, read 2026-10-08.
  • The five APIs' own price pages, linked in the table above, read 2026-10-08.

Stackrig