How many tokens per second should an LLM serve on one GPU?
One H100 serves an 8B model at 1,300 to 26,000 output tokens/s in aggregate, but one user sees far less. How to read tok/s and price it.
One NVIDIA H100 serving an 8B model at full load produces about 1,300 to 26,000 output tokens per second in total, depending on prompt length, while any single user sees only a few dozen to a few hundred, capped by memory bandwidth. On an H100 SXM listed at $2.60/hr on the Nodus pricing page, NVIDIA’s published 14,992 tokens per second for Llama 3.1 8B works out to about $0.05 per million output tokens, and serving one user at a time costs about 72 times more. So the right target is not one number: set a per-user floor for chat (around 30 to 50 tokens per second), then pack in as many users as the GPU allows while that floor holds.
What does “tokens per second” actually measure?
Section titled “What does “tokens per second” actually measure?”It measures two different things, and most confusion comes from mixing them. NVIDIA’s LLM benchmarking metrics guide defines both.
Total tokens per second (TPS) per system is all output tokens across every simultaneous request, divided by the benchmark’s wall time. This is what decides your cost.
TPS per user is one request’s output length divided by its end-to-end latency. It approaches 1 divided by the inter-token latency (ITL), the gap between consecutive tokens. This is what a person watching a chat window feels.
The guide states the tradeoff plainly: as concurrent requests increase, total system TPS rises while TPS per user falls. Two more numbers complete the picture:
| Metric | What it tells you | Who cares |
|---|---|---|
| Time to first token (TTFT) | Queueing plus prefill plus network before the first token | Chat users, voice agents |
| Inter-token latency (ITL) | Time between tokens while streaming | Chat users |
| TPS per user | Output speed for one request | Chat, agents waiting on a reply |
| Total TPS per system | Output across all requests | Batch jobs, whoever pays the bill |
Definitions from NVIDIA’s metrics guide. Tools differ on whether TTFT is included in ITL, so compare numbers only when the definitions match.
Why is one user’s speed capped by memory bandwidth?
Section titled “Why is one user’s speed capped by memory bandwidth?”Generating each new token means reading every model weight from GPU memory once. With a single request, there is nothing else to share that read with, so the ceiling is roughly memory bandwidth divided by the bytes of weights. Compute barely matters at this point.
Here is that ceiling worked out for Llama 3.1 8B at 16-bit precision, about 16 GB of weights, using bandwidth from NVIDIA’s own spec pages:
- L4: 300 GB/s / 16 GB = 18.75 tokens per second
- L40S: 864 / 16 = 54 tokens per second
- A100 80 GB SXM: 2,039 / 16 = 127.4 tokens per second
- H100 SXM: 3,350 / 16 = 209.4 tokens per second
- H200: 4,800 / 16 = 300 tokens per second
Bandwidth figures are from NVIDIA’s pages for the L4, L40S, A100, H100 and H200. These are upper bounds we calculated, not measurements. Real decoding also reads the KV cache and pays kernel overhead, so expect less.
Single-stream speed tracks memory bandwidth, so an H200 can stream one user about 16 times faster than an L4.
The same arithmetic gives two quick rules. Halving the bytes per weight, for example by serving in 8-bit (FP8), doubles the ceiling: 418.75 tokens per second for the 8B model on an H100. A 70B model in FP8 is about 70 GB, which leaves an H100 a ceiling of 3,350 / 70 = 47.9 tokens per second for one user, before any KV cache traffic.
How many tokens per second does one GPU serve with batching?
Section titled “How many tokens per second does one GPU serve with batching?”Far more than the single-user ceiling, because a batch of requests shares each read of the weights. NVIDIA’s TensorRT-LLM performance overview publishes total output throughput at maximum load. For Llama 3.1 8B in FP8 on one H100 SXM:
| Input, output tokens per request | Output tokens per second, one H100 |
|---|---|
| 128, 128 | 26,401 |
| 1,000, 1,000 | 14,992 |
| 5,000, 500 | 3,276 |
| 20,000, 2,000 | 1,341 |
Source: TensorRT-LLM performance overview, throughput tables, checked November 18, 2026. NVIDIA describes these as reference points, not peak performance.
Long prompts cut the same GPU’s output throughput by up to 20 times, so ask any benchmark what input length it used.
Three things in that table matter more than the headline number. Prompt length dominates: long inputs spend GPU time on prefill and fill memory with KV cache, so fewer requests fit in a batch. Bigger models scale down fast: the same page lists Llama 3.3 70B FP8 at 4,181 tokens per second on two H100s at 1,000 in and 1,000 out, or about 2,091 per GPU. A faster GPU helps less than its spec suggests: the H200 row for 8B at 1,000 and 1,000 is 17,162, only 14% above the H100.
These are maximum-throughput runs, and they hide the user’s experience. The example output on the same page reports 5,531 total output tokens per second alongside a per-user output speed of 5.47 tokens per second and an average TTFT of 52.7 seconds. That is a fine result for an overnight batch job and an unusable one for chat. NVIDIA’s parameters guide explains why: throughput saturates near the maximum batch size while latency keeps rising.
Is 40 tokens per second good?
Section titled “Is 40 tokens per second good?”For one person reading a streamed chat reply, yes. At NVIDIA’s rule of thumb of about 0.75 English words per token, 40 tokens per second is 30 words per second, with 25 ms between tokens. That is comfortably faster than people read, so pushing a chat endpoint much higher per user mostly costs money.
It is not good in two other cases. Reasoning models and agents generate tokens nobody reads: a 2,000-token reasoning trace at 40 tokens per second is 50 seconds of waiting before the answer starts. For those, per-user speed is latency, and faster is worth paying for.
It is also a poor number if it is your whole GPU. One H100 at $2.60/hr producing 40 tokens per second costs $2.60 x 1,000,000 / (40 x 3,600) = $18.06 per million output tokens. The same GPU at NVIDIA’s 14,992 tokens per second costs $0.05.
The decision rule we use:
- Chat and voice: pick a per-user floor (30 to 50 tokens per second is comfortable for reading) and a TTFT limit, then raise concurrency until either one breaks.
- Agents and reasoning: set the floor from the total wait you can accept. Hidden tokens divided by tokens per second is seconds of waiting.
- Batch jobs: ignore per-user speed and TTFT. Maximize total throughput and judge by cost per million tokens.
How do you convert tokens per second into cost per million tokens?
Section titled “How do you convert tokens per second into cost per million tokens?”Divide the hourly price by the tokens produced in an hour. We use the H100 SXM listed at $2.60/hr on the Nodus pricing page, checked November 18, 2026, with NVIDIA’s published 8B throughput:
- 26,401 tokens per second: $2.60 x 1,000,000 / (26,401 x 3,600) = $0.027 per million output tokens
- 14,992 tokens per second: $0.048
- 3,276 tokens per second: $0.22
- 1,341 tokens per second: $0.54
- One user at the 209.4 tokens per second bandwidth ceiling: $3.45
Batching, not the hourly rate, sets the price: the same H100 hour costs 72 times more per token when it serves one user.
The 70B case costs more because it needs two GPUs: two H100s at $5.20/hr and 4,181 tokens per second come to $0.35 per million output tokens. The H200, listed at $3.593/hr, comes to $0.058 at 17,162 tokens per second, slightly more per token than the H100 here because its price rises faster than this throughput.
These figures assume the GPU stays saturated and count output tokens only. Real traffic has idle hours and input tokens. Our post on LLM inference cost per million tokens covers input, cached and reasoning token pricing, and whether self-hosting is cheaper than an API works through break-even volume and how busy the GPU must be.
How do you measure tokens per second for your own workload on Nodus?
Section titled “How do you measure tokens per second for your own workload on Nodus?”Run a concurrency sweep with your real prompt and output lengths, not synthetic 128-token prompts. NVIDIA’s parameters guide recommends controlling load by concurrency, the number of requests in flight, and recording TTFT, ITL and total TPS at each level. The useful point is the highest concurrency where your per-user floor and TTFT limit still hold. Total TPS at that point, divided into the hourly price, is your real cost per token.
To stand up the server itself, our guide to hosting your own LLM behind an OpenAI-style API has the vLLM commands, and vLLM vs SGLang compares the two engines. On Nodus, a Sweep runs the same Job across GPU types and parameter values, then reports each cell’s cost, wall time and throughput, so a single nodus apply -f sweep.yaml can be set up to compare, say, an L40S against an H100 at several concurrency levels under a maxCostUSD cap. We have not published such a sweep ourselves yet. Throughput there counts the units your program reports, so log one unit per request and compute tokens per second from your benchmark client.
Why do published tokens per second numbers disagree so much?
Section titled “Why do published tokens per second numbers disagree so much?”They usually differ in input and output length, precision (16-bit, FP8, FP4), engine, concurrency and whether they report per-user or total throughput. NVIDIA’s own H100 table spans 1,341 to 26,401 tokens per second for one model on one GPU just by changing lengths.
Does a B200 with FP4 lower the cost per token for a 70B model?
Section titled “Does a B200 with FP4 lower the cost per token for a 70B model?”In NVIDIA’s published numbers, yes. The same TensorRT-LLM page lists Llama 3.3 70B in FP4 at 6,434 output tokens per second on one B200 at 1,000 in and 1,000 out. At the B200’s listed $5.50/hr on Nodus, that is $0.24 per million output tokens, against $0.35 for FP8 on two H100s, provided FP4 quality passes your evals.
Can a small GPU like an L4 serve chat?
Section titled “Can a small GPU like an L4 serve chat?”For a small model and few users, yes, if the model is quantized. The 16 GB 8B model has an 18.75 tokens per second single-user ceiling on an L4, below a comfortable chat floor, while 8-bit weights double that to 37.5. The L4 is listed at $0.430/hr on the Nodus pricing page.
More from the blog
api8 min read
How do you host your own LLM behind an OpenAI-style API?
Run vLLM's OpenAI-compatible server on a GPU that fits the model, lock it down, and change one base_url. Commands, GPU costs and a decision rule.
Read the postcapacity9 min read
What are the best Shadeform alternatives for buying GPU time?
Shadeform alternatives compared: Nodus, SkyPilot, Vast.ai and Runpod, with H100 prices, recovery after a lost machine and a rework break-even rule.
Read the post