How do you measure latency for streaming LLM responses?

Nov 3, 2026 - 7 min read

Measure streaming LLM latency from the application’s request start to three distinct events: first visible content, first task-usable output, and the completed response. Then report whether acceptable results arrive before the task’s deadline, with failures and timeouts included. A fast first token is useful feedback, but it does not establish that the user got a usable answer sooner.

Nodus’s hosted inference API (Beta) supports streaming chat responses. The measurement approach below helps you decide whether a streaming experience meets your application’s needs. It is not a Nodus performance benchmark or a claim that Nodus provides these measurements automatically.

Which latency should determine your decision?

Choose the event that lets the customer move forward.

For a reader following a long explanation, a meaningful opening sentence may be useful before the answer finishes. For a tool consuming a complete decision, partial text may have no value. An apparently valid opening can also become incorrect once the rest of an answer arrives.

Keep these boundaries separate:

Measurement	Start and finish	What it tells you
Time to first content	Request begins to first answer content received by the application	How long the initial silence lasts
Time to first usable output	Request begins to the first output that meets a predefined task requirement	When the user can safely make progress
Time to completed response	Request begins to the end of the response	When response production has finished, not whether the answer is correct
Time to accepted task result	Original task begins to a complete result that passes the application’s checks	The wait for success, including necessary retries or corrections

The first measurement is often called time to first token, or TTFT. State your definition: the first nonempty answer content is not the connection opening, response headers, or an empty event. If you measure displayed content instead of received content, label it that way. Rendering can add a further wait.

For all boundaries, use elapsed time observed by the same application. A service-side duration and a browser-observed duration describe different portions of the experience.

Define “usable” before comparing responses

A usability boundary needs a task-specific rule, not a generous interpretation after seeing the results.

For a writing assistant, the rule might be a complete opening paragraph that a reviewer judges relevant and accurate in context. For a classification task, it might require the complete permitted category plus successful validation. For a generated tool action, do not treat an unfinished suggestion as permission to execute it.

Sometimes the first usable output is the entire accepted response. That is an honest result. Streaming still may reassure the user that work is happening, but reassurance and readiness for action are different benefits.

Record when candidate output arrives, then judge whether it actually met the predefined rule. If later content undermines an opening claim, do not count that opening as usable merely because it looked promising at the time.

Worked example: the faster stream can miss the deadline

The following numbers are invented for illustration, not measured Nodus performance. Imagine a support application comparing two response configurations on the same 100 requests. It requires a complete, acceptable reply within 10 seconds. Partial replies are visible but cannot be sent to customers.

Result	Configuration A	Configuration B
Median time to first content, across responses that produced content	0.4 seconds	0.9 seconds
Median completion time, across responses that completed	7 seconds	5 seconds
Acceptable complete replies within 10 seconds	78 of 100	91 of 100
Acceptable replies completed after the deadline	12 of 100	4 of 100
Completed but unacceptable replies	6 of 100	3 of 100
Failed or timed-out requests without a completed reply	4 of 100	2 of 100

The four outcome rows add to 100 for each configuration. The conditional medians do not include requests that never reached their respective event; those requests remain visible in the outcome counts.

A starts showing content sooner. B delivers more acceptable complete replies before the application’s deadline: 91%, compared with 78%. For this task, B is the stronger candidate despite its slower first content. This example is a decision illustration, not enough evidence to establish a reliable performance advantage.

If the product instead required useful explanation while a reader waits, compare the predefined usable-output event too. Do not select whichever metric makes a preferred configuration look best.

Compare under the request traffic you intend to serve

A quiet single-request test does not answer how an application behaves when several users arrive together. Separate results for ordinary traffic and the bursts you actually expect.

Keep the input set, answer requirements, output limits and retry policy comparable. If you deliberately change models or request settings, describe that as a comparison of whole configurations, not an isolated test of streaming. Balance test order so one candidate does not always receive the easier traffic period.

Also preserve the arrival pattern. A test that waits for each request to finish before starting the next sends less traffic when responses slow down. That can hide the very congestion your users would experience. State whether the test used a fixed arrival schedule or a fixed number of requests in progress.

Report the median and a tail measure such as the 95th percentile, with sample counts and the population defined. A 95th percentile from a small test is unstable; repeat the comparison before treating it as a service expectation. Do not average several groups’ 95th percentiles and call the result the overall 95th percentile.

Keep interrupted streams and retries in the result

A stream can emit convincing text and still stop before the response completes. Receiving content is therefore not proof of completion, and completion is not proof of task success.

The Nodus inference guide distinguishes completed chat streams from interrupted or errored streams. That distinction matters when counting outcomes. Do not silently turn an interrupted response into a successful short answer, or exclude it from a chart because it has no completion time.

For retries, report both attempt-level measurements and original-task outcomes. Start the task timer before the first attempt, and include waiting, retries and validation until an acceptable result arrives or the task deadline is reached. Otherwise, a fast successful retry can conceal the user’s earlier failed wait.

Nodus documents request identifiers and receipts for reviewing token usage and charges. These can support a usage review alongside your latency experiment; they do not replace application-observed timing or answer-quality checks.

FAQ

Does streaming make the model finish sooner?

Not necessarily. Streaming changes when content becomes available to the caller. Measure completion time separately rather than inferring a speedup from earlier visible text.

Should failed requests receive a latency of zero?

No. Keep them as failures with their observed elapsed time or timeout boundary. Report their count alongside latency statistics, not as instant successful responses.

Is tokens per second enough to compare user experience?

No. It leaves out the initial wait, answer length, interruptions and whether the result was usable. It can be a diagnostic measure, but not the sole acceptance criterion.

What should go in the release decision?

Use acceptable results before the task deadline as the primary measure when complete answers are required. Add first-content timing for perceived responsiveness and tail latency for slow experiences. Inspect important task types separately so a good overall result does not hide a critical failure.

Where Nodus fits

Nodus offers hosted model inference in Beta, including streaming chat responses. Check the current inference guide for model capabilities and availability before testing. Set your acceptance boundary first, then evaluate the actual customer experience; this article promises no latency target or service guarantee.

If you want to discuss your inference needs and whether Nodus fits its response requirements, book a conversation.