Streaming LLM Responses: TTFT, Perceived Latency, and UX That Feels Fast

A 500-token answer at 60 tokens per second takes over 8 seconds to finish. Show nothing until it is done and your product feels broken. Stream it and the same 8 seconds feels responsive, because the user starts reading at the half-second mark and the text stays ahead of their eyes. Streaming does not make the model faster; it moves the moment the user starts getting value from the end of generation to the start of it, which is the difference that perception actually measures.

What actually determines how fast an LLM response feels?#

Two numbers dominate perceived speed: time to first token, which decides how long the user stares at nothing, and tokens per second, which decides whether reading keeps pace with generation. Total completion time matters far less, because people read along instead of waiting for the end.

The research thresholds are published; the pipeline figures are modeled examples labeled as such.

Time to first token, TTFT, is the sum of everything before generation starts: network hops, queueing, prompt processing. Long prompts raise it, which is one of the quieter reasons prompt caching improves UX and not just cost: a cached prefix skips reprocessing and the first token lands sooner. Tokens per second only needs to beat reading speed, roughly 4 to 6 words per second for skimming, and most current models clear that comfortably.

The modeled table shows why optimizing total time first is usually backwards. Cutting completion from 8.3 to 6 seconds changes little; cutting TTFT from 2.5 to 0.5 seconds changes how the product feels in the first impression it makes.

How does token streaming actually work?#

The API returns the completion as a stream of small events, usually server-sent events, each carrying a few tokens. Your backend forwards chunks to the browser as they arrive, and the UI appends them to the message. The hard part is that every layer in between must forward, not buffer.

Every hop must forward instead of buffer, or the pipeline silently degrades back into a spinner.

Server-sent events are the transport nearly every LLM provider settled on: a long-lived HTTP response where each data: line is a delta event with a few tokens of text. Source Your server reads that stream and writes its own chunked response onward, and the browser appends deltas into the visible message.

The failures are almost always buffering. A proxy that waits for the full response before forwarding, a serverless platform configured to buffer function output, a compression layer that batches small chunks, or a framework helper that collects the body before returning it: any one of these silently converts your streaming pipeline back into a spinner with extra steps. The debugging method is unglamorous and works: log timestamps at each hop for 1 request and find the layer where the first chunk's arrival time jumps.

What breaks when you add streaming?#

Streaming complicates everything that assumed a complete response: JSON parsing fails on fragments, moderation has to run on text you already showed, retries happen after the user watched half an answer, and tool calls pause the stream in ways the UI has to explain.

Each item is invisible in a happy-path demo and shows up in the first week of real traffic.

Structured output is the clearest conflict. Half a JSON object is syntactically invalid, so anything machine-readable wants the complete response, validated, before use. The workable compromise for user-facing structure is to stream a human-readable status while the structured call runs, then render the result once it parses, the same separation that makes structured output with validation reliable in workflows.

Mid-response tool calls are the second trap. When an agent stops generating prose to call a tool, the token stream pauses, and an unexplained pause reads as a hang. The fix is naming the pause: "Searching the docs" while the call runs beats frozen text by a wide margin. Partial-output retries round out the list: if the stream dies at token 300, replacing half an answer the user already read with a fresh attempt needs an explicit "regenerating" state, not a silent splice, and your latency and cost instrumentation should count those restarts, because users experience them as doubled wait time.

Which UX patterns make streaming feel right?#

The pattern set is small: show a status line the moment the request leaves, stream tokens as soon as they exist, name tool calls while they run, keep an abort button visible, and pin the scroll only while the user has not scrolled up. Each one removes a specific moment of doubt.

The scroll rule is the one most chat UIs get wrong. Auto-scroll should follow the stream only while the user is already at the bottom; the moment they scroll up to reread something, the stream must stop yanking the viewport, and a small "jump to latest" affordance takes over. Markdown rendering has a similar subtlety: re-parsing the whole message on every token is wasteful and makes half-finished tables and code fences flicker through broken states. Render incrementally, and hold back a fence until it closes.

The abort button is not decoration. Generation you can cancel converts a wrong-direction answer from an 8-second tax into a 1-second correction, and the cancelled tokens are tokens you do not pay for, a small but real term in the token cost math.

When should you not stream?#

Skip streaming when nobody is watching: background agents, batch jobs, and machine-consumed structured output gain nothing from partial text. Short answers under roughly 2 seconds also stream badly, because the animation lasts longer than the wait it was meant to soften.

Streaming is a latency treatment. Response types without felt latency get nothing from it.

The nobody-watching case is bigger than it looks. Most pipeline calls, classification steps, and scheduled jobs have no human at the other end, and for those the right optimization is throughput and price, which is exactly what a batch API at half price trades latency away for. Streaming infrastructure adds moving parts, and adding them to a code path with no audience is pure cost.

The short-answer case is subtler. A 30-token reply arrives in well under a second; letting it type out character by character is theater, and theater that slows the user down. Several products deliberately render short responses whole and reserve the stream for long-form answers, which is the right instinct: streaming is a latency treatment, and you do not treat latency the user never felt.

How do you decide for each response type?#

Route by audience and shape: a human watching plus prose means stream, a human watching plus structured output means stream a status but deliver the result whole, and no human watching means no streaming at all. The flowchart below is the whole policy in 3 checks.

Three checks route every response type: who is watching, what shape the output is, and whether the wait is even noticeable.

The policy earns its keep by being boring. Teams get into trouble when streaming is a global toggle rather than a per-endpoint decision, because the JSON endpoints end up with fragile incremental parsers and the batch jobs end up holding SSE connections nobody reads. Classify each response type once, wire the 3 routes, and the argument never comes back.

What should you build first?#

Instrument time to first token before optimizing anything, since you cannot improve a number you do not record. Then stream your longest prose responses, add the tool-status line, and leave structured outputs unstreamed. Most chat products get 80% of the perceived-speed win from exactly those steps.

The instrumentation order matters because TTFT regressions come from everywhere: a longer system prompt, a new proxy, a model swap, a growing conversation history. With the number on a dashboard, each regression is a same-day fix. Without it, the product just gradually starts feeling worse and nobody can say when it began. Fast is a feature users cannot articulate but always notice, and the first token is where they notice it.