Malhar Ujawane

Blog

Explainer·(updated October 31, 2026)

LLM latency explained: TTFT, tokens per second and ITL

LLM latency is time to first token (TTFT) plus output tokens times inter-token latency. What each metric measures, what moves it, with a Python model.

LLM latency explained: TTFT, tokens per second and ITL

LLM latency is two numbers, not one: time to first token (TTFT), which is how long you wait before anything appears, and inter-token latency (ITL), the gap between each token after that. The total time for a response is roughly TTFT plus the number of output tokens times ITL, and tokens per second is just ITL turned upside down. Because the second term scales with output length, the length of the answer usually matters more than any other knob.

I ran the script below on 2026-10-31 with Python 3.13.11 and no packages. Its stream is simulated, and the inputs are assumptions I state in the code, not measurements of any provider.

What do TTFT, ITL and tokens per second mean?

The cleanest definitions I’ve found are in NVIDIA’s NIM benchmarking docs, and most benchmark tools use the same ones:

  • Time to first token (TTFT) “measures how long you wait before seeing the model’s output”. It runs from sending the request to receiving the first token, and “generally includes request queuing time, prefill time, and network latency”.
  • End-to-end latency is the time from sending the request to receiving the full response: TTFT plus generation time.
  • Inter-token latency (ITL) is “the average time between consecutive tokens”, computed as (e2e_latency - TTFT) / (output_tokens - 1). Some papers call it time per output token (TPOT).
  • Tokens per second (TPS) in a server benchmark means total output tokens across all simultaneous requests, divided by the test duration. That’s a throughput number for the whole system, not the speed one user sees.

That last point causes most confusion. A provider page that says “1,000 tokens per second” might mean one stream decoding fast, or a hundred streams at 10 tokens per second each. For a single user, the number that matters is 1 / ITL.

Why are there two latencies?

Because generation has two phases with different bottlenecks, which the DistServe paper (OSDI 2024) maps directly onto the two metrics: TTFT measures the prefill phase, and TPOT measures decoding.

Prefill reads your whole prompt in one parallel pass and fills the KV cache. It’s heavy on arithmetic and grows with prompt length, so long prompts raise TTFT.

Decode produces one token per forward pass, and each pass reads all the model’s weights from memory to make a single token. That makes it limited by memory bandwidth rather than compute; the speculative decoding paper describes this as the common case for autoregressive models. ITL is set by how fast the hardware can stream the weights (plus the growing KV cache) through the chip, and by how many other requests share that chip.

DistServe’s point is that running both phases on the same GPUs makes them interfere: a long prefill from a new user stalls the decode steps of everyone already streaming. Separating them let the authors serve “7.4x more requests or 12.6x tighter SLO” while keeping over 90% of requests within their latency targets.

How do I calculate LLM latency?

Here is a script that measures the metrics from a token stream, the way a benchmark client does, then asks which change moves the total most, then computes a hard ceiling on decode speed from memory bandwidth.

# latency.py: TTFT, ITL and tokens per second from a token stream, plus a decode ceiling.
# Python 3 standard library only. The stream is simulated; the inputs are assumptions.
import random

random.seed(3)

# 1. A streamed response: one arrival timestamp per output token (seconds after send).
def fake_stream(n_out, ttft=0.45, itl=0.022, jitter=0.004):
    t, stamps = ttft, []
    for _ in range(n_out):
        stamps.append(t)
        t += max(0.001, random.gauss(itl, jitter))
    return stamps

def metrics(stamps):  # same definitions as NVIDIA's NIM benchmarking docs
    ttft, e2e, n = stamps[0], stamps[-1], len(stamps)
    itl = (e2e - ttft) / (n - 1)
    return ttft, itl, e2e, n / e2e, 1 / itl

print(f"{'out tokens':>10}{'TTFT s':>8}{'ITL ms':>8}{'e2e s':>8}{'tok/s e2e':>11}{'tok/s decode':>14}")
for n in (20, 200, 2000):
    ttft, itl, e2e, tps, dps = metrics(fake_stream(n))
    print(f"{n:>10}{ttft:>8.2f}{itl * 1000:>8.1f}{e2e:>8.2f}{tps:>11.1f}{dps:>14.1f}")

# 2. What moves end-to-end latency: e2e = TTFT + (n_out - 1) x ITL.
base = dict(ttft=0.45, itl=0.022, n_out=500)
e2e = lambda ttft, itl, n_out: ttft + (n_out - 1) * itl
print(f"\nbaseline e2e: {e2e(**base):.2f} s")
for label, change in [("halve TTFT", {"ttft": 0.225}), ("halve ITL", {"itl": 0.011}),
                      ("halve output tokens", {"n_out": 250})]:
    print(f"{label:<20} -> {e2e(**{**base, **change}):.2f} s")

# 3. Decode ceiling at batch size 1: each token reads every weight once,
#    so tokens/s <= memory bandwidth / bytes of weights (ignores KV cache reads).
bandwidth = 120e9  # bytes/s, Apple's spec for the M4 MacBook Air
print("\n8B-parameter model on 120 GB/s:")
for name, bytes_per_param in [("16-bit", 2), ("8-bit", 1), ("4-bit", 0.5)]:
    weights = 8e9 * bytes_per_param
    print(f"  {name:<7} weights {weights / 1e9:>4.0f} GB  ceiling {bandwidth / weights:>5.1f} tok/s")
out tokens  TTFT s  ITL ms   e2e s  tok/s e2e  tok/s decode
        20    0.45    21.8    0.86       23.1          45.9
       200    0.45    21.5    4.72       42.4          46.6
      2000    0.45    22.1   44.55       44.9          45.3

baseline e2e: 11.43 s
halve TTFT           -> 11.20 s
halve ITL            -> 5.94 s
halve output tokens  -> 5.93 s

8B-parameter model on 120 GB/s:
  16-bit  weights   16 GB  ceiling   7.5 tok/s
  8-bit   weights    8 GB  ceiling  15.0 tok/s
  4-bit   weights    4 GB  ceiling  30.0 tok/s

Three things fall out.

“Tokens per second” depends on what you divide by. The same stream reads 23 tokens per second for a 20-token answer if you divide by total time, and 46 if you divide by decode time. On short answers TTFT dominates, and an end-to-end tokens-per-second figure hides that.

For a 500-token answer, TTFT barely matters. Halving it saved 0.23 seconds out of 11.43. Halving ITL or halving the output length each roughly halved the total. OpenAI’s latency optimization guide states the same rule of thumb from production: “cutting 50% of your output tokens may cut ~50% of your latency”, while cutting half the prompt may improve latency by only 1 to 5%.

Memory bandwidth caps decode speed. At batch size 1, every token reads every weight, so tokens per second can’t exceed bandwidth divided by model size in bytes. My laptop is an M4 MacBook Air, which Apple’s tech specs list at 120GB/s. For an 8B-parameter model, that’s a ceiling of 7.5 tokens per second at 16-bit and 30 at 4-bit. Real numbers come in below the ceiling because the KV cache is read too and no kernel hits peak bandwidth. This arithmetic is why quantization speeds up local models: fewer bytes per weight, fewer bytes to move per token. I use it again in how much VRAM an LLM needs.

What makes TTFT slow?

TTFT is queue time, plus prefill, plus network. The levers:

  • Prompt length. Prefill scales with input tokens. RAG systems that paste 30 chunks pay for it here.
  • Prompt caching. If the start of the prompt is the same across requests, a provider can reuse its KV cache and skip most of the prefill. Anthropic’s prompt caching post reports time to first token for a 100,000-token cached prompt falling from 11.5s to 2.4s. See what is prompt caching.
  • Load. Queueing is part of TTFT. A busy endpoint at peak hours can add more than prefill does.
  • Hidden thinking. With reasoning models, the first visible token arrives after the reasoning tokens. OpenAI’s reasoning docs say those tokens are not visible via the API but are billed as output tokens. They also have to be decoded, one at a time, at the model’s ITL. So a “slow first token” on a reasoning model is often decode time in disguise.

What makes tokens per second slow?

  • Model size and precision. Bytes of weights per token, as the ceiling above shows. A smaller or routed model is the biggest single change.
  • Batching. Serving engines batch many users onto one GPU, raising total throughput while each user’s ITL creeps up. Throughput and per-user speed trade against each other.
  • Context length. Each decode step reads the KV cache for the whole conversation, so ITL grows in long agent sessions.
  • Decoding tricks. Speculative decoding lets the big model accept several tokens per forward pass, which lowers ITL without changing outputs, mostly at low concurrency.

Why does latency matter for cost?

On hosted APIs you pay per input and output token, and output tokens are the ones that cost time. So the same move, asking for shorter output, cuts both bill and wait. Reasoning tokens make this sharper: they’re billed and decoded like any other output token, even when you never see them. When I compare models for a feature, I look at latency and price per completed task, not per token, because a cheaper model that writes three times as much can lose on both.

What I would do

I’d start by measuring on my own traffic, from the client side: record the send time and the arrival time of every streamed token, and compute TTFT, ITL and end-to-end latency at p50 and p95. Benchmarks from a vendor’s region with an empty queue don’t predict what a user on another continent sees at peak.

Then, in order:

  1. Stream everything a user watches. The OpenAI guide calls streaming “the single most effective approach” for perceived waiting. It doesn’t change end-to-end time; it changes what the user experiences.
  2. Cut output tokens. Ask for the format you need, cap max_tokens, set reasoning effort as low as the task allows. This is the lever with a near-linear payoff.
  3. Make the prompt cacheable. Put stable content (system prompt, tool definitions, documents) first and changing content last.
  4. Route. Send easy requests to a small, fast model and keep the large one for the hard ones.
  5. Only then tune infrastructure, such as speculative decoding, quantization or a different serving engine like vLLM, if you self-host.

FAQ

What is a good time to first token for an LLM?

It depends on the product. For chat, the OpenAI guide says streaming “cuts the waiting time to a second or less”, which is a sensible bar for the first visible token. For background jobs it doesn’t matter, and end-to-end time or throughput is the number to watch.

What is the difference between ITL and TPOT?

They describe the same thing: the average time between output tokens after the first. NVIDIA’s docs call it inter-token latency; DistServe and other papers call it time per output token.

Why is the first token slower than the rest?

The first token waits for queueing, network and prefill, which processes the whole prompt. Every later token only needs one decode step that reuses the KV cache.

Do input tokens affect LLM latency?

They affect TTFT, through prefill, but much less than output tokens affect the total. OpenAI’s guide says halving the prompt may improve latency by only 1 to 5%, while halving output can cut it by about half.

How many tokens per second can a laptop generate?

At batch size 1, at most memory bandwidth divided by the model’s size in bytes. For an 8B model in 4-bit on a 120GB/s M4 MacBook Air, that ceiling is 30 tokens per second; measured speed will be lower.

Sources

Get new posts in your inbox

© 2026 Malhar Ujawane. Built with Astro.