LLM Latency Explained: TTFT, ITL, and How to Speed Up Inference

You type a question into an AI chatbot. You hit enter. Then... nothing happens for three seconds. That pause is frustrating. It breaks the flow of conversation. But what actually happened in those three seconds? Why did the model take so long to start talking?

Most people assume Large Language Models (LLMs) are just "thinking" hard. The reality is more mechanical. Your request traveled across networks, got converted into numbers, processed through massive mathematical layers, and then began generating text one tiny piece at a time. This delay is called prompt-to-response latency. Understanding it isn't just about patience; it's about knowing how to build faster, cheaper, and better AI applications.

The Two Faces of Latency: TTFT and ITL

To fix slowness, you first have to measure it correctly. In the world of LLM inference, there are two main metrics that matter. If you only look at "total time," you miss the real problem.

1. Time to First Token (TTFT): This is the silence after you hit enter. It measures the time from sending your prompt until the very first character appears on your screen. High TTFT feels like lag. It makes the app feel broken or unresponsive.
2. Inter-Token Latency (ITL): Also known as Time Per Output Token (TPOT), this is the speed of the typing effect once the answer starts flowing. Low ITL means the words appear quickly, creating a smooth reading experience.

Imagine ordering coffee. TTFT is the wait while the barista takes your order and grinds the beans. ITL is how fast they pour the espresso once the machine starts. A short TTFT makes you feel heard immediately. A low ITL gets you your drink faster. Both matter, but for different reasons.

For chatbots, TTFT is critical. Users expect instant acknowledgment. For background tasks, like summarizing a long document where the user walks away, ITL matters less because the user isn't watching the cursor blink.

What Happens During TTFT? The KV Cache Build-Up

Why does TTFT take time? Because the model has to read everything you sent before it can say anything back.

When you send a prompt, the LLM doesn't just "read" it. It converts every word into numerical vectors. Then, it processes these vectors through dozens or hundreds of neural network layers. During this process, it builds something called a Key-Value (KV) Cache.

The KV cache is a memory structure that stores the context of your prompt. The model needs this cache to ensure coherence. When it generates the second word, it needs to remember the first. When it generates the third, it needs to remember the first and second. Building this cache requires heavy computation. The longer your prompt, the bigger the cache, and the longer the TTFT.

Research from Proxet benchmarking GPT-3.5-turbo showed this clearly. A 500-token prompt took about 1.0 second of pure processing time (excluding network delays). As prompts grew to 4,000 tokens, the median processing time increased proportionally. There is no shortcut here. The model must process the entire input sequence before outputting a single token.

The Sequential Bottleneck: Why ITL Exists

Once the KV cache is built, the model starts generating. You might think it could generate the whole sentence at once. It can't. This is the fundamental constraint of the transformer architecture.

LLMs work by predicting the next token based on all previous tokens. This is called autoregressive generation. To predict the second word, it needs the first. To predict the third, it needs the first two. This creates a strict sequential dependency.

This sequential nature means you cannot parallelize the output generation. Each token generation step must complete before the next begins. This is why even powerful GPUs can't make an LLM generate a 1,000-word essay instantly. They are limited by the speed of the forward pass for each individual token.

Hardware and Infrastructure: The Physical Limits

Software optimizations help, but hardware sets the ceiling. The speed of your GPU determines how fast the matrix multiplications happen inside the model.

NVIDIA H100 and A100 processors are industry standards for high-performance inference. They offer high memory bandwidth, which is crucial because LLMs are memory-bound, not just compute-bound. Moving data from memory to the processor cores takes time. Faster interconnects, like NVIDIA NVLink, reduce communication overhead when sharding models across multiple GPUs.

But hardware alone isn't enough. System load plays a huge role. If a server is handling 1,000 requests simultaneously, your request sits in a queue. This adds variable delay to both TTFT and ITL. Autoscaling helps, but it introduces its own latency spikes during scale-up events.

Inference engines use batching to maximize throughput. By grouping multiple requests together, they keep the GPU busy. However, larger batch sizes can increase ITL for individual users because the GPU splits its attention among more sequences. It's a trade-off: higher Requests Per Second (RPS) for the system, potentially slower per-request latency for the user.

The Cost-Latency Trap

There is a direct financial link between latency and cost. Providers like OpenAI bill per token. Longer prompts mean more input tokens, which costs money. They also mean longer TTFT, which costs user patience.

This creates a dual incentive to optimize prompt size. Every unnecessary word in your prompt increases both the bill and the wait time. Few-shot prompting-where you provide examples in the prompt to guide the model-is powerful for accuracy but expensive for latency. Each example adds tokens to the KV cache build-up phase.

Techniques like P-tuning offer a middle ground. Instead of stuffing instructions into the prompt, P-tuning uses a small trainable module to encode task-specific information. This keeps the prompt shorter, reducing TTFT, while maintaining control over the model's behavior. It shifts some computational load from the inference phase to a pre-processing step, which can be optimized separately.

Practical Strategies to Reduce Latency

You can't change the physics of transformers, but you can optimize around them. Here are actionable steps:

• Trim Prompts Aggressively: Remove filler words, redundant instructions, and excessive context. Keep only what is strictly necessary for the model to understand the task.
• Use Streaming: Always enable streaming responses. This allows the client to display tokens as they arrive, masking ITL delays. The user sees progress immediately after TTFT ends.
• Optimize Batching: Tune `max_num_seqs` and `max_num_batched_tokens` in your inference engine. Higher values improve throughput but may hurt tail latency. Test for your specific workload.
• Leverage Caching: If users ask similar questions, cache the KV states. Some advanced systems support prefix caching, where common parts of prompts (like system instructions) are reused, drastically cutting TTFT for repeated patterns.
• Choose the Right Model Size: Don't use a 70-billion parameter model if a 7-billion parameter model solves the problem. Smaller models have smaller KV caches and faster forward passes, resulting in lower TTFT and ITL.

Future Outlook: Beyond Sequential Generation

The research community knows sequential generation is a bottleneck. New architectures are being explored that allow non-autoregressive generation, where multiple tokens are predicted in parallel. Speculative decoding is another promising technique, where a smaller, faster model drafts tokens, and the larger model verifies them. This can effectively double or triple generation speed without changing the core model weights.

As we move into 2026, latency remains a critical constraint for real-time applications like voice assistants and live translation. The gap between human speech rates (~150 words per minute) and LLM generation speeds is closing, but it hasn't disappeared yet. Optimizing for latency isn't just a technical detail; it's a requirement for usability.