TL;DR
- Time to first token (TTFT) measures the delay between sending a prompt and seeing the first output token appear.
- TTFT is driven by prefill (processing your prompt), not by how fast the model writes the rest of the answer.
- TTFT, TPOT, throughput, and end-to-end latency are related but different metrics, and mixing them up leads to the wrong optimization.
- Benchmarked TTFT rarely matches what users feel, because production requests pass through gateways, retrieval, tool calls, and agent orchestration before they ever reach the model.
- Reducing TTFT requires work at the model layer and the application layer.
- For agents, model-level TTFT is not enough. You need observability that traces the whole request path, which is where a control plane comes in.
A user sends a message to your AI product. Nothing happens for two seconds.
Then the reply starts typing itself out, word by word, and everything feels fine again.
That two-second gap has a name: time to first token, or TTFT. It is the single number that decides whether your AI product feels instant or feels stuck. And it is one of the most misunderstood metrics in LLM performance, because teams keep treating it as a pure model-speed benchmark when, in production, it is something closer to a symptom.
This guide walks through what time to first token actually measures, how it is calculated, what causes it to spike, and why the fastest model on a leaderboard is not always the fastest thing your users experience. By the end, you will know how to reduce TTFT where it can be reduced, and how to think about it where it can’t.
What Does Time to First Token Actually Mean?
Time to first token is the delay between when a request reaches an LLM and when the first output token is generated.
Time to first token, or TTFT, measures the time between a user submitting a request and an AI model generating its first output token, with the clock starting the moment a prompt arrives and stopping when the first output token is generated.
It sounds simple. It gets confusing because of what happens in that gap.
AI inference consists of two stages: prefill and decode. During prefill, the model processes the entire prompt and builds the key-value (KV) cache, a stored set of attention values the model reuses instead of recomputing them for every new token. Decode is the second stage, where the model produces output one token at a time using that cache.
TTFT includes the entire prefill stage and the generation of the first output token, but not the time taken to complete the rest of the response. Everything after that first token belongs to a different metric, TPOT (time per output token), which decode governs.
That distinction matters because prefill and decode behave nothing alike computationally. Prefill processes every prompt token in parallel and builds the KV cache; it is compute-bound and sets TTFT. Decode then generates one token per step, reading the whole cache each time; it is memory-bandwidth-bound and sets TPOT. Fixing a slow TTFT and fixing a slow TPOT are two different engineering problems.
How Is TTFT Calculated, and How Does It Compare to Other Metrics?
TTFT is calculated as the timestamp of the first output token minus the timestamp the request was sent: TTFT = t(first token) minus t(request). TTFT is the delay between issuing a request and receiving the first token, ATL (average token latency) is the total generation time divided by the number of tokens, and generation latency multiplies ATL by the token count to recover the full generation time.
The confusion starts when people use TTFT, TPOT, throughput, and end-to-end latency as if they were interchangeable. They are not, and each one tells you something different about where time actually goes.
TTFT vs TPOT vs Throughput vs End-to-End Latency
| Metric | What it measures | What a spike tells you |
| TTFT | Time from request to first output token | Prefill, queueing, or routing is slow |
| TPOT / ITL | Average gap between tokens after the first | Decode is memory-bandwidth constrained or batching is aggressive |
| Throughput (tokens/sec) | Sustained token output rate under load | System capacity, not per-user responsiveness |
| End-to-end latency | Time from request to final token | Approximately TTFT + (output tokens × TPOT) |
End-to-end latency is the total time from request to the last token, roughly approximated as TTFT plus output tokens multiplied by TPOT, while throughput is the total tokens or requests a system handles per second, a measure of sustained capacity rather than responsiveness.
A system can look excellent on paper and still feel slow. A system can have excellent throughput and cost per token and still feel slow if TTFT is high, because the user is staring at a blank screen while the prefill phase works through a long prompt. This is the trap: throughput dashboards look green while the actual product feels broken.
What Actually Drives TTFT Up or Down?
TTFT moves based on four overlapping forces: prompt size, model and hardware, system load, and infrastructure configuration. None of them act alone.
Prompt length is the most direct lever, because prefill scales with the number of input tokens. In a standard serving setup, the model finishes processing the full prompt before generating the first output token, so longer prompts generally mean higher TTFT. This is also why RAG and agent workloads run hotter on TTFT than a plain chatbot. Longer prompts and larger context windows increase the amount of work during prefill, which is why retrieval-augmented generation (RAG), agentic AI, and applications with long conversation histories often have higher TTFT than simpler chatbot interactions.
System load matters just as much as prompt size, sometimes more. Requests sitting in the queue due to resource limits raise TTFT if requests are consistently waiting, and if the queue is empty but TTFT is still high, the delay is coming from the prefill phase itself, meaning the server is compute-bound. Two requests with identical prompts can produce very different TTFT numbers depending purely on what else the server is doing at that moment.
Factors That Influence TTFT
| Factor | Direction of effect | Why |
| Prompt length | Longer prompt → higher TTFT | More tokens to process in prefill |
| Model size | Larger model → higher TTFT | More compute per prefill pass |
| Request queue depth | Deeper queue → higher TTFT | Requests wait before prefill even starts |
| Quantization | Lower precision → lower TTFT | Less compute and memory per forward pass |
| Batching policy | Prefill-prioritized → lower TTFT, higher TPOT variance | Trade-off between first-token speed and steady decode |
| Reasoning mode | Extended thinking → much higher TTFT | Hidden reasoning tokens generated before the visible answer |
Reasoning models deserve a specific callout here. Some reasoning models generate longer or more complex intermediate traces before emitting the first visible token, which can inflate TTFT, and in those cases the delay goes beyond standard prefill latency. A model advertised as fast can still post a TTFT in the tens of seconds once extended reasoning is switched on.
Why Benchmarked TTFT Rarely Matches What Users Feel
TTFT is often blamed on the model. In production, the model is usually only one leg of the trip.
A public benchmark measures a clean, direct path: request in, model processes it, first token out. That is a fair way to compare raw inference speed, and it is genuinely useful for choosing between providers. It is also not what your users experience.
A real production agent looks more like this: user input, application logic, an API gateway, an orchestration layer that decides what to do next, a retrieval step against a vector database, one or more tool or API calls, and only then a call to the model. The model’s own TTFT might be excellent. The user is still watching a loading indicator for three or four seconds because of everything that happened before the model was even called.
Comparative benchmarks also change fast enough that any specific number goes stale within months, so treat published rankings as a snapshot, not a permanent truth. As one mid-2026 latency comparison noted, “Gemini 2.5 Flash and Claude Haiku 4.5 deliver the lowest time-to-first-token among mainstream LLM APIs, both consistently under 600ms on medium-length prompts”, while a larger reasoning-oriented model from the same period, tested under different conditions, showed a dramatically higher figure. According to Artificial Analysis, GPT-5.5 (high) achieves 62 tok/s with a 27.9s TTFT and scores 59 on the Intelligence Index. Same generation of models, wildly different TTFT, because reasoning depth and benchmark conditions were not the same.
That gap is exactly why any TTFT figure you read needs its model, provider, date, and test conditions attached. Without those, a “fastest model” claim is not comparable to anything you’re about to build.
How to Measure TTFT Yourself
Measuring TTFT means recording the timestamp of the first content chunk in a streamed response, not the timestamp of the full response. Time To First Token (TTFT) measures the elapsed time between sending a prompt request and receiving the first token in the response stream, making it fundamentally different from traditional HTTP response time, which captures only when the final byte arrives.
In practice, that means hooking into whatever streaming protocol the provider uses. Whether you use Python, Node.js, or Apache JMeter with a plugin, the measurement approach is the same: hook into the SSE stream, record the timestamp of the first content chunk, and treat everything after that as token throughput territory.
A minimal Python pattern looks like this against a self-hosted vLLM deployment:
import time
start = time.time()
first_token_time = None
for chunk in client.stream_completion(prompt):
if first_token_time is None:
first_token_time = time.time()
ttft = first_token_time – start
print(f”TTFT: {ttft:.3f}s”)
If you’re running vLLM directly, the engine already tracks this for you. vLLM exposes vllm:time_to_first_token_seconds as a Histogram metric for Time to first token (TTFT), alongside separate histograms for prefill time, decode time, and end-to-end latency, so you can see exactly which phase is responsible for a given spike.
For LLM observability platforms, TTFT is usually derived from a completion_start_time field on the generation span. The “time to first token” metric reflects the model’s generation time, while the actual time observed by the client includes additional overheads like network transit, which is exactly why the number a tracing tool reports and the number your user experiences can legitimately differ.
How to Reduce Time to First Token
Reducing TTFT means attacking it at two layers: the model and infrastructure layer, and the application layer around it. Neither one alone gets you all the way there.
At the model and infrastructure layer, three levers do most of the work. Quantization cuts the compute and memory needed per prefill pass, directly lowering TTFT. Chunked prefill and disaggregated prefill/decode configurations let you tune TTFT and inter-token latency somewhat independently. Disaggregated prefilling puts the prefill and decode phase of LLM inference inside different instances, giving you the flexibility to tune TTFT without affecting inter-token latency, or to tune inter-token latency without affecting TTFT. And scheduling policy itself is a trade-off: vLLM provides the lowest TTFT because it schedules a prefill on the first available opportunity, at the cost of steadier decode timing.
At the application layer, prompt length is the lever you control directly. Reducing prompt length is a direct lever: fewer tokens means shorter prefill, which means lower TTFT. Semantic caching helps for repetitive workloads. Semantic caching works best for workloads with high query repetition, such as customer support bots, frequently asked question systems, and internal knowledge assistants, though hit rates stay low for highly creative or unique queries.
One thing streaming does not do is lower TTFT itself. It changes how the wait feels, not how long the wait actually is. TTFT dominates perceived responsiveness because streaming UIs hide generation time behind reading time; once tokens flow faster than a person reads, the user never waits on the model again, so optimizing TTFT buys more perceived speed than raising throughput. Streaming is a UX technique layered on top of a real TTFT number, not a substitute for reducing it.
Why Model-Level TTFT Isn’t Enough for Agents
None of the levers above touch what happens before your model call fires. And for agents, that’s often where most of the time goes.
TTFT is a stack problem. Several delays add up before the first token can be produced. A retrieval step against a vector database, a tool call to a CRM or inventory system, an orchestration layer deciding which sub-agent handles the request, a gateway doing authentication and rate limiting: each one sits in front of the model, invisible to any model-level TTFT number, and each one adds real, felt latency.
This is also why the calculus flips for agents that don’t stream intermediate output. An AI agent doesn’t read as it goes; each step blocks on the full completion, so end-to-end latency dominates and TTFT barely matters in that specific step, even though TTFT still matters enormously for the final, user-facing response.
TTFT tells you when the model started responding. It doesn’t tell you why the user had to wait. That’s the difference between measuring latency and understanding it.
From Measuring Latency to Understanding It
Understanding latency, rather than just measuring it, requires seeing the entire path a request travels, not just the model’s slice of it.
This is the gap between an LLM observability dashboard and genuine agent observability.
This is the specific problem an AI Control Plane is built to address. A control plane provides step-by-step traces on every agent run, with per-step latency and cost showing where time and money go before it becomes a production incident, and traces tied to agent registry and identity showing which agent, owned by whom, produced a given result.
Lyzr’s Control Plane applies that model to agents built on any framework. It’s framework agnostic, accepting agents built with LangGraph, CrewAI, Strands, the Lyzr SDK, or proprietary code, with all of them moving through the same path and appearing in one shared catalog. Instead of a TTFT number sitting alone in a model provider’s dashboard, latency shows up as one piece of a full run trace, correlated against the retrieval step, the tool calls, and the orchestration logic that ran before the model was ever invoked. That context is what turns a latency number into an answer you can act on, not just a stat you file away for a postmortem.
Platform teams managing latency, cost, and governance across multiple agents and frameworks face a specific version of this problem: too many dashboards, not enough correlated context.
Where This Leaves You
TTFT is a real, measurable, worth-optimizing number. It is also, on its own, an incomplete answer to the question your users are actually asking, which is simply “why did I wait?”
Fix the model-level TTFT you can control. Then look at what actually happens before your model call fires. If you can’t currently see queue time, retrieval time, and tool call time as separate, correlated pieces of the same request, that’s the next thing worth building, not another leaderboard comparison. Start by tracing one slow agent run end to end and see how much of the wait was never the model’s fault at all.
Frequently Asked Questions
Time to first token (TTFT) is the delay between sending a request to an LLM and the moment the first output token is generated. It captures prefill and any queueing time, but not the rest of the response.
Prompt length, model size, request queue depth, quantization, batching policy, and reasoning mode all affect TTFT. Longer prompts and busier servers push TTFT up, while quantization and prefill-prioritized scheduling tend to push it down.
TTFT stands for time to first token, the standard metric for how quickly an LLM begins responding to a request. It is one of several core LLM latency metrics, alongside TPOT and end-to-end latency.
There’s no single universal number, since it depends heavily on workload and prompt length, but general response-time thresholds are a useful anchor. The classic response-time thresholds still hold up: 0.1 seconds feels instantaneous, 1 second keeps the user’s flow of thought intact, and 10 seconds risks losing attention entirely. Interactive chat and coding tools generally target well under a second.
TTFT measures only the wait for the first token, while end-to-end latency measures the wait for the entire response. End-to-end latency is roughly TTFT plus the number of output tokens multiplied by TPOT.
TTFT covers the prefill phase, before any output exists. TPOT (time per output token) covers the decode phase, measuring the average gap between tokens once generation has started.
No, streaming does not reduce the actual TTFT value. It changes the user’s perception of the wait by showing partial output immediately, but the underlying prefill time is unchanged.
You reduce TTFT by shortening prompts, applying quantization, using prefill-optimized serving configurations, caching repeated queries, and reducing queue depth through better capacity planning. Each lever works on a different part of the stack, so most real gains come from combining several of them.
You benchmark LLM latency by fixing prompt length and output length, then measuring TTFT, TPOT, and end-to-end latency separately under controlled load. Comparing numbers without matching these conditions produces misleading results.
High TTFT in production is usually caused by something outside the model itself, such as request queueing, long prompts, or upstream steps like retrieval and tool calls that run before the model is even invoked. Check queue depth first, since a healthy queue with still-high TTFT points to a compute-bound prefill phase instead.
Yes, larger models generally require more compute per prefill pass, which raises TTFT compared to smaller models processing the same prompt. This is why latency-sensitive tasks often default to smaller, faster models even when a larger model would produce a marginally better answer.
Book A Demo: Click Here
Join our Slack: Click Here
Link to our GitHub: Click Here
