Time to First Token measures how long a streaming system takes before the first piece of generated output appears. It is useful as the latency users actually feel when they judge whether an assistant is responsive.
Whilst TTFT is easy to put on a slide, in practice it is mostly queueing, retrieval, and prompt size — not only GPU speed. For example, a “faster model” still feels slow because hybrid search and re-ranking run on every turn. We often recommend splitting TTFT from tokens-per-second, then cutting context and first-stage work before buying more hardware.
In streaming inference, users feel speed at first token, then at steady token rate. TTFT bundles queue time, retrieval, re-ranking, prompt assembly, and model prefill — not only GPU decode. Levers: smaller first-stage retrieval, async prefetch, prompt caching, fewer re-rank candidates, tighter context, routing. Report TTFT p50/p95 beside quality metrics so latency wins do not silently trash answer faithfulness. If you add HyDE or heavy re-ranking, put their cost on the same dashboard.