Time to first token, tokens per second and latency percentiles
Each request now shows how long its first token took and how fast the answer came, for every call upstream. The project Overview and Dispatch show p50 and p95 time to first token and latency, and median output tokens per second, for each model and provider.
Time to first token is measured on streamed requests, from sending the request upstream to the first event coming back. A non-streamed response arrives whole, so it has none. Output tokens per second is completion tokens over generation time, which starts at the first token for a stream and at dispatch otherwise.
Logs shows TTFT under the latency of streamed requests and can filter on it, as in ttft > 1s. GET /v1/requests/:id returns ttft_ms, duration_ms and output_tokens_per_second for each attempt.
More in Speed: TTFT and tokens per second.