Output Tokens per Second
Output tokens per second measures how fast a model creates output tokens after decoding starts. The report must name the tokenizer, time range, grouping method, and number of requests running together.
Another example
A test sends 500 requests with one tokenizer version. It records output counts and time after the first token. It tests concurrency at 1 and 32. The report shows per-request and total tokens per second with percentiles. Input speed, requests per second, and TTFT are different measures.
Main parts
- 01output-token count
- 02decode interval
- 03concurrency and aggregation
Use it when
Use this term when comparing decoding speed with the same model, hardware, tokenizer, output length, batching, and concurrency.
Do not use it when
Do not give tokens per second without saying whether it is for one request or all requests. Different tokenizers may put different amounts of text in one token.
Name used in code
output token count / decode seconds Before you ship
Check the model and prompt versions, source data, versioned evaluation set, and measures. Verify links to evidence, tool permissions, privacy, common failures, decline-to-answer and fallback behavior, human review, monitoring, cost, speed, and rollback.
Outcome: Use or evaluate Output Tokens per Second to make AI behavior measurable and tied to evidence. User context: A test sends 500 requests with one tokenizer version. It records output counts and time after the first token. It tests concurrency at 1 and 32. The report shows per-request and total tokens per second with percentiles. Input speed, requests per second, and TTFT are different measures. AI method or concept: Output Tokens per Second. Why it fits: Use this term when comparing decoding speed with the same model, hardware, tokenizer, output length, batching, and concurrency. Do not use it when: Do not give tokens per second without saying whether it is for one request or all requests. Different tokenizers may put different amounts of text in one token. AI requirements: Define the input and source evidence. Set model and tool permissions. Use versioned evaluation data and measures. Define failure, decline-to-answer, privacy, speed, and cost limits. Operational safeguards: Keep a clear trace and hide sensitive log data. Show users a safe fallback. Mark steps that need human review. Define how to roll back the model or prompt. Acceptance criteria: Record baseline and target measures on a versioned evaluation set. Test edge cases and hostile inputs. Verify fallback, monitoring, permissions, and rollback. Evidence and limits (evidence boundary): It is official for that source. Other platforms or teams may use the term in another way. Unknowns to confirm: Target task, model and version, evaluation owner, source data, risk limit, tool permissions, and production fallback.
Check this request
Review the current use of Output Tokens per Second. Definition: Output tokens per second measures how fast a model creates output tokens after decoding starts. The report must name the tokenizer, time range, grouping method, and number of requests running together. Release checks: Check the model and prompt versions, source data, versioned evaluation set, and measures. Verify links to evidence, tool permissions, privacy, common failures, decline-to-answer and fallback behavior, human review, monitoring, cost, speed, and rollback. Before changing code, report the evidence you found, gaps, severity, and the smallest safe fix.