Inference throughput

Knowledge graph concept · also referred to as tok/s, serving throughput, useful serving throughput, throughput ceiling, tokens per second, token throughput, compute throughput, decode throughput, throughput, prefill tok/s, TTFT, median decode performance, aggregate tok/s, profiling sweep, engine performance, real-life performance, decode tok/s, single session, parallel

Mentioned in

• Local AI will favor LPDDR and MoE X thread · 2026-08-10
• Theoretical limits can expose inference problems X thread · 2026-08-09
• How diffusion language models generate text X thread · 2026-07-31
• Benchmarking Laguna S 2.1 on DGX Spark X thread · 2026-07-24
• Report Per-Session and Aggregate Token Throughput X thread · 2026-07-21
• Upper Bounds for Local Model Throughput X thread · 2026-07-17
• Theoretical Upper Bounds for LLM Performance post · 2026-07-14
• Open source AI needs cheap local speed X thread · 2026-07-01

Related concepts

• LLM inference 6
• Memory capacity 4
• Speculative decoding 4
• Inference scaling 3
• KV cache 3
• Local model deployment 3
• Memory bandwidth 3
• Model quantization 3