Inference throughput

Knowledge graph concept · also referred to as tok/s, serving throughput, useful serving throughput, throughput ceiling, tokens per second, token throughput, compute throughput, decode throughput, throughput, prefill tok/s, TTFT, median decode performance, aggregate tok/s, profiling sweep, engine performance, real-life performance, decode tok/s, single session, parallel, LLM Throughput, speed, decoding throughput

Mentioned in

• Qwen3.8 27B compressed to 5.95 GB, running in a browser X thread · 2026-09-18
• DeepSeek V4.1 Flash is built for datacenters X thread · 2026-09-10
• Local AI will favor LPDDR and MoE X thread · 2026-08-10
• Theoretical limits can expose inference problems X thread · 2026-08-09
• How diffusion language models generate text X thread · 2026-07-31
• Benchmarking Laguna S 2.1 on DGX Spark X thread · 2026-07-24
• Report Per-Session and Aggregate Token Throughput X thread · 2026-07-21
• Upper Bounds for Local Model Throughput X thread · 2026-07-17
• Theoretical Upper Bounds for LLM Throughput post · 2026-07-14
• Open source AI needs cheap local speed X thread · 2026-07-01

Related concepts

• LLM inference 6
• Memory capacity 5
• Speculative decoding 5
• Local model deployment 4
• Model size 4
• Inference scaling 3
• KV cache 3
• Memory bandwidth 3