Inference batching

Knowledge graph concept · also referred to as batching, batch size, batch, batched decoding, active sessions, concurrency, batched decode, many active sessions, parallel sessions, 16 sessions in parallel

Mentioned in

• How diffusion language models generate text X thread · 2026-07-31
• Write-up of Reiner Pope's Lecture: How GPT, Claude, and Gemini Are Actually Trained and Served post · 2026-07-14
• Theoretical Upper Bounds for LLM Performance post · 2026-07-14

Related concepts

• LLM inference 3
• Inference throughput 2
• KV cache 2
• Memory bandwidth 2
• Roofline model 2
• AI inference pricing 1
• Denoising 1
• Diffusion language model 1