KV cache

Knowledge graph concept · also referred to as KV fetches, per-conversation state, context traffic, KV, per-session context traffic, kv cache size per session, KV per thousand tokens, KV per thousand tokens of context, key-value cache, KV caches

Mentioned in

• Report Per-Session and Aggregate Token Throughput X thread · 2026-07-21
• Upper Bounds for Local Model Throughput X thread · 2026-07-17
• Write-up of Reiner Pope's Lecture: How GPT, Claude, and Gemini Are Actually Trained and Served post · 2026-07-14
• Theoretical Upper Bounds for LLM Performance post · 2026-07-14

Related concepts

• Inference throughput 3
• LLM inference 3
• Memory bandwidth 3
• Speculative decoding 3
• Inference batching 2
• Inference scaling 2
• Memory capacity 2
• Roofline model 2