Entries for July 14, 2026

Note: this post is an AI-assisted write-up of the blackboard lecture Reiner Pope gave on Dwarkesh Patel’s podcast. Watch the original video: How GPT, Claude, and Gemini are actually trained and served (YouTube, 2h13m).¹

Pope is the CEO of the chip startup MatX and previously worked on TPU architecture at Google. With two rules of thumb (a roofline model of a GPU rack, and “set competing costs equal to each other”), he derives why batching makes tokens up to 1000x cheaper, why frontier models may be over-trained ~100x beyond Chinchilla-optimal, and how much of a lab’s serving stack you can reverse-engineer from its public API prices. The figures below are redrawn from the blackboard.

I have tried to stay faithful to the original throughout, converting the dialogue into prose and keeping all the numbers as stated. Any errors introduced in the conversion are mine.

The question that motivates everything

Dwarkesh opens with a pricing puzzle. Companies like Anthropic, OpenAI, and Cursor offer a “fast mode” that streams tokens at roughly 2.5x the speed for 6x the price. What is mechanically going on that makes this trade possible? Could you pay 100x more and go even faster? And could there be a “slow mode” where you wait minutes and pay much less?

Pope’s answer is that the dominant effect is batch size, and the rest of the lecture quantifies exactly what batching does to latency and cost. (A second effect, speculative decoding / multi-token prediction, is set aside.)

The whole analysis rests on two simplifications:

1. A roofline model of the hardware. For a cluster like an NVIDIA Blackwell NVL72 rack (72 GPUs), only two numbers matter: memory bandwidth and compute throughput (FLOPs).
2. Two numbers for the model. The time to operate on the weights, and the time to operate on the context (the KV cache).

The KV cache is the per-conversation state the model keeps in memory. During decode, each new token runs a full forward pass through all the weight matrices, and its attention mechanism looks back at an internal representation of every previous token. That stored representation is the KV cache, and reading it is dominated by memory fetches rather than matrix multiplies.

The two-line roofline

The time for one decode step is bounded below by whichever is slower, the memory system or the compute:

The compute side has to multiply a batch of $B$ tokens by all the active parameters:

(The attention compute is ignored; it is small in comparison.) Note the distinction between active and total parameters: in a mixture-of-experts model like DeepSeek V3, about 37B parameters are active per token out of roughly 700B total.

The memory side has to fetch all the weights once per step, plus the KV cache of every sequence in the batch:

These two lines are enough to draw the latency picture:

The weight fetch is a constant floor: no matter how small the batch, you must stream all total parameters from HBM into the chips once per token, and if you use all your memory bandwidth you cannot beat that. This is the latency lower bound, and it already answers the fast-mode question: for a given hardware configuration there is a floor on how fast tokens can come out, and paying more only helps until you hit it.

Cost is a different plot. Renting the GPUs for one step costs the same regardless of batch size, but the step produces $B$ tokens, so the cost per token is $t/B$:

At batch size 1 the weight fetches are not amortized over anything and the economics are up to a thousand times worse. As the batch grows, the weight-fetch hyperbola vanishes and the compute term becomes a hard cost floor. This also answers the “slow mode” question: a hypothetical Claude Code Slow would live on that floor, and it would not be much cheaper than normal serving, because the compute and the KV fetches are unique to each request and cannot be amortized further.

The magic batch size

Where is the balance point where memory time equals compute time? Ignoring the KV term for a clean answer and equating the weight fetch with the weight multiply:

The first factor is purely a hardware constant. Counted in FP4 multiplies (half a byte each), it comes out around 300 on most GPUs, and it has stayed roughly stable from A100 to H100 to B100 because FLOPs and memory bandwidth grew together. The second factor is the sparsity of the model. So:

For DeepSeek, which activates 32 of 256 experts (sparsity 8), that gives a batch of about 2,400 sequences. In practice people run double or triple that, since real-world efficiency is worse than the roofline. Including the KV fetch would push the optimal batch higher still. Remarkably, this result depends only on sparsity, never on model scale.

Trains departing every 20 milliseconds

How does a batch fill up with real users? Pope’s model is a train schedule. The server starts a new batch every ~20 ms whether or not it is full: any requests that are ready board the train, and a request that arrives just after departure waits for the next one. Worst-case queueing latency is therefore about 40 ms.

The 20 ms itself comes from a separate design principle: you want to read your entire HBM capacity once per forward pass, so the natural step time is capacity divided by bandwidth. On the Rubin generation that is 288 GB / 20 TB/s ≈ 15 ms, and the number has hovered around 20 ms across many HBM generations. There is no point going slower, because reading the read-only weights or the KV cache twice per token does nothing for you.

A batch of ~2,000 at ~64 steps per second is ~128,000 tokens per second per rack. Google has bragged about Gemini traffic in the hundreds of millions of tokens per second worldwide, so one rack’s economical batch is about one-thousandth of Gemini. That is the economy of scale in inference: real, but reachable by any serious provider.

Does sparsity hurt quality?

The roofline says sparsity is nearly free performance, so the follow-up is empirical: how much quality do you lose? From the paper “Unified Scaling Laws for Routed Language Models”, with an older MoE technique, a 64-expert model with 370M active parameters matched a dense 1.3B model. That is a 64x increase in total parameters for a 4x effective gain, a huge parameter cost for a modest efficiency win.

And yet from the systems side it is still nearly a pure win: the extra weight fetches amortize over a larger batch, so you keep increasing sparsity until you run out of simultaneous users. The real price is memory capacity, which is what the next sections are about.

Laying out a mixture of experts on a rack

An MoE layer has a router that sends each token to a small fraction of the experts (each expert being an ordinary MLP), an all-to-all “dispatch” of tokens to their experts, an all-to-all “combine” that sums the results, and a residual connection around the whole thing.

The standard practice is expert parallelism: different experts live on different GPUs. DeepSeek’s 256 experts on a Blackwell rack (using 64 of the 72 GPUs for divisibility) means 4 experts per GPU. Since the router’s decisions are data-dependent, any GPU may need to send tokens to any other GPU.

This all-to-all traffic pattern is a perfect fit for how a rack is wired. In NVIDIA’s design the GPUs sit on the outside of the rack and NVSwitches in the middle, with every GPU cabled to every switch, so any GPU reaches any other in two hops. This is the scale-up network (NVLink). Leaving the rack means taking the scale-out network through a NIC and a data-center switch, which is typically about 8x slower.

If you spread one expert layer across two racks, half of every all-to-all crosses the slow rack-to-rack boundary and becomes the bottleneck. So one rack bounds the size of an expert layer, and this is what has been driving interconnect domains bigger: Hopper had 8 GPUs in a scale-up domain, Blackwell 72, Rubin 500-something (some of that is Jensen math, but there is a genuine ~4x from a much harder rack design). The physical constraint is mundane: cable density. Doubling the GPUs in a rack literally doubles the density of cables that must be routed to the switches, against limits of space, weight, power, cooling, and the bend radius of the cables.

This is also a lens on model scaling history. GPT-4 (2023) was rumored to be over a trillion parameters, and models only clearly exceeded that scale once racks with tens of terabytes of fast memory arrived. Google’s TPU deployments have had very large scale-up domains for a long time, which may be part of why Gemini’s pre-training scaled successfully early. The summary: active parameters are limited by compute cost, and total parameters are limited by scale-up size.

Pipeline parallelism

Expert parallelism uses up one rack. To use more racks, the remaining options are data parallelism and pipeline parallelism (tensor parallelism has become irrelevant now that experts are small). Pipelining means putting different layers on different racks: a token flows through rack 0 for the first stage of layers, then hops to rack 1, and so on.

Is the hop a bottleneck? Compare the time spent on scale-up traffic to the time on scale-out traffic. Crossing racks sends each token once per stage, while inside the rack each token fans out to every activated expert, twice (dispatch and combine), for every layer in the stage:

The 1/8 is the bandwidth ratio. With 8+ activated experts and a few layers per stage, the inequality is easily satisfied, so an entire pipeline of racks, one stage each, is communication-feasible.

Dwarkesh brings up Ilya’s remark that “as we now know, pipelining is not wise,” and the architectural constraints it imposes (e.g. Kimi’s attention to layers a few back is awkward to pipeline). Pope’s framing: pipelining is a massive hassle with real but narrow benefits. It saves no runtime at all (the memory fetches just happen on a different rack), but it divides the weight storage per rack, which matters if memory capacity is your constraint.

The catch is micro-batching. To keep four racks busy, you need four micro-batches in flight, each wrapping around for its next decode step as soon as it finishes:

In inference this is natural and the bubble costs nothing; latency is identical to running unpipelined on one rack. In training there is a hard stop between the forward and backward passes of a batch, which creates a genuine bubble of idle time (the literature has zero-bubble and one-forward-one-backward schemes to interleave around it; as Dwarkesh notes, you could also mine Bitcoin in it):

The training batch size itself is a trade-off: smaller batches are always better for ML convergence (fresher gradients), larger batches are better for systems throughput, and the optimum sits in between.

The memory wall and why the KV cache won’t shard

Here Dwarkesh raises the macro puzzle. Memory is the scarce commodity of the moment: Dylan Patel claims hyperscalers are spending half of their CapEx on memory, and consumer devices are getting squeezed. Yet the pipelining analysis just said racks have a memory surplus, since a trillion-parameter model needs only ~1 TB against a rack’s tens of TB. Why is Jensen shoving all that HBM in?

Write down the memory demand across the whole system:

Sharding across $E$ GPUs of expert parallelism and $P$ racks of pipelining, the per-GPU requirement is this divided by $E \cdot P$. But the global batch is (number of micro-batches) × (micro-batch size), and the number of micro-batches needed to fill the pipeline equals $P$, while the micro-batch size $b$ is pinned near $300 \times \mathrm{sparsity}$ by the roofline. Substituting $B = P \cdot b$, the $P$‘s cancel in the KV term:

More pipeline stages keep shrinking the weight footprint, but the KV footprint per GPU stays constant: each extra stage requires proportionally more sequences in flight to stay busy. The KV cache can’t be amortized across the batch (it is unique per user), and it can’t be sharded across pipeline stages either. It loses on both fronts, and once you pipeline even a little, it becomes the dominant use of memory.

So what do labs actually run? Per the DeepSeek paper: expert parallelism up to the scale-up domain size, then very little pipelining (maybe none, maybe 2 stages so the weights aren’t an issue). Frontier inference essentially lives inside a single scale-up domain. Each rack hop would also add on the order of milliseconds of latency per token, which stacks across stages in sequential decode.

The last piece of the scale-up story is bandwidth. The weight-fetch latency is

where $S$ is the scale-up size, because all GPUs in the domain load the weights in parallel. Per-GPU HBM bandwidth improves maybe 1.5–2x per generation, but $S$ jumped 8x from Hopper to Blackwell. Pipelining solves the capacity problem; big scale-up domains solve the bandwidth problem, which is what actually lets you serve at low latency and long context.

Over-trained 100x beyond Chinchilla

Chinchilla scaling tells you the compute-optimal ratio of model size to training data. But a lab does not minimize training compute; it minimizes total compute across pre-training, RL, and inference for all its users. Pope’s heuristic: when minimizing a sum of competing costs, the minimum tends to sit where the costs are equal (true for $x + 1/x$, for $e^x + e^{-x}$, and generally for power laws). So set all three equal.

Using the 6ND rule (6 FLOPs per parameter per token for forward+backward, 2 for forward only):

RL sits between 2 and 6 because you generate every rollout but may not train on all of it, and it carries an extra inefficiency factor (~30%) because RL involves a lot of decode, which runs at lower MFU than training. The active parameter count divides out entirely. Working through the arithmetic on the board, the equal-cost condition lands at roughly

In words: the number of pre-training tokens, RL tokens, and lifetime inference tokens should all be about the same, within factors the analysis can’t resolve. (Dwarkesh’s gloss: every model should stream out roughly the sum of human knowledge that was streamed into it.)

Now plug in real-world guesses. Global traffic of ~500M tokens/s, cut by 5–10x for one specific model in a family, gives ~50M tokens/s; times a two-month deployment life, that is roughly $2.6 \times 10^{14}$, call it 200T inference tokens. The rumor mill says frontier pre-training is ~150T tokens, which matches. With ~100B active parameters, Chinchilla would prescribe only ~2T tokens. The ratio is about 100x over-trained, derived almost from first principles. As Pope puts it, approximate everywhere, set A equal to B, and it’s kind of empowering how far that gets you.

One asymmetry he flags: if your model might miss the frontier and get thrown away, the expected inference tokens shrink, so you should derate the inference term and err toward less over-training.

Reading the infrastructure off API prices

Since providers are incentivized to price close to cost (otherwise someone scoops them), public price sheets leak infrastructure details.

The 200k context surcharge

Gemini 3.1 charges 50% more per token beyond 200k context. Redraw the roofline as a function of context length at a fixed large batch: compute cost is flat (the attention FLOPs slope is negligible until millions of tokens), while memory cost grows linearly with the KV cache. The provider wants to be profitable at every context length, so a two-tier price is laid over a kinked cost curve, and the price bump should sit near the crossover where the model flips from compute-bound to memory-bound:

Assuming the crossover is at 200k, you can solve for the model’s KV bytes per token. Setting KV fetch time equal to compute time and cancelling the batch size:

Is ~2 kB/token plausible? The KV size is (number of unique attention contexts) × 2 × $d_{\mathrm{head}}$ × (KV heads). With $d_{\mathrm{head}} = 128$ and 8 KV heads and a single global context shared across all layers (the Character AI trick, also used in Gemma), you get exactly 2 kB. Sparse attention gets there with bigger raw numbers divided by the sparsity. So the pricing page is consistent with a real architecture, if maybe a little on the small side.

Input vs output prices

Output tokens cost 3–5x more than input tokens. The two phases differ in tokens per forward pass: decode processes one new token per pass, prefill processes the whole prompt in one pass. Dividing the same roofline by the tokens per pass (len_pass) gives the cost per token: the compute term is flat, and the memory term is a hyperbola that only bites when len_pass is small.

Prefill is compute-bound, decode is memory-bandwidth-bound, and a 5x price gap says decode at the provider’s operating point is deeply memory-bound: they are paying ~5x more per output token in memory time than the compute floor.

This also explains the context length plateau. Contexts jumped from ~8k (GPT-3 era) to 100–200k around GPT-4 and have hovered there for a year or two, which suggests that is the balanced cost point. The barrier to 100M-token contexts (the “in-context learning is enough for AGI” scenario) is memory bandwidth and capacity, and HBM is not getting hugely better. Sparse attention (DeepSeek published one mechanism that effectively puts a square root on the KV term) is a big one-time improvement, but going too sparse costs quality, so it is a get-out, not a solution.

Cache pricing and memory tiers

Providers charge much less for cached input tokens, and charge different rates for keeping a cache alive 5 minutes vs 1 hour. There are two ways to produce a KV cache for a token: rematerialize it from scratch (a forward pass: pure compute cost, nothing to store) or hold it in some memory tier (near-zero retrieval cost, but you occupy capacity that scales with hold time). Each tier down (HBM → host DDR → flash → spinning disk) is cheaper to occupy and slower to retrieve from.

Which tier backs which price? Pope’s rule: a storage tier is well-matched to hold times around its drain time, capacity divided by bandwidth, the same ratio that gave 20 ms for HBM. DDR drains in seconds, flash in about a minute, spinning disk in about an hour. So a 5-minute cache tier and a 1-hour cache tier probably map to flash and spinning disk, which surprised him: “I’m kind of shocked to see spinning disk being used at all.”

Convergent evolution with cryptography

The sit-down portion covers Pope’s blog post on how neural nets and ciphers evolved similar shapes. Both need to thoroughly mix information across all their inputs, and even stirring cake batter alternates directions for the same reason. But they optimize in opposite directions. A neural net is kept differentiable in a useful way: residual connections and LayerNorm exist to keep the derivative simple and meaningful for gradient descent. A cipher is designed so that its derivative is useless: differential cryptanalysis attacks a cipher by differentiating it (over the field of two elements), and a well-designed cipher makes a small input difference blow up into a huge output difference, the avalanche effect. Adversarial examples in image models are exactly the avalanche property showing up where it is not wanted.

Building ciphers out of neural nets is a bad idea (99% of new ciphers get broken), but one construction has productively flowed the other way. A Feistel network turns any non-invertible function $f$ into an invertible two-input block:

To invert, read off $x$ from the second slot, then recover $y = z - f(x)$.

The 2017 RevNets paper imported this into deep learning: make each layer a Feistel block (which turns out to look like a residual connection from two layers back) and the whole network becomes invertible. Training normally has to write every layer’s activations to HBM on the forward pass so that the backward pass can read them, a memory footprint linear in depth and often the largest one in training. An invertible network stores none of it: during the backward pass it undoes the forward pass in lockstep, rematerializing activations as needed.

That is spending compute to save memory, the exact mirror image of the KV cache, which spends memory to save compute. Given where hardware is, the KV cache direction is usually the profitable one, which is a fitting last word for a lecture that is mostly about the price of memory.

Note: this post is AI generated, adapted from my working notes behind the Our Models calculator. It is a work in progress and may lack rigor in places. If you find an issue in the formulation, please email me at [email protected].

How fast can a machine serve a large language model? I built Our Models, a database of open models, AI hardware, and audited model profiles, to answer that question for any hardware and model pair. This post derives, from first principles, the math the calculator runs.

The math gives a ballpark ceiling on how many tokens per second a given hardware and model pair can serve, a roofline the pair cannot exceed under a stated set of assumptions. Nothing in it is specific to local machines. The decode bound needs only two numbers from the hardware, memory capacity and memory bandwidth, and the prefill bound adds a third, compute throughput, so the same formulas cover a MacBook and a datacenter GPU node. The post focuses on local hardware because local machines are the weak ones, and the weaker the machine, the harder good throughput is to get and the more the ceiling matters. A real implementation lands somewhere below these ceilings.

The argument builds in layers, each created by a problem the previous layer cannot solve. A memory system is two numbers, capacity and bandwidth, and their product turns out to be a natural measure of what the system can fund. That product yields a clean but loose throughput ceiling for batched decoding. The loose ceiling ignores per-session context traffic, so it is too generous for real serving, and repairing it gives the bound the calculator actually uses. The repaired bound is then filled in per architecture by small adapters and checked against real hardware. That covers decode, the phase that generates tokens. A second part runs the same program for prefill, the phase that processes the prompt, where the limiting resource switches from memory bandwidth to compute.

Interactive figures accompany the derivation. They all use the same toy setup, a 32B-class dense model at 4-bit (18 GB of weights, 6 GB of runtime overhead, 0.26 GB of KV per thousand tokens of context) on a 128 GB machine with 800 GB/s of sustained bandwidth, so the numbers stay comparable from figure to figure.

Four questions

Serving an LLM raises four separate questions, and mixing them up is an easy way to be wrong about a machine.

The third question is the hard one for decode. A machine can fit many sessions in memory and still be too slow per session at that concurrency, so the calculator must report both whether sessions fit and whether the fitting sessions are fast enough to be worth running. The fourth question belongs to prefill, runs on a different resource, and gets its own part of the post.

Every number produced here is an upper bound. A real system can fall below it for reasons the model deliberately ignores, among them kernel quality, quantization overhead, scheduling, CPU involvement, paging, interconnects, and thermal throttling. The decode bounds also ignore compute, which is safe because decode is memory-bound. Prefill is usually compute-bound, so its bounds have to include it. The value of a clean upper bound is that it tells you the best case you are allowed to hope for, and therefore how much room an implementation still has.

Memory power

Before any model-specific detail, ask what a memory system fundamentally offers. When people compare accelerators for local inference they list many specs, from capacity and bandwidth through compute throughput, cache hierarchy, PCIe lanes, and thermals. For the decode phase of autoregressive generation, two of these recur in almost every bound. How much state can the memory hold, and how fast can it move that state? We start there.

Two numbers and their product

Model an idealized memory system by exactly two quantities.

Capacity is a stock, the amount of resident state that can exist at once. Bandwidth is a flow, the amount of state that can be moved per second. They have different units and answer different questions, so any single product of them needs justifying before we rely on it.

Define memory power as

If $C$ is in GB and $R$ is in GB/s, then $D$ is in $\mathrm{GB}^2/\mathrm{s}$. Power is meant in the colloquial sense of capability, as in computing power, and no watts appear anywhere in this model. The units look strange, so the rest of this section explains why this particular combination of $C$ and $R$ is the right scalar.

For a feel of the scale, a 24 GB GPU at 1000 GB/s has $D = 24 \times 1000 = 24{,}000\ \mathrm{GB}^2/\mathrm{s}$. One thing to be careful with throughout is that every memory quantity in a given calculation must use the same unit system. The equations are identical in bytes, GB, or bits, and only the numeric value of $D$ changes.

Feasibility theorem

Here is the toy problem that justifies $CR$. The models in this formulation are deliberately simple, and they are meant as rules of thumb, useful when deciding on hardware for a model or when checking whether an existing setup is getting the most out of the hardware it runs on.

A pure memory workload is a pair $w = (h, r)$, where $h$ is the resident information it must keep alive and $r$ is the information flow rate it must sustain. The workload is feasible on the system $M(C, R)$ when the memory can both hold the state and carry the traffic. There are exactly two ways to fail.

The first is a capacity failure. If a workload needs more resident state than the memory can store, it cannot fit, and no scheduling trick repairs that.

The second is a flux failure. If a workload needs more traffic per second than the interface can deliver, it cannot be sustained, and no surplus capacity repairs that.

In the idealized model these two conditions are also sufficient. If $h \le C$ and $r \le R$, allocate $h$ of state and stream at rate $r$. Therefore the feasible set is precisely the rectangle

whose area is

So memory power has a concrete meaning. It is the measure of the feasible workload region. A workload lives at a point $(h, r)$ in the plane, the system can serve every workload inside its rectangle and none outside, and the size of that rectangle is $CR$.

A caution before building on this. The rectangle is a toy model, and the world is messier in both directions. Real hardware delivers only a fraction of its catalog bandwidth. Even a perfectly sequential read falls short of the spec number, and the achieved fraction depends on the access pattern, on whether enough parallel work is in flight to hide memory latency, and on the hardware itself. The sufficiency claim is idealized too, because the rate a machine reaches depends on which bytes are read and in what order. The bounds below survive this, since real inefficiency only pushes a machine further below its ceiling. The cost lands on comparisons between machines. If one machine sustains 85% of its spec bandwidth and another 60%, ceilings computed from spec numbers make the second machine look better than it really is. Read $R$ as sustained bandwidth where a measured number exists, and read every comparison in this post as approximate.

A caveat on the metric

One misreading is worth heading off. The product $CR$ measures the workload-feasibility region, the set of jobs the device can support at an instant. A different question, “how many distinct memory histories can this device produce over a time $T$”, has a different answer, namely the resident information plus the information streamed over that window,

The feasibility-region reading is the one an inference-throughput bound will need, because serving means keeping sessions resident in memory and streaming data for them.

A first decode bound

The feasibility theorem describes static workloads. Decoding is dynamic, emitting tokens over time. This section turns memory power into a throughput ceiling for batched decoding, and in doing so shows where the product $CR$ enters a real performance bound.

A toy decoder

Consider the decode phase of an autoregressive transformer, simplified to the memory system alone. Let

let $b$ be the batch size (number of concurrent sequences), and let $s$ be the number of decode steps per second. Each decode step emits one token per active sequence, so aggregate output throughput is

Throughput is a product of two things, how many sequences run in parallel and how fast the shared model can be swept, and the memory system caps each factor separately.

Capacity limit

The model weights must be resident once, and every active sequence needs its own KV cache, the per-conversation attention state that grows with context length. So the resident state is $W + bK$, and it must fit.

This is the capacity limit, and it sets the maximum parallelism. It is exactly the $h \le C$ condition of the feasibility theorem, with $h = W + bK$.

Bandwidth limit

In a dense transformer, each decode step must apply the model weights. In the memory-bound idealization, applying the weights means streaming roughly $W$ of data per step. With bandwidth $R$, the step rate obeys

This is the bandwidth limit, and it sets the maximum step rate. It is the $r \le R$ condition, with the per-step traffic playing the role of the flux.

Memory-power decode bound

Multiply the two caps. Throughput is parallelism times step rate, and each is separately bounded, so

Substituting $D = CR$ and factoring out $C$ exposes the memory-power term.

When the model is much smaller than memory, $W \ll C$, the correction vanishes and the bound collapses to the memorable form

This is the memory-power decode bound. The product $CR$ appears because maximum throughput genuinely factors into maximum parallelism times maximum step rate.

The numerator $D = CR$ is the machine. The denominator $KW$ is the workload, the per-session state times the model sweep size. The bound reads as throughput is memory power divided by memory cost per active model-token.

Scope of the bound

The memory-power decode bound governs batched throughput. Set $b = 1$ and it degenerates to

so for a single session only bandwidth matters and capacity is merely a fit constraint. That is the correct behavior, and it makes clear what each metric is for.

Memory power is the right scalar precisely for the third row. The next section explains why even that bound is too optimistic.

Missing traffic

The memory-power decode bound assumes the only per-token memory traffic worth counting is the model sweep $W$, shared across the batch. Real decoding also reads each session’s growing KV cache, and that traffic is private, so it does not amortize over the batch. Ignoring it makes the bound promise throughput that long-context serving can never reach. This section introduces the correct per-token accounting and shows the memory-power bound falls out of it as a loose corollary.

Universal resource bound

Step back to the most general statement, which holds regardless of architecture. If each output token costs at least $q_{\min}$ of unavoidable memory traffic and at least $a_{\min}$ of unavoidable compute, and the device delivers at most $R$ bytes/s and $F$ FLOP/s, then over all setups that fit,

Two rooflines, and the workload lives under the lower of them. For decode I assume the memory roofline is the lower of the two, which is the usual case for local serving, and keep only its half of the minimum,

where $q$ is the memory traffic per output token. Everything in the decode part now reduces to estimating $q$ honestly. The compute half of the minimum is not discarded, though. It returns as the binding limit in the prefill part, which computes where the two rooflines cross. Dropping it here is also practical. A hardware catalog can collect capacity and bandwidth consistently across consumer and workstation devices, while comparable sustained-compute numbers are much harder to obtain, a price the prefill bound will have to pay.

Bytes per token

Split the per-token traffic into the two kinds that behave differently under batching. The model (or active-expert) weights are shared, since one sweep serves the whole batch, so their per-token cost is divided by the batch. The KV/context read is private, since each session reads its own cache, so its per-token cost is not divided at all. With $W_{\mathrm{active}}$ the shared weight traffic per iteration, $\rho$ the tokens emitted per session per iteration (one for ordinary decoding), and $K_{\mathrm{read}}(L)$ the private context traffic per output token at active context length $L$,

The first term shrinks as the batch grows, because more sessions share each weight sweep. The second term does not move. A larger batch does not make any session’s context cheaper to read, and this asymmetry is the main reason long-context serving behaves differently from short-context serving.

Memory-power bound as a corollary

Treat $q_{\text{simple}}$ as an optimistic lower estimate of the true bytes per token, $q_{\text{simple}}(b) \le q_{\text{actual}}(b)$. Dividing $R$ by a smaller denominator gives a larger quotient, so substituting $q_{\text{simple}}$ keeps the result an upper bound.

Here the maximization runs over batches that fit in memory, with $W_{\mathrm{resident}}$ the resident model footprint, $K_{\mathrm{store}}$ the KV memory stored per session, and $O$ the runtime overhead. This is the honest simple bound, bandwidth divided by shared-per-token cost plus private-per-token cost, maximized over fitting batches.

Now recover the previous section’s bound by deliberately throwing information away. Since $K_{\mathrm{read}}(L) \ge 0$, dropping it only loosens the denominator.

The capacity constraint caps the batch at $b \le (C - W_{\mathrm{resident}} - O)/K_{\mathrm{store}}$, so

This is the memory-power decode bound again. It is what you get by discarding the private context term and using the largest batch that fits. That is why it can sit far above achievable throughput while remaining a true ceiling. The ordering is

The memory-power bound is the orientation line. The KV-aware bound is the one to serve from, and the next section develops it into the operational calculator.

KV-aware bound

This section turns the simple bytes-per-token bound into the model the calculator runs. Three refinements are needed. Context must be split into the part that controls memory and the part that controls speed. The shared weight traffic must be allowed to grow with the batch, which matters for mixture-of-experts (MoE) models, models that route each token through a small subset of their weights. And the batch must be filtered so that we never count concurrency at which every session has become uselessly slow.

Two context lengths

A serving system usually reserves KV space for a long maximum context but reads, on average, a shorter active context. These two lengths drive different parts of the bound, so we keep them separate.

The allocation length controls how much KV memory each session reserves, and therefore how many sessions fit. The read length controls how much context each output token must stream, and therefore per-token cost. Collapsing them into one number either overcharges memory or overcharges speed.

Model quantities

A model contributes five quantities. Two are about fitting, two are about speed, and one is about decoding style.

The split between $K_{\mathrm{alloc}}$ and $K_{\mathrm{read}}$ mirrors the split between the two context lengths. Allocation controls how many sessions fit, read controls how fast each one decodes. Note that $W_{\mathrm{batch}}$ takes the iteration’s total token routings as its argument, $n = b\rho$ for a decode iteration, for a reason the adapter section explains. Prefill will later feed the same function far larger token counts.

Memory-fit batch

The first gate is whether sessions fit. Load the model, reserve overhead, and divide the remainder by the per-session allocation.

This is memory-fit concurrency only. It is necessary but not sufficient, and it is exactly the trap that makes a machine look like it can serve a hundred sessions when it cannot serve them usefully.

Aggregate and per-session ceilings

The per-token traffic is the shared weight sweep amortized over the emitted tokens, plus the private context read.

Then the memory roofline $T \le R/q$ gives the aggregate ceiling at batch $b$,

and dividing by the batch gives the per-session rate,

As $b$ grows, the aggregate $T$ rises but the per-session $r$ falls. That tension is the whole serving tradeoff, and it is why a fit-only bound is not enough.

Usable-batch correction

The fix is to refuse batches at which a session would crawl. Impose a per-session floor $r_\star$, the minimum useful tokens/s/session, and solve $r(b) \ge r_\star$ for $b$. Replacing the batch-dependent $W_{\mathrm{batch}}(b\rho)$ by its shared lower bound $W_{\mathrm{active}}$ keeps a closed form. Because $W_{\mathrm{batch}}(b\rho) \ge W_{\mathrm{active}}$, the substitution only weakens the condition, so the implication runs one way, and the closed form is a necessary condition on the admissible batch rather than a sufficient one.

which defines a rate-limited batch

The usable batch is whichever gate binds first.

Because $b_{\mathrm{rate}}$ comes from a necessary condition, $b_{\mathrm{usable}}$ is itself an upper bound on the truly admissible batch, and the calculator applies the exact floor test with the true $W_{\mathrm{batch}}(b\rho)$ in the next step. This is what stops the “hundred sessions” illusion. As context grows, $K_{\mathrm{read}}(L)$ grows, so $b_{\mathrm{rate}}$ falls quickly even while $b_{\mathrm{mem}}$ stays large. The KV slots fit while the useful rate does not.

The bound the calculator uses

Collecting the pieces, define the usable batch set as the fitting batches that also clear the floor,

and take the best aggregate over that set.

This is the KV-aware bound, the main practical formulation. In words, try every batch that fits, reject the ones too slow per session, and for the rest take bandwidth divided by bytes per output token, keeping the best.

The looser memory-power bound is its corollary, obtained as before by dropping the private term and using the largest fitting batch.

The two stand in a fixed relation, which is the main result of the derivation, written first by name and then in full.

The gap across these terms is the point of the whole derivation. The KV-aware line is the tight, practical bound. The memory-power line shows the memory system’s large theoretical capacity-bandwidth product, and the distance between them comes from private context traffic, expert diversity, and the per-session floor. The final term drops the resident-model factor as well, so the right-hand side is exactly the simplified $D/(KW)$ from the memory-power decode bound, now with $K = K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$ and $W = W_{\mathrm{active}}$. It is the loosest, most optimistic reading, since the resident model and overhead always claim a real share of $C$.

Single session

Set $b = 1$ to recover the latency-style bound for one conversation. There is no batch to amortize the weight sweep over.

and for ordinary decoding ($\rho = 1$) this is just $R / (W_{\mathrm{batch}}(1) + K_{\mathrm{read}}(L_{\mathrm{read}}))$. Capacity has dropped out except as the gate that decides whether the model fits at all, consistent with the earlier observation that bandwidth governs single-session speed.

Speculative decoding

Speculative decoding lets $\rho > 1$. A draft model proposes several tokens and the target verifies them in one iteration, so $\rho$ is the expected number of accepted tokens per session per verification step, bounded by the draft length $\gamma$ as $1 \le \rho \le \gamma + 1$. The temptation is to multiply throughput by $\rho$ and stop. That is wrong, because the draft model and verification are not free. Their traffic belongs in $W_{\mathrm{batch}}(b\rho)$ or in the per-token term. The safe rule is to never scale by $\rho$ without charging the draft cost in the denominator. With both effects included, speculative decoding moves through the same KV-aware formula unchanged.

The two bounds side by side

The gap between the memory-power bound and the KV-aware bound is easiest to see as memory traffic. Below, two copies of the same machine decode side by side. Each board is the machine’s usable memory, with the weights packed into an orange container of equal-sized cells and each session’s blue KV cells packed into a small container of its own.

Each decode iteration must move every byte that its accounting charges, at the same bandwidth on both machines, so the charged cells light up one by one and the board resets when the iteration completes. The left board charges only the shared weight cells, so it resets quickly and its token counter races ahead. The right board also charges the read cells of every session’s context, so its iterations stretch as the batch and the context grow. A real iteration takes milliseconds, so time runs in slow motion here.

Both boards run on the same silicon at the same bandwidth, and only the bookkeeping differs between them. The left counter is the memory-power accounting at the chosen batch, and the right one is what the KV-aware bound admits once private context reads are charged.

The sliders show the two context lengths at work. The reserved context sets how many cells each session’s container holds and can push the machine past its capacity, so raising it eventually makes the containers stop fitting. The read context sets how many of those cells light up every iteration, and the longer it gets the smaller the weights’ share of each iteration becomes. It follows the reservation at an adjustable fraction, 90 percent by default, with the unread rest capped at 32k, and the sliders for all of this sit under Advanced. Growing the model itself slows both boards down in step, while the orange container eats the room the blue ones need.

The full memory-power bound goes one step further than the left board. It grows the batch until memory is completely full of KV cells, which is exactly what the default maximize batch mode does, and the line under the left board reports that number. Shrinking the reserved context makes it explode.

At a 4k reservation about a hundred sessions fit and the bound climbs past 4,000 tok/s on this toy machine, and at a 1k reservation it would pass 17,000. Those numbers are true ceilings and useless forecasts at the same time. What stops a real machine long before then is reading each session’s context, which is exactly the traffic the right board charges.

Prefill

Everything so far bounds decode, which produces tokens one iteration at a time. Before decode can start, the machine has to push the whole prompt through the model once to build the KV cache. That phase is prefill, and its limits work differently. A decode iteration re-reads the weights for every token it emits, while prefill reads them once and serves every prompt token in the batch with that single sweep. The weight term that dominated the decode denominator nearly vanishes, and the limit moves to the compute roofline $F/a$ that the decode part dropped. The one new hardware number is $F$, the device’s sustained compute throughput in FLOP/s, and prefill quantities carry a pf subscript throughout this part.

Prefill workload

A prefill batch holds $b$ prompts of $L$ tokens each, so one iteration processes $bL$ input tokens. The prompt length $L$ is a third length next to the decode part’s two. $L_{\mathrm{read}}$ counted the average context a decode step reads back, while $L$ counts the tokens of the prompt being pushed through now. Write $T_{\mathrm{pf}}$ for aggregate prefill throughput in prompt tokens per second, matching the decode part’s use of $T$. If the batch takes $t_{\mathrm{pf}}$ seconds, then

and $t_{\mathrm{pf}}$ is the wait for the first token. Dividing $L$ by the aggregate rate would understate that wait by a factor of $b$, since all $b$ prompts share the machine, and the two readings agree only at $b = 1$. Queueing, tokenization, sampling, and the first decode step sit outside this model.

Prefill capacity gate

Everything the batch needs has to fit in memory before speed matters. The gate is the decode gate plus a workspace term,

where $\Omega_{\mathrm{pf}}$ is the temporary activation memory prefill holds in flight. A runtime that grows the KV cache on demand instead of reserving it replaces $b,K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$ with $b,K_{\mathrm{store}}(L)$, the cache actually written for an $L$-token prompt. For a conventional KV cache each prompt token writes

bytes, so $K_{\mathrm{store}}(L) = \kappa_{\mathrm{KV,w}}, L$. This is the per-token form of the KV-allocation formula in the adapter section below. The workspace has no decode counterpart, and this post models it only roughly, as a few activation-sized tensors over the tokens in flight. Leaving it out of the gate is ceiling-safe, because a looser gate only admits more batches and a maximum over more batches can only rise.

Traffic per prompt token

The decode part defined $W_{\mathrm{batch}}(n)$ as the shared weight traffic of an iteration with $n$ token routings. A one-shot prefill iteration routes $bL$ tokens, so its sweep is $W_{\mathrm{batch}}(bL)$, and each token’s share is $W_{\mathrm{batch}}(bL)/(bL)$. For a dense model the sweep is one full read of the active weights however large $bL$ grows, which is why prefill costs so much less memory traffic per token than decode. Adding the KV bytes each token writes and the kernel-dependent terms gives a minimal traffic model,

where $q_{\mathrm{attn}}^{\mathrm{HBM}}$ is the attention kernel’s memory traffic and $q_{\mathrm{act}}$ the traffic from activations. The memory-side ceiling is $R / q_{\mathrm{pf}}$.

One warning applies to the attention term. In decode, every output token reads the whole cache, so the decode bound charges $K_{\mathrm{read}}(L_{\mathrm{read}})$ per token. Prefill does not work that way. All the tokens of one prompt share the same keys and values, and how many bytes actually move depends on the kernel. FlashAttention never writes the full $L \times L$ attention matrix out to memory, so this traffic stays small even though the amount of math stays quadratic. The safest loose ceiling drops the two kernel-dependent terms entirely, which only raises $R/q_{\mathrm{pf}}$ and keeps it a true ceiling.

Mixture-of-experts models need one correction. The adapter section below derives the expected distinct-expert count $m(n)$, which applies with $n = bL$, the token routings of the prefill iteration, where decode fed it $n = b\rho$. Prefill routes so many tokens at once that $m(bL)$ reaches all $E$ experts almost immediately, so an honest prefill sweep reads every expert,

whereas decode at modest batch sizes touches far fewer.

Compute per prompt token

Every prompt token has to be multiplied through the model’s weight matrices. Let $P_{\mathrm{lin,active}}$ count the parameters in those multiplications, meaning the attention projections and whichever MLP or expert matrices are active, but leaving out embedding lookups and any output projection that runs once per request. This is deliberately narrower than the MoE section’s $P_{\mathrm{active}}$, which was introduced for weight accounting rather than as a FLOP count. Counting one multiply-add as two FLOPs, the linear layers cost $A_{\mathrm{linear}} = 2, b L, P_{\mathrm{lin,active}}$ FLOPs per batch.

Attention adds the part that grows with context. In a causal sequence of length $L$, each token attends to itself and everything before it, which makes $L(L+1)/2$ query-key pairs per layer, and computing both $QK^\top$ and $PV$ costs about $4d$ FLOPs per pair, where $d = N_q d_h$ is the query-side hidden width and $N_q$ the query-head count. Summing over layers and adding a remainder term $A_{\mathrm{extra}}$ for softmax, normalization, rotary embeddings, routing, dequantization, and logits,

and dividing by $bL$ gives FLOPs per prompt token,

The compute-side ceiling is $F / a_{\mathrm{pf}}$. For short and medium prompts the weight multiplications dominate and the ceiling is roughly $F / (2 P_{\mathrm{lin,active}})$. For long prompts the attention work takes over, growing with $L$ per token and with $L^2$ per request, and layers that only attend to a sliding window stop growing once $L$ passes the window width $w$, the same split the adapter section below uses for $K_{\mathrm{read}}$.

One refinement matters when precisions mix, because $F$ is really several numbers on one chip. The linear layers can run at the weight precision’s tensor rate, while the attention matmuls run at the cache and activation precision, usually BF16, and the two families share the same tensor cores, so their times add rather than overlap. A tighter compute floor charges each term at its own rate,

Charging everything at the fastest path’s rate stays a valid ceiling, only a looser one, and the gap opens exactly where attention dominates, at long context. The worked example below charges attention at the BF16 rate for this reason.

Combined ceiling and crossover

Putting the two sides together, prefill at a fixed batch and prompt length obeys

for any batch that passes the capacity gate. In time terms, with total traffic $Q_{\mathrm{pf}} = bL,q_{\mathrm{pf}}$ and total compute $A_{\mathrm{pf}} = bL,a_{\mathrm{pf}}$, the batch cannot finish faster than the slower of its two jobs,

Which side is the limit comes down to arithmetic intensity, the ratio of math to bytes. At short context, one weight sweep of $W_{\mathrm{batch}}(S)$ bytes serves $S$ tokens costing $2 P_{\mathrm{lin,active}}$ FLOPs each, so the two rooflines cross where the sweep time equals the math time,

For a dense model the sweep is the active parameters themselves, so the parameters cancel and $S^\star \approx e_w F / (2R)$. At $F = 100$ TFLOP/s, $R = 800$ GB/s, and 4-bit weights that is about 30 tokens, so dense prefill turns compute-bound almost immediately. A sparse MoE does not cancel. Prefill sweeps every expert while each token only computes through its active parameters, so

the dense figure times the sparsity ratio. A model with 14 times more total than active parameters crosses in the thousands of tokens instead of 30, and the worked example below lands between about 2,100 and 8,300 tokens per sweep. So compute is the usual limit for dense prefill, while a sparse MoE processed in modest chunks on bandwidth-poor hardware can stay memory-bound. Decode sits below the crossover in either case, which is why the decode part could ignore compute entirely.

One formula for both phases

Real servers rarely run a prompt in one piece. Let $\ell$ be how much of the prompt is already cached and $c$ how many new tokens this iteration processes, so one iteration handles $bc$ tokens and the weight sweep $W_{\mathrm{batch}}(bc)$, including the MoE expert count $m(bc)$, amortizes over all of them. Each new token attends to the $\ell$ cached tokens plus the new ones before it, which makes $c\ell + c(c+1)/2$ attention pairs per request, so the compute per new token generalizes to

and an optimistic memory model to

The $K_{\mathrm{read}}(\ell)/c$ term assumes the cache of the old prefix is read about once per chunk and shared by the chunk’s $c$ new tokens. Averaged over a prompt split into equal chunks, the prefixes run $0, c, 2c, \ldots$, so the average prefix is $(L-c)/2$, while the compute average $\ell + (c+1)/2$ sums to $(L+1)/2$ however the prompt is divided, since chunking changes the rereads and the scheduling but never the total attention arithmetic.

The earlier one-shot forms are the case $c = L,\ \ell = 0$, where the reread term vanishes because there is no earlier chunk. At $c = 1$ the formula reproduces the decode bound of the first part, up to the KV write term $\kappa_{\mathrm{KV,w}}$, which the decode formulation drops as negligible next to $K_{\mathrm{read}}$ (a tenth of a megabyte against gigabytes at long context, and dropping it only loosens that ceiling). Speculation has no counterpart before the first output token, so $\rho$ stays a decode-only quantity. One formula covers both phases, and the chunk width $c$ is what separates them.

Chunk size belongs to the serving software, so a ceiling for a hardware and model pair must range over it. Within this model the answer is simple. Shrinking the chunk only adds rereads and shrinks the weight amortization, while the total attention arithmetic never changes, so the combined ceiling is largest at $c = L$ and the one-shot case is the true ceiling. A number computed at a fixed smaller chunk is a policy ceiling, the best a scheduler that caps its chunks at $c$ can reach, and a one-shot implementation may legitimately exceed it. What pushes real systems toward smaller chunks sits outside the bound, in the workspace $\Omega_{\mathrm{pf}}$ and in the waiting decode requests that Sarathi-Serve style scheduling lets cut in between chunks.

TTFT floor

The decode bound rejects batches where a session falls below $r_\star$ tokens per second. The prefill counterpart is a budget $\tau_\star$ on the wait for the first token. A setting is now a batch and a chunk size, and the settings that fit in memory and finish within the budget form the set

and the prefill ceiling at prompt length $L$ is the best throughput over the admissible settings,

The budget is optional. Without one, the set is simply every setting that fits, and a caller who supplies $\tau_\star$ tightens it without hiding the unrestricted result. One-shot prefill maximizes throughput and minimizes the wait at the same time, so the maximum lands at $c = L$ whenever it fits. As with the decode floor, passing the test does not guarantee the budget is met, since scheduling and kernel overhead can still push the real wait past $\tau_\star$.

Ceilings across prompt lengths

$T_{\mathrm{pf,max}}(L)$ is a ceiling for one prompt length, and prefill has no length-independent speed. Short prompts amortize the weight sweep over fewer tokens, long prompts pay more attention work, and the fastest ceiling can sit at an intermediate length. A summary that avoids asking for $L$ takes the range over every prompt length that fits,

where $L_{\mathrm{model,max}}$ is the model’s context limit. Both extremes must carry the prompt length where they occur, and $T_{\mathrm{pf,lo}}$ is still an upper ceiling on throughput at its length, never a lower bound on what real software achieves.

Ordering of the prefill bounds

The decode part arranged its bounds in one chain, each looser than the last, because every bound lived on the same resource and each step dropped a positive term from the same denominator. Prefill cannot be arranged that way. The memory ceiling $R/q_{\mathrm{pf}}$ and the compute ceiling $F/a_{\mathrm{pf}}$ are incomparable, since neither is smaller everywhere and which one is the limit flips at the crossover $S^\star$. What survives is a chain within each resource. Dropping the kernel-dependent traffic terms loosens the memory side toward a memory-power analogue,

obtained by multiplying the capacity cap on $b$ with the sweep-rate cap $R/W_{\mathrm{active}}$, and dropping the attention term loosens the compute side toward $F/(2P_{\mathrm{lin,active}})$. The full ordering is

two independent chains hanging off a shared minimum, a lattice rather than a line. The practical consequence is that prefill has no single loosest bound to serve as an orientation line, and a loose bound from the wrong chain can sit orders of magnitude above the true ceiling. The memory-power analogue above is exactly such a bound. The decode part was the special case where one resource was known in advance to always win, and that knowledge is what collapsed the lattice into a chain.

Model adapters

The KV-aware bound is architecture-agnostic, and an architecture enters only through a short list of quantities. So each model family is captured by a small adapter that supplies them, and the prefill bound extends the same adapter with two compute entries described at the end of this section.

Three adapters cover the catalog. They handle dense transformers, mixture-of-experts, and hybrid/sliding/recurrent attention.

Dense transformers

For a dense model with $P_{\mathrm{total}}$ parameters at $e_w$ bytes each, all weights are touched every step, so the resident footprint and the per-step sweep coincide.

With $N_{\mathrm{layers}}$ layers, $N_{\mathrm{kv}}$ key/value heads, head dimension $d_h$, and KV byte widths $e_{\mathrm{KV,store}}$ and $e_{\mathrm{KV,read}}$, full-context attention reserves and reads

where the factor $2$ counts keys and values. Weight precision and KV precision are independent settings. NVFP4 weights (NVIDIA’s 4-bit floating-point format) do not imply an NVFP4 cache, so $e_w$ and $e_{\mathrm{KV}}$ are tracked separately.

Mixture-of-experts

An MoE model is where the constant-$W_{\mathrm{batch}}$ assumption breaks, and fixing it is the single most important adapter correction. Let $P_{\mathrm{total}}$ be total parameters, $P_{\mathrm{active}}$ the active parameters per token, $E$ the number of routed experts, and $k$ the experts selected per token. Assuming uniformly sized routed experts, each routed expert holds

and the always-on remainder (dense trunk, shared experts, embeddings, attention) is

The naive model assumes a batch touches the same active experts every session, keeping $W_{\mathrm{batch}}$ constant. That is false. Independent sessions route to different experts, so a larger batch touches more distinct experts. With $n = b\rho$ token routings, each independently missing a given expert with probability $1 - k/E$, the expected number of distinct experts touched is

and the per-iteration shared traffic is the fixed part plus the touched experts.

At $n = 1$ this reduces to the active-parameter footprint, and as $n \to \infty$ it saturates at all $E$ experts. This rising $W_{\mathrm{batch}}(n)$ is why MoE batching does not amortize for free, and why the MoE rows in the worked table reach their throughput optimum at modest batch sizes.

One caveat applies here. Every other traffic term in the bound is a deliberate under-estimate of real traffic, which is what makes $R/q$ a true ceiling. The expert count $m(n)$ is different. It is an expectation under independent, uniform routing rather than a lower bound. Real routing is correlated, since load-balancing losses push toward uniform while hot experts and topically similar sessions pull the other way, and correlated routing touches fewer distinct experts than the formula predicts. In that case the modeled traffic overstates the actual traffic, and the computed ceiling can sit below the true one. When a guaranteed ceiling is required, replace $m(n)$ by its minimum $k$, which replaces $W_{\mathrm{batch}}(n)$ by $W_{\mathrm{active}}$. The expectation form is the better estimate, the floor form is the safe bound.

Hybrid, sliding, and recurrent attention

Models with local or sliding-window attention, compressed or latent attention, or linear/recurrent state must not use the full-KV formula blindly, because their cache does not grow linearly in $L$ everywhere. Split both KV terms into global, local, and fixed-state parts.

A simple read approximation with sliding-window width $w$ is

Full-attention layers pay for the whole context, sliding-window layers pay only up to the window, and recurrent or latent state adds a fixed or slowly growing term. The same shape covers Gemma-style local/global attention and DeepSeek-style compressed/sparse attention, with only the coefficients changing.

Prefill adapter fields

The prefill bound reuses almost the whole decode adapter. $W_{\mathrm{resident}}$ and $K_{\mathrm{alloc}}$ serve the same capacity gate, $W_{\mathrm{batch}}(n)$ is the same function fed the iteration’s token count, $K_{\mathrm{read}}$ reappears as the chunked prefix reread, and the write coefficient $\kappa_{\mathrm{KV,w}}$ is derivable from the allocation fields. Two entries are new. The linear FLOP count $P_{\mathrm{lin,active}}$ prices the weight multiplications, and a layered attention-FLOP function prices the context work,

with $\bar L$ the average attended context of the iteration. Each coefficient is $4d$ FLOPs per pair times the number of layers in that group, so this is the FLOP mirror of $K_{\mathrm{read}}(L) = \kappa_{\mathrm{global}} L + \kappa_{\mathrm{local}} \min(L, w) + K_{\mathrm{state}}$ above, and the same hybrid, sliding, and recurrent splits carry over unchanged. One decode caveat softens here. The MoE adapter had to distinguish the expected expert count from its worst-case floor, but prefill token counts saturate $m$ so quickly that the expectation and the worst case are the same number, all $E$ experts. The decode field with no prefill counterpart is $\rho$. The extended adapter is

Calculator procedure

The bound is now ready to compute. Because the same computation runs for every hardware-and-model pair, it is worth stating once as a procedure.

The inputs are a hardware row $(C, R, F, O)$, with $F$ needed only for prefill, a model adapter $(W_{\mathrm{resident}}, W_{\mathrm{batch}}(\cdot), K_{\mathrm{alloc}}(\cdot), K_{\mathrm{read}}(\cdot), \rho, P_{\mathrm{lin,active}}, a_{\mathrm{attn}}(\cdot))$, and workload assumptions $(L_{\mathrm{alloc}}, L_{\mathrm{read}}, r_\star)$, plus a prompt length $L$ and an optional TTFT budget $\tau_\star$ for prefill.

1. Compute the resident margin $C - W_{\mathrm{resident}} - O$. If it is negative, the model does not fit, so stop.
2. Compute $K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$ and the memory-fit batch $b_{\mathrm{mem}}$.
3. For each integer batch $1 \le b \le b_{\mathrm{mem}}$, compute $W_{\mathrm{batch}}(b\rho)$ and $q_{\mathrm{KV}}(b, L_{\mathrm{read}})$.
4. Compute the aggregate ceiling $R / q_{\mathrm{KV}}$ and the per-session ceiling $R / (b, q_{\mathrm{KV}})$ at each batch.
5. Keep the batches whose per-session ceiling is at least $r_\star$.
6. Among the kept batches, choose the one with the largest aggregate ceiling, and report it as the KV-aware result with its batch as $b^\star$.
7. Separately compute the memory-power ceiling for orientation.