Note: This post is AI-assisted. I developed and checked the formulation. AI tools helped draft parts of the prose from my working notes, but I reviewed and edited the equations, technical claims, and conclusions. The prose reads as AI-written in places, and I am aware of it. The post is still a work in progress and may contain errors. If you find one, please email me at [email protected].

How fast can a machine serve a large language model? I built Our Models, a database of open models, AI hardware, and audited model profiles, to answer that question for any hardware and model pair. This post derives, from first principles, the math the calculator runs.

The math gives a ballpark ceiling on how many tokens per second a given hardware and model pair can serve, a roofline the pair cannot exceed under a stated set of assumptions. Nothing in it is specific to local machines. The decode bound needs only two numbers from the hardware, memory capacity and memory bandwidth, and the prefill bound adds compute rates for the arithmetic formats used by the runtime, so the same formulas cover a MacBook and a datacenter GPU node. The post focuses on local hardware because local machines are the weak ones, and the weaker the machine, the harder good throughput is to get and the more the ceiling matters. A real implementation lands somewhere below these ceilings.

The argument builds in layers, each created by a problem the previous layer cannot solve. A memory system is two numbers, capacity and bandwidth, and their product turns out to be a natural measure of what the system can fund. That product yields a clean but loose throughput ceiling for batched decoding. The loose ceiling ignores per-session context traffic, so it is too generous for real serving, and repairing it gives the bound the calculator actually uses. The repaired bound is then filled in per architecture by small adapters and checked against real hardware. That covers decode, the phase that generates tokens. A second part runs the same program for prefill, the phase that processes the prompt, where the limiting resource can switch from memory bandwidth to compute.

Interactive figures accompany the derivation. They all use the same toy setup, a 32B-class dense model at 4-bit (18 GB of weights, 6 GB of runtime overhead, 0.26 GB of KV per thousand tokens of context) on a 128 GB machine with 800 GB/s of sustained bandwidth, so the numbers stay comparable from figure to figure.

Four questions

Serving an LLM raises four separate questions, and mixing them up is an easy way to be wrong about a machine.

Question	What it asks
Resident fit	Can the model plus runtime overhead be held in memory at all?
Single-session speed	What is the memory-side ceiling for one active conversation?
Useful serving throughput	Across many active sessions, how many tokens per second can the device produce while each session stays above a minimum useful rate?
Prompt wait	How long does a user wait for the first token while their prompt is processed?

The third question is the hard one for decode. A machine can fit many sessions in memory and still be too slow per session at that concurrency, so the calculator must report both whether sessions fit and whether the fitting sessions are fast enough to be worth running. The fourth question belongs to prefill, also considers compute, and gets its own part of the post.

The formulas give upper bounds when resource rates are upper limits and memory and compute costs are lower limits. Expected expert routing or measured sustained rates give estimates instead. A real system can fall below a bound for reasons the model deliberately ignores, among them kernel quality, quantization overhead, scheduling, CPU involvement, paging, interconnects, and thermal throttling. The decode bounds also ignore compute, which is safe because dropping the compute constraint only loosens the bound. Prefill is usually compute-bound, so its bounds have to include it. The value of a clean upper bound is that it tells you the best case you are allowed to hope for, and therefore how much room an implementation still has.

Memory power

Before any model-specific detail, ask what a memory system fundamentally offers. When people compare accelerators for local inference they list many specs, from capacity and bandwidth through compute throughput, cache hierarchy, PCIe lanes, and thermals. For the decode phase of autoregressive generation, two of these recur in almost every bound. How much state can the memory hold, and how fast can it move that state? We start there.

Two numbers and their product

Model an idealized memory system by exactly two quantities.

$$C = \text{memory capacity before subtracting overhead}, \qquad R = \text{memory bandwidth}.$$

Capacity is a stock, the amount of resident state that can exist at once. Bandwidth is a flow, the amount of state that can be moved per second. They have different units and answer different questions, so any single product of them needs justifying before we rely on it.

Define memory power as

$$D = C R .$$

If $C$ is in GB and $R$ is in GB/s, then $D$ is in $\mathrm{GB}^2/\mathrm{s}$. Power is meant in the colloquial sense of capability, as in computing power, and no watts appear anywhere in this model. The units look strange, so the rest of this section explains why this particular combination of $C$ and $R$ is the right scalar.

For a feel of the scale, a 24 GB GPU at 1000 GB/s has $D = 24 \times 1000 = 24{,}000\ \mathrm{GB}^2/\mathrm{s}$. One thing to be careful with throughout is that every memory quantity in a given calculation must use the same unit system. The equations are identical in bytes, GB, or bits, and only the numeric value of $D$ changes.

Feasibility theorem

Here is the toy problem that justifies $CR$. The models in this formulation are deliberately simple, and they are meant as rules of thumb, useful when deciding on hardware for a model or when checking whether an existing setup is getting the most out of the hardware it runs on.

A pure memory workload is a pair $w = (h, r)$, where $h$ is the resident information it must keep alive and $r$ is the information flow rate it must sustain. The workload is feasible on the system $M(C, R)$ when the memory can both hold the state and carry the traffic. There are exactly two ways to fail.

The first is a capacity failure. If a workload needs more resident state than the memory can store, it cannot fit, and no scheduling trick repairs that.

$$h \le C .$$

The second is a flux failure. If a workload needs more traffic per second than the interface can deliver, it cannot be sustained, and no surplus capacity repairs that.

$$r \le R .$$

In the idealized model these two conditions are also sufficient. If $h \le C$ and $r \le R$, allocate $h$ of state and stream at rate $r$. Therefore the feasible set is precisely the rectangle

$$F(C, R) = \{(h, r) : 0 \le h \le C,\ 0 \le r \le R\},$$

whose area is

$$\mu(F) = C R .$$

So memory power has a concrete meaning. It is the measure of the feasible workload region. A workload lives at a point $(h, r)$ in the plane, the system can serve every workload inside its rectangle and none outside, and the size of that rectangle is $CR$.

A caution before building on this. The rectangle is a toy model, and the world is messier in both directions. Real hardware delivers only a fraction of its catalog bandwidth. Even a perfectly sequential read falls short of the spec number, and the achieved fraction depends on the access pattern, on whether enough parallel work is in flight to hide memory latency, and on the hardware itself. The sufficiency claim is idealized too, because the rate a machine reaches depends on which bytes are read and in what order. The bounds below survive this, since real inefficiency only pushes a machine further below its ceiling. The cost lands on comparisons between machines. If one machine sustains 85% of its spec bandwidth and another 60%, ceilings computed from spec numbers make the second machine look better than it really is. Read $R$ as the hardware bandwidth limit for a ceiling, or as measured sustained bandwidth for an estimate, and read every comparison in this post as approximate.

A caveat on the metric

One misreading is worth heading off. The product $CR$ measures the workload-feasibility region, the set of jobs the device can support at an instant. A different question, “how much data can this device hold and transfer over a time $\Delta t$”, has a different answer, namely the resident information plus the information streamed over that window,

$$C + R\,\Delta t .$$

This volume can count the same bytes repeatedly. It does not count distinct histories or new information. The feasibility-region reading is the one an inference-throughput bound will need, because serving means keeping sessions resident in memory and streaming data for them.

A first decode bound

The feasibility theorem describes static workloads. Decoding is dynamic, emitting tokens over time. This section turns memory power into a throughput ceiling for batched decoding, and in doing so shows where the product $CR$ enters a real throughput bound.

A toy decoder

Consider the decode phase of an autoregressive transformer, simplified to the memory system alone. Let

$$W = \text{model weight footprint}, \qquad K = \text{per-session KV/state footprint},$$

let $b$ be the batch size (number of concurrent sequences), and let $s$ be the number of decode steps per second. Each decode step emits one token per active sequence, so aggregate output throughput is

$$T = b \, s .$$

Throughput is a product of two things, how many sequences run in parallel and how fast the shared model can be swept, and the memory system caps each factor separately.

Capacity limit

The model weights must be resident once, and every active sequence needs its own KV cache, the per-conversation attention state that grows with context length. So the resident state is $W + bK$, and it must fit.

$$W + bK \le C \quad\Longrightarrow\quad b \le \frac{C - W}{K} .$$

This is the capacity limit, and it sets the maximum parallelism. It is exactly the $h \le C$ condition of the feasibility theorem, with $h = W + bK$.

Bandwidth limit

In a dense transformer, each decode step must apply the model weights. In the memory-bound idealization, assume applying the weights means streaming $W$ of data per step, equal to the resident footprint in this first model. With bandwidth $R$, the step rate obeys

$$s W \le R \quad\Longrightarrow\quad s \le \frac{R}{W} .$$

This is the bandwidth limit, and it sets the maximum step rate. It is the $r \le R$ condition, with the per-step traffic playing the role of the flux.

Memory-power decode bound

Multiply the two caps. Throughput is parallelism times step rate, and each is separately bounded, so

$$T = b\, s \le \frac{C - W}{K}\cdot\frac{R}{W} = \frac{R\,(C - W)}{K W} .$$

Substituting $D = CR$ and factoring out $C$ exposes the memory-power term.

$$T \le \frac{D}{K W}\left(1 - \frac{W}{C}\right).$$

When the model is much smaller than memory, $W \ll C$, the correction vanishes and the bound collapses to the memorable form

$$T \lesssim \frac{D}{K W} .$$

This is the memory-power decode bound. The product $CR$ appears because maximum throughput genuinely factors into maximum parallelism times maximum step rate.

$$\underbrace{\frac{C}{K}}_{\text{how many sessions resident}}\times \underbrace{\frac{R}{W}}_{\text{how many sweeps per second}} = \frac{CR}{KW} = \frac{D}{KW} .$$

The numerator $D = CR$ is the machine. The denominator $KW$ is the workload, the per-session state times the model sweep size. The bound reads as throughput is memory power divided by memory cost per active model-token.

Scope of the bound

The memory-power decode bound governs batched throughput. Set $b = 1$ and it degenerates to

$$T \lesssim \frac{R}{W},$$

so for a single session only bandwidth matters and capacity is merely a fit constraint. That is the correct behavior, and it makes clear what each metric is for.

Use case	The metric that governs it
Single-user local chat	bandwidth $R$, with capacity $C$ as a fit gate
Largest model that fits	capacity $C$ first, then bandwidth
Maximum batched decode throughput	memory power $D = CR$

Memory power is the right scalar precisely for the third row. The next section explains why even that bound is too optimistic.

Missing traffic

The memory-power decode bound assumes the only per-token memory traffic worth counting is the model sweep $W$, shared across the batch. Real decoding also reads each session’s growing KV cache, and that traffic is private, so it does not amortize over the batch. Ignoring it makes the bound promise throughput that long-context serving can never reach. This section introduces the correct per-token accounting and shows the memory-power bound falls out of it as a loose corollary.

Universal resource bound

Step back to the most general statement, which holds regardless of architecture. If each output token costs at least $q_{\min}$ of unavoidable memory traffic and at least $a_{\min}$ of unavoidable compute, and the device delivers at most $R$ bytes/s and $F$ FLOP/s, then over all setups that fit,

$$T_{\max} \le \max_{\text{setup fits}} \min\!\left(\frac{R}{q_{\min}},\ \frac{F}{a_{\min}}\right).$$

Two rooflines, and the workload lives under the lower of them. For decode I keep only the memory half of the minimum, which gives a looser bound even when compute is the limiting resource,

$$T_{\max} \le \frac{R}{q},$$

where $q$ is the memory traffic per output token. Everything in the decode part now reduces to estimating $q$ honestly. The compute half of the minimum is not discarded, though. It returns as a possible limit in the prefill part, which computes where the two rooflines cross. Dropping it here is also practical. A hardware catalog can collect capacity and bandwidth consistently across consumer and workstation devices, while comparable sustained-compute numbers are much harder to obtain, a price the prefill bound will have to pay.

Bytes per token

The following decode formulas use $\rho=1$ and private caches without cross-session prefix sharing. Speculation uses $q_{\mathrm{spec}}$ below. Split the per-token traffic into the two kinds that behave differently under batching. The model (or active-expert) weights are shared, since one sweep serves the whole batch, so their per-token cost is divided by the batch. The KV/context read is private, since each session reads its own cache, so its per-token cost is not divided at all. With $W_{\mathrm{active}}$ the shared weight traffic per iteration and $K_{\mathrm{read}}(L)$ the private context traffic per output token at active context length $L$,

$$q_{\text{simple}}(b) = \frac{W_{\mathrm{active}}}{b} + K_{\mathrm{read}}(L).$$

The first term shrinks as the batch grows, because more sessions share each weight sweep. The second term does not move. A larger batch does not make any session’s context cheaper to read, and this asymmetry is the main reason long-context serving behaves differently from short-context serving.

Memory-power bound as a corollary

Treat $q_{\text{simple}}$ as an optimistic lower estimate of the true bytes per token, $q_{\text{simple}}(b) \le q_{\text{actual}}(b)$. Dividing $R$ by a smaller denominator gives a larger quotient, so substituting $q_{\text{simple}}$ keeps the result an upper bound.

$$T_{\max} \le \max_{b\,:\,W_{\mathrm{resident}} + b K_{\mathrm{alloc}} + O \le C}\ \frac{R}{\dfrac{W_{\mathrm{active}}}{b} + K_{\mathrm{read}}(L)} .$$

Here the maximization runs over batches that fit in memory, with $W_{\mathrm{resident}}$ the resident model footprint, $K_{\mathrm{alloc}}$ the KV memory reserved per session, and $O$ the runtime overhead. This is the honest simple bound, bandwidth divided by shared-per-token cost plus private-per-token cost, maximized over fitting batches.

Now recover the previous section’s bound by deliberately throwing information away. Since $K_{\mathrm{read}}(L) \ge 0$, dropping it only loosens the denominator.

$$\frac{R}{\dfrac{W_{\mathrm{active}}}{b} + K_{\mathrm{read}}(L)} \le \frac{R}{\dfrac{W_{\mathrm{active}}}{b}} = \frac{b\,R}{W_{\mathrm{active}}} .$$

The capacity constraint caps the batch at $b \le (C - W_{\mathrm{resident}} - O)/K_{\mathrm{alloc}}$, so

$$T_{\max} \le \frac{R\,(C - W_{\mathrm{resident}} - O)}{K_{\mathrm{alloc}}\,W_{\mathrm{active}}} = \frac{D}{K_{\mathrm{alloc}}\,W_{\mathrm{active}}}\left(1 - \frac{W_{\mathrm{resident}} + O}{C}\right).$$

This is the memory-power decode bound again. It is what you get by discarding the private context term and using the largest batch that fits. That is why it can sit far above achievable throughput while remaining a true ceiling. The ordering is

$$\text{actual throughput} \;\le\; \text{simple (KV-aware) bound} \;\le\; \text{memory-power bound}.$$

The memory-power bound is the orientation line. The KV-aware bound is the one to serve from, and the next section develops it into the operational calculator.

KV-aware bound

This section turns the simple bytes-per-token bound into the model the calculator runs. Three refinements are needed. Context must be split into the part that controls memory and the part that controls speed. The shared weight traffic must be allowed to grow with the batch, which matters for mixture-of-experts (MoE) models, models that route each token through a small subset of their weights. And the batch must be filtered so that we never count concurrency at which every session has become uselessly slow.

Two context lengths

A serving system usually reserves KV space for a long maximum context but reads, on average, a shorter active context. These two lengths drive different parts of the bound, so we keep them separate.

$$\begin{aligned} L_{\mathrm{alloc}} &= \text{reserved (maximum) context}, \\ L_{\mathrm{read}} &= \text{average active context}, \\ L_{\mathrm{read}} &\le L_{\mathrm{alloc}}. \end{aligned}$$

The allocation length controls how much KV memory each session reserves, and therefore how many sessions fit. The read length controls how much context each output token must stream, and therefore per-token cost. Collapsing them into one number either overcharges memory or overcharges speed.

Model quantities

A model contributes five quantities. Two are about fitting, two are about speed, and one is about decoding style.

Symbol	Role
$W_{\mathrm{resident}}$	Full resident footprint (for MoE, all resident weights, including the inactive experts)
$W_{\mathrm{batch}}(n)$	Shared weight traffic for one iteration that routes $n$ tokens in total, evaluated at $n = b$ in ordinary decode
$K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$	KV/cache memory reserved per session, which controls concurrency
$K_{\mathrm{read}}(L_{\mathrm{read}})$	Private context traffic per output token, which controls decode cost
$\rho$	Tokens emitted per session per iteration ($\rho = 1$ ordinary, $\rho > 1$ speculative)

The split between $K_{\mathrm{alloc}}$ and $K_{\mathrm{read}}$ mirrors the split between the two context lengths. Allocation controls how many sessions fit, read controls how fast each one decodes. Note that $W_{\mathrm{batch}}$ takes the iteration’s total token routings as its argument, $n = b$ for an ordinary decode iteration, for a reason the adapter section explains. Prefill will later feed the same function far larger token counts.

Memory-fit batch

The first gate is whether sessions fit. Load the model, reserve overhead, and divide the remainder by the per-session allocation.

$$b_{\mathrm{mem}}(L_{\mathrm{alloc}}) = \left\lfloor \frac{C - W_{\mathrm{resident}} - O}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})} \right\rfloor .$$

This is memory-fit concurrency only. It is necessary but not sufficient, and it is exactly the trap that makes a machine look like it can serve a hundred sessions when it cannot serve them usefully.

Aggregate and per-session ceilings

The per-token traffic is the shared weight sweep amortized over the emitted tokens, plus the private context read.

$$q_{\mathrm{KV}}(b, L_{\mathrm{read}}) = \frac{W_{\mathrm{batch}}(b)}{b} + K_{\mathrm{read}}(L_{\mathrm{read}}) .$$

Then the memory roofline $T \le R/q$ gives the aggregate ceiling at batch $b$,

$$T(b, L_{\mathrm{read}}) \le \frac{R}{q_{\mathrm{KV}}(b, L_{\mathrm{read}})},$$

and dividing by the batch gives the per-session rate,

$$r(b, L_{\mathrm{read}}) = \frac{T(b, L_{\mathrm{read}})}{b} \le \frac{R}{W_{\mathrm{batch}}(b) + b\,K_{\mathrm{read}}(L_{\mathrm{read}})} .$$

As $b$ grows, the aggregate $T$ rises but the per-session $r$ falls. That tension is the whole serving tradeoff, and it is why a fit-only bound is not enough.

Usable-batch correction

The fix is to refuse batches at which a session would crawl. Impose a per-session floor $r_\star$, the minimum useful tokens/s/session, and solve $r(b) \ge r_\star$ for $b$. Replacing the batch-dependent $W_{\mathrm{batch}}(b)$ by its shared lower bound $W_{\mathrm{active}}$ keeps a closed form. Because $W_{\mathrm{batch}}(b) \ge W_{\mathrm{active}}$, the substitution only weakens the condition, so the implication runs one way, and the closed form is a necessary condition on the admissible batch rather than a sufficient one.

$$r(b) \ge r_\star \quad\Longrightarrow\quad b \le \frac{R / r_\star - W_{\mathrm{active}}}{K_{\mathrm{read}}(L_{\mathrm{read}})} ,$$

which defines a rate-limited batch

$$b_{\mathrm{rate}}(L_{\mathrm{read}}, r_\star) = \left\lfloor \frac{R / r_\star - W_{\mathrm{active}}}{K_{\mathrm{read}}(L_{\mathrm{read}})} \right\rfloor .$$

The usable batch is whichever gate binds first.

$$b_{\mathrm{usable}} = \min\!\big(b_{\mathrm{mem}}(L_{\mathrm{alloc}}),\ b_{\mathrm{rate}}(L_{\mathrm{read}}, r_\star)\big).$$

Because $b_{\mathrm{rate}}$ comes from a necessary condition, $b_{\mathrm{usable}}$ is itself an upper bound on the truly admissible batch, and the calculator applies the exact floor test with the true $W_{\mathrm{batch}}(b)$ in the next step. This is what stops the “hundred sessions” illusion. As context grows, $K_{\mathrm{read}}(L)$ grows, so $b_{\mathrm{rate}}$ falls quickly even while $b_{\mathrm{mem}}$ stays large. The KV slots fit while the useful rate does not.

The bound the calculator uses

Collecting the pieces, define the usable batch set as the fitting batches that also clear the floor,

$$\mathcal{B}(L_{\mathrm{alloc}}, L_{\mathrm{read}}, r_\star) = \left\{ b : 1 \le b \le b_{\mathrm{mem}}(L_{\mathrm{alloc}}),\ \frac{1}{b}\,\frac{R}{q_{\mathrm{KV}}(b, L_{\mathrm{read}})} \ge r_\star \right\},$$

and take the best aggregate over that set.

$$T_{\max}(L_{\mathrm{alloc}}, L_{\mathrm{read}}, r_\star) \le \max_{b \in \mathcal{B}}\ \frac{R}{\dfrac{W_{\mathrm{batch}}(b)}{b} + K_{\mathrm{read}}(L_{\mathrm{read}})} .$$

This is the KV-aware bound, the main practical formulation. In words, try every batch that fits, reject the ones too slow per session, and for the rest take bandwidth divided by bytes per output token, keeping the best.

The looser memory-power bound is its corollary, obtained as before by dropping the private term and using the largest fitting batch.

$$T_{\max} \le \frac{D}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})\,W_{\mathrm{active}}}\left(1 - \frac{W_{\mathrm{resident}} + O}{C}\right), \qquad D = CR.$$

The two stand in a fixed relation, which is the main result of the derivation, written first by name and then in full.

$$\begin{aligned} T_{\max} \;&\le\; \text{KV-aware bound} \;\le\; \text{memory-power bound} \;\le\; \text{large-memory limit} \\ T_{\max} \;&\le\; \max_{b \in \mathcal{B}} \frac{R}{\dfrac{W_{\mathrm{batch}}(b)}{b} + K_{\mathrm{read}}(L_{\mathrm{read}})} \\ \;&\le\; \frac{D}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})\,W_{\mathrm{active}}} \left(1 - \frac{W_{\mathrm{resident}} + O}{C}\right) \\ \;&\le\; \frac{D}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})\,W_{\mathrm{active}}} \end{aligned}$$

The gap across these terms is the point of the whole derivation. The KV-aware line is the tight, practical bound. The memory-power line shows the memory system’s large theoretical capacity-bandwidth product, and the distance between them comes from private context traffic, expert diversity, and the per-session floor. The final term drops the resident-model factor as well, so the right-hand side is exactly the simplified $D/(KW)$ from the memory-power decode bound, now with $K = K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$ and $W = W_{\mathrm{active}}$. It is the loosest, most optimistic reading, since the resident model and overhead always claim a real share of $C$.

Single session

Set $b = 1$ to recover the latency-style bound for one conversation. There is no batch to amortize the weight sweep over.

$$T_{1,\max} \le \frac{R}{W_{\mathrm{batch}}(1) + K_{\mathrm{read}}(L_{\mathrm{read}})},$$

and capacity has dropped out except as the gate that decides whether the model fits at all, consistent with the earlier observation that bandwidth governs single-session speed.

Speculative decoding

Speculative decoding lets $\rho > 1$. A draft model proposes $\gamma$ tokens, the target verifies all of them in one forward pass, and $\rho$ is the expected number of tokens emitted, including the correction or bonus token, per session per iteration, bounded as $1 \le \rho \le \gamma + 1$. If the target accepts each draft token with the same conditional probability $\alpha<1$ given all earlier acceptances, the expected count is $\rho = (1 - \alpha^{\gamma+1})/(1 - \alpha)$, with $\rho=\gamma+1$ at $\alpha=1$. A measured marginal acceptance rate alone need not satisfy this assumption. The formulas of the previous sections apply only at $\rho = 1$, and this section gives $\rho > 1$ its honest accounting, which changes the earlier formulas in both directions.¹

The first change charges costs the earlier formulas skip. The draft must be resident, so its footprint and its own cache allocation join the capacity gate, $W_{\mathrm{resident}} + W_{\mathrm{draft,res}} + O + b,(K_{\mathrm{alloc}} + K_{\mathrm{draft,alloc}}) \le C$. For a sequential autoregressive draft, proposing $\gamma$ tokens sweeps its weights $\gamma$ times and reads its own cache $\gamma$ times, and every iteration carries that traffic. The sweep of the target grows too, since all $b(\gamma+1)$ candidates route through the model whether or not they survive verification, so the MoE expert count takes $b(\gamma+1)$ routings rather than $b\rho$.

The second change runs the other way and removes an overcharge. Keeping the ordinary cache term at $\rho > 1$ charges $K_{\mathrm{read}}$ once per emitted token, so $\rho$ times per iteration. A verification pass does less work than that, because it scores all $\gamma + 1$ candidates against the session’s cache in one sweep, the same way a prefill chunk reads a prefix once for many tokens. The honest private charge is one read per iteration, or $K_{\mathrm{read}} / \rho$ per emitted token. The overcharge matters because it pushes the computed ceiling below what a clean speculative implementation can reach at long context, and a bound an implementation can beat has stopped being an upper bound.

Collecting both changes, the traffic per emitted token is

$$q_{\mathrm{spec}}(b) = \frac{W_{\mathrm{batch}}(b(\gamma+1)) + \gamma\, W_{\mathrm{draft}}}{b\,\rho} + \frac{K_{\mathrm{read}}(L_{\mathrm{read}}) + \gamma\, K_{\mathrm{draft,read}}}{\rho},$$

where $W_{\mathrm{draft}}$ is the draft’s weight sweep, shared across the batch like the target’s, and $K_{\mathrm{draft,read}}$ is the draft’s per-step cache read, private like the target’s. At $\gamma = 0$ there is no draft, $\rho = 1$, and the formula collapses to the ordinary $q_{\mathrm{KV}}(b)$. For fixed draft costs and length, whether speculation improves this memory-side estimate depends on the acceptance rate, since $\rho$ rises with it while every $\gamma$-proportional cost stays fixed. One more limit applies at large batches. A verification pass pushes $b(\gamma+1)$ tokens through the model at once, which is as much work as a small prefill chunk. The compute roofline of the prefill part below therefore applies to the pass, and the same crossover test decides whether memory or compute limits it. Parallel draft methods such as DFlash need their own draft cost model instead of $\gamma$ sequential sweeps.

The two bounds side by side

The gap between the memory-power bound and the KV-aware bound is easiest to see as memory traffic. Below, two copies of the same machine decode side by side. Each board is the machine’s usable memory, with the weights packed into an orange container of equal-sized cells and each session’s blue KV cells packed into a small container of its own.

Each decode iteration must move every byte that its accounting charges, at the same bandwidth on both machines, so the charged cells light up one by one and the board resets when the iteration completes. The left board charges only the shared weight cells, so it resets quickly and its token counter races ahead. The right board also charges the read cells of every session’s context, so its iterations stretch as the batch and the context grow. A real iteration takes milliseconds, so time runs in slow motion here.

Both boards run on the same silicon at the same bandwidth, and only the bookkeeping differs between them. The left counter is the memory-power accounting at the chosen batch, and the right one is what the KV-aware bound admits once private context reads are charged.

The sliders show the two context lengths at work. The reserved context sets how many cells each session’s container holds and can push the machine past its capacity, so raising it eventually makes the containers stop fitting. The read context sets how many of those cells light up every iteration, and the longer it gets the smaller the weights’ share of each iteration becomes. It follows the reservation at an adjustable fraction, 90 percent by default, with the unread rest capped at 32k, and the sliders for all of this sit under Advanced. Growing the model itself slows both boards down in step, while the orange container eats the room the blue ones need.

The full memory-power bound goes one step further than the left board. It grows the batch until memory is completely full of KV cells, which is exactly what the default maximize batch mode does, and the line under the left board reports that number. Shrinking the reserved context makes it explode.

At a 4k reservation about a hundred sessions fit and the bound climbs past 4,000 tok/s on this toy machine, and at a 1k reservation it would pass 17,000. Those numbers are true ceilings and useless forecasts at the same time. What stops a real machine long before then is reading each session’s context, which is exactly the traffic the right board charges.

Prefill

Everything so far bounds decode, which produces tokens one iteration at a time. Before decode can start, the machine has to push the whole prompt through the model once to build the KV cache. That phase is prefill, and its limits work differently. A decode iteration re-reads the weights for every batch of tokens it emits, while one-shot prefill reads them once and serves every prompt token in the batch with that single sweep. The weight term that dominated the decode denominator nearly vanishes for large prompts, and the limit can move to the compute roofline $F/a$ that the decode part dropped. The common compute rate is $F$, in FLOP/s, and prefill quantities carry a pf subscript throughout this part.

Prefill workload

A prefill batch holds $b$ prompts of $L$ tokens each, so one iteration processes $bL$ input tokens. The prompt length $L$ is a third length next to the decode part’s two. $L_{\mathrm{read}}$ counted the average context a decode step reads back, while $L$ counts the tokens of the prompt being pushed through now. Write $T_{\mathrm{pf}}$ for aggregate prefill throughput in prompt tokens per second, matching the decode part’s use of $T$. If the batch takes $t_{\mathrm{pf}}$ seconds, then

$$T_{\mathrm{pf}} = \frac{bL}{t_{\mathrm{pf}}}, \qquad t_{\mathrm{pf}} = \frac{bL}{T_{\mathrm{pf}}},$$

and $t_{\mathrm{pf}}$ is the prompt-processing time. Dividing $L$ by the aggregate rate would understate that time by a factor of $b$, since all $b$ prompts share the machine, and the two readings agree only at $b = 1$. Queueing, tokenization, sampling, and delivery also contribute to client-observed time to first token and sit outside this model.

Prefill capacity gate

Everything the batch needs has to fit in memory before speed matters. The gate is the decode gate plus a workspace term,

$$W_{\mathrm{resident}} + O + b\,K_{\mathrm{alloc}}(L_{\mathrm{alloc}}) + \Omega_{\mathrm{pf}} \le C, \qquad \Omega_{\mathrm{pf}} = 0,$$

where $\Omega_{\mathrm{pf}}$ is the temporary activation memory prefill holds in flight. A runtime that grows the KV cache on demand instead of reserving it replaces $b,K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$ with $b,K_{\mathrm{store}}(L)$, the cache actually written for an $L$-token prompt. For a conventional KV cache each prompt token writes

$$\kappa_{\mathrm{KV,w}} = 2\, N_{\mathrm{layers}}\, N_{\mathrm{kv}}\, d_h\, e_{\mathrm{KV,store}}$$

bytes, so $K_{\mathrm{store}}(L) = \kappa_{\mathrm{KV,w}}, L$. This is the per-token form of the KV-allocation formula in the adapter section below. This theoretical bound sets temporary workspace to zero for every batch and chunk size. Leaving it out of the gate is ceiling-safe, because a looser gate only admits more batches and a maximum over more batches can only rise.

Traffic per prompt token

The decode part defined $W_{\mathrm{batch}}(n)$ as the shared weight traffic of an iteration with $n$ token routings. A one-shot prefill iteration routes $bL$ tokens, so its sweep is $W_{\mathrm{batch}}(bL)$, and each token’s share is $W_{\mathrm{batch}}(bL)/(bL)$. For a dense model the sweep is one full read of the active weights however large $bL$ grows, which is why prefill costs so much less memory traffic per token than decode. Adding the KV bytes each token writes and the kernel-dependent terms gives a minimal traffic model,

$$q_{\mathrm{pf}}(b, L) = \frac{W_{\mathrm{batch}}(bL)}{bL} + \kappa_{\mathrm{KV,w}} + q_{\mathrm{attn}}^{\mathrm{HBM}} + q_{\mathrm{act}},$$

where $q_{\mathrm{attn}}^{\mathrm{HBM}}$ is the attention kernel’s memory traffic and $q_{\mathrm{act}}$ the traffic from activations. The memory-side ceiling is $R / q_{\mathrm{pf}}$.

One warning applies to the attention term. In decode, every output token reads the whole cache, so the decode bound charges $K_{\mathrm{read}}(L_{\mathrm{read}})$ per token. Prefill does not work that way. All the tokens of one prompt share the same keys and values, and how many bytes actually move depends on the kernel. FlashAttention never writes the full $L \times L$ attention matrix out to memory, so this traffic stays small even though the amount of math stays quadratic. The safest loose ceiling drops the two kernel-dependent terms entirely, which only raises $R/q_{\mathrm{pf}}$ and keeps it a true ceiling.

Mixture-of-experts models need one correction. The adapter section below derives the expected distinct-expert count $m(n)$, which applies with $n = bL$, the token routings of the prefill iteration, where ordinary decode fed it $n = b$. Prefill routes so many tokens at once that $m(bL)$ approaches $E$ under independent uniform routing, so the expected prefill sweep is approximately a full-expert read, excluding $P_{\mathrm{embed}}$ parameters in the untied input embedding (zero when tied),

$$W_{\mathrm{batch}}(bL) \approx e_w\, (P_{\mathrm{total}} - P_{\mathrm{embed}}),$$

whereas decode at modest batch sizes touches far fewer in expectation. This is an estimate, not a guaranteed minimum sweep.

Compute per prompt token

Every prompt token has to be multiplied through the model’s weight matrices. Let $P_{\mathrm{lin,active}}$ count the parameters in those multiplications, meaning the attention projections and whichever MLP or expert matrices are active, but leaving out embedding lookups and any output projection that runs once per request. This is deliberately narrower than the MoE section’s $P_{\mathrm{active}}$, which was introduced for weight accounting rather than as a FLOP count. Counting one multiply-add as two FLOPs, the linear layers cost $A_{\mathrm{linear}} = 2, b L, P_{\mathrm{lin,active}}$ FLOPs per batch.

Attention adds the part that grows with context. In a causal sequence of length $L$, each token attends to itself and everything before it, which makes $L(L+1)/2$ query-key pairs per layer, and computing both $QK^\top$ and $PV$ costs about $4d$ FLOPs per pair, where $d = N_q d_h$ is the query-side hidden width and $N_q$ the query-head count. Summing over layers and adding a remainder term $A_{\mathrm{extra}}$ for softmax, normalization, rotary embeddings, routing, dequantization, and logits,

$$A_{\mathrm{pf}}(b, L) = 2 b L P_{\mathrm{lin,active}} + 2 b N_{\mathrm{layers}} d L (L+1) + A_{\mathrm{extra}},$$

and dividing by $bL$ gives FLOPs per prompt token, with $a_{\mathrm{extra}} = A_{\mathrm{extra}}/(bL)$,

$$a_{\mathrm{pf}}(L) = 2 P_{\mathrm{lin,active}} + 2 N_{\mathrm{layers}}\, d\, (L + 1) + a_{\mathrm{extra}}.$$

The compute-side ceiling is $F / a_{\mathrm{pf}}$. For short and medium prompts the weight multiplications dominate and the ceiling is roughly $F / (2 P_{\mathrm{lin,active}})$. For long prompts the attention work takes over, growing with $L$ per token and with $L^2$ per request, and layers that only attend to a sliding window stop growing once $L$ passes the window width $w$, the same split the adapter section below uses for $K_{\mathrm{read}}$.

One refinement matters when precisions mix, because $F$ is really several numbers on one chip. Let $F_{\mathrm{hw}}(\phi)$ be the hardware’s compute rate limit in FLOP/s for arithmetic mode $\phi$. Each mode specifies the operand and accumulation formats and whether the rate assumes dense or sparse arithmetic. The model and runtime settings select modes $\phi_{\mathrm{linear}}$ and $\phi_{\mathrm{attention}}$ for the linear layers and attention. Weight and cache storage formats alone do not determine these modes. The two compute rates are lookups,

$$F_{\mathrm{linear}} = F_{\mathrm{hw}}(\phi_{\mathrm{linear}}), \qquad F_{\mathrm{attention}} = F_{\mathrm{hw}}(\phi_{\mathrm{attention}}).$$

This two-rate form assumes one arithmetic mode for each path. If a path uses several modes, split its FLOP count by mode and charge each part at its matching rate. If a required rate is unknown, report the compute result as unavailable. The time sum below assumes both paths share the same execution units and cannot overlap. A tighter compute floor charges each term at its own rate,

$$t_{\mathrm{compute}} \ge \frac{A_{\mathrm{linear}}}{F_{\mathrm{linear}}} + \frac{A_{\mathrm{attention}}}{F_{\mathrm{attention}}}.$$

Here $A_{\mathrm{attention}}$ is the batch’s attention FLOP count, and $F_{\mathrm{linear}}$ and $F_{\mathrm{attention}}$ are the rates looked up for the arithmetic used by the linear layers and attention. The calculator adds the two processing times and omits $A_{\mathrm{extra}}$. The single-rate expressions $F/a_{\mathrm{pf}}$ below are the special case $F_{\mathrm{linear}} = F_{\mathrm{attention}} = F$. Charging everything at the fastest path’s rate stays a valid ceiling, only a looser one, and the gap opens exactly where attention dominates, at long context. The worked example below charges attention at the BF16 rate for this reason.

Combined ceiling and crossover

Putting the two sides together, prefill at a fixed batch and prompt length obeys

$$T_{\mathrm{pf}}(b, L) \le \min\left(\frac{R}{q_{\mathrm{pf}}(b, L)},\ \frac{F}{a_{\mathrm{pf}}(L)}\right),$$

for any batch that passes the capacity gate. In time terms, with total traffic $Q_{\mathrm{pf}} = bL,q_{\mathrm{pf}}$ and separate compute totals $A_{\mathrm{linear}}$ and $A_{\mathrm{attention}}$, the batch cannot finish faster than the slower of its two jobs,

$$t_{\mathrm{pf},\min}(b, L) = \max\left(\frac{Q_{\mathrm{pf}}}{R},\ \frac{A_{\mathrm{linear}}}{F_{\mathrm{linear}}} + \frac{A_{\mathrm{attention}}}{F_{\mathrm{attention}}}\right).$$

Which side is the limit comes down to arithmetic intensity, the ratio of math to bytes. At short context, one weight sweep of $W_{\mathrm{batch}}(S)$ bytes serves $S$ tokens costing $2 P_{\mathrm{lin,active}}$ FLOPs each, so the two rooflines cross where the sweep time equals the math time, using $F=F_{\mathrm{linear}}$ in this approximation that omits attention,

$$S^\star = \frac{W_{\mathrm{batch}}(S^\star)\, F}{2\, P_{\mathrm{lin,active}}\, R}.$$

For a dense model the sweep is the active parameters themselves, so the parameters cancel and $S^\star \approx e_w F / (2R)$. At $F = 100$ TFLOP/s, $R = 800$ GB/s, and 4-bit weights that is 31.25 tokens, or about 30, so dense prefill turns compute-bound almost immediately. A sparse MoE does not cancel. Prefill sweeps every expert while each token only computes through its active parameters, so

$$S^\star \approx \frac{e_w F}{2R} \cdot \frac{P_{\mathrm{total}} - P_{\mathrm{embed}}}{P_{\mathrm{lin,active}}},$$

the dense figure times the sparsity ratio. A model with 14 times as many swept as active linear parameters crosses at $14 \times 31.25 = 437.5$ tokens with those hardware rates, and the worked example below uses different rates and lands between about 2,100 and 8,300 tokens per sweep. So compute is the usual limit for dense prefill, while a sparse MoE processed in modest chunks on bandwidth-poor hardware can stay memory-bound. Large decode batches and speculative verification can also cross this threshold. Dropping compute still gives a looser upper bound.

One formula for both phases

Real servers rarely run a prompt in one piece. Let $\ell$ be how much of the prompt is already cached and $c$ how many new tokens this iteration processes, so one iteration handles $bc$ tokens and the weight sweep $W_{\mathrm{batch}}(bc)$, including the MoE expert count $m(bc)$, amortizes over all of them. Each new token attends to the $\ell$ cached tokens plus the new ones before it, which makes $c\ell + c(c+1)/2$ attention pairs per request, so the compute per new token generalizes to

$$a_{\mathrm{pf}}(c, \ell) \approx 2 P_{\mathrm{lin,active}} + 4 N_{\mathrm{layers}}\, d \left(\ell + \frac{c+1}{2}\right) + a_{\mathrm{extra}},$$

and an optimistic memory model to

$$q_{\mathrm{pf}}(b, c, \ell) \approx \frac{W_{\mathrm{batch}}(bc)}{bc} + \frac{K_{\mathrm{read}}(\ell)}{c} + \kappa_{\mathrm{KV,w}} + q_{\mathrm{attn}}^{\mathrm{HBM}} + q_{\mathrm{act}}.$$

The $K_{\mathrm{read}}(\ell)/c$ term assumes the cache of the old prefix is read about once per chunk and shared by the chunk’s $c$ new tokens. For full attention, averaged over a prompt split into equal chunks that divide $L$, the prefixes run $0, c, 2c, \ldots$, so the average prefix is $(L-c)/2$, while the compute average $\ell + (c+1)/2$ sums to $(L+1)/2$ however the prompt is divided, since chunking changes the rereads and the scheduling but never the total attention arithmetic. For sliding windows, average the individual capped prefix reads instead of capping the average prefix. For unequal chunks, sum traffic and FLOPs over chunks and divide by the total prompt-token count.

The earlier one-shot forms are the case $c = L,\ \ell = 0$, where the reread term vanishes because there is no earlier chunk. At $c = 1$ the formula reproduces the decode bound of the first part, up to the KV write term $\kappa_{\mathrm{KV,w}}$, which the decode formulation drops as negligible next to $K_{\mathrm{read}}$ (a tenth of a megabyte against gigabytes at long context, and dropping it only loosens that ceiling). Speculation has no counterpart before the first output token, but its verification pass is itself a chunk, $c = \gamma + 1$ positions yielding $\rho$ emitted tokens on average, and that is where the speculative section’s division of the cache read by $\rho$ comes from. One formula covers both phases, and the chunk width $c$ is what separates them.

Chunk size belongs to the serving software, so a ceiling for a hardware and model pair must range over it. At fixed batch with workspace omitted, the answer is simple. Shrinking the chunk only adds rereads and shrinks the weight amortization, while the total attention arithmetic never changes, so the combined ceiling is largest at $c = L$ and the one-shot case is the true ceiling. A number computed at a fixed smaller chunk is a policy ceiling, the best a scheduler that caps its chunks at $c$ can reach, and a one-shot implementation may legitimately exceed it. What pushes real systems toward smaller chunks sits outside the bound, in the workspace $\Omega_{\mathrm{pf}}$ and in the waiting decode requests that Sarathi-Serve style scheduling lets cut in between chunks.

TTFT floor

The decode bound rejects batches where a session falls below $r_\star$ tokens per second. The prefill counterpart is a budget $\tau_\star$ on the wait for the first token. A setting is now a batch and a chunk size, and the settings that fit in memory and finish within the budget form the set

$$\mathcal{B}_{\mathrm{pf}} = \left\lbrace (b, c) :\ W_{\mathrm{resident}} + O + b K_{\mathrm{alloc}} \le C,\quad t_{\mathrm{pf},\min}(b, c) \le \tau_\star \right\rbrace,$$

and the prefill ceiling at prompt length $L$ is the best throughput over the admissible settings,

$$T_{\mathrm{pf,max}}(L) \le \max_{(b, c) \in \mathcal{B}_{\mathrm{pf}}} \frac{bL}{t_{\mathrm{pf},\min}(b,c)}.$$

The budget is optional. Without one, the set is simply every setting that fits, and a caller who supplies $\tau_\star$ tightens it without hiding the unrestricted result. At fixed batch with workspace omitted, one-shot prefill maximizes throughput and minimizes the wait at the same time, so the maximum lands at $c = L$ whenever it fits. As with the decode floor, passing the test does not guarantee the budget is met, since scheduling and kernel overhead can still push the real wait past $\tau_\star$.

Ceilings across prompt lengths

$T_{\mathrm{pf,max}}(L)$ is a ceiling for one prompt length, and prefill has no length-independent speed. Short prompts amortize the weight sweep over fewer tokens, long prompts pay more attention work, and the fastest ceiling can sit at an intermediate length. A summary that avoids asking for $L$ takes the range over every prompt length that fits,

$$\mathcal{L} = \left\{L : 1 \le L \le L_{\mathrm{model,max}},\ W_{\mathrm{resident}} + O + K_{\mathrm{alloc}}(L) \le C\right\},$$

where $L_{\mathrm{model,max}}$ is the model’s context limit, and reports the two ends of the curve,

$$T_{\mathrm{pf,lo}} = \min_{L \in \mathcal{L}} T_{\mathrm{pf,max}}(L), \qquad T_{\mathrm{pf,hi}} = \max_{L \in \mathcal{L}} T_{\mathrm{pf,max}}(L).$$

Both extremes must carry the prompt length where they occur, and $T_{\mathrm{pf,lo}}$ is still an upper ceiling on throughput at its length, never a lower bound on what real software achieves.

Ordering of the prefill bounds

The decode part arranged its bounds in one chain, each looser than the last, because every bound lived on the same resource and each step dropped a positive term from the same denominator. Prefill cannot be arranged that way. The memory ceiling $R/q_{\mathrm{pf}}$ and the compute ceiling $F/a_{\mathrm{pf}}$ are incomparable, since neither is smaller everywhere and which one is the limit flips at the crossover $S^\star$. What survives is a chain within each resource. Dropping the kernel-dependent traffic terms loosens the memory side toward a memory-power analogue,

$$T_{\mathrm{pf,MP}} := \frac{L\, D}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})\, W_{\mathrm{active}}} \left(1 - \frac{W_{\mathrm{resident}} + O}{C}\right),$$

obtained by multiplying the capacity cap on $b$ with the sweep-rate cap $R/W_{\mathrm{active}}$, and dropping the attention term loosens the compute side toward $F/(2P_{\mathrm{lin,active}})$. The full ordering is

$$\text{actual} \le \min\!\left(\frac{R}{q_{\mathrm{pf}}},\ \frac{F}{a_{\mathrm{pf}}}\right) \le \frac{R}{q_{\mathrm{pf}}} \le T_{\mathrm{pf,MP}}$$

on the memory side and

$$\min\!\left(\frac{R}{q_{\mathrm{pf}}},\ \frac{F}{a_{\mathrm{pf}}}\right) \le \frac{F}{a_{\mathrm{pf}}} \le \frac{F}{2 P_{\mathrm{lin,active}}}$$

on the compute side, two independent chains hanging off a shared minimum, a lattice rather than a line. The practical consequence is that prefill has no single loosest bound to serve as an orientation line, and a loose bound from the wrong chain can sit orders of magnitude above the true ceiling. The memory-power analogue above is exactly such a bound. The decode part kept only the memory resource, which collapsed the lattice into a chain.

Model adapters

The KV-aware bound is architecture-agnostic, and an architecture enters only through a short list of quantities. So each model family is captured by a small adapter that supplies them, and the prefill bound extends the same adapter with two compute entries described at the end of this section.

Here $M$ means the model with the selected weight storage, cache, and decoding settings. Runtime overhead $O$ is supplied separately and is not a hardware property. Capacity $C$ is measured before subtracting $O$, so the memory gate subtracts it only once.

$$\mathrm{Adapter}(M) = \big[\, W_{\mathrm{resident}},\ W_{\mathrm{batch}}(n),\ K_{\mathrm{alloc}}(L_{\mathrm{alloc}}),\ K_{\mathrm{read}}(L_{\mathrm{read}}),\ \rho \,\big].$$

Three adapters cover the catalog. They handle dense transformers, mixture-of-experts, and hybrid/sliding/recurrent attention.

Dense transformers

For a dense model with $P_{\mathrm{total}}$ parameters at $e_w$ bytes each, the resident footprint and the per-step sweep need not coincide: an untied input embedding reads selected rows, not the full table. Let $P_{\mathrm{embed}}$ count the parameters in that untied input embedding, or zero when it is tied to the output projection. With uniform weight storage and one sweep, omitting the selected embedding rows gives

$$W_{\mathrm{resident}} \approx P_{\mathrm{total}}\, e_w, \qquad W_{\mathrm{batch}}(n) = W_{\mathrm{active}} = (P_{\mathrm{total}} - P_{\mathrm{embed}})\, e_w .$$

The resident footprint is approximate because quantization metadata, padding, and mixed tensor precision can change the stored bytes.

With $N_{\mathrm{layers}}$ layers, $N_{\mathrm{kv}}$ key/value heads, head dimension $d_h$, and KV byte widths $e_{\mathrm{KV,store}}$ and $e_{\mathrm{KV,read}}$, full-context attention reserves and reads

$$K_{\mathrm{alloc}}(L) = 2\, N_{\mathrm{layers}}\, N_{\mathrm{kv}}\, d_h\, e_{\mathrm{KV,store}}\, L, \qquad K_{\mathrm{read}}(L) \approx 2\, N_{\mathrm{layers}}\, N_{\mathrm{kv}}\, d_h\, e_{\mathrm{KV,read}}\, L,$$

where the factor $2$ counts keys and values. Weight precision and KV precision are independent settings. NVFP4 weights (NVIDIA’s 4-bit floating-point format) do not imply an NVFP4 cache, so $e_w$ and $e_{\mathrm{KV}}$ are tracked separately.

Mixture-of-experts

An MoE model is where the constant-$W_{\mathrm{batch}}$ assumption breaks, and fixing it is the single most important adapter correction. Let $P_{\mathrm{total}}$ be total parameters, $P_{\mathrm{active}}$ the active parameters per token, $E$ the number of routed experts, and $k$ the experts selected per token. For this decomposition, both parameter counts include the full input embedding. Assuming uniformly sized routed experts, each routed expert holds

$$p_{\mathrm{expert}} = \frac{P_{\mathrm{total}} - P_{\mathrm{active}}}{E - k},$$

and the always-on remainder (dense trunk, shared experts, embeddings, attention) is

$$P_{\mathrm{fixed}} = P_{\mathrm{active}} - k\, p_{\mathrm{expert}} .$$

The naive model assumes a batch touches the same active experts every session, keeping $W_{\mathrm{batch}}$ constant. That is false. Independent sessions route to different experts, so a larger batch touches more distinct experts. With $n$ token routings, each independently missing a given expert with probability $1 - k/E$, the expected number of distinct experts touched is

$$m(n) = E\left(1 - \left(1 - \frac{k}{E}\right)^{n}\right),$$

and the per-iteration shared traffic is the fixed part minus the untied input embedding, plus the touched experts. The embedding stays in the resident footprint.

$$W_{\mathrm{batch}}(n) = e_w\big[\, P_{\mathrm{fixed}} - P_{\mathrm{embed}} + p_{\mathrm{expert}}\, m(n) \,\big].$$

At $n = 1$ this reduces to the active-parameter footprint minus the untied input embedding, $W_{\mathrm{active}} = e_w(P_{\mathrm{active}} - P_{\mathrm{embed}})$, and as $n \to \infty$ it saturates at all $E$ experts. This rising $W_{\mathrm{batch}}(n)$ is why MoE batching does not amortize for free, and why the MoE rows in the worked table reach their throughput optimum at modest batch sizes.

One caveat applies here. Every other traffic term in the bound is a deliberate under-estimate of real traffic, which is what makes $R/q$ a true ceiling. The expert count $m(n)$ is different. It is an expectation under independent, uniform routing rather than a lower bound. Real routing is correlated, since load-balancing losses push toward uniform while hot experts and topically similar sessions pull the other way, and correlated routing can touch fewer distinct experts than the formula predicts. In that case the modeled traffic overstates the actual traffic, and the computed ceiling can sit below the true one. When a guaranteed ceiling is required, replace $m(n)$ by its minimum $k$, which replaces $W_{\mathrm{batch}}(n)$ by $W_{\mathrm{active}}$. The expectation form gives estimates, including during prefill. The floor form is the safe bound when the active-weight bytes are a valid traffic lower bound.

Hybrid, sliding, and recurrent attention

Models with local or sliding-window attention, compressed or latent attention, or linear/recurrent state must not use the full-KV formula blindly, because their cache does not grow linearly in $L$ everywhere. Split both KV terms into global, local, and fixed-state parts.

$$K_{\bullet}(L) = K_{\mathrm{global},\bullet}(L) + K_{\mathrm{local},\bullet}(L) + K_{\mathrm{state},\bullet}, \qquad \bullet \in \{\mathrm{alloc}, \mathrm{read}\}.$$

A simple read approximation with sliding-window width $w$ is

$$K_{\mathrm{read}}(L) = \kappa_{\mathrm{global}}\, L + \kappa_{\mathrm{local}}\,\min(L, w) + K_{\mathrm{state}} .$$

Full-attention layers pay for the whole context, sliding-window layers pay only up to the window, and recurrent or latent state adds a fixed or slowly growing term. The same shape covers Gemma-style local/global attention and DeepSeek-style compressed/sparse attention, with only the coefficients changing.

Prefill adapter fields

The prefill bound reuses almost the whole decode adapter. $W_{\mathrm{resident}}$ and $K_{\mathrm{alloc}}$ serve the same capacity gate, $W_{\mathrm{batch}}(n)$ is the same function fed the iteration’s token count, $K_{\mathrm{read}}$ reappears as the chunked prefix reread, and the write coefficient $\kappa_{\mathrm{KV,w}}$ is derivable from the allocation fields. Two entries are new. The linear FLOP count $P_{\mathrm{lin,active}}$ prices the weight multiplications, and a layered attention-FLOP function prices the context work for chunk width $c$ and prefix length $\ell$, with the window including the current token,

$$a_{\mathrm{attn}}(c,\ell) = \alpha_{\mathrm{global}}\, \bar L + \frac{\alpha_{\mathrm{local}}}{c}\sum_{j=1}^{c}\min(\ell+j,w) + \alpha_{\mathrm{state}}, \qquad \bar L = \ell + \frac{c+1}{2},$$

with $\bar L$ the average attended context of the iteration. Each global or local coefficient sums $4d$ FLOPs per pair over the layers in that group, using each layer’s query width, so this is the FLOP mirror of $K_{\mathrm{read}}(L) = \kappa_{\mathrm{global}} L + \kappa_{\mathrm{local}} \min(L, w) + K_{\mathrm{state}}$ above, and the same hybrid, sliding, and recurrent splits carry over, with each local length capped before averaging and the state cost supplied by the architecture. One decode caveat still applies here. The MoE adapter had to distinguish the expected expert count from its worst-case floor, and large prefill token counts do not make the expectation a guaranteed minimum. All tokens can still select the same $k$ experts. The decode field with no prefill counterpart is $\rho$. The extended adapter is

$$\mathrm{Adapter}(M) = \big[\, W_{\mathrm{resident}},\ W_{\mathrm{batch}}(n),\ K_{\mathrm{alloc}},\ K_{\mathrm{read}},\ \rho,\ P_{\mathrm{lin,active}},\ a_{\mathrm{attn}}(\cdot,\cdot) \,\big].$$

Calculator procedure

The ordinary-decode bound ($\rho=1$) is now ready to compute. Because the same computation runs for every hardware-and-model pair, it is worth stating once as a procedure.

The inputs are a hardware row $(C, R, F_{\mathrm{hw}}(\cdot))$, runtime overhead $O$, a model adapter $(W_{\mathrm{resident}}, W_{\mathrm{batch}}(\cdot), K_{\mathrm{alloc}}(\cdot), K_{\mathrm{read}}(\cdot), \rho, P_{\mathrm{lin,active}}, a_{\mathrm{attn}}(\cdot,\cdot))$, and workload assumptions $(L_{\mathrm{alloc}}, L_{\mathrm{read}}, r_\star)$, plus a prompt length $L$ and an optional TTFT budget $\tau_\star$ for prefill. For prefill, the model and runtime also select arithmetic modes $(\phi_{\mathrm{linear}}, \phi_{\mathrm{attention}})$ for the hardware-rate lookups. Ordinary decode needs no compute rate.

1. Compute the resident margin $C - W_{\mathrm{resident}} - O$. If it is negative, the model does not fit, so stop.
2. Compute $K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$ and the memory-fit batch $b_{\mathrm{mem}}$.
3. For each integer batch $1 \le b \le b_{\mathrm{mem}}$, compute $W_{\mathrm{batch}}(b)$ and $q_{\mathrm{KV}}(b, L_{\mathrm{read}})$.
4. Compute the aggregate ceiling $R / q_{\mathrm{KV}}$ and the per-session ceiling $R / (b, q_{\mathrm{KV}})$ at each batch.
5. Keep the batches whose per-session ceiling is at least $r_\star$.
6. Among the kept batches, choose the one with the largest aggregate ceiling, and report it as the KV-aware result with its batch as $b^\star$.
7. Separately compute the memory-power ceiling for orientation.

For speculation, use $q_{\mathrm{spec}}$ and its draft-inclusive capacity gate instead. The output is a stack of gates, and the right phrasing depends on which gate bound.

State	Meaning
Resident fit	The model plus overhead fits in memory
Session fit	At least one reserved-context session fits
Floor fit	Some fitting batch clears $r_\star$
No floor	Sessions fit, but no batch clears $r_\star$

The common invalid reading is that fitting in memory implies serving usefully. A model can pass resident fit and session fit and still have an empty usable batch set, because every fitting batch is below the floor. The honest report for that case is “fits, but no batch satisfies the floor”, a distinct verdict from a true fit failure. Keeping the two apart is the reason the floor gate exists.

For prefill, first look up $F_{\mathrm{linear}}=F_{\mathrm{hw}}(\phi_{\mathrm{linear}})$ and $F_{\mathrm{attention}}=F_{\mathrm{hw}}(\phi_{\mathrm{attention}})$. The prefill computation runs the same way over batch and chunk settings, with $\Omega_{\mathrm{pf}}=0$ in every capacity check. For each fitting $(b,c)$, sum traffic over the prompt’s chunks to get $Q_{\mathrm{pf}}$, set $A_{\mathrm{linear}}=2bL P_{\mathrm{lin,active}}$, and sum $bc,a_{\mathrm{attn}}(c,\ell)$ over the chunks to get $A_{\mathrm{attention}}$, using the actual width of any shorter final chunk. Compute the time and throughput as

$$\begin{aligned} t_{\mathrm{pf},\min}(b,c) &= \max\left(\frac{Q_{\mathrm{pf}}}{R},\ \frac{A_{\mathrm{linear}}}{F_{\mathrm{linear}}} + \frac{A_{\mathrm{attention}}}{F_{\mathrm{attention}}}\right), \\ T_{\mathrm{pf}}(b,c) &= \frac{bL}{t_{\mathrm{pf},\min}(b,c)}. \end{aligned}$$

Report the largest $T_{\mathrm{pf}}$ with its batch, chunk, both resource ceilings, and the latency floor. At fixed batch with workspace omitted, the search over $c$ collapses, since the optimum sits at $c = L$. The central formula is

$$T_{\mathrm{pf,max}}(L) \le \max_{\text{setting fits}} \frac{bL}{t_{\mathrm{pf},\min}(b,c)},$$

the prefill counterpart of the KV-aware decode bound, with a supplied $\tau_\star$ restricting the settings inside the maximum.

Worked examples

Now check the theory against real hardware. Consider two 128 GB machines, one bandwidth-rich and one bandwidth-poor, which isolate the effect of $R$ at fixed $C$.

Two machines

NVIDIA’s DGX Spark, a small desktop AI machine, carries 128 GB of LPDDR5x unified memory at 273 GB/s. An Apple M5 Max with a 40-core GPU reaches 614 GB/s and is configurable to 128 GB of unified memory. At equal capacity their memory powers are

Hardware	$C$	$R$	$D = CR$
DGX Spark	128 GB	273 GB/s	34,944 $\mathrm{GB}^2/\mathrm{s}$
Apple M5 Max 128GB	128 GB	614 GB/s	78,592 $\mathrm{GB}^2/\mathrm{s}$

Both bandwidth numbers are catalog figures rather than measured sustained rates, so the earlier caveat applies. If one machine sustains a larger share of its spec than the other, the comparison will make the other machine look better than it really is.

Three models

Three MoE models, modeled with simplified adapter parameters, without re-measurement.

• Qwen3.6-35B-A3B, with 35B total / 3B active parameters, $E = 256$ routed experts, $k = 8$ routed (plus one shared) per token, and weights quantized NVFP4.
• Gemma 4 26B-A4B-it, with 26B total / 4B active, $E = 128$, top-8 routing, hybrid local/global attention, and NVFP4 weights.
• DeepSeek V4 Flash (DS4), with 284B total / 13B active, $E = 256$ routed plus one shared, $k = 6$ per token, million-token context via compressed/sparse attention, and weights at a Q2-style mixed quantization.

All rows use the original Our Models defaults, namely reserved context $L_{\mathrm{alloc}} = 100{,}000$, active context $L_{\mathrm{read}} = 32{,}000$, per-session floor $r_\star = 20$ tok/s/session, ordinary decoding $\rho = 1$, runtime overhead $O = 8$ GB, and the memory roofline only. The numbers are memory-side estimates from the simplified adapters and expected expert routing, not guaranteed hardware ceilings.

The following inputs complete the calculation at those context lengths. They are the original illustrative adapter values, not audited tensor sizes or cache measurements. Use the parameter and expert counts above in the MoE formula, which includes shared experts in $P_{\mathrm{fixed}}$. These three illustrative adapters set $P_{\mathrm{embed}}=0$ as an approximation, since their rounded inputs do not separate the embedding. Memory values use decimal GB. Results are rounded only after choosing the batch.

Model	$e_w$ (bytes/parameter)	$W_{\mathrm{resident}}$ (GB)	$K_{\mathrm{alloc}}(100{,}000)$ (GB/session)	$K_{\mathrm{read}}(32{,}000)$ (GB/token)
Qwen3.6-35B-A3B	0.5	17.5	1.025	0.3268992
Gemma 4 26B-A4B-it	0.5	13	0.618497	0.269504
DeepSeek V4 Flash (Q2)	0.25	71	0.31410256	0.016

Results

Hardware	Model	Single-session estimate	$b^\star$	KV-aware aggregate estimate	Memory-power estimate
DGX Spark	Qwen3.6-35B-A3B	149 tok/s	17	346 tok/s	18.2k tok/s
DGX Spark	Gemma 4 26B-A4B-it	120 tok/s	16	333 tok/s	23.6k tok/s
DGX Spark	DeepSeek V4 Flash (Q2)	84 tok/s	7	155 tok/s	13.1k tok/s
Apple M5 Max 128GB	Qwen3.6-35B-A3B	336 tok/s	50	1,008 tok/s	40.9k tok/s
Apple M5 Max 128GB	Gemma 4 26B-A4B-it	271 tok/s	66	1,323 tok/s	53.1k tok/s
Apple M5 Max 128GB	DeepSeek V4 Flash (Q2)	188 tok/s	22	448 tok/s	29.5k tok/s

Two things stand out. First, with capacity held equal, the higher M5 Max bandwidth lifts the single-session estimates in proportion to $R$, and the batched estimates by even more, because the extra bandwidth also lets more sessions clear the per-session floor. The bandwidth-rich machine wins exactly where the theory says it should, in batched throughput. Second, the memory-power column sits one to two orders of magnitude above the KV-aware column. That gap is the cost of private context traffic and expert diversity, and showing it is the point of the derivation.

Forced concurrency

What if concurrency is fixed by policy rather than chosen at the floor-satisfying optimum? On DGX Spark, pushing past $b^\star$ buys aggregate throughput at the cost of per-session rate.

Model	Batch $b$	Aggregate estimate	Per-session estimate
Qwen3.6-35B-A3B	32	397 tok/s	12.4 tok/s/session
Qwen3.6-35B-A3B	64	482 tok/s	7.5 tok/s/session
Gemma 4 26B-A4B-it	32	434 tok/s	13.6 tok/s/session
Gemma 4 26B-A4B-it	64	581 tok/s	9.1 tok/s/session

This is the serving tradeoff in numbers. For DGX Spark under these assumptions, 32 and 64 sessions are estimated to be too high if the goal is around 20 tok/s/session, exactly the regime the usable-batch correction is built to reject, and the reason $b^\star$ for these models settles near 16.

A sanity check against a real run

A reported but unverified DGX Spark run served Gemma at concurrency 16 at roughly 16 to 18 tok/s/session, an aggregate of $16 \times 16 = 256$ to $16 \times 18 = 288$ tok/s. The KV-aware aggregate estimate for Gemma at this batch is 333 tok/s, so the reported throughput is

$$\frac{256}{333} \approx 77\% \quad\text{to}\quad \frac{288}{333} \approx 86\%$$

of the simplified estimate. That ratio alone does not establish proximity to the memory-side roofline. It does not prove the quantization is optimal. The bound omits compute, scheduler behavior, kernel details, and exact cache traffic, and proving optimality would require profiler evidence of bandwidth saturation with no compute, scheduler, or CPU stalls.

Prefill on real hardware

The prefill example uses the DGX Spark again, paired with Poolside’s Laguna S 2.1 NVFP4 checkpoint, because a real prefill benchmark exists for the pair. For $F_{\mathrm{hw}}$, NVIDIA quotes the GB10 chip at 1 PFLOP of FP4 with sparsity, and this example assumes ideal rates of 500 TFLOP/s of dense FP4 and 125 TFLOP/s of dense BF16. The attention math is assumed to run on the BF16 path, so it is charged at 125 TFLOP/s throughout, and only the linear term ranges over the 125-to-500 bracket, since it is not obvious how much of that math runs on the fast path.

The bound needs three things from the model, its sweep size, its linear FLOP count, and its KV write coefficient. The model card, configuration, and tensor index at revision b482b5d57fda6e4e562a652869bde24ba2a57c92 supply the inputs. Laguna S 2.1 has 117.6B total and 8.5B active parameters, 48 layers of which 12 use full attention and 36 use a 512-token sliding window, 48 query heads in full-attention layers and 72 in sliding-window layers, each of dimension 128 (so $d = 6144$ and $d = 9216$, respectively), 256 routed experts with 10 picked per token, and an FP8 KV cache. From those numbers, the 71.9 GB NVFP4 checkpoint carries a swept weight payload of 71.28 GB (71.8987 GB of tensor data minus the 0.6166 GB input embedding), subtracting the embedding and output head from the active parameters leaves $P_{\mathrm{lin,active}} \approx 7.9$B, and the FP8 cache takes 98,304 bytes of writes per prefilled token.

The expert formula decides the memory side. The expected distinct experts $m(c)$ already reach 254 of 256 at a 128-token chunk, so the expected-routing model approximates realistic chunks as full sweeps of 71.28 GB, and at 273 GB/s the model gives about 3.8 sweeps per second. A scheduler that caps chunks at 2048 tokens is therefore estimated near $3.8 \times 2048 \approx 7{,}800$ tokens per second before prefix rereads. One-shot prefill sweeps once per prompt instead and lifts the memory estimate to 31k tok/s at an 8k prompt, 120k at 32k, and 425k at 128k. The crossover confirms the regime, since $S^\star$ for this pair sits between about 2,100 and 8,300 tokens per sweep depending on the compute bracket, so a 2048-token chunk is still memory-bound at the FP4 bracket while a one-shot prompt can pass the crossover.

The compute side uses the layered attention form with 12 full and 36 sliding-window layers. The tables use $P_{\mathrm{lin,active}} = 7.9$B and omit $A_{\mathrm{extra}}$. At $L=8192$, the attention cost is $4[12\times6144\times4096.5 + 36\times9216\times(512-512\times511/(2\times8192))] \approx 1.866$ GFLOP per token, giving $15.8+1.866=17.666 \approx 17.7$ GFLOP per token. The estimate, at batch 1 with one-shot prefill:

Prompt $L$	$a_{\mathrm{pf}}$ per token	Compute estimate	Memory estimate	Combined estimate	Modeled prefill time
8,192	17.7 GFLOP	7.1k to 21.5k tok/s	31k tok/s	7.1k to 21.5k tok/s	0.4 to 1.2 s
32,768	21.3 GFLOP	5.9k to 13.2k tok/s	120k tok/s	5.9k to 13.2k tok/s	2.5 to 5.6 s
131,072	35.8 GFLOP	3.5k to 5.2k tok/s	425k tok/s	3.5k to 5.2k tok/s	25 to 38 s

The 2048-token-chunk policy is the same formula with $c$ pinned:

Prompt $L$	Memory estimate	Combined estimate	Modeled prefill time
8,192	7.8k tok/s	7.1k to 7.8k tok/s	1.0 to 1.2 s
32,768	7.8k tok/s	5.9k to 7.8k tok/s	4.2 to 5.6 s
131,072	7.6k tok/s	3.5k to 5.2k tok/s	25 to 38 s

The weight multiplications, 15.8 GFLOP per token, dominate the compute cost until the context gets long. At 128k the attention work reaches 20.0 GFLOP per token and overtakes them, and under the assumed BF16 path it pulls even the high bracket down to 5.2k, where compute is the limit under either policy.

A measured run gives a comparison. A vLLM 0.25.1 NVFP4 benchmark on the GB10 reported 2,348 prompt tokens per second by dividing about 8,192 input tokens by the median 3.49-second time to first token. This includes work outside the prefill kernels. It used DFlash with 15 draft tokens and max_num_seqs=32. With a 2048-token batch budget, vLLM’s scheduler rule reserves $32\times(15-1)$ draft slots, giving max_num_scheduled_tokens=1600 for chunked prefill. Neither table matches that policy exactly. The earlier 77% to 86% decode comparison used Gemma, not this Laguna workload. These ratios do not measure idle hardware or identify the cause of the gap.

Omitted rooflines

The memory and compute rooflines are now both explicit, and real throughput is still the minimum over further limits,

$$T_{\max} \le \min\big(T_{\mathrm{memory}},\ T_{\mathrm{compute}},\ T_{\mathrm{kernel}},\ T_{\mathrm{scheduler}},\ T_{\mathrm{interconnect}}\big),$$

where the terms this post computes can be undercut by dequantization kernels, attention kernels and KV layout, prefill/decode phase mixing, scheduler overhead and request churn, CPU and PCIe involvement, multi-GPU communication, allocator fragmentation, thermal and power limits, tokenization and sampling, and speculative rejection rates. Each belongs as its own limit term. Prefix sharing can instead reduce traffic and storage. The private-KV formulas here assume no cross-session sharing. The two derived rooflines remain useful because they make the first unavoidable ceilings explicit and cheap to compute.

The compute roofline also carries a weakness the memory one does not. The memory bound remains a true ceiling only when its traffic terms are lower bounds. Expected expert routing gives an estimate, while a ceiling computed from spec-sheet peak FLOP/s is far too high, because real kernels on quantized weights often reach less than half of peak and the fraction changes with kernel and precision. For bandwidth this post could note the sustained-versus-spec gap and move on. For $F$ it decides whether the number means anything, and measuring believable sustained FLOP/s across consumer devices is the hard open problem of the prefill extension. Apple silicon illustrates it, since Apple publishes no comparable dense BF16 or FP16 GPU rate, and Neural Engine TOPS or estimates from GPU core counts are no substitute.

One recurring caution applies to models with recurrent or linear state. A model with tiny fixed state and tiny private read traffic produces an enormous memory-side aggregate at high concurrency, because almost nothing in the denominator grows with the batch. That is precisely the signal that compute, kernel, scheduler, and recurrent-state details must be added before the aggregate number is treated as realistic. The memory bound describes the best case the hardware allows, and reaching it is the implementation’s job.

Cheat sheet

This section collects the whole formulation in one place, so it can be read on its own. A machine is two memory numbers and a table of compute rates, a model adapter is five decode quantities plus two prefill ones, and the workload adds its assumptions.

Symbol	Meaning
$C$, $R$	Hardware memory capacity before subtracting overhead, and memory bandwidth
$F_{\mathrm{hw}}(\phi)$	Hardware compute rate limit in FLOP/s for mode $\phi$, including operand and accumulation formats and dense or sparse arithmetic
$O$	Runtime memory overhead, subtracted from $C$ once
$W_{\mathrm{resident}}$	Full resident weight footprint, which must fit in memory
$W_{\mathrm{batch}}(n)$	Shared weight traffic for an iteration of $n$ token routings, $n = b$ in ordinary decode, equal to $W_{\mathrm{active}}$ for dense models and growing with $n$ for MoE
$W_{\mathrm{active}}$	Active-weight sweep size, the lower bound $W_{\mathrm{batch}}(1)$
$K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$	KV memory reserved per session
$K_{\mathrm{read}}(L_{\mathrm{read}})$	Private context traffic per output token
$\rho$	Tokens emitted per session per iteration, one for ordinary decoding
$\gamma$, $W_{\mathrm{draft}}$, $K_{\mathrm{draft,read}}$, $K_{\mathrm{draft,alloc}}$	Draft length, draft weight sweep, draft per-step cache read, and draft cache allocation for speculative decoding
$L_{\mathrm{alloc}}$, $L_{\mathrm{read}}$	Reserved and average read context, $L_{\mathrm{read}} \le L_{\mathrm{alloc}}$
$r_\star$	Minimum useful tokens/s per session
$\phi_{\mathrm{linear}}$, $\phi_{\mathrm{attention}}$	Arithmetic modes selected by the model and runtime for linear layers and attention
$F_{\mathrm{linear}}$, $F_{\mathrm{attention}}$	Derived rates $F_{\mathrm{hw}}(\phi_{\mathrm{linear}})$ and $F_{\mathrm{hw}}(\phi_{\mathrm{attention}})$ in FLOP/s
$F$	Common compute rate in the single-rate derivation
$\Omega_{\mathrm{pf}}$	Temporary prefill workspace, set to zero in this theoretical bound
$L$, $c$, $\ell$	Prompt length, chunk width, and cached prefix length in prefill
$\kappa_{\mathrm{KV,w}}$	KV bytes written per prompt token
$P_{\mathrm{lin,active}}$, $d = N_q d_h$	Active linear parameters, and the query-side hidden width
$N_{\mathrm{layers}}$	Transformer layer count
$\tau_\star$	Optional TTFT budget for the prefill floor

Everything descends from the two rooflines, $T_{\max} \le \min(R/q,\ F/a)$ for a token costing $q$ bytes of traffic and $a$ FLOPs. These are bounds when resource rates are upper limits and costs are lower limits. Expected expert routing or measured sustained rates give estimates instead. The ordinary-decode formulas here use $\rho=1$ and keep only the memory constraint,

$$T_{\max} \le \frac{R}{q} .$$

The first gate is whether sessions fit. Load the weights, reserve the overhead, and divide what is left by the per-session KV allocation.

$$b_{\mathrm{mem}}(L_{\mathrm{alloc}}) = \left\lfloor \frac{C - W_{\mathrm{resident}} - O}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})} \right\rfloor .$$

Per-token traffic is the shared weight sweep amortized over the batch plus the private context read, which no batch size amortizes.

$$q(b, L_{\mathrm{read}}) = \frac{W_{\mathrm{batch}}(b)}{b} + K_{\mathrm{read}}(L_{\mathrm{read}}) .$$

The roofline gives the aggregate ceiling at batch $b$, and dividing by the batch gives the per-session rate. The aggregate rises with $b$ while the per-session rate falls.

$$T(b) \le \frac{R}{q(b, L_{\mathrm{read}})}, \qquad r(b) = \frac{T(b)}{b} \le \frac{R}{W_{\mathrm{batch}}(b) + b\,K_{\mathrm{read}}(L_{\mathrm{read}})} .$$

Imposing the floor $r_\star$ rejects the batches where every session crawls, and the usable batch is whichever gate binds first. The closed form below uses $W_{\mathrm{active}}$ as a lower bound and requires positive $K_{\mathrm{read}}$. For MoE it is a necessary condition, and the full batch test below is still required.

$$b_{\mathrm{rate}}(L_{\mathrm{read}}, r_\star) = \left\lfloor \frac{R / r_\star - W_{\mathrm{active}}}{K_{\mathrm{read}}(L_{\mathrm{read}})} \right\rfloor, \qquad b_{\mathrm{usable}} = \min\big(b_{\mathrm{mem}},\ b_{\mathrm{rate}}\big) .$$

The usable batch set holds the batches that fit and clear the floor, and the KV-aware bound is the best aggregate over it.

$$\mathcal{B} = \left\{ b : 1 \le b \le b_{\mathrm{mem}},\ \frac{R}{b\, q(b, L_{\mathrm{read}})} \ge r_\star \right\}, \qquad T_{\max} \le \max_{b \in \mathcal{B}} \frac{R}{q(b, L_{\mathrm{read}})} .$$

Setting $b = 1$ in the same formula gives the single-session ceiling. Dropping the private context term and taking the largest fitting batch gives the looser memory-power ceiling,

$$T_{\max} \le \frac{D}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})\,W_{\mathrm{active}}}\left(1 - \frac{W_{\mathrm{resident}} + O}{C}\right), \qquad D = CR .$$

With speculation the private read divides by $\rho$, because one verification pass reads the cache for all emitted tokens, and the draft adds its own costs,

$$q_{\mathrm{spec}}(b) = \frac{W_{\mathrm{batch}}(b(\gamma+1)) + \gamma\, W_{\mathrm{draft}}}{b\,\rho} + \frac{K_{\mathrm{read}}(L_{\mathrm{read}}) + \gamma\, K_{\mathrm{draft,read}}}{\rho},$$

with the draft’s residency and cache joining the memory gate. For ordinary decoding with upper resource limits and lower costs, the three levels order the same way.

$$\text{actual throughput} \;\le\; \text{KV-aware bound} \;\le\; \text{memory-power bound} .$$

Prefill runs on the same adapter plus $F_{\mathrm{linear}}$ and $F_{\mathrm{attention}}$, looked up from the hardware table using the selected arithmetic modes, with $\Omega_{\mathrm{pf}}=0$ in the capacity gate. If a required rate is unknown, the compute result is unavailable. Per prompt token, an iteration of chunk width $c$ over a cached prefix $\ell$ moves

$$q_{\mathrm{pf}}(b, c, \ell) \approx \frac{W_{\mathrm{batch}}(bc)}{bc} + \frac{K_{\mathrm{read}}(\ell)}{c} + \kappa_{\mathrm{KV,w}}$$

bytes of memory traffic and spends

$$a_{\mathrm{pf}}(c, \ell) \approx 2 P_{\mathrm{lin,active}} + 4 N_{\mathrm{layers}}\, d \left(\ell + \frac{c+1}{2}\right)$$

FLOPs of compute. For sliding and recurrent layers, replace the full-attention term with $a_{\mathrm{attn}}(c,\ell)$ above. Sum traffic over the prompt’s chunks to get $Q_{\mathrm{pf}}$, set $A_{\mathrm{linear}}=2bL P_{\mathrm{lin,active}}$, and sum $bc,a_{\mathrm{attn}}(c,\ell)$ over the chunks to get $A_{\mathrm{attention}}$, using the actual width of any shorter final chunk. Assuming one arithmetic mode per path and that both paths share the same execution units and cannot overlap, charge each compute total at its own rate and add the times. Split a path’s FLOPs by mode if it uses several modes.

$$t_{\mathrm{pf},\min}(b,c) = \max\left(\frac{Q_{\mathrm{pf}}}{R},\ \frac{A_{\mathrm{linear}}}{F_{\mathrm{linear}}} + \frac{A_{\mathrm{attention}}}{F_{\mathrm{attention}}}\right).$$

When both compute rates equal $F$, the sum of the processing times is $A_{\mathrm{pf}}/F$, with $A_{\mathrm{extra}}=0$. The ceiling at prompt length $L$ maximizes throughput over settings that pass the same memory gate,

$$T_{\mathrm{pf,max}}(L) \le \max_{\text{setting fits}} \frac{bL}{t_{\mathrm{pf},\min}(b,c)},$$

with the maximum at one-shot $c = L$ at fixed batch when workspace is omitted, fixed-chunk evaluations serving as policy ceilings, and a supplied TTFT budget $\tau_\star$ restricting the settings through the wait $t_{\mathrm{pf}} = bL/T_{\mathrm{pf}}$. Ordinary decode uses the $c = 1$ memory formula with the KV write term omitted, the rooflines cross near $S^\star = W_{\mathrm{batch}}(S^\star) F_{\mathrm{linear}} / (2 P_{\mathrm{lin,active}} R)$ tokens per sweep when attention is omitted, and a range over prompt lengths reports $T_{\mathrm{pf,lo}}$ and $T_{\mathrm{pf,hi}}$ with the length at each extreme. With both resources included, decode and prefill’s loosened bounds form two chains hanging off the shared minimum, one per resource, so no single orientation line exists.

When resource rates are bounded above and per-token costs below, these formulas give an upper bound built from memory capacity, memory bandwidth, and for prefill compute throughput. Expected-routing calculations are estimates. Real implementations under the stated assumptions land below a valid bound, and kernel quality, software overhead, and interconnects can only lower the ceiling further.

When reporting a result, always state the assumptions that move it, namely $O$, $L_{\mathrm{alloc}}$, $L_{\mathrm{read}}$, $\rho$, $r_\star$, the weight precision, and the KV-cache precision or attention adapter, and for prefill also the prompt length, the chunk mode, $\phi_{\mathrm{linear}}$, $\phi_{\mathrm{attention}}$, their hardware rate entries and sources, and $\Omega_{\mathrm{pf}}=0$. State whether the calculation uses resource limits or measured rates and minimum or expected expert traffic. Without them, a single tok/s number is not reproducible.

To see these bounds computed live for hundreds of audited model profiles against a catalog of local hardware, try Our Models. An independent implementation of the decode bound runs at the Localmaxxing decode calculator.