---
title: "Upper Bounds for Local Model Throughput"
date: 2026-07-17
canonical: https://solmaz.io/x/2078193844966080981/
x_url: https://x.com/onusoz/status/2078193844966080981
license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
---

Request for Review

I spent a lot of time trying to profile local models, and am bothered by the fact that I don't have a mathematical framework to reason about the maximum token throughput I can expect from a model (to my knowledge)

I did some exploration based on a toy model of a GPU. My main goal is to find guaranteed upper bounds for throughput and other performance, to act as a rule of thumb while trying to optimize inference

Like, if I wanted to serve at 50 tok/s, how many parallel sessions can I do with this GPU, running a specific model's specific quantization with a specific architecture?

Ideally, one should be able to plug in memory capacity and bandwidth of the GPU, and model specific parameters, like total model size, number of parameters or active parameters, etc. This is what I tried to do here and I think I have a good first try

I am not sure if I am reinventing the wheel, so please tell me if this formulation already exists somewhere. The goal is to find an upper bound in terms of memory, so it makes the assumption that inference is bottlenecked by memory read/write and ignores the case where computation is a bottleneck

I present 2 closed form upper bound formulas. The bigger one is to draw an absolute ceiling on token throughput from a GPU, which is:

(memory capacity * memory bandwidth)
/
(model weight size * kv cache size per session)

which is not practical and gives too high numbers due to ignoring overhead of KV operations, but it is a theoretical upper limit for the given numbers

And then I introduce architecture specific formulas which take into account KV operations, and give much closer results. But I will not go into the details of that here, please refer to the post for that

Speculative decoding speedup also appears as a simple multiplier rho

If you are profiling local models, I would appreciate if you took time to look at this formulation and review it!🙏 Please let me know in the replies if you find any issues, because there most certainly are some!

Post:

Using the formulation, I calculated theoretical upper bounds for all the models on Hugging Face that have downloads over 100k, matched against a database of consumer GPUs. You can find the link to that in the tweet below 👇

The video here is a visualization of the 2 upper bounds I introduce, showing the bits of memory that is on the critical path of one cycle of inference, as if they are on a single pipeline

*Part 1/2 of a thread; root: https://solmaz.io/x/2078193844966080981/*