Entries for August 9, 2026

urgh typo, I meant 35-40 tok/s

Apparently Alibaba did all this work, published a paper, but did not create a public backup of repos used in the review tasks arxiv.org/html/2601.19494v2 Then keycloak and nodejs repos got force pushed, so the commits for 6 of the tasks got lost :( I recovered 4 of them, but 2 commits are still missing: keycloak/keycloak#35645 460f8008f86d3fa8f62da63e26d8bdc306af60b2 nodejs/node#56185 b2255442712cb6db83d112deb6ba61197d06a5f3 Would anyone happen to have them backed up locally or somewhere in a fork?

I have made an update to my theoretical upper bound calculation to also predict prefill speed Prefill relaxes the assumption we make for decode, that it is only be memory bottlenecked. So prefill can be both compute or memory bottlenecked. I use the FLOP limits reported by hardware producers for the estimates: These estimates will also be available in ourmodels.cc for indexed model and hardware in a couple days, once a long running job finishes Blog post: solmaz.io/llm-performance-upper-bounds

You can use the calculator in localmaxxing.com/en now to predict theoretical upper bounds from solmaz.io/llm-performance-upper-bounds 🙌 A little note, pay attention to the speculative decoding coefficient rho. If an upper bound is calculated using rho=1, then dspark might push it well above that limit. For example, the formulation predicts an upper bound of 28 decode tok/s, but dspark pushes it up to 35-50 tok/s in github.com/0xSero/deepseek-v4-flash-0731-spark-… by @0xSero I am curious whether we will observe a phenomenological law between real-life engine performance and the upper bound, like real-life performance maxes out at e.g 80% of the upper bound in most cases That would be very useful! Then we would be able to tell when something is wrong with an inference engine, if it doesn't reach at least 80% of the upper bound, ignoring spec. decoding