Entries for July 24, 2026

  1. Portrait of Onur Solmaz
    @onusoz · /2026/07/24 · View on
    Was surprised to see no results for @poolsideai Laguna S 2.1 on DGX spark on localmaxxing.com/en, so I submitted what I get with setups based on official recipes on my end. (Hello World @LottoLabs) Interesting, vLLM had 3x higher prefill and 3x shorter TTFT. Could be partially or fully due to native kernels on NVFP4. Unsurprising that it is faster, but 3x seems too much, maybe I am doing something wrong on my end Also interesting that median decode performance increase is around 10%, even with DFlash (if not due to it). 19 -> 21 is negligible The model is out since a few days, and things should increase in the long run My theoretical upper bound formulation estimates ~25 tok/s decode at 4 bit quantization (ignoring speculative decoding) Also, getting this model to work at 4 bit quantization with 70 gb weights in the 128 gb dgx spark was a challenge (this model is not the only thing I am running on my machine) I am guessing a lot of people are running into the infamous OOM freeze the spark suffers from right now, while experimenting with this (If you suffer from this, osolmaz/infer-guard might help you) A 2-bit quantization might prove to be more ergonomic for this model in the long run, if it doesn't affect the performance that much (+ you would be able to run it on 64gb memory machines) (Compare with Poolside's reported numbers: huggingface.co/poolside/Laguna-S-2.1-NVFP4) vLLM NVFP4+DFlash: 21.6 tok/s decode, ~2300 prefill tok/s Localmaxxing: localmaxxing.com/en/models/poolside/Laguna-S-2.… Recipe: github.com/vllm-project/recipes/blob/f63b326a77… llama.cpp Q4_K_M: 19.3 tok/s decode, ~790 prefill tok/s Localmaxxing: localmaxxing.com/en/models/poolside/Laguna-S-2.… Recipe:
  3. Portrait of Onur Solmaz
    @onusoz · /2026/07/24 · View on
    PSA. This little button collapses the sidebar in @herdrdev so you can see the session full width Must have been there for some already but I have updated only recently…