Entries for August 24, 2026

@onusoz · /2026/08/24 · 04:13 PM View on

I was not expecting this to receive so much attention! There was a lot of constructive feedback to what was basically an outcry after battling the complexity of the benchmarking space for 2 months straight I agree with the points that models are post-trained to be efficient in a certain harness, so an objective measurement must use that But I also see that it is very convenient for proprietary model vendors for everyone to accept that the model is inseparable from the harness The main issue is that model population (as well as harness) is about to explode, and there must be an effort to standardize things. That should be done by an industry-wide consortium and/or a neutral third party I am not going to propose a harness in this post, but just share an idea that, if we had agreed on one, how we would reconcile models' baselines across different harnesses Idea: We could run each benchmark on at least 2 harnesses: Let S be the score of the model on a given benchmark using the *standardized test harness* and N be the score of the model over the *native harness* which the vendor post-trained the model with Define "Standard Harness Sensitivity"[*] as: SHS = (N - S) / (N + S) Which gives normalized score between -1 and 1 that measures how sensitive a model is to being put in a different harness other than the recommended one. 0 means that the model is neutral to which harness it's in (at least for the given 2) 1 means it is much worse outside the native harness, for that benchmark -1 means for some reason, it functions much better in the standard harness than the native harness, which might imply that the vendor messed something up So just by running a benchmark 2 times and reporting 3 quantities S/N/SHS, you can get a lot of info: - How well a model would perform if you were to use the native harness, - how well it would perform if you were to use a more simple and neutral harness, - and how sensitive it is to change of harness Running benchmarks is EXPENSIVE, and the cheaper we can extract this signal (i.e. with 2 harnesses instead of N), the better --- [*] I would call it just Harness Sensitivity, but apparently that term is already used

@onusoz · Aug 23, 2026

We need to normalize measuring and judging models against a standardized test harness "Oh but model X performs best in their own proprietary harness" I could not care less. When I take exams, I go to the standardized classroom, get the standardized pencil and exam sheet, and have to solve it under 2 hours This system arose because we have a LOT of people to test Guess what? We now have a LOT of models, and they are multiplying by the day "Oh but model X performs substantially better in ARC-AGI-3 with a custom harness" I don't care... Then imbue model X with enough knowledge so that it can reconstruct that harness on the spot The main harness could be mini-swe-agent, terminus 2, vanilla pi or something along those lines It needs to be simple, and stay roughly the same over time There is already too much complexity in the benchmarking space right now, and I feel like not enough people are putting their feet down to cut away some part of it