Post
  1. Portrait of Onur Solmaz

    Measuring agents is the next essential skill

    @onusoz · /2026/09/07 · View on
    Can we show my wife @qrlow some love 🤗🤗🤗 She's just started dipping her toes in evaluating LLMs in her own field I helped her out just with running the benchmark and she solved some of the issues pretty creatively, like how to come up with synthetic tasks that match what usually happens inside a trading house She is on Hugging Face too 🤗 No better place than to publish a benchmark I have been encouraging all my close circle to go towards benchmarking models as well Literally every company that uses AI in any way will have to measure their performance, on internal metrics Especially if they start leveraging their data, training their own models and such So every employee will benefit from such know-how The most important one that comes after knowing how to use agents is knowing how to measure them If you are worried of becoming obsolete, one of the highest leverage things you could learn to do would be to try out running some benchmarks, learn what @harborframework does, converting your own companies' mundane tasks into evals, and so on
    @qrlow ·
    I started building a benchmark for AI's performance on the tasks done on a commodities trading desk The biggest challenge was figuring out where to source the tasks The commodities industry is a gate-kept one, where most data and systems are proprietary, and most workflows can only exist within the infrastructure of a firm. Building an eval independently meant not having access to anything I had to recreate simulated environments based on how I’ve seen these workflows operate in practice And that wasn’t the only challenge. As a non-technical person, I learned that running benchmarks is pretty complicated with a lot of moving parts (e.g. infrastructure issues, rate limits) Thankfully, I could figure things out with the help of agents Btw, if you are a commodities analyst/trader/operator etc and have a workflow in mind you want to test if AI agents can do, feel free to DM me a description of it to run a eval on Blog: qrlow.me/blog/introducing-deepdesk-bench/