Yup. I recall that at some point maybe a year ago, pretty much every other LLM thread that had people speculating whether some LLM could do X or Y / improved or worsened for Z / etc., would have Paul show up and comment something along the lines, "Actually, I've benchmarked this thoroughly in my work on Aider; here's <link to data and analysis>" or such. Those were usually some of the most insightful comments in the whole thread.

I found those comments, and the work they linked to, especially valuable because it's rare to see advanced work on LLM applications done and talked about in the open. Everyone else doing equivalent work seems to want to make a startup out of it, so all we usually get is weekend hacks that stall out quickly.