Auto-research with codex: How I achieved a 232x Faster Kernel

In the last couple of days I wanted to try out the new definitive DeepSeek v4 releases. I gave it the repository of a semi-abandoned video compression codec and I told it to perform the usual benchmark -> profile -> verify -> research -> improve loop. I specifically chose this codec because the authors include a verifier for the bitstream to make sure you don't break stuff if you want to try your own implementation. I gave the agents access to the compiler's profiler and also Intel's VTune, which has fantastic output. In a couple of hours the LLM generated SSE and AVX implementations of the compression and decompression algorithms that almost doubled performance with a single core. Then I asked it to create a CUDA implementation using NVIDIA's NSIGHT profiler as a guide and it also started doing some good work.

Personally, I believe that LLMs should be treated like an advanced version of Prolog or linear programming: you give the constraints, you have a way of verifying correctness, and you give it a clear goal. If the LLM can verify itself and course-correct you can basically leave it on autopilot

One thing worth to note in the competition is that 8 out of the 10 top solutions, which all happened to be optimized this way completely broke at any other input than the competition ones.

The only solutions that did not break when tested with OOD shapes were made by experts who know a lot about GPU programming and that did not create 25k lines of CUDA but followed and adjusted their solution in reasonable bounds.

The takeaway from this is that these approaches will always solve for specificity, but it's a much harder task to steer the model into making general solutions. So if you're an inference provider for some specific model shape, fantastic, go for it. If you are a maintainer of a open-source library, this is not useful.

Meta commentary but it felt fresh to read a long wall of text that didn't seem to be AI generated. Thanks.

It's been fascinating doing a custom variant for GFQL, the first OSS embeddable Cypher property graph query engine for CPU+GPU -

- accelerated launch of our new backends like polars, including a new lazy mode & planner, which are fundamentally new paths

- while we initially aimed for top GPU benchmark scores, we now also maintain top CPU scores too!

Long-term, more interesting to me is this opens rethinking what it means to be a query engine. Right now we are making it the fastest in general, especially on workloads from our own use, major industry benchmarks, and our users. At the same time, similar to jit and multistage computing, we're looking at new ahead-of-time optimization techniques users can do that are more interesting than plugging in custom indexes. Essentially, if our agents can do fast specializations, there should be safe hooks that we can expose to our user's agents too!

Training material seems to be especially rich re GPU kernels and SIMD.

I wonder if there is extra effort put into this because they are useful for the researchers working on the models or just a sub-domain that language models are a great fit for and humans have trouble with?

Mirrors my experience: LLMs are really good at optimizing, better than most humans. But also, they tend to not reach absolute peak performance where people made an effort to optimize something.

Since most problems see fairly little optimization, that's still a big win most of the time.

Isn't cholesky - used to substitute householder at a point - faster but less stable in some cases? I'm just recalling from memory since I had done a small project on qr decomposition with householder for an exam this year. I mean, if it is faster than the standard torch operation probably there are good reasons for which it is not the default standard torch operation. Might as well be wrong, I'm not sure

some of these submissions seem to be omitting the actual rules. the #1 on edinh has a line that says "bypass ban check"

Damn! If a solo engineer can do this, it makes the most around OAI/Anthropic start to look pretty weak.

This is really cool - I really like the beam search idea,

[flagged]

Every step here has an oracle: wall-clock, the profile, pass or fail from the verifier. I had an agent-built app audited task by task, 10 came back done and 7 worked, and the three misses were the ones needing a credential or a setting on someone else's dashboard. Nothing in the loop could tell the agent it had failed, so it said done and moved on.

[dead]

[dead]

People are always going to hate auto-research and "loop engineering". Because it's got 2 properties:

1) it's the only way to get something out of models (or people for that matter) that they don't know yet.

2) it's harder to do with an LLM than without. Not easier.

3) and when you fuck it up, half the time the LLM (or other ML technique) makes a fool out of you and you spent $1000 to find the quickest way to get a robot leg on the ground is just to crash it into the ground.

Achieving a 232x speedup on a kernel via automated tooling is an incredible engineering feat. Fascinating read on optimization.