I rewrote PostHog's SQL parser, 70x faster, while barely looking at the code

I’ve had very good success in similar setups where you have some sort of “oracle” and can generate enormous corpuses of test data, such that you really, really trust the LLM code must work for the inputs you expect it’ll ever need to handle.

Makes me think of all the algorithms we specify in proof languages and then hand-implement in production languages - this setup could maybe let you just specify the proof of an algorithm and then let LLMs derive efficient implementations with the (slow) proof as an oracle

Well despite my current anti AI sentiment, I have to admit that after reading the article, It was a good use of AI, done by someone with good technical skills. Still I have the feeling that this only works because of the vast accumulated knowledge pre-AI, and if everybody keeps going in this path, it will end up making everyone not advancing their knowledge at the pace they did before. I feel that this AI immersion is really about selling our soul to the devil for short term gains.

This is the type of problem for which LLM generation is great for.

If you have an oracle, and your problem is largely just a pure function, it's pretty good at generating something that both works and is fast.

I cannot believe they're sticking to their guns on this website design. It's awful.

Great loop spotting!

Recently I was messing around with parquet files in Python and ended up needing to ship the results on Windows, without a Windows machine to test on.

Shipping Python to end users is half mad already, and doing it on Windows is exactly the kind of thing I don't want to spend my life maintaining.

So I figured I'd rewrite it in Go. But that meant embedding a DLL, and how would I test it? I could spin up a VM, sure. But GitHub Actions already has a Windows environment, and there was my loop: let the agent push to the repo, run tests in GHA, rinse and repeat.

In under an hour it had a full rewrite of my Python, passing every test and producing row-for-row copies of my Parquet output. And it does work on the user machine!

Spotting a loop like that is as satisfying as noticing you can walk your chess opponent into a smothered mate. Truly empowering.

The key parts of this is how not vibecoded it is. Feels like a model of how you should do software with AI. Now that we can easily set up property testing, fuzzing, etc. there's almost no reason not to.

That's great but I really wish you guys would do something about the llm integration, I tried using it two days ago to create a cohort of users using a sql query, and I was surprised to see that it said that it could not create cohorts for me and i had to resort to exporting data from a sql insight as a cohort cannot use a sql query. However the worst part was it just writing in the text input slowed down my m4 pro chip to less than 1 fps after 2 prompts and it really left a bad taste in my mouth.

Perhaps the next target for a 100x improvement

This must the most compelling look I’ve seen at how software might work with LLMs doing a ton of heavy lifting.

There’s something kind of amazing here in that having read about property based testing I’m pretty confident I could apply it if I had a good use case.

The thing I would have liked to know is why they don't use an existing fast SQL parser. Was being slightly incompatible with all existing SQL dialects a product requirement?

Dunno about the parser, but you broke scrolling on your fancy website without noticing it also ;-)

You have a grammar file in a formal language, and want to generate a faster parser in another formal language.

What's wrong with the source language that it's better to use a sufficiently smart random code generator for the target language, and then fuzz the hell out of the output of it until it behaves the same as the slow translated code, than to create a sufficiently smart compiler from the source to target languages?

I mean this sounds like if we replaced GCC with a really smart random assembly generator and a fuzzer for the output.

ha, try to keep going. Run it under samply and Gungraun (need AMD64 for this)

Could the agent traces from this be used to improve sqlglot?

tobymao/sqlglot: Python SQL Parser and Transpiler; with tests and support for 30+ dialects: https://github.com/tobymao/sqlglot

Ibis depends upon sqlglot: https://github.com/tobymao/sqlglot/network/dependents

How long did this take?

There is no such thing as a legally-required cookie banner. You can read the GDPR, or ask an LLM to read it for you if you can’t read anymore.

"I didn't rewrite"

[dead]

[flagged]

[dead]

Good read, but "70x" is always misleading.