logoalt Hacker News

user_7832 • today at 5:16 AM • 5 replies • view on HN

I am increasingly more convinced every day that every LLM company is just making their latest model to be the best at (coding) benchmarks, at the cost of pretty much everything else.

I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.

Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).


Replies

vintermann • today at 8:46 AM

I also use LLMs for a difficult non-coding task (historical HTR), and my experience is that Gemini Argon is much better than its competitors on that.

Its main failure modes are silently stopping with the first column of a multi column layout, and ignoring explicit formatting instructions and polluting the transcription with letter variants and ligatures (such as long s, ū, double-dotted y and æ in Latin). It's a bad habit it's picked up from HTR datasets.

Still, those are much better failure modes than hallucinating, which powerful models are still prone to.

SyneRyder • today at 7:00 AM

Are you able to give hints to some of those use cases? Recent Opus and Fable seem to have made big leaps in art and music, and I don't think all of that is the longer context window since Sonnet 3.x. But those are both things that can be created with code and mathematical thinking as well. I definitely wouldn't want to use recent models for writing though (not that I ever wanted them to write for me anyway).

incognition • today at 5:29 AM

Oddly flash 3.8 for me is good. they really pushed carbon to some other extreme

➕ show 1 reply
auyez • today at 6:18 AM

At this point it might make sense to put all their resources into improving coding and math skills, because it might lead them to a state where they keep accelerating their capabilities. Even in the worst scenario where LLM approach would hit some wall, they would have technology to quickly explore all other approaches.

➕ show 1 reply
wellcolormesurp • today at 6:34 AM

They are cracking software and math, not finance, law or medicine or whatever it is you do.

Those two domains have leverage, the others do not. It makes perfect sense. If you and I had the goals they have we would start with code and math as well. The other dominos will fall automatically once you got the engineering automated. It makes no sense to stop focusing on that until you either run out of money or achieve your goal.

➕ show 1 reply