Training a 3.8B LLM to 0.384 CORE for $998

More and more such experiments. I felt sad for a couple months when I realized that writing code will not be the same since. Now I am on the other side.

LLMs are interesting in their own ways but as an engineer, this is a way to unlock a new way of building software.

I recently build a Claude-assisted Excel/CSV parser for a US based property management system (tax compliance). Uses Haiku and has a lot of deterministic code to extract column/row combinations to check known formats and finally handing out the headers to Haiku to give us a translation plan to our support columns.

These would eventually become part of the software, in a tiny LLM. The gap between training (such tiny LLMs) and inference will shrink. We can consult Claude for edge cases, create sample dataset and train a the tiny LLM on demand so we go to Claude less.

The tooling that a project needs is really important. Something I have been feeling as well. Not just in LLM building projects, but regular software projects that are LLM generated.

This is really neat! I'd be tempted to try this again targetting ~1B params and the entire cookbook of "current" small model ideas: gated delta nets (or is ~2K context too short to benefit?), per layer or n-gram embeddings, gated residuals etc.

I don’t blame you for using an LLM to write an article about an LLM that you built. I’ve been reading so much LLM output that now I see it everywhere. I wonder if humans will start writing more like LLMs?

Very cool. I wonder what's the minimum amount of money you'd need to get it to score well on AIME

[flagged]

[dead]

[flagged]

[flagged]