alt Hacker NewsI'd say try the nanogpt speedrun. It's much easier to train, and gives you a better comparison vs optimized systems.
Replies
nickpsecurity • today at 1:27 AM
Labs were also competing to train BERT's for $20 or less. People still use them a lot, too.
https://www.databricks.com/blog/mosaicbert
I'll add they should do a number of small, training runs with different architectures and data mixes. That proves generalization.
The linked paper tested nanoGPT with this new transformer:
https://www.techrxiv.org/users/685780/articles/1375955-topol...