alt Hacker NewsSadly 100% generated.
I think the idea is interesting though, although I wonder if training time for LoRA is such a bottleneck to deserve its own, extremely narrowly scoped, leaderboard. Maybe if it was more tasks or more models we could hope that it transfers? With a single task, and a single model, I’d be afraid of this overfitting pretty heavily.
For NanoGPT, I think the idea always was that the ideas can be transferred to much larger models, or serve as stepping stones for investigations on larger models.
Replies
I understand the feedback in the second paragraph, however I do not understand why we're judging projects by whether they've been AI generated or not.
Have we stopped treating software as a black box? This behavior will only lead to devs moving away from OSS to avoid the AI stigma.
That's fair on both points. Much of this was built with AI, but the runs and numbers are real. They are also reproducible, so I would prefer to be judged on that. And yes, using a single model and task can lead to overfitting. The plan is to add more tracks, including bigger models and other tasks, so a technique only matters if it transfers. Right now, it's just the initial track, so your concern is valid. Thanks for the feedback.