alt Hacker NewsMy quick review of the 3.7B model (because I was interested) is that it's not to be trusted for coding.
It failed my basic test I like to ask models and generated incorrect code. When prompted about the bug, it preceded to start hallucinating non-existent APIs. After doing that it got caught in a loop trying to desk check the solution that didn't work.
Replies
dotancohen • today at 7:55 PM
I'd you have some tips for coming up with such tests, I would love to hear them. My Gmail username is the same as my HN username. Thank you!
➕ show 1 reply
xienze • today at 8:28 PM
Not sure a model that small is really supposed to be used for any real coding. At that size you're usually using the model to do simple tasks like summarization.
➕ show 1 reply
7B produced 2 answers, 1 was correct though more expensive and the second was incorrect.
The first attempt with 7B the model got stuck in an infinite loop.