alt Hacker NewsYou can commit statistical analysis on frontier models and still use commercial applications as an identifier & comparison.
Criticism is not vitriol - it's possible to make a wider point about being taken aback by the lack of education within AI to the point that there's a critical mass of people using them for calorie counting; but there are many studies on effects of LLMs on psychology etc that are far more effective.
But for me - this is like creating a study that performing algebra & calculus is innacurate on LLMs. That should be common knowledge
Replies
Well, for me the comments that insist we don't need to study X because everybody knows LLMs can't do that is a very good justification to study exactly X.
Not to mention that this is now a standard thought-terminating cliché, where someone points out a use case where LLMs don't work at all well and irrate responses protest that LLMs aren't meant to be used in that way. Says who? If you ask an LLM a question and it answers it- then that's an LLM use case. If you can ask the same question many times and evaluate the results then that's an evaluation that is perfectly fine to make.
It is not uncommon to study things that are considered common knowledge.