alt Hacker NewsI like these tests
In 6 months an LLM will get it to 80%
people will exclaim negatively “it was trained on the eval [to do this incredibly useful thing]”
meanwhile a problem is solved
I like these tests
In 6 months an LLM will get it to 80%
people will exclaim negatively “it was trained on the eval [to do this incredibly useful thing]”
meanwhile a problem is solved
Well in this case that’s a valid complaint. The tax code changes yearly. If the primary reason an llm can get through a return is because it’s been trained on some example returns then it’ll work for a year. Which I don’t disagree that’s useful if you can retrain yearly but it doesn’t mean the problem has been solved naively.