alt Hacker News> nearly as accurate as a human book keeper
Anything to avoid using the metric system.
Though seriously, what is this metric? Why would I care if an LLM is accurate as a human bookkeeper? Humans aren't exactly known for perfect recall.
Replies
infecto • yesterday at 7:16 PM
It’s a service that provides an AI bookkeeper so it’s a pretty relevant metric.
wat10000 • yesterday at 6:59 PM
You'd care if you had a human bookkeeper and you were considering replacing them with this company's AI bookkeeper.
➕ show 1 reply
Hey, author of the blog post here.
I was one of the human book-keepers for this benchmark (the preparer; my co-founder verified the VAT submission once ready), and given that at the time of doing this I knew I was eventually going to use this data for evaluating the models, I was super careful. So I guess this is a "good book-keeper". In the previous company our book-keepers made lots of mistakes; some serious enough that we had to restate our company's accounts.