alt Hacker NewsThis shouldn't be ignored in the discussion here:
The job performed by the humans was broader than what was requested of the model in this benchmark: humans also had to find the relevant invoices (searching through mailboxes, or requesting them from providers) and reason through any circumstances which cannot be inferred from the bank feed and invoices/receipts on their own. In the benchmark these circumstances are presented to the model as “user notes."
And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices: "cool, Claude logged that it found the May 6th bill from the paper supplier, I am sure it didn't just make something up arbitrary, then compound on the error by agentically iterating over the made-up invoice lurking in its reasoning traces. I checked the first 30 times and there were no problems!"
Replies
Hahaha non-deterministic accounting probably won’t fly well with the IRS
My wife (head of accounting for a small business) has been working on automating large parts of her job using AI.
It's not completely reliable and the human cannot be taken out of the loop, but the number of menial tasks she's been able to automate has been really cool. A lot of processing data that arrives in non-standard formats, generating documents based on that data, etc.
She still has to review everything, but her workload is way down, and when her assistant quit she automated away his whole position.
Hey, the author of the benchmark here.
The benchmark data was prepared in April 2026 (when I was manually doing our VAT return with my co-founder). The invoices were indeed found manually.
Currently we're using a custom "invoice searcher" built on Kimi 2.6 (in our testing several weeks ago it outperformed Opus 4.7; it was just more persistent).
Ultimately, I still verify everything manually after the model is finished fetching invoices for the month -- but it's a great help to have all the invoices already found (usually correctly).
> And I would be pretty nervous about asking any of the frontier LLMs to retrieve invoices:
I watched an accountant YouTuber reviewing a new AI-driven personal finance app the other day (I really need to touch grass), and it started out just fine. He had seeded the account with a bunch of his data and was able to ask questions about which categories had the most spend, etc.
About half a dozen questions in, he asked it to calculate a certain segment of his spend (and being an accountant, he had his numbers memorized), and he immediately got back a calculation that he did not expect. So he asked for an itemized response and it hallucinated line items that never appeared in his account data, which he pointed out to viewers. He followed up with the chatbot with "where did line item X come from?" and the bot acknowledged that it wasn't legit. He immediately noped out after that, and who could blame him?
AI helps automate things that didn't already have rigorous formatting and structures available as input... and that's really all it does (99% of the time).
Doesn't matter how many more nines you add, rigorous formatting is still required. In some cases, it has teeth with compliance standards. Those standards cannot be compromised because there are already a lot of other layers contributing inaccuracy. It all adds up.
In most situations, you could just hire a junior dev (or an intern! remember those?) write some CSV scripts and call it a day. Cheaper and auditable too. Those scripts can't change anyway until standards are revised.
I'm still not seeing the benefit outside of solopreneur efforts and shady businesses wanting to launder blame.
[flagged]
If and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era.
edit: There's already a number of LLM which are intended for outgoing data loss protection to redact or prevent PII from escaping. Is anyone specifically working on a training set and agent that is specialized in reviewing "is this legit to pay", as a sub-task or filtering step in an AP workflow? I suppose it's a GIGO problem, as it would work best only if you have suppliers enrolled in some kind of existing db, with a specific contracted format for invoices, and correlating with project numbers/cost codes.