alt Hacker NewsIf and when a large number of companies blindly turn over their accounts payable workflow to some AI agent system, it'll be very interesting to see the "social engineer the LLM" methods that fraud people use to get money sent to them. Basically the same idea as the ancient "send a fax with a bill for an unsolicited delivery of copier toner to 30,000 businesses" but taken into the modern era.
edit: There's already a number of LLM which are intended for outgoing data loss protection to redact or prevent PII from escaping. Is anyone specifically working on a training set and agent that is specialized in reviewing "is this legit to pay", as a sub-task or filtering step in an AP workflow? I suppose it's a GIGO problem, as it would work best only if you have suppliers enrolled in some kind of existing db, with a specific contracted format for invoices, and correlating with project numbers/cost codes.
Replies
i specialize in invoice related analytics and business processes. AI can be useful for data extraction and saves me some typing time. even so, i dont blindl trust it because it sometimes makes very reasonable mistakes because invoices, quotes, and POs are sometimes structured in very informal ways. people misuse the lines, sublines, totals, and other data fields. they are often technically incorrect but when you look at them you know what they mean. i am not sure how to hand that off to something that has plausible deniability to guess even if it doesnt know
sometimes details are just in notes at the bottom and are applied to selectively applied to line items. sometimes the charge doesnt exist anywhere on the bill but there is an understanding (due to a separate agreement) of additional charges to be paid as a result of the invoice.
taxes sometimes are or are not explicitly stated
tariffs sometimes are or are not explicitly stated
when things are not explicitly stated or line item'd, they will usually still appear in the invoice total. so you have item 1 - $500, item 2 $500. total: $1300
At the end of the day invoices are often part of an ongoing communication / conversation between two organizations and they are created with an assumption that a rational and reasonable human who is in the loop with that conversation is going to read it.
You can fix this simply by using normal controls.
That's why we have purchase orders that can only be entered by buyers. Product is received and approved by buyer. Invoice goes to accounting, who can't approve it unless there's a matching purchase order and receiver.
Yes, letting agents do whatever they want leads to disaster. But humans are gullible stochastic token generators as well. And that's why the problem is already solved.
Sounds like Enron was just a bit early. They could have just blamed the AI system instead of becoming a meme with "Enron math"
With AI you can scale the protection against social engineering. Where with humans you have to start from scratch each time and they are more likely to mess up.
I think it would be absolutely insane to hand over a serious-sized company's books to an LLM.
As a small consultancy though, looking forward to my next filing, and having just moved to a new and better-specced jurisdiction, I'm sorely tempted to outsource to Claude.
I've had mixed experience with accountants in the past. No horror stories, but I often feel I'm not getting everything laid out clearly, and that I don't fully understand the process.
I've got plenty of reasons to dislike LLMs in my own work, but when dealing with well-scoped but professionally gatekept things like tax or property transactions, they're an absolute godsend.