AI chatbots give wrong answers to financial queries 'most of the time'

https://archive.is/WD6Vy

Article is just a vague summary of https://www.saturnos.com/report/artificial-authority

Anecdotally, current models seem to be decent at general personal finance principles - certainly better than the majority of personal finance education that people get exposed to unless they seek it out and read a variety of books and sources. But I wouldn't trust them with direct decision making with actual money due to the training lag time on current tax policy, etc.

This will be the same story for every industry again and again. AI is not good in x=Finance because models were not RL trained heavily on x=Finance capabilities yet. This is only because Big Labs have finance benchmarks lower in their priority list. Their first priority was solving programming because that gives the best leverage at this stage. As a side-effect they were able to solve a Millennium Prize problem, since theorem proving was also code.

So finance advisors in the comments section of FT are falling for the classical pitfall. They assume there is something fundamentally wrong with “AI chatbots” that they can’t do finance ever. They mistake the current products in the market for the technology itself. In near future someone will release “Claude x=Finance” and their world will shatter.

Given most financial advisors tend to vend out suboptimal advice and steer customers in favour of products they receive a kickback for, I'm happy to be accepting of an unbiased LLM that's trained on bogleheads.org.

The problem with LLMs in finance is the same as it is in writing, design, and many other disciplines: it isn’t code.

Code objectively does what it‘s intended to do or it doesn’t (and passes certain tests or not) which gives coding agents an indication on whether their solution is adequate.

This is much harder in almost any other discipline.

Pass-fail tests in other disciplines are much less useful. You can tell an AI to not use certain words or not write sentences longer than X, but those rules are insufficient.

At no point can a piece of writing or a design be evaluated to “work” the way code does.

Single shot or with reasoning enabled? My experience is that reasoning dramatically reduces hallucinations and improves output quality. I don't trust models without it.

This seems like undisclosed paid stealth-advertising for Thomson Reuters' new model.

And they hallucinate errors in the millions and struggle with financial data that is in a layout that isn’t in the training data. Ie balance sheet etc.

Been trying to add more AI to my workflow but it just doesn’t work (yet) - not in the same way as vibe coding does

The technical references lookups work though. Looking up regulations etc

I think this depends a lot on what context you give it. I've had solid answers on financial stuff when I give it the right info. But, I wouldn't trust it to guess the missing pieces. I'd be curious how much data or context the model's had in the test.

From the official report:

> Since LLMs can give different answers to the same question, each question was run five times. That means, each LLM was tested 600 times, and in total over 10,000 questions and answers were assessed.

> All models were given the same zero-shot format. They were not given worked examples, previous conversations, hints or an opportunity to correct their answers. This is to make it as similar as possible to a response to a question from consumers.

As for the evaluation itself:

> Responses were checked against this (using an LLM-as-a-judge), and was only given a pass if every element was met; otherwise it was assessed as a fail. This all-pass approach was intentionally strict, so that the score measures whether an answer is complete enough to meet the expert legal standard, rather than how many individual points it gets right.

It's just AI slop and it should be taken with a mountain of salt.

I would prefer to use agent-assisted python scripts that chatbot.

So I downloaded that report which of course doesn't contain the most relevant information (the questions) but it contains some examples of wrong answers.

I fed the first question to Grok (which they claimed they tested as well) and it answered it correctly in detail.

I repeated it with another one - again correct answer. I then selected the question they said Grok specifically answered incorrectly and it again answered it correctly.

I am sticking with my first intuition: people are terrible at testing tools and probably wanted them to answer incorrectly/not fully (the questions are constructed in a way to make it difficult as well). They also have vested interest in the conclusion (they are financial advisory firm) so there is that to consider.

People reading ft will now think chat boxes are bad at answering financial questions while they are pretty good at it. Zero consequences for spreading fake news for Financial Times there but good for financial advisors I guess.

Now, compare this to a recent story that seemed to claim the opposite:

https://news.ycombinator.com/item?id=49139102

I don't have the time to review the underlying research and decide which one is more correct. My personal biases make me want to believe the current one. Your personal biases may be pulling you in the other direction. How do we make the conversation more intelligent than that?

Not a surprise, so do financial advisors: garbage in = garbage out.

Well duh! If it's not using tools to look up the state of the market empirically it's not likely to be accurate financially.

A company selling combined human + AI financial advice finds that AI advice alone is unreliable? Color me surprised.

I like how FT makes me accept cookies from their 46 “technology” (advertising) partners before showing me that the article is behind a paywall anyway.

These models do pretty well in benchmarks and real world so I'm highly suspicious of this article. Further more, in the original report, the examples of bad answers are from Haiku - at least 7 out of 10. Anyone who knows anything about LLMs know that haiku shouldn't be used for anything pretty much.

There's no reproducible set either. I'm not gonna trust this report.

Tool calling solves this

Another summary report here, for those who can't get past the paywall:

https://www.financialreporter.co.uk/ai-models-give-wrong-fin...

Much of the testing is on Haiku and Luna, and criticizing the quality of free AI (!). But they do claim Opus 5 with reasoning still failed 39% of their financial questions.

[dead]

[dead]

In my experience AI chatbots give wrong answers to all queries most of the time.

Getting results requires a harness like in coding and objective metrics, like tests.