AI Chatbots Fail Most Money Questions in 10,000-Answer Test

Saturn, a UK fintech firm, reports that mainstream AI models got 57% of personal finance answers wrong, with failure rates hitting 88% on harder, multi-step questions.

The findings land as consumers increasingly turn to chatbots for financial guidance.

Harder Questions Broke Almost Every Model

The study put 121 questions to 18 free and paid models from providers such as ChatGPT, Gemini, Claude, and Copilot, with each question repeated up to five times for over 10,000 answers.

The test judged an answer a failure if it had a factual mistake, missed something important, left out a required warning, or was otherwise incomplete.

Free models did worst, failing 63% of answers versus 49% for paid ones, and hit a 93% failure rate on the toughest questions.

Claude Opus 5 was the best performer, yet it still failed 39% of the time, with errors including miscalculations, missed tax changes, and made-up rules.

One pension tax answer could have left a saver with a £17,500 bill from HM Revenue and Customs.

“Millions of people are trusting the AI models for money advice, but they are getting wrong answers that can lose them money,” Amal Jolly, Saturn chief executive, said.

Trust in AI Chatbots Keeps Climbing Anyway

Usage has grown across regions and age groups, with a global EY survey of 18,000 people finding 49% had turned to AI for savings and investment advice.

In August, Britain’s financial regulator said four in five less experienced investors have used AI for help with investing, and 56% of those surveyed trust these tools more than television or radio, which came in at 47%.

The same research also found 44% wrongly think AI-generated financial information is regulated.

A PensionBee survey of 1,000 US adults found nearly six in ten would act on money guidance from a chatbot without double-checking it, and about one in four said a chatbot had already given them wrong financial information.

Jolly said AI financial advice remains unregulated, leaving consumers without the compensation rights they would get from a human adviser, and he called on the FCA to act quickly.