AI advice made people three times less accurate but twice as confident, researchers found

Researchers from three French and Italian universities found that access to AI advice collapsed people’s willingness to say “I don’t know” from 44% to 3%. Accuracy dropped from 27% to 9%. Confidence, meanwhile, rose from 30% to 76%. “People became much worse, the accuracy was only one third, but they were twice as confident,” said Valerio Capraro, associate professor at the University of Milano-Bicocca.

The study, authored by Capraro with Chiara Marcoccia of École Normale Supérieure and Walter Quattrociocchi of Sapienza University of Rome, deliberately used questions where AI models typically fail: visual details from films, such as the colour of a team’s uniform in Bend It Like Beckham. The researchers used Step 3.5 Flash, a model that was usually wrong on these questions, precisely so any reduction in judgment could not be explained as sensible delegation to a reliable tool. Some participants who would have answered correctly on their own asked the AI and became wrong.

Monetary incentives helped, but not much. Willingness to admit ignorance rose from 3% to 8% and accuracy from 9% to 16%, both still well below the no-AI baselines of 44% and 27%. Wharton researchers coined the term “cognitive surrender” earlier this year to describe the same phenomenon: people accepting incorrect AI answers 80% of the time while reporting higher confidence than those working without AI. The new study adds a sharper data point. It is not just that people trust wrong AI answers. It is that the mere availability of AI suppresses the cognitive habit of recognising what you do not know.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol’ founder Boris, and some questionable AI art. It’s free, every week, in your inbox. Sign up now!

“For humans, the capacity to say ‘I don’t know’ is very important because it represents the recognition of the limits of our own knowledge,” Capraro said. He is particularly concerned about children, who are growing up with these systems before they have developed critical thinking skills. Google’s AI search overhaul replaced links with confident AI-generated summaries, and Common Sense Media this week called that design an “unacceptable risk” for students. The pattern is consistent: AI products are designed to answer, never to say “I don’t know.” The humans using them are learning to do the same.