AI advice erodes critical thinking: study shows accuracy drops as confidence soars

A new study delivers a sharp warning for anyone building AI products: giving users access to AI advice can make them three times less accurate while simultaneously making them twice as confident. The AI advice impact on critical thinking goes beyond simple overreliance. It suppresses the cognitive habit of recognizing what you do not know, a phenomenon researchers have termed 'cognitive surrender.'

What happened

Researchers from the University of Milano-Bicocca, École Normale Supérieure, and Sapienza University of Rome designed an experiment to isolate the effect of AI availability on human judgment. They used questions that AI models typically fail at: visual-detail questions about films, such as the color of a team's uniform in Bend It Like Beckham. The team deliberately chose the Step 3.5 Flash model, which is usually wrong on these questions, to ensure any decline in performance could not be explained as sensible delegation to a reliable tool.

The results were stark. When AI was available, participants' willingness to say 'I don't know' collapsed from 44% to 3%. Accuracy dropped from 27% to 9%. Confidence, meanwhile, rose from 30% to 76%. Some participants who had answered correctly on their own asked the AI and became wrong.

Monetary incentives helped only marginally. Willingness to admit ignorance inched up from 3% to 8%, and accuracy from 9% to 16%. Both remained well below the no-AI baselines of 44% and 27%.

Why AI builders should care

These findings are not an academic curiosity. They speak directly to how users interact with AI-powered search, coding assistants, agent copilots, and educational tools. The study adds a precise data point to the concept of cognitive surrender, earlier coined by Wharton researchers to describe people accepting incorrect AI answers 80% of the time while feeling more confident than those working without AI.

For product teams, the implication is uncomfortable: the mere availability of AI advice can erode users' metacognitive skills. Users stop self-checking. They stop recognizing the limits of their own knowledge. Valerio Capraro, the study's lead author, highlighted particular concern for children, who are growing up with these systems before developing critical thinking skills. Common Sense Media this week called Google's AI-generated search summaries an 'unacceptable risk' for students.

Practical implications

Design choices matter. AI interfaces that present answers with full confidence and no uncertainty indicators actively suppress user skepticism. Product teams should consider:

• Uncertainty signals: Show confidence scores, alternative answers, or explicit 'I don't know' responses when appropriate.
• Verification prompts: Build in steps that encourage users to verify or explain their answer before accepting an AI suggestion.
• Training or onboarding: Educate users on when to trust and when to question AI outputs, especially in high-stakes domains like education or decision support.

The data shows that even financial incentives barely move the needle. The solution likely lies in interface design and user experience patterns that preserve user agency and awareness.

Caveats

The study uses a specific experimental setup. It focuses on visual-detail questions and a deliberately unreliable model (Step 3.5 Flash), so results may not generalize to all AI tools or domains. The long-term effects of repeated AI use on education and critical thinking require further study. However, the direction of the effect is consistent with earlier work on cognitive surrender and should give AI builders concrete reason to audit their products for overreliance patterns.