The nicer AI's work looks, the less we check it
Why do I trust polished AI output more than I should?
Polished output lowers your guard.
Two documents
One is a messy bullet list in a chat bubble.
The other is a formatted report with headings, a chart, a tidy table and footnotes.
Which one do you double-check?
Most people check the messy one. The polished one looks finished.
Anthropic measured it
In 2026, Anthropic looked at how people actually work with Claude. When Claude produced a finished-looking piece of work, people were less likely to:
- notice that it was missing context
- check its facts
- question its reasoning
And checking facts was already rare: it happened in fewer than 1 in 10 conversations.
Artifact: Claude's word for a standalone piece of work it makes, like a document, a chart or a small app, shown in its own panel.
Why it works on us
Polish used to be a sign of effort. A formatted report with charts meant someone had spent days on it, and probably checked the numbers along the way.
With AI, polish costs nothing. A chart takes the same few seconds as a sentence.
The signal broke, but our instinct to trust it didn't.
Automation bias: our tendency to trust what a machine produces more than we should, especially when it looks authoritative.
A building front in Auckland, held up by steel while the building behind it was replaced. From the street, it looks like a building.
The AU$440,000 report
In 2025, Deloitte Australia delivered a report to the Australian government, reportedly worth about AU$440,000.
It cited papers that didn't exist and included a made-up quote from a court judgment. Deloitte agreed to refund part of the fee.
It looked like a consulting report, footnotes and all.
Same with research tools
"Deep research" features produce multi-page reports with headings and dozens of citations. They look like a week of work from an analyst.
OpenAI's own documentation for its deep research tool warns that it can make "convincing mistakes".
Try this
Check the substance before the style. Before you admire the formatting, pick the three claims that matter most and check them.
Ask for the plain version first. Get the facts right as a rough list, then ask for the polished version.
Ask yourself before you send it on: "Would I put my name to every number in here?" If you haven't checked them, you're passing the checking on to whoever reads it next.
Sources
- Anthropic, "The AI Fluency Index", Feb 2026: academy.claude.com/tutorials/the-ai-fluency-index
- The Guardian, "Deloitte to pay money back to Albanese government after using AI in $440,000 report", Oct 2025
- OpenAI deep research system card, via simonwillison.net/2025/Feb/25/deep-research-system-card/
- Nielsen Norman Group, "AI chatbots discourage error checking": nngroup.com/articles/ai-chatbots-discourage-error-checking/