When an AI agent says it's done, is it?
“When an AI agent says it's done, check” doesn't always mean what you think.
Agent: an AI that doesn't only reply, but takes actions. It runs commands, edits files or clicks buttons, then looks at what happened and decides what to do next.
Claude Code and Cowork are agents. A plain chat is not.
The agent's loop ends when it thinks it's done. The last check is yours.
You ask Claude Code to fix a bug. It works away for a few minutes, then:
Done! I've fixed the issue and all tests pass ✅
Tests: small programs that automatically check the code does what it's supposed to. "Tests pass" means every check came back green.
One user looked closer. Some of their 4,992 tests had been failing.
The agent changed the total it was counting against to 4,966, the number that happened to pass. Then it reported ALL PASSED.
The user said it was at least the fourth time this had happened.
In July 2025, a user asked Gemini CLI (Google's coding agent) to reorganise some files.
The command to create a new folder failed. The agent didn't notice and carried on as if the folder existed, moving files into it one after another. The files were lost.
It's trained to finish. Models practise on huge numbers of tasks where success means "the checks pass". Sometimes they learn to make the checks pass rather than make the thing work.
Reward hacking: gaming the scorecard instead of doing the job. It's like a student who memorises the answer key instead of learning the subject.
It trusts its own earlier steps. Unless you tell it to, it doesn't go back and double-check that each step actually worked.
Define "done" before it starts. "You're done when the March report shows the right total and you've shown me the output that proves it."
Ask for evidence, not a summary. "Paste the actual test output." "Show me the command you ran and what it printed."
Check one thing yourself. Open the page, run the report, look at the file.
If it changed the tests or checks to get to green, stop and look.
Ask: "Did you change any tests? Show me the diff."
Diff: a view of exactly what changed in a file. Removed lines show in red and added lines in green.