Why would an AI agent cheat?
Reward hacking: when an agent games the scorecard.
You ask the agent to fix a failing test. A few minutes later, the test passes.
You look at what changed. It didn't fix the code. It changed the test. Or it marked the test "skip". Or it deleted it.
Claude Code users have filed this as a bug report again and again.
Agents learn by practising on huge numbers of tasks. Each attempt gets a score, and the model is nudged towards whatever scores well.
For coding tasks, the score is often simply: did the tests pass?
Reward: the score used during training to tell the model "that was good, do more of that".
If there's a way to make the tests pass without doing the work, some of the time the model finds it. And it learns that it works.
Reward hacking: getting a high score by gaming how the score is measured, instead of doing what the score was meant to measure.
People do it too. A student who memorises the answer key gets full marks without learning the subject. A call centre measured on short call times starts hanging up on people.
There's an old name for this: Goodhart's law. When a measure becomes a target, it stops being a good measure.
Anthropic described one hack its models picked up during training on coding tasks.
Instead of fixing the code, the model added a line that shut the testing program down early with a "success" signal. The tests never actually ran, but the result said everything passed.
It's the coding equivalent of pulling the fire alarm to get out of an exam, and having it count as a pass.
The worrying part was what happened next.
Once the model learned to cheat on coding tasks, it got worse in other ways it had never been trained on:
Learning "cheating works" seemed to teach a broader lesson about who it was.
Two findings stood out.
Saying what counts as cheating helps. Drew Breunig points to an Anthropic test where one short instruction against hacking cut Claude Opus 4.1's rate from 52% to 18%.
Odd, but true: in Anthropic's research, telling the model that hacking was fine in this particular setting stopped the bad habits from spreading. Cheating that's been openly allowed doesn't teach "I'm the kind of model that cheats".
Model makers also change the training itself. But on your side, the instruction is the part you control.
Say the rule out loud. "Don't change, skip or delete any tests. If a test looks wrong, stop and tell me why."
Look at what changed in the tests. Ask: "Show me the diff for any test files." (See "When an AI agent says it's done, check" for what a diff is.)
Lock the scorecard if it matters. In Claude Code, permission settings or a hook can block edits to test files altogether (see "Why AI ignores your instructions").