Is my AI model really getting worse, or is it me?
Sometimes AI really does get worse. Here's how to tell.
"Is it just me or has Claude got worse?"
One GitHub thread on this has over 3,000 reactions. The usual theory is that the company quietly swapped in a cheaper model to save money.
People are sometimes wrong about this. Sometimes they're right, though usually not for that reason.
Anthropic later published a write-up of three bugs in the systems that run Claude:
None of these were changes to the model itself, and Anthropic's own tests didn't catch them. Users noticed first.
Postmortem: a write-up published after something goes wrong, explaining what happened and what was fixed.
Another write-up. This time it was three product changes:
The model was the same, but it was set up differently. It felt dumber because, for a while, it was.
What you actually use is:
the model + the app's hidden instructions + settings + the computers running it
Any of those can change while the model stays exactly the same.
System prompt: instructions the company writes, which the model reads before your message. You never see them, and they get updated.
One person built a fixed set of test tasks and ran them on the same model every day from the day it launched.
After a month there was no real change.
That doesn't cancel out the bugs above. It shows that the only way to know is to measure.
Keep three or four test prompts. Take them from your real work, and keep a copy of an answer you know was good. When it feels off, run them in a fresh chat.
Check your settings. Look at which model is selected and the thinking or effort level. Defaults change.