What do the scores in AI launch announcements really tell me?
How to read the scores in AI announcements.
A new model comes out. There's a chart. The new model's bar is the tallest on every test. Words like state of the art everywhere.
A month later another company does the same, with a taller bar.
They can't all be the best. So what are those charts measuring?
Benchmark: a standard test that lots of models take so their scores can be compared. It's like an exam everyone sits.
There are benchmarks for maths, coding, general knowledge and reasoning. Each one tests a narrow slice. None of them tests your work.
One popular leaderboard, Chatbot Arena, works by asking people to compare two anonymous answers and vote for the better one.
In 2025, researchers found that some big companies tested many private versions there before launch and published only the best one. Before releasing Llama 4, Meta tested 27 private variants.
If you sit the exam 27 times and report only your best score, your score looks great.
The same study found the playing field wasn't level. Google and OpenAI each received around a fifth of all the Arena data. 83 open models combined got about 30%.
Getting more of that data could raise scores by up to 112% (a relative gain) on the Arena itself, without making the model better at anything else.
Benchmarks are published, often online. The internet ends up in training data. So the test questions, and sometimes the answers, can end up in what the model learned.
Contamination: when a model has seen the test questions during training. A high score might mean it memorised the answers rather than learned the skill.
That's why some benchmarks, like LiveBench, keep adding fresh questions.
Even a fair, clean benchmark measures something specific. Your work is messier: your documents, your jargon, your edge cases.
A model that tops the coding chart might still be worse than the runner-up at your kind of coding. Scores can also be good at hard things and bad at easy ones (see "Why AI is great at hard things and bad at easy ones").
Treat launch charts as advertising. Interesting, not decisive.
Keep your own small test. Use three to five real tasks from your work with answers you know are good (see "Is my AI getting worse?"). Run them on a new model before switching.
Look for independent tests. Prefer people who test models the same way every time to the company's own chart.