Test AI on five real examples

A test you can build in an afternoon, using your own work.

Benchmarks aren't your job

Every new model launches with a chart of scores on tests with names like MMLU or GPQA.

Benchmark: a standard set of test questions used to compare AI models, a bit like a school exam that every model sits.

They're useful for comparing models in general. They tell you nothing about whether it can write your weekly report or tidy your data.

For that, you need your own test. It can be small.

What you need

• Five real examples from your work. Use real inputs, not made-up ones.
• At least two awkward ones: the messy file, the unusual request, the case that went wrong last time.
• For each one, a short list of yes/no checks that describe a good answer.

A spreadsheet is plenty.

An example: meeting summaries

12 of 15 checks passed. And now you know where it fails.

Made-up example.

Use yes or no, not 1 to 5

It's tempting to score answers out of 5. Don't.

What's the difference between a 3 and a 4? You won't score it the same way on Tuesday as you did on Monday, and nor will a colleague.

"Did it name an owner for every action? Yes or no" is a question anyone can answer the same way.

Read the answers yourself

Don't just tick the boxes. Read every output, at least the first few times.

You'll spot problems you didn't think to check for, like an odd phrase it repeats, or a habit of inventing a date. Add a check for each one.

Evals experts call this looking at your data. It's the step most people skip.

Your idea of "good" will change

As you read real outputs, you'll realise you care about things you never wrote down. Maybe it's tone, or whether it mentions risks.

That's normal. Researchers call it criteria drift. Update your checks and run the test again.

When to run it

This is the grown-up version of the test prompts from "Is my AI getting worse?".

Try this

This week, pick one task you use AI for regularly. Save five real inputs, including two awkward ones.

Write three yes/no checks for what good looks like.

Run it and record passes and fails. Keep the sheet, and run it again whenever something changes.

Sources

• Hamel Husain and Shreya Shankar, "Evals FAQ": hamel.dev/blog/posts/evals-faq/
• Hamel Husain, "Creating a LLM-as-a-judge that drives business results": hamel.dev/blog/posts/llm-judge/
• Shankar et al., "Who validates the validators?" (2024), on criteria drift