The test set is the product: shipping AI features without guessing

Why demos mislead

You try a handful of inputs, the answers look good, and you ship. Then a customer finds the input you never tried.

With ordinary software, that's a bug you fix. With an AI feature, fixing it for one input can quietly break three others, and you won't know unless you check.

An evaluation set, usually just called an eval, is how you check. It's a list of realistic inputs, each with a description of what a good answer looks like.

The smallest useful eval

• Real inputs. Pull them from support tickets, emails and past requests. Invented examples are always too clean.
• Fifty to start. Include the easy ones, the awkward ones, and the ones where the right answer is "I don't know" or "ask a person".
• A clear pass or fail. Where you can, check by rule: right category, right number, required field present. Where you can't, use a person with a short rubric. A model can help score, but spot-check it.
• Version everything. When you change the prompt, the model or the retrieval, run the set again and compare.

What it buys you

You can switch models without fear. You can change a prompt and know whether it helped. You can tell the business how often the feature is right, instead of how it feels. And when something goes wrong in production, you add that case to the set so it can't fail the same way twice.

What to do this month

1. Collect 50 real inputs. Give it one afternoon. It's the best use of that week.
2. Write the expected outcome next to each. Include a few "should refuse" cases.
3. Run it before every release. Make it part of the deploy, not a ritual.
4. Track the pass rate over time. A number on a chart changes the conversation with leadership.
5. Feed failures back in. Every miss in production becomes a new test.