Most AI features fail the same way. Not in production — in the founder's head, weeks earlier, the moment someone decides the thing works because a demo said so.
A demo proves an AI feature can succeed once. It says nothing about how often it fails, or how badly. And "how often, how badly" is the entire question. So before a team writes the feature, I make them write the eval that will judge it.
The eval comes first. The feature is what you build to pass it.
The demo lies, the eval doesn't
A demo is a single sample, hand-picked, run in good conditions. Of course it works. You chose the input that made it work. That's not evidence — it's a magic trick you performed on yourself.
An eval is a set of inputs you did not hand-pick, scored against what a correct answer looks like. Twenty of them, fifty, a hundred. Real ones, messy ones, the ugly edge that shows up on a Tuesday. The number that comes out the other end is the first honest thing anyone has said about the idea.
I've killed features at this stage that everybody loved in the demo. I've also greenlit unglamorous ones that scored 94% on inputs nobody wanted to look at. Both were the right call, and neither was possible without the eval.
What goes in the set
An eval set is not a benchmark you download. It's a small, deliberate collection of your problem, and building it is where the real product thinking happens.
I want four kinds of cases in there.
The obvious wins. Clean inputs, unambiguous answers. If the feature can't nail these, stop now. This is the floor, and it should be boring.
The realistic middle. The inputs your actual users will actually send — abbreviated, half-formatted, context missing. This is the band the feature lives or dies in, and it should be the biggest slice of the set.
The known-hard. The cases you already suspect will break it. Ambiguous requests, two valid answers, information that isn't in the input at all. You're not trying to pass these on day one. You're measuring how the feature fails when it fails.
The must-not-fail. The handful where a wrong answer is expensive — a mis-stated number, a confident answer to a question with no answer, anything that costs trust or money. These don't get graded on a curve. One failure here can outweigh a hundred wins everywhere else.
Assembling those four buckets forces the argument you were going to have eventually — what does "correct" even mean here? — into week one, when it's cheap, instead of into a production incident, when it isn't.
Score it before you love it
Here's the discipline that makes this work: write the scoring rule before you see the results.
It's easy to look at an output, decide it's "basically fine," and move on. That's how standards erode one lenient judgment at a time. So the rule gets fixed in advance. Exact match where you can. A rubric where you can't. A more capable model as an automated grader for the fuzzy stuff, spot-checked by a human so you trust the grader too.
Then you run the set and read the number without flinching. Sixty percent on the realistic middle is not "we're most of the way there." It's "four in ten users get a wrong answer," and you decide with that sentence in front of you, not the demo you remember fondly.
The decision the number buys you
The eval doesn't just tell you whether to build. It tells you how.
A feature that scores well everywhere except the known-hard bucket doesn't need a better model. It needs a design that detects the hard case and steps back — asks a clarifying question, hands off to a human, says "I'm not sure." A feature that fails the obvious wins has a foundation problem no amount of UX will paper over. Same number, different diagnosis, different plan. The eval is what separates the two.
And it becomes the one artifact that survives the whole project. The set you built to decide if is the set you rerun on every prompt change, every model swap, every refactor, for as long as the feature lives. You never wonder again whether a change made things quietly worse. You run the eval and you know.
The rule I hold teams to
No AI feature gets a roadmap slot until someone has built its eval set and shown me the first score. Not a plan to build one. The set, and the number.
It takes a day, maybe two. It has killed more of my bad ideas than any review meeting ever has, and it has saved the good ones from dying next to them. The feature is the easy part. The eval is the part that tells you the truth — so build it first.
FAQ
How big does an eval set need to be to be useful? Smaller than people think. Twenty to fifty well-chosen cases across the four buckets — obvious wins, realistic middle, known-hard, must-not-fail — will tell you more than a thousand random ones. You're after coverage of the ways it can fail, not statistical significance. You can grow the set once the feature is real.
Isn't building the eval first just slower? It front-loads a day or two, then saves weeks. The eval kills bad ideas before you build them and turns "does this still work?" into a command you run instead of a debate you have. On any feature that ships and keeps changing, it's the fastest path, not the slowest.
Who should write the eval — engineering or product? Both, together, which is the point. Deciding what counts as a correct answer is a product decision; measuring it reliably is an engineering one. Building the set forces that conversation early, when it's a whiteboard, instead of late, when it's an outage.
What if I can't define what "correct" looks like? Then you're not ready to build the feature — and that's the most valuable thing the eval could have told you. If you can't score it, your users can't tell when it's wrong either. Sharpen the definition until it's measurable, or narrow the feature until it is.
I'm Ravi Jadav, Chief Product Officer and Co-Founder at Sunbots Innovations and Co-Founder at Xwits Developers. Get in touch.