How We Evaluate GenAI Features Before They Reach Production

Every generative-AI demo looks finished. A model drafts a document, summarises a case file, or answers a question in natural language, and the room nods — it looks like software that could ship tomorrow. The gap between that moment and a feature we’re comfortable putting in front of a paying client’s production system is where most of our applied-AI work actually happens, and it rarely has anything to do with picking a bigger model.

This is the evaluation discipline we hold every GenAI feature to before it leaves a pilot and becomes part of a client’s day-to-day system — whether that’s a document-review assistant inside an immigration case-management tool or a complaint-triage model sitting in front of a support queue.

Start with the decision, not the model

The first question we ask is never “which model should we use.” It’s “what specific decision does this feature change, and who is accountable for that decision today.” A feature that drafts a first-pass summary for a human reviewer to edit is a very different engineering problem from a feature that auto-approves something. Naming the decision up front tells us how much error the system can tolerate, what the fallback path looks like when it’s wrong, and who needs visibility into its confidence — before a single prompt is written.

If we can’t say precisely what decision a model is influencing, we’re not ready to evaluate it — we’re just admiring output.

Build the eval before the feature

Once the decision is scoped, we build the evaluation set before we finish the feature. Concretely, that means:

• A labelled test set pulled from real, anonymised examples the client already has — not synthetic cases that happen to make the model look good.
• A rubric a human can apply consistently, scoring outputs on correctness, completeness and tone, not just “looks right.”
• A regression suite that reruns automatically whenever we touch a prompt, a retrieval step, or swap a model version — so an improvement in one case can’t quietly become a regression in another.
• A disagreement log between the model’s output and what a human reviewer would have produced, which becomes the backlog for the next iteration.

This is slower than shipping the first version that “looks good in the demo.” It’s also the only way we’ve found to catch the failure modes that only show up on the client’s actual data — inconsistent formatting in source documents, edge cases their domain experts handle from memory, language mixing across English and Bangla source text.

Where a human still has to be in the loop

For anything touching a compliance-sensitive or high-stakes decision — an immigration document check, a financial reconciliation, a health-adjacent workflow — the model’s job in our systems is to prepare and prioritise, not to conclude. We design a confidence threshold into the workflow itself: high-confidence, low-risk outputs move forward with a visible audit trail; anything below the threshold, or above a defined risk tier, routes to a human reviewer with the model’s reasoning attached, not just its answer.

What we watch after launch

Shipping isn’t the end of the evaluation — it’s where the real data starts arriving. We keep a small set of production metrics on every GenAI feature we operate: the human override rate (how often a reviewer changes the model’s suggestion), output drift against the original eval set as source data shifts over time, latency and cost per call against the volume the client is actually running, and a lightweight monthly sample review by someone who wasn’t on the build team, specifically to catch the kind of complacency that sets in once a feature has been “working fine” for a while.

None of this is exotic. It’s closer to how we’d treat any other production system with real consequences — the difference with GenAI is that the failure modes are quieter and more confident-sounding than a stack trace, which is exactly why the discipline around them has to be louder.