Write the eval before the prompt
Most AI features ship with no way to tell whether the next change improved them. That is a design gap, not a testing one. In a regulated product the eval is also the audit trail.
A model demo takes an afternoon. A model feature that is still accurate, affordable and explainable eighteen months later is an engineering problem, and the difference between the two is almost entirely measurement.
The pattern to avoid is familiar: someone tunes a prompt, the output looks better on the three examples they tried, and it ships. Six weeks later nobody can say whether a change helped, because there was never a number to move.
A scored dataset, before anything else
It does not need to be large, but it does need to be honest, and it needs to include the cases you are afraid of.
Once it exists, every subsequent decision becomes an experiment with a result instead of an argument between preferences: model choice, retrieval strategy, prompt structure, whether to fine tune at all.
Ground it in something you own
In a regulated setting, a model that cannot show its source is a liability regardless of how good the answer is. Retrieval over a corpus you control turns an assertion into a citation, and turns an audit conversation from a discussion about model behaviour into one about documents.
It also constrains the failure mode. A grounded system that finds nothing should say so; that is a far better outcome than a fluent answer with no provenance.
Cost is a quality attribute
Inference cost is usually treated as an operational detail and discovered at scale. It belongs next to accuracy in the same table, measured per outcome rather than per token, because the two trade against each other constantly.
Model routing, caching and the decision to fine tune are all economic choices. Without the eval, they are guesses about quality as well.
The gate
The point of all this is a deployment gate: a change ships when the eval says it is at least as good, and a regression blocks it. That single rule converts an AI feature from something that drifts into something you can maintain.
Leave with the approach, the risks and what it would take
Ninety minutes with the engineers who would do the work, on one problem you name: a processor replacement, an onboarding flow, a ledger reconciliation.