Related question
How much testing should an AI build actually get?
Short answer
More than you think, and more than most builders are doing. In 22 years running a software company we spent more time testing than programming, because a change that breaks something else in the larger system costs more than the feature was worth.
Quality is job one, and the definition of quality has not changed just because the build method did.
One failure pattern deserves naming because it is so common right now: going straight to production, so that every user experiences whatever problems the update introduced. That was never acceptable in the software business and it should not be acceptable now, particularly when the system is making decisions rather than displaying information.
For an executive buying a build, the practical requirements are straightforward. Ask what the testing plan is and who executes it. Insist that testing use real data rather than AI-generated sample data, because sample data does not contain the variability that breaks things. Require a staging environment before production. And define an acceptance standard beyond "it returns a reasonable answer."
The bar should be that it delivers real value to the people who have to use it.