Core question
Who is responsible when AI produces a wrong answer?
Short answer
A person, always. AI can draft, analyze, and recommend, but accountability cannot be handed to a model. Define in advance who reviews consequential output, who approves action, and who owns the result. If nobody can answer that question, the problem is not the model. Your governance is incomplete.
"The AI made a mistake" is not an accountability structure. It is a description of an outcome nobody had assigned.
The level of review should scale with what happens if the output is wrong. An internal draft that a person is going to rewrite anyway needs very little supervision. A commitment to a customer, a legal communication, a personnel decision, a financial recommendation, or anything touching safety needs a named human between the output and the action. Most organizations have never drawn that line, which means it gets drawn under pressure by whoever is closest when something goes wrong.
Three questions settle most of it. Who owns this workflow? Where in it does a person have to approve before anything leaves the building? What is supposed to happen when the output looks questionable, and does that person have the authority to stop it?
That last one matters more than it sounds. Plenty of organizations have someone nominally reviewing output who has no practical ability to halt the process, which is not oversight. It is a signature.
If your guardrails are not enforced, they are not guardrails. They are guidelines. And if no one owns the output, the policy is broken regardless of how well it reads.