Related question
How should oversight tighten as AI systems gain more capability?
Short answer
In proportion to what the system can do without asking. A tool that drafts text carries different exposure than one that changes records, reaches customers, or starts a workflow. As authority grows, permissions narrow, logging increases, approval points appear, and you need a way to stop the thing quickly.
Governance written once at the start of AI adoption will not survive contact with what these systems become. It is an operating discipline rather than a document.
The threshold that changes everything is the move from generating information to taking action. As long as a person has to read the output and decide what to do with it, that person is the control. Once the system can write to a database, send a message, or trigger a process on its own, the control has to be built rather than assumed.
At that point the questions get concrete. What can this system reach? What can it change, as opposed to only read? Which actions require a person to approve before they happen? What activity is logged, and does anyone actually look at the log? How would you notice an anomaly before a customer did?
And then the one that gets skipped. One security professional put it plainly: you need a big red button, and without that, do not put anything into an operational setting. A hard stop has been recommended as a standard part of the security framework for these systems, and it is not a dramatic precaution. Manufacturing equipment has emergency procedures. Financial systems have transaction controls. An automated workflow that can act on its own deserves the same thinking.
Autonomy without the ability to interrupt it is risk you did not need to take.