Most enterprise AI policies now contain the sentence “a human remains in the loop.” Very few can produce, on request, the record that shows which human, holding what authority, looked at which output, and what happened on the occasions they disagreed with it.
Regulators wrote oversight as a set of capabilities, not a job title
The European Union’s Artificial Intelligence Act spells out what oversight has to mean in practice. Article 14 requires high-risk systems to be designed so they can be “effectively overseen by natural persons during the period in which they are in use,” then lists what those persons must be enabled to do: understand the system’s capacities and limitations and detect anomalies; remain aware of the tendency to over-rely on its output, which the text names as automation bias; correctly interpret the output; decide not to use the system, or to disregard, override, or reverse what it produced; and intervene or stop it through a “stop” button or an equivalent procedure. For remote biometric identification the Act goes further: no decision may rest on a match unless at least two people with the necessary competence, training and authority have separately verified it, with a narrow exemption for law-enforcement, migration, border-control and asylum uses.
The deployer duties in Article 26 turn those design properties into organizational facts. Deployers must “assign human oversight to natural persons who have the necessary competence, training and authority,” monitor the system against the provider’s instructions, retain automatically generated logs for at least six months, and inform affected workers before the system is used on them. Financial institutions already subject to EU financial-services governance rules are treated as meeting the monitoring duty through those rules, which tells you how the legislator sees it: AI oversight is an extension of the controls discipline a regulated firm already runs, not a separate ritual.
The United States arrived at the same place by a different road. The National Institute of Standards and Technology’s (NIST) AI Risk Management Framework 1.0, published in January 2023, asks organizations to “define and differentiate roles and responsibilities for human-AI configurations and oversight of AI systems,” to ensure “processes for human oversight are defined, assessed, and documented,” and to keep mechanisms in place to “supersede, disengage, or deactivate AI systems.” It warns in its opening pages that “humans may assume that AI systems work – and work well – in all settings.”
For banks, the Federal Reserve, the Office of the Comptroller of the Currency (OCC), and the Federal Deposit Insurance Corporation (FDIC) issued SR 26-2 on April 17, 2026, replacing SR 11-7 after fifteen years. Its anchor is still effective challenge: “the critical analysis conducted by objective experts who evaluate model risk,” exercised by people with expertise, independence, and enough standing to effect change. A footnote places generative and agentic AI outside the guidance’s formal scope because those models are “novel and rapidly evolving,” so the burden of showing that a human can and does challenge them rests, for now, on the bank’s own governance.
The research says the person you assign will defer, and training alone will not fix it
“A human reviews every output” is weak evidence because the human-factors literature has documented for decades what happens to reviewers who sit behind a mostly-correct machine. In their 2010 synthesis in Human Factors, Parasuraman and Manzey reviewed the empirical record on complacency and automation bias and reached conclusions every oversight policy should be written against. Complacency “occurs under conditions of multiple-task load” and is “found in both naive and expert participants and cannot be overcome with simple practice.” Automation bias produces omission errors (missing what the system missed) and commission errors (acting on what the system got wrong) when decision aids are imperfect, and it “cannot be prevented by training or instructions.” Both, the authors argue, are overlapping phenomena with attention at the centre: the overseer’s attention drifts to whatever else competes for it, and the automated channel stops being checked.
A systematic review by Goddard, Roudsari and Wyatt in the Journal of the American Medical Informatics Association screened 13,821 papers, retained 74, and defined automation bias plainly as “the tendency to over-rely on automation.” The mediators it identified are the ordinary conditions of enterprise work: workload, task complexity, time constraints, trust, confidence, and the reviewer’s experience with the decision tool. The mitigators are implementation and design choices, not exhortations: training paired with emphasized user accountability, where the advice sits on the screen, confidence levels attached to the output and kept current, and, in our experience the one most often ignored, giving the reviewer information rather than a recommendation to accept. An interface that presents a conclusion beside a confirm button is engineered to produce rubber stamps.
A human in the loop is a claim. A human who could, did, and demonstrably would again intervene is a control.
A human in the loop is a claim. A human who could, did, and demonstrably would again intervene is a control.
What an examiner can test is whether the human could, did, and would again
Put the regulatory text and the research together and the shape of an auditable oversight control is clear. It has three layers of evidence, and an assessor will ask for all three.
- Could. The named overseers appear in the inventory entry for the system with their role, the authority delegated to them, and dated training on the system’s limitations and failure modes, as Article 26 and the NIST framework both expect. The stop or override mechanism exists, and a dated test shows it was exercised and brought the system to a safe state. The interface has been reviewed against the design mitigators: output presented as information with current confidence signals, not as a decision awaiting a click.
- Did. Retained logs record each output, the reviewer who handled it, the time spent, and the disposition: accepted, modified, rejected, or escalated. The disagreement rate is tracked and reported, because a queue in which thousands of outputs pass with no overrides is the signature of complacency the research predicts, not proof of a well-performing model. A sample of the overrides is quality-reviewed, so the firm can show interventions were correct and not merely present.
- Would again. Workload and queue metrics demonstrate reviewers have time to review, since the literature ties automation bias directly to multiple-task load. Authority is re-attested and training refreshed when the model or its use changes. And the oversight design itself receives independent challenge from someone with the standing SR 26-2 describes.
Notice what is absent from that list: a policy sentence. The policy is necessary and nearly worthless on its own. Everything an examiner can test lives in the inventory record, the logs, the metrics, and the dated artifacts around them. In our experience the gap is rarely one of intent. The evidence was simply never designed to exist: reviewer identity is not captured, overrides are possible but not logged as overrides, training is a one-time onboarding slide, and no one owns the disagreement rate.
The remediation is tractable and has owners. The system owner named in the inventory closes the logging gaps and publishes the disagreement rate within a quarter; the second line, the risk function that validates controls, validates the stop mechanism and reviewer workload in the same quarter; internal audit tests all three layers on a sample in the following one.
“Human in the loop” was always a promise about behavior, and the research is unambiguous that the behavior degrades under exactly the conditions enterprises create: high volume, mixed tasks, mostly-correct systems. Regulators and standards bodies on both sides of the Atlantic have responded by writing oversight as a list of capabilities and records. Build the records first. A reviewer who cannot be shown to have intervened will, in the examination that matters, be treated as if they were never there.