# The Approve Button Is Not Oversight

I was half-listening to a talk from an Amazon engineer a while back. The kind of internal-flavoured session where someone who ships agentic systems at scale says something more honest than the marketing allows. One line stuck. I'm paraphrasing, because I didn't write it down: the more your reviewers approve, the less they're actually reviewing. After enough green checkmarks, "approve" stops being a decision. It becomes a keystroke.

I've thought about that sentence more than I expected to. Because it quietly dismantles the thing everyone reaches for when someone asks "but is it safe to let the agent do that?" The answer is always the same. **We'll keep a human in the loop**. Said with confidence, like it settles the matter.

It doesn't settle anything. It just moves the failure somewhere harder to see.

## **What "human in the loop" is supposed to do**

The premise is reasonable. You have a system that's right most of the time but not all of the time, and the failures are expensive. So you put a person at the decision point. The model proposes, the human disposes. The agent drafts the refund, moves the money, merges the PR, changes the infra. A human signs off before it lands.

On a whiteboard this is airtight. The machine handles volume. The human handles judgment. The rare catastrophic case gets caught by the one component in the system capable of understanding context. Two failure modes, two very different mechanisms. Defence in depth.

The whole design rests on one assumption: that the human in the loop is actually looking. Not present. Looking. And that assumption is exactly where it falls apart, because it assumes the one behaviour the rest of the system is busy destroying.

## **The reliability paradox**

Here's the part nobody wants on the slide. The better your agent gets, the worse your reviewer gets. Those aren't two trends. They're the same curve read from opposite ends.

This isn't a hot take. It's decades old. Raja Parasuraman and Dietrich Manzey laid it out in their 2010 review, *Complacency and Bias in Human Use of Automation*. Automation complacency shows up reliably when a person is juggling multiple tasks and one of them is "watch the reliable machine." Two findings sting. It appears in experts, not just novices. And it doesn't wash out with practice. You cannot train your way to permanent vigilance against a system that's usually right. The reliability is the thing sedating you.

Aviation has known this forever. It's why they don't just tell pilots to "stay alert." Alertness isn't a policy you can declare. It decays. A human watching a system that's correct 999 times out of 1000 will, by the thousandth event, have been conditioned by 999 confirmations that watching is a waste of time. And they'd be statistically correct to feel that way. That's the trap. The rational response to a highly reliable system is to stop checking it, which is precisely when it needs you.

There's a name for what your reviewer becomes at that point: automation bias. It produces two failures, not one. Omission, where the system stays quiet when it should have flagged something and so does the human. Commission, where the system gives bad advice and the human follows it over their own correct judgment. That second one should bother you more than it does.

## **The numbers are worse than the vibe**

I don't want to argue this on intuition alone, because intuition is exactly what automation bias corrupts.

A 2012 systematic review of clinical decision-support systems found that when the system gave wrong advice, it raised the odds of the clinician making an incorrect decision by 26 percent. Risk ratio of 1.26. Read that again. These are trained clinicians reversing their own correct answers because a machine disagreed. Across the studies, somewhere between 6 and 11 percent of correct human decisions got flipped to wrong ones after the system chimed in. That's not the system failing. That's the oversight layer failing, in the exact direction it was installed to prevent.

Software has its own version, and it's newer. Stanford researchers found that developers using an AI coding assistant wrote less secure code, and were more confident it was secure. That combination is the real problem. When the output looks clean and structured, you review it less. The NYU study on Copilot's early output put roughly 40 percent of suggestions in security-relevant contexts as containing vulnerabilities. Not because the tool was bad. Because polished-looking wrongness is the hardest kind to catch, and a tool that's right most of the time makes every output look like the times it was right.

Anyone who's approved a stack of agent-generated PRs knows the feeling in their gut. The fifth one gets less scrutiny than the first. The twentieth gets a glance and a merge. You tell yourself you reviewed it. You didn't. You pattern-matched it against the last nineteen that were fine.

## **The part that should actually worry you**

So the loop degrades into a rubber stamp. Annoying, fixable, you'd think. But there's a second thing happening, and it's structural, not cognitive.

Madeleine Clare Elish has a phrase for it: **the moral crumple zone.** In a car, the crumple zone absorbs the crash to protect the human. In an automated system, the human absorbs the blame to protect the system. When the agent does something catastrophic and a person clicked approve somewhere upstream, the accountability doesn't attach to the model, the training data, the eval gap, or the org that shipped it under-tested. It attaches to the nearest human who touched it. They become, in Elish's framing, a liability sponge.

This is the quiet reason "human in the loop" is so popular in decks and compliance docs. It isn't always there to catch errors. Sometimes it's there to have someone to blame when the errors aren't caught. That's a use, I suppose. It just isn't oversight. It's a name on a form, positioned to take the hit. If your loop can't slow the system down, can't say no without a fight, and gets blamed when the thing it was never empowered to stop goes wrong, that's not a safeguard. That's a designated fall guy with a keyboard.

And regulators are about to make this everyone's problem. The EU AI Act mandates human oversight for high-risk systems under Article 14, with enforcement for high-risk obligations landing in August 2026. A lot of teams are going to satisfy that clause by bolting an approve button onto an agent and calling it oversight. The letter of the law, met. The spirit, a human with the authority, time, expertise, and independence to actually push back, nowhere in sight.

## **So what actually works**

I won't pretend I've fully solved this in my own systems. I haven't. But the failed approaches share a shape, and so do the ones that hold up.

What doesn't work is making the human review everything. That's the fast road to the rubber stamp. Volume is the enemy of vigilance. If your reviewer sees a hundred approvals a day, ninety-nine of them fine, you have not built oversight. You've built a clicking exercise with a human-shaped bottleneck, and you've trained that human to click.

What seems to actually help, roughly in order of how much it moves the needle:

**Make the human sample, not screen.** A reviewer who checks a well-chosen 5 percent carefully catches more than one who "checks" 100 percent at a glance. Vigilance is a fixed budget. Spend it deep, not wide. Randomised deep audits also break the pattern-matching, because you can't coast if you don't know which one is graded.

**Route by uncertainty and blast radius, not by default.** The agent should escalate the cases where it's unsure or where the downside is large, and quietly handle the boring middle. A human forced to look at obvious approvals stops looking by the time the dangerous one arrives. Reserve human attention for where it's actually differential.

**Make "no" cheap and real.** If pausing the system costs the reviewer a meeting with their skip-level, they will not pause it. Oversight that's socially expensive to exercise isn't oversight. This is an org-design problem wearing a UI costume, and no amount of front-end polish fixes it.

**Give the human something the machine didn't generate.** Independent context, a second signal, the raw inputs. Not just the model's confident summary of its own work. Asking someone to check an agent using only what the agent chose to show them is theatre. They're grading the essay with the answer key the student wrote.

The pattern underneath all of it: aviation didn't fix cockpit complacency by telling pilots to try harder. It built Crew Resource Management. Structure, training, simulated failures, and an explicit culture where the junior officer is expected to challenge the captain. They turned oversight from a personality trait into an operational discipline. That's the bar. Anything less, and "keep a human in the loop" is a sentence you say to feel better, not a control that does anything.

## **The line I keep coming back to**

The loop was never the safeguard. The human's attention was the safeguard. And attention is the one resource a reliable system is guaranteed to consume. You built something good enough to make watching it feel pointless, then stationed a person to watch it. Of course they stopped watching. You designed the incentive that way.

So before you write "human in the loop" in your next design doc, ask the only question that matters. What would it cost that human, right now, today, to say no? If the answer is "nothing, they'd click the other button and everyone would thank them," good. You might have oversight. If the answer is anything else, you've got a rubber stamp, and you've already decided whose name goes on the incident report.

Approve, approve, approve. At some point that isn't judgment. It's just latency with a person attached.