The Human in the Loop Who Was Never Really in It

The human in the loop who was never really in it.

Twenty-six thousand families accused of fraud, and a human being signed off on almost every one.

Between roughly 2013 and 2019, the Dutch tax authority ran a fraud-detection system against applications for childcare benefit. The system scored families for risk, and one factor that fed into a family’s score, at points in the programme, was whether they held a second nationality alongside Dutch citizenship. Families flagged as high risk were investigated, and under the rules in force at the time, even a small paperwork error could mean the entire benefit had to be repaid, in full, with no proportionality built in.

By the time a 2020 parliamentary inquiry reported, roughly 26,000 families had been wrongly accused. Many were forced into debt, some into bankruptcy or eviction; caseworkers later described being pressured to close cases quickly rather than review them properly. The Dutch cabinet resigned over it in January 2021, a rare and direct case of an algorithmic governance failure reaching all the way to a government’s fall.

The oversight that existed on paper

This is not a story about a machine acting alone. A caseworker was formally required to review each flagged case before a decision went out. Human oversight, in the literal, org-chart sense, was built into the process from the start.

What later reporting and the inquiry itself found was different in practice: caseload, time pressure, and a system that presented its risk score as the starting assumption rather than one input among several -> meant that review, for many cases, meant confirming what the algorithm had already decided.

The box was ticked.

The judgement behind it, for a meaningful share of cases, was not really exercised.

Why “a human signs off” is not the same claim as “a human decided”

This is the gap EN 18274’s human oversight and societal-impact competences are built to close, and it is a harder standard than it sounds. A reviewer needs three things to make oversight real rather than nominal:

• enough time per case to actually look,
• the standing to overrule the system without it counting against them, and
• a channel to escalate a pattern, not just a single case, once one becomes visible.

Take away any one of the three and the review still happens, the box still gets ticked, and the oversight is real only in the sense that the word appears in the process document.

The caseworkers involved were given a job that had been quietly redefined to something a person in their position could not actually do…

What to check for, on a system you are responsible for

Not whether a human reviews the output. Almost every system claims that.

• Ask instead: how long does a reviewer actually spend per case, on average, and is that number known to anyone.
• Ask what happens to a reviewer’s numbers if they overturn the system more often than their peers.
• Ask whether a reviewer who spots a pattern across many cases, not just one, has anywhere real to take it.

The org self-audit asks a version of this directly: whether your organisation has a real channel for raising a concern, used at least once. It takes five minutes and it is free, which is a small ask against finding out the hard way.

Written by us at Ethics Directive, drawing on the Dutch parliamentary inquiry’s published findings (the Bijlmerramp met Beleid /”Ongekend Onrecht” report, December 2020) and subsequent public reporting on the childcare benefits scandal. Summarised in our own words, not reproduced from any single source. If anything here needs correcting, we will say so in the open, dated.