Four Failures, Four Honest Disagreements

Four failures, four honest disagreements.

None of these four are stories about villains. Each is a real, documented system, a real harm, and a value someone chose over another value, in good faith, on a deadline.

It is tempting to sort every AI failure into “ethical” or “unethical” and move on.

Most real failures are not that clear. They are a clash between two things that each deserve weight, decided in favour of one of them, usually without anyone framing it as a choice at all. Naming which values were actually in tension is the first act of ethics here.

Four well-documented cases show the pattern clearly enough to recognise it elsewhere:

COMPAS: efficiency against dignity

COMPAS was a risk-scoring tool used in US courts to help predict whether a defendant would reoffend, feeding into decisions on bail and sentencing. In 2016, journalists at ProPublica examined its outcomes and found that Black defendants were substantially more likely than white defendants to be wrongly flagged as high risk, while white defendants were more often wrongly flagged as low risk.

The tool was never designed to be racist, and its overall accuracy looked reasonable on paper. The clash sits underneath: an efficiency argument (better, faster risk triage than a judge alone) against a dignity argument (that error should not fall unevenly on people by race, whatever the aggregate accuracy).

Auditing error rates by group, not just accuracy overall, is the direct answer to this one, and it is why many of the international standards ‘Skill domain’ includes assessing societal impact rather than trusting an accuracy number at face value.

Clearview AI: security against consent

Clearview AI built a facial-recognition tool by scraping billions of photos from public web pages without asking anyone, then sold search access to police departments and other agencies. The pitch was straightforwardly welfare-shaped: more solved cases, more found people.

Consent sits on one side, and scale sits on the other, which is the imbalance: a decision nobody could reasonably have agreed to, made once, then applied to everyone whose face was ever posted online. Data minimisation and purpose limitation exist as principles precisely to stop “it would be useful” from quietly becoming sufficient justification on its own.

The Tempe crash: safety against smoothness

In 2018, a self-driving test vehicle operated by Uber struck and killed a pedestrian, Elaine Herzberg, in Tempe, Arizona. Investigators found the system had detected her in time, but its emergency braking had been suppressed, reportedly to reduce false alarms and keep the ride from feeling jerky during testing. That is safety traded against a smoother product experience.

It is also a clean case of the responsibility gap: a system, a safety driver, and a company each had a piece of the decision, and not a single one of them was positioned to catch what the other two had assumed was covered.

Meta in Ethiopia: safety against voice

During the Tigray conflict in Ethiopia -> reporting in 2021, Meta’s moderation systems failed with/in Amharic and other local languages: missing content that incited violence, while at times removing posts from activists and journalists documenting atrocities.

Moderation was built and tuned mostly around English-language, Western-context speech does not transfer cleanly, and a single global policy applied at that scale does not fail evenly.

Safety pulls one way here, voice the other, and it depends on which side of the moderation check you were standing on: censored if you were trying to be heard, endangered if you were not being protected.

What the four have in common

Every one of these failures could have been asked, in advance:

• who benefits here,
• who is harmed,
• who actually decided this,
• could anyone affected challenge it, and
• at what scale does one bad call repeat.

None of the four organisations involved lacked intelligent people. What they lacked, in each case, was someone positioned to ask that question before launch and be heard, not after the reporting had already been published. That is the job this whole site is trying to help you get right, and the five-question gut check on the practice shelf is the fast version of it.

Written by us at Ethics Directive, drawing on public reporting: ProPublica’s 2016 analysis of COMPAS, reporting on Clearview AI’s data practices, the US National Transportation Safety Board’s findings on the 2018 Tempe crash, and reporting on Meta’s content moderation during the Tigray conflict. Summarised in our own words, not reproduced from any single source. If anything here needs correcting, we will say so in the open, dated.