Explainable Is Not Interpretable

Explainable is not interpretable, and the difference matters.

A model can produce a fluent, specific-sounding reason for its own decision and still be a black box to the person asking. Those are two different claims, and vendors have every incentive to let you hear only one of them.

Ask a complex model why it made a particular call and, increasingly, it will tell you: a confident sentence naming the factors that mattered, in an order that sounds like reasoning. That sentence is an explanation. Whether it is an accurate account of the actual computation that produced the decision is a separate question, and for most of the models generating these explanations, the honest answer is that nobody knows, including the people who built it…

Two different claims wearing one word

Explainability, in the sense most products sell it, means the system can generate a human-readable account of a decision after the fact. It is a communication feature. It can be produced by a second model whose only job is to describe the first model’s output plausibly, with no guarantee, and often no mechanism, connecting the description to the actual internal computation. Interpretability means something stricter and much harder to build: that a person can trace, mechanically, how specific inputs produced a specific output, using the model’s actual structure rather than a generated narrative about it. A linear regression is interpretable; you can read the coefficients. A large neural network with a natural-language explanation bolted onto its output is explainable. Very few systems in production today are both, and the ones that are tend to be simple enough that nobody needed an explanation layer to understand them in the first place. The gap between the two is not a technical footnote. An explanation that sounds right but is not causally connected to the decision is worse than no explanation, because it creates the confident appearance of accountability. A person denied a loan, told a plausible-sounding reason, has no way to know whether that reason is the model’s actual logic or a separately generated story that happens to sound reasonable. Contesting a decision requires knowing what actually drove it, not what the system was willing to “say.”

Where this actually bites

The EU AI Act’s transparency obligations, and most internal AI governance policies modelled on them, ask for meaningful information about the logic involved in a decision. That phrase is regularly satisfied, in practice, by an explainability layer that was never validated against the model’s real decision process. In most cases; the explanation genuinely reads as meaningful. It just is not verified, and “sounds meaningful” and “is accurate” are not the same property, however much they resemble each other from the outside. The practical competence this demands of an ethicist is not learning to build interpretable models, that is a specialised technical skill most roles do not require. It is learning to ask the one question vendors rarely volunteer an answer to: has this explanation ever been checked against the model’s actual behaviour. The one worth knowing before an explanation gets relied on for a decision that affects someone.

Written by us at Ethics Directive. If anything here needs correcting, we will say so in the open, dated.