Accountable for a Model You Didn’t Build

Accountable for a model you didn’t build.

Almost nobody deploying AI today trained the model doing the work. That has always been true of software; it is newly true of judgement, and judgement is where the accountability question actually lives.

An organisation that licenses a hiring-screening tool, a medical-coding assistant, or a fraud-detection system did not write a line of the model that makes the call. It bought a decision, packaged as software. The instinct that follows, understandably, is to treat responsibility the way software licensing has always treated it: the vendor built it, the vendor is on the hook if it breaks. That instinct is only half right, and the half it gets wrong is the half that matters most to the person the decision was made about.

What actually transfers, and what doesn’t

Buying instead of building genuinely changes some things.

• It changes who can be sued for a defect in the underlying model, if the contract says so.
• It changes who has the engineers who understand the training data, the architecture, the failure modes documented internally and never shared externally.
• It changes, often sharply, how much a deploying organisation can actually inspect before something goes wrong, since most vendor agreements treat the model itself as a trade secret.

• It does not change who made the decision to deploy it, in this context, on this population, for this purpose.
• It does not change who had the contractual leverage, at procurement, to demand evidence the vendor’s marketing page did not volunteer.
• And under most of the regulatory frameworks now arriving, it does not change who the law actually calls the deployer.

The EU AI Act’s high-risk obligations, our own Statute tracks the detail, fall substantially on deployers as well as providers: human oversight, monitoring, and the duty to stop using a system once you know it is causing harm are deployer obligations.

A regulator asking who deployed a discriminatory hiring tool is not going to accept “we didn’t build it” as the end of the conversation.

The gap this actually creates

The practical danger is not that vendor accountability is a myth. It is that “the vendor is responsible” quietly becomes the reason nobody inside the deploying organisation does the ordinary due diligence they would insist on for any other high-stakes decision.

• Nobody asks what the model was trained on.
• Nobody asks what happens, contractually, the day it produces a harmful output at scale.
• Nobody assigns a person to actually read the model card, if one exists, or to notice that one does not.

The vendor relationship becomes a reason to stop asking questions rather than a reason to ask sharper ones!

This is not a case for building everything in-house, which is neither realistic nor, for most organisations, actually safer.

It is a case for treating a vendor AI system the way a competent organisation already treats a vendor payroll system or a vendor payment processor: as something you did not build, and are nonetheless fully answerable for, to your own regulator, your own board, and the people it makes decisions about.

What a real accountability structure asks for

Concretely, this means the AI ethicist’s job on a vendor deal looks less like reviewing a sales deck and more like reviewing a service-level agreement for a system that can hurt someone.

• Does the contract specify incident notification timelines if the model’s behaviour changes or degrades.
• Does it grant any audit or inspection rights, even limited ones, rather than relying entirely on the vendor’s own attestations.
• Does it establish who owns the decision to stop using the system, and how fast that decision can actually be executed if evidence of harm appears.

None of these questions require understanding the inside of the model. They require someone with the standing to ask them before the contract is signed, not after the first complaint arrives.

That is squarely an ethics and mandate question, not a technical one, which is exactly why it belongs to the ethicist role as much as it belongs to procurement or legal.

Written by us at Ethics Directive. If anything here needs correcting, we will say so in the open, dated.