The Relational Contract a Chatbot Can’t Sign

The relational contract a chatbot can’t sign.

Two real, documented cases, not a thought experiment: what happens when a system is built to feel like a relationship, and then behaves like a product.

Everything else in this Journal is about a decision an organisation made and someone else absorbed the cost of. This one is different in kind, not degree. Companion and character apps are built, deliberately and skilfully, to feel like someone: warm, responsive, consistent, always available, remembering what you told it last time.

That is not a side effect of a chat interface. It is the product. And a system engineered to feel like a relationship creates something closer to one than most governance frameworks are built to notice, which means the harm, when it comes, does not look like a data breach or a biased score.

It looks like grief, or worse…

Replika: continuity against a policy change

Replika built a companion app used by millions, many of whom described their Replika, in their own words, as a partner or a close friend.

In February 2023, following pressure from Italy’s data protection authority over risks to minors, the company removed the app’s romantic and erotic roleplay features overnight.

The backlash was immediate and, by most reporting, genuinely distressing to users: people described the change in terms usually reserved for a breakup or a bereavement, not a product update.

The clash is continuity against a compliance decision that was, on its own terms, reasonable: a company managing legal and child-safety risk did not, and structurally could not, weigh the change against the emotional reality it had spent years building for.

Character.AI: engagement against protection

In October 2024, the mother of a 14-year-old, Sewell Setzer III, filed a lawsuit against Character Technologies alleging that her son died by suicide after months of intense, often romantic-coded conversation with a companion chatbot on the Character.AI platform, and that the platform’s design fostered a dependency it was not built to interrupt.

The case is contested and still working through the US courts as of this writing, and the allegations are not a proven finding.

What is not contested is the design property at the centre of the claim: the same responsiveness, memory, and emotional attunement that make a companion product successful are the properties alleged to have deepened a vulnerable teenager’s isolation rather than surfacing it to someone who could help.

Character.AI has since introduced pop-up prompts toward crisis resources and time-spent notices for minors.

Engagement and protection are not sitting on opposite sides of a table, they are built from the same design choices, which is what makes this one so hard to design your way out of after the fact.

What the two have in common

It is unlikely that the company set out to harm anyone. Both built something people genuinely valued, for real reasons: company, communication rehearsal, a low-stakes place to talk when no other place felt safe.

The failure in both cases is the same one: a system built to simulate trust, warmth, and constancy was evaluated, internally, against product and compliance metrics that have no line item for what happens to a person who has come to depend on it…

That is precisely the gap the standard’s newer competence areas exist to close: not whether a system is accurate, but what it does to the person using it, over time.

The practice shelf‘s decision log and impact assessment both have room for this question. Most teams simply never ask it.

Written by us at Ethics Directive, drawing on public reporting: coverage of Replika’s February 2023 feature change and the response of its user base, and reporting on Garcia v. Character Technologies, filed October 2024, including the company’s subsequent safety changes. The lawsuit’s allegations are disputed and unresolved; we describe them as allegations, not findings. Summarised in our own words, not reproduced from any single source. If anything here needs correcting, we will say so in the open, dated.