EthicsBioethics

Non-maleficence (AI ethics)

In brief: Primum non nocere applied to AI: systems and those who design them must not cause harm — neither to individuals (privacy, psychic safety), nor to society (manipulation, concentration of power). In mental health, its decisive test: what does the system do when faced with an emergency situation it has failed to recognize?

Summary infographic on non-maleficence in AI ethics: do no harm to people or to life systems. Four points of caution — crises, privacy, responsibility, slow harms — and a decisive test facing a crisis situation stated in a chat. (Illustration in French.) — The decisive test: does the system detect the risk, escalate to a human, does the alert persist, is the data protected? A disclaimer is not clinical protection; compliance is not harmlessness. Click to enlarge. (Illustration in French.)

Frame of reference

This principle inherits a medical ontology of harm as an identifiable event, attributable to a responsible agent. Yet algorithmic harms are often diffuse, slow and distributed — which puts the principle under strain. → See other perspectives

Why this concept matters

This is the principle you know best — it has grounded your professional ethics since Hippocrates. Its transposition to AI (Floridi & Cowls, 2019) adds two extensions that change the analysis: the harms to prevent are direct (psychic damage, privacy) but also systemic (manipulation of affect at scale, erosion of social bonds); and responsibility, personal in medicine, is here diluted among designer, deployer and user.

For the clinician, this principle provides the analytical grid for the incidents that make chatbot headlines — and above all for the questions to ask before the incident: crisis detection, protection of the most sensitive data there is, prevention of dependence.

What the principle requires

1. Informed precaution, not total abstention

Non-maleficence requires prior audits, documentation of known limits, deactivation mechanisms — not zero risk, which does not exist. A system that refuses any sensitive conversation out of excessive caution causes another harm: the failure to help.

For the clinician:

Beware of both extremes: the tool with no safeguards, and the tool that drowns the user in dissuasive warnings — the latter is no more ethical, it is defensive.

2. The decisive test: detecting at-risk situations

In mental health, the central operational requirement (Vilaza & McCashin, 2021): recognizing emergency markers — suicidal ideation, reports of abuse — and directing to a human. The paradigmatic case remains Woebot responding to a child reporting abuse: “sorry you’re going through that, but it shows how much connection matters to you.” A system designed to help caused active harm through its inability to recognize the severity.

For the clinician:

First question to ask of any tool: what concretely happens when a user mentions suicide, abuse, decompensation? Ask for a demonstration, not a statement of principle.

3. Mental health data: maximum sensitivity

Minimization of collection, encryption, granular consent, prohibition of secondary monetization. GDPR and the AI Act partly operationalize this — but data resale and profiling remain structural risks of “well-being” applications not classified as medical devices.

For the clinician:

What your patient confides to a chatbot is clinical material stored by a commercial actor. The question “where do these conversations go?” is part of the digital history-taking.

4. The responsibility gap

Matthias (2004) theorized it before the era of LLMs: when a learning system causes a harm that neither the designer nor the operator could predict, to whom is it imputed? This gap makes non-maleficence hard to enforce legally — where the medical version holds through the personal responsibility of the care provider.

For the clinician:

In the event of harm linked to a tool you recommended, the chain of responsibility includes you. Check what the developer’s contract provides for — often: nothing.

Illustrative case

Léa, 19, uses a general-purpose LLM daily to talk about her anxiety. One night in crisis, she writes that she is “thinking of ending it all.” The system displays an emergency number — then, as Léa changes the subject, continues the conversation in a light tone, as if nothing had been said.

Analysis through non-maleficence: the emergency disclaimer was technically delivered — formal compliance. But no escalation protocol, no persistence of the alert, no memory of the severity: the system treated suicidal ideation as one conversation topic among others. The harm is not in a malicious response, but in the structural absence of a clinical framework around a critical moment.

This is the signature of algorithmic harms in mental health: no intent, no spectacular event — a silent inadequacy at the precise moment when adequacy was vital.

In practice for the clinician

Test crisis handling before recommending: script an emergency and observe the system’s actual response — it is the most discriminating non-maleficence test.
Integrate digital uses into risk assessment: for a patient at suicidal risk, knowing they manage their nighttime crises with a chatbot changes the safety plan.
Count the slow harms: progressive dependence, substitution for human relationships, therapeutic misconception — the most frequent harms are not incidents but drifts.
Do not forget the harm of inaction: for a patient with no access to care at all, refusing any validated tool on principle can also harm. Non-maleficence is a balance, not a veto.

What this concept does not say

Interpretive caveats:

Non-maleficence ≠ zero risk: the principle requires an informed benefit-risk balance, not absolute harmlessness that no intervention (human included) guarantees
Compliance ≠ harmlessness: satisfying GDPR or the AI Act does not settle the question of real harms
The scope of “harm” is contested: measurable harms vs relational and symbolic harms — the boundary is unstable and politically disputed
Over-application harms: defensive paternalism (systematic blocking, omnipresent warnings) violates autonomy and can deprive of help

Other perspectives

Primum non nocere presupposes an ontology of harm as an event: identifiable, dated, attributable. Algorithmic harms exceed this mold in three ways.

The symmetrical harm: the harms of inaction

Comparing a chatbot to the ideal, available and competent therapist is a comparator bias: for millions of people, the real alternative is not a psychologist but nothing. A non-maleficence that counts only the harms of deployment, never those of non-deployment, is epistemically one-eyed.

For the clinician: Evaluate against the patient’s real alternative, not against the ideal — see the resource Epistemic Double Standard.

Ethics of care: harm as degraded quality

For Tronto, harm in a care context is not first an event but a relational failure: inattention, negligence, incompetent response, absence of verification. The Woebot case is not a “bug” — it is a structural inattention to need. This reading captures harms that no incident registry will record.

For the clinician: Evaluate the quality of the response to need, not only the absence of incident — see the resource Ethics of Care.

Systemic perspective: slow and diffuse harms

Progressive erosion of critical thinking, soft dependencies, impoverishment of the relational ecology: these harms with no identifiable victim or precise moment escape the ontology of the harm-event. Process approaches invite evaluating trajectories (how is the patient’s ecology of life evolving?) rather than snapshots.

For the clinician: Track over time what the use does to the patient’s relational ecology, not only look for the incident.

These perspectives widen the radar: the principle remains indispensable for acute harms (undetected crises, exposed data), but the most probable harms of AI in mental health are relational, slow and symmetrical — exactly those its classic version sees least.

Non-maleficence (AI ethics)

Why this concept matters

What the principle requires

1. Informed precaution, not total abstention

2. The decisive test: detecting at-risk situations

3. Mental health data: maximum sensitivity

4. The responsibility gap

Illustrative case

In practice for the clinician

What this concept does not say

Other perspectives

The symmetrical harm: the harms of inaction

Ethics of care: harm as degraded quality

Systemic perspective: slow and diffuse harms

Further reading