Explainability (Explainable AI, XAI)
Matthieu Ferry ⇄ IAIn brief: The capacity of an AI system to make its decisions understandable to a human. It is the fifth principle of the AI ethics framework — the one that makes the four others verifiable. But explainability is a continuum, not a state: current techniques provide after-the-fact approximations, not access to the real mechanism.
Frame of reference
This resource adopts the dominant XAI framework, which treats the explanation as a technical property of the system that could be extracted and delivered. This objectivist presupposition is contested: an explanation exists only for someone, in a context and for a use. → See other perspectives
Why this concept matters
The AI tools arriving in the mental health field no longer merely converse: they score (suicidal risk, severity), prioritize (waiting lists, alerts) and synthesize (reports, hypotheses). Each of these outputs pre-formats a clinical decision.
To exercise your judgment — and not abdicate it — you must be able to understand why the tool suggests what it suggests. And for your patient to give informed consent to a device that concerns them, they must be able to receive an explanation within their reach.
This is why Floridi and Cowls (2019) make explicability the fifth ethical principle of AI, the only one without an equivalent in classic bioethics: without it, it is impossible to establish that a system is unjust, harmful or that it erodes autonomy. It is the principle that makes the others verifiable — see the resource AI Ethics Frameworks.
What explainability (really) covers
1. A continuum, not a switch
A system can be partially explainable (one identifies the main factors of a prediction) without being transparent (one does not understand the complete mechanism). A useful image: frosted glass — one makes out the silhouettes, never the detail.
For the clinician:
When a developer claims their tool is “explainable,” ask: explainable to what degree, for whom, and in what form?
2. Two faces: intelligibility and accountability
The AI4People framework (Floridi et al., 2018) breaks the principle into intelligibility (how does the system work?) and accountability (who is answerable for its effects?). A system can be technically opaque but backed by a clear chain of responsibility — and vice versa.
For the clinician:
Of the two faces, accountability is often the more decisive in practice: in the event of harm, is there an identifiable responsible party — developer, deployer, prescriber?
3. Post hoc techniques: approximations
The most widespread methods (SHAP, LIME, attention maps) produce an explanation after the fact: they identify the variables that weighed on this result, by local approximation. Nothing guarantees that this explanation reflects the model’s real process.
For the clinician:
“High risk score: ideation + isolation + history” is a plausible approximation of the computation — not the computation. Demanding it remains essential; relying on it blindly is a mistake.
4. Counterfactual explanations
An approach proposed by Wachter, Mittelstadt and Russell (2018): rather than opening the black box, state what would have needed to change to obtain a different result (“without the social-isolation item, the score would have gone from high to moderate”). Actionable and understandable without access to the mechanism.
For the clinician:
The counterfactual question is an excellent test to put to any tool: “what would make this recommendation change?”. If no one can answer, be wary.
The central trap: the illusion of understanding
A plausible explanation is not a true explanation. The documented risk of XAI interfaces is to produce a feeling of understanding that reinforces trust in the system instead of calibrating it: the explanation becomes a cognitive sales argument.
This mechanism worsens automation bias: a clinician who receives a score accompanied by an apparently sensible justification is even less inclined to verify than a clinician who receives a raw score. Misunderstood explainability can therefore produce the opposite of what it promises — less vigilance, not more.
Illustrative case
A remote-monitoring service equips its psychologists with a tool that analyzes patients’ messages between sessions and produces a weekly risk score, accompanied by three explanatory factors generated automatically.
For Mr. R., the tool displays: rising risk — factors: negative tone, mention of sleep disturbances, drop in message frequency. The explanation seems clinically coherent; the lead psychologist initiates a call. A good decision — but in session, she discovers that the “drop in frequency” was due to a week of vacation, and that the model had mainly reacted to nighttime sending times, a factor that did not appear in the explanation.
The post hoc explanation was plausible, partly false, and impossible to distinguish from an exact explanation without human verification. It served well — as an alert to be scrutinized, not as a diagnosis. That is exactly the status to give XAI outputs in clinical practice.
In practice for the clinician
- Demand explainability, without relying on it blindly: treat any automatic explanation as a hypothesis to be scrutinized clinically, never as an established fact.
- Ask for the type of explanation: a model interpretable by construction, or a black box dressed up with post hoc explanations? The difference (Rudin) changes the credit to give the displayed justifications.
- Ask the counterfactual question: “what would make this score, this alert, this recommendation change?” — the fastest test of the developer’s real mastery of its own system.
- Think of the patient: informed consent requires an explanation within the patient’s reach, not only the professional’s — see the resource Informed Consent and AI.
- Distinguish intelligibility and accountability: when the technical explanation reaches its limits, the question that remains decisive is “who is answerable for the effects of this system?”.
What this concept does not say
Interpretive caveats:
- Explainability ≠ clinical validity: a system can explain itself perfectly well and be wrong — the explanation says nothing about correctness
- Explanation ≠ causation: the factors highlighted are influential correlations in a computation, not clinical causes
- In mental health, we explain poorly what we understand poorly: psychiatric diagnosis is itself multifactorial and debated — demanding of a tool a univocal explanation of a phenomenon that is not is an epistemic trap
- Robust explainability is expensive: rigorous methods (validated counterfactuals) are rarely deployed in real-time production — the marketing label “XAI” often covers the bare minimum
Other perspectives
The dominant XAI framework treats the explanation as a property of the system, to be extracted by techniques. This objectivist presupposition carries blind spots that other perspectives illuminate.
Rudin: the black box is not inevitable
For Cynthia Rudin (2019), explaining black boxes is the wrong battle: for high-stakes decisions, one should demand models interpretable by construction — often just as performant. Opacity is frequently an economic choice (intellectual-property protection), not a technical necessity.
For the clinician: Faced with an opaque tool, the right question is not only “explain it to me” but “why choose a black box for a high-stakes clinical decision?”
Explanation as a relational act
For the social sciences of explanation, to explain is not to exhibit a mechanism but to answer someone’s question, in a context, for specific purposes. The “right” explanation of the same score differs for the regulator, the clinician and the patient — there is no explanation in itself.
For the clinician: Evaluate a tool’s explanations as you evaluate a feedback session for an assessment: adapted to the recipient, or bolted on?
The clinical mirror: rationalization
A justification produced after the fact, coherent but disconnected from the real process: clinical work knows this under the name of rationalization. Humans too have no transparent access to their own decision-making processes — what we require of them is not mechanistic transparency, but accountability: supervision, professional ethics, justification before peers.
For the clinician: This parallel calibrates the requirement: ask of the system what we ask of a professional — not “show me your neurons,” but “to whom are you answerable, and by what rules?” See the resource Epistemic Double Standard.
Technical explainability is one piece of the apparatus, not the whole: without institutional accountability (audits, responsibility, recourse), the best explanation remains a display. And conversely, a solid chain of responsibility can compensate for partial intelligibility.
Further reading
The ↩ arrows link back to the passage of the resource that cites the reference.
The fifth principle: Floridi, L. & Cowls, J. (2019). A Unified Framework of Five Principles for AI in Society. Harvard Data Science Review, 1(1). DOI ↩
Intelligibility + accountability: Floridi, L., Cowls, J., Beltrametti, M. et al. (2018). AI4People — An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations. Minds and Machines, 28(4), 689-707. DOI ↩
Counterfactuals: Wachter, S., Mittelstadt, B. & Russell, C. (2018). Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law & Technology, 31(2), 841-887. DOI ↩
The counter-position: Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1, 206-215. DOI ↩
All concepts
Last updated: July 2026