29 June 2021 - reading time: 20 min
Explainable AI is not a model type, but a toolkit for finding out what a model is based on. An overview of ante-hoc and post-hoc methods, LIME and SHAP, and what regulations expect from explanations.
Explainable artificial intelligence (XAI) is the collection of methods that make it understandable why a machine learning model produced the output it did. It is not a separate model type: either we choose a model that is understandable by design, or we explain the black box decision afterwards with separate tools.
Interest in XAI has grown dramatically since 2015, and the reason is not theoretical. As machine learning moves into healthcare, lending, recruitment and public administration, the question of what stands behind a model’s decision comes up more and more often. A model that works well on its own is not enough if no one can defend its result in front of a client, a regulator or their own organisation.
In the literature, two concepts get mixed up, and there is still no full consensus on how to use them.
In practice, the two words are often used as synonyms, and most tools answer both questions to some extent. At the start of a project, though, it is worth clarifying which one you need, because understanding the whole model and justifying a single decision require different toolkits.
The literature lists five goals, and all five have a direct business counterpart.
There is also a sixth, purely practical aspect: debugging. An explanation often reveals before deployment that a model is relying on the wrong signal.
Which goal matters most largely depends on the model’s role in the process. If the model makes decisions instead of a human, the explanation is about trust and auditability. If the human makes the decision and the model only supports them, then the explanation itself is the deliverable.
XAI tools fall into two broad groups. The ante-hoc approach chooses a model whose operation is transparent from the start. The post-hoc approach adds an explanation afterwards to an already trained, possibly complex model. Among post-hoc methods, you also need to distinguish between global methods, which describe the whole model, and local ones, which explain a single decision.
| Method | Type | What does it show? | When should you use it? |
|---|---|---|---|
| Linear and logistic regression | Ante-hoc | The weight and direction of each variable | If explainability matters more than the last few percentage points of accuracy |
| Decision tree, rule-based model | Ante-hoc | The chain of splits by which the decision is made | If you want to turn the model into a human rule |
| k nearest neighbour | Ante-hoc | Which earlier cases the new case is similar to | If the expert thinks in examples |
| Permutation importance | Post-hoc, global | Which variables make the model perform worse if removed | A quick first look at any model |
| Surrogate model | Post-hoc, global | A simple model that mimics the complex one | If you need an easy-to-follow picture of the black box |
| PDP and ICE curves | Post-hoc, global and local | How the output changes along one variable | If the question is the direction and shape of the effect |
| LIME | Post-hoc, local | A local explanation around one specific decision | For customer-level justifications |
| SHAP | Post-hoc, local, aggregated to global | The contribution of each factor to a decision | If you need both individual and aggregated views with the same tool |
Two questions decide the choice: who the explanation is for, and whether you need to justify one decision or present the whole model.
The two most widely used local methods answer the same question in different ways.
LIME looks at what happens to the model’s output if we slightly change the input around the specific case. It fits a simple, interpretable model to the resulting points, and returns the explanation of this local model. It is fast, works with any model, but the explanation is sensitive to how the neighbourhood was generated.
SHAP is based on Shapley values from game theory: it splits the model output among the input variables according to how much each one contributed compared with the average output. It is slower, but more consistent, and the individual explanations can be aggregated, so you get a global picture from the same tool.
One important limitation applies to both: the explanation is about the model, not about the world. It shows what the model relies on, not what causes the phenomenon.
Expectations arrive from two directions, and the two are not the same.
Article GDPR 22. restricts solely automated decision-making when it has a significant effect on the person concerned, and the information rules require meaningful information about the logic involved. There is debate about exactly how much explanation this means, but there is no debate that an unjustifiable decision carries risk.
The EU AI Act is still a proposal: the Commission published its draft in April 2021. It is risk-based: some systems would count as prohibited practices, others would be classified as high-risk, and the rest would be subject to lighter or no special rules. According to the draft’s annex, the high-risk category would include applications such as assessing the creditworthiness of natural persons, recruitment and selection, or supporting education admission decisions.
What matters most from the XAI perspective is this: the draft requires a transparency obligation for high-risk systems, meaning the operation must be understandable enough for the user to interpret the output, and it also requires human oversight and detailed technical documentation. In practice, that is exactly what explainability tools provide.
It is important to add that the proposal is still at an early stage of the legislative process, and the final text and deadlines may still change. Anyone starting to build a system that looks high-risk today should not aim for literal compliance yet, but should make sure that explainability and documentation are built into development from the start. Doing it later is much more expensive.
XAI’s best-known dilemma is that simpler, inherently interpretable models usually perform worse, while more complex ones are more accurate but opaque. This tension is real, but three points put it in perspective.
One: the difference depends on the task, and is often smaller than expected. On tabular business data, a well-built logistic regression often comes surprisingly close to more complex models, and then interpretability comes for free.
Two: there is no single scale for how good an explanation is. We can measure accuracy, but not really the usefulness of an explanation, so the quality of the explanation has to be validated with the user, not closed off with a number.
Three: explanations are not infallible. They can mislead just as easily as a badly evaluated model, which is why explainability does not replace proper evaluation. Common model evaluation mistakes are still there even if we get a nice chart about factor contributions.
Business secret is a separate issue. A model’s internal workings are often valuable, and full transparency is not always desirable. The solution is usually not all or nothing, but deciding how much detail to show to whom.
Four decisions are enough to make sure explainability is not an afterthought.
For us, these four questions come up during use-case design, even before development starts, and the same questions accompany the production system build phase as well. Experience shows that the bigger the process a model affects, the less you can avoid this question, because more people will make decisions based on its output.
What is the difference between interpretability and explainability? Interpretability is about how understandable the model’s general operation is, while explainability is about the reasons for a specific decision. The literature often uses the two as synonyms, but in a project it is worth handling them separately because they require different tools.
Do all models need to be explained? No. If the decision has no meaningful impact on individuals and the cost of an error is low, good evaluation is enough. Explanation becomes mandatory practice where the decision must be defended in front of someone, or where the insight itself is the result.
Does SHAP show what causes the phenomenon? No. SHAP shows what the model relies on, not the cause-and-effect relationship in reality. If the model is built on a co-moving signal, SHAP will highlight that signal, and interpreting it as causation would be misleading.
How much accuracy do you lose if you choose an interpretable model? That depends on the task, so it is worth measuring. A common approach is to build the interpretable model as a baseline, compare it with the best available model, and decide whether the black box is worth it based on the difference.
Is explainability enough for regulatory compliance? No, but without it you cannot do it. Alongside transparency, human oversight, documentation and risk management are also required, and explainability is one of their building blocks.
If you have a model whose decision you cannot justify today, or you are planning one where that will be a requirement, it is worth deciding before development starts what kind of explanation it will need to provide.
Every year, roughly 80,000 young people finish without a secondary-school qualification. In a joint project, using DPMK’s domain knowledge, Microsoft’s technology and Dmlab’s data analysis, we looked at three levels at what data can add: what happened, why it happened, and what will happen.
Data projects do not fail because of technology, but because of human and organisational reasons. The seven most common pitfalls, the seventh success criterion you rarely hear about, and the three steps that can reduce the risk to a fraction.
The six phases of CRISP-DM are not theory, but the sequence in which a data project stays on track. We walk through a bank credit card campaign, show where the time goes, and what has changed in the deployment phase in the age of MLOps.
A complex data collection, portfolio management and trading system in the energy sector
Campaign reporting in real time, without a single manual step