The need for explainable AI

Most blogs, papers, and articles within the field of AI start by explaining what AI is. I will assume that the reader of this piece knows more about AI than what would be possible to put into one paragraph, but for the sake of completeness, I will refer to AI as a statistical model which will recognize patterns in data to make predictions. This blog will look more specifically into explainable AI.

The first step that executives seek when using AI is to solve meaningful business problems. In order to fulfill this step, a data scientist often aims to produce the best possible model to make an accurate prediction. So far, so good. At the end of the process is a model that delivers satisfactory prediction results relating to the said business problem. It seems simple enough, but there is more to it. Having a prediction is often just one part of what the end-user asks for. The second question will always be “why?” and just like a child who always asks “why?” this question needs to be addressed.

The need for explainable AI

Over the last years, we have seen a rising quest for AI explainability (in machine learning, deep-learning, NLP, etc.) Business owners, end-users, and even regulators continue asking for more explainable models.

The reasons for the AI explainability craze are diverse. Some want to have control over the models and test them based on gut feelings. This quest is often raised by business owners or even data scientists themselves during building stages. One has to compare predictions with the reality (split test) and check whether the reasons for the prediction make sense. For example, a text routing model that always sends texts to one user when the text is converted from word does not make sense. The routing should usually be based on the content.

Eradicate unethical predictions and decision making

Another need for AI explainability is to mitigate the risk of false or unethical predictions/decisions. This request often comes from internal or external regulators, but might also be on the board’s agenda. As seen during the last two years, regulatory bodies such as the EU commission and national organizations have released ethical AI guidance and recommendations. These guidelines often include a quest for transparency and explainable models. One has to be able to say why a model has made a specific decision. In finance, where statistical credit rating and risk models have been standard for decades, this is not new. All current models running in production (whether AI or statistics) need to be explainable. If not, most FSAs would not allow institutions to use them. This proves that model explainability is a must to harvest the value from AI in an organization fully.

In the next two paragraphs, I want to give the reader a short and non-technical overview of what type of AI explainability is possible so far.

The possibilities with AI explainability

The first group is direct explainability. Models in this mathematics can be explained very easily. For example, direct explainability is the case for OLS regressions, which are common in economics and is what most readers might be familiar with or have at least heard of during their studies. Other models, where predictions can be directly explained, are decision tree models. Here the notes and breakpoints can be presented as a graphical tree (see figure 1).

Figure 1. Example decision tree animal classification

When defining directly explainable model types, it is essential to address those not directly explainable. These types of models are usually more complex, such as neural networks or boosted tree models. Both model types are applied to complex problems such as image recognition and are often referred to as black-box models. With this type, it is clear that the model made correct predictions like correctly detecting damage to a car, for example. However, with black-box models, it can be unclear as to why a prediction is made. Going back to the car example, the prediction for damage should be because of actual damage on an image, not because the card was red.

Recent research shows that the data science community has overcome some of the problems and released frameworks that bring light to these models. Depending on the application, python libraries such as [SHAP; LIME; Gradcam; eli5; etc.] give us a hint on variable impact. These frameworks are often model neutral and can be used for standardization. They furthermore allow data scientists to deliver reasons for each prediction. So, If you were to ask a model about the value of your house, it will tell you, for example, 400.000 EUR and the reason is the home’s location in a quiet area with a garden, etc.

A unique role play framework that tests for bias

One prominent example is [AI Fairness 360] from IBM. These frameworks aim to test whether certain critical variables, such as gender, have a significant impact on the model and lead to differing results. There is much more to this field, and it is a growing and complex area. I just wanted to name it for the sake of completeness but without going too much into detail.

In conclusion, we can say that with the named model explainability frameworks, we can: standardize code, explain the overall model for a sanity check, check for bias, explain the reason for each prediction and, with this, give the end-user an answer to his/her question, but also use this to document for regulators and risk managers answering why a model has done sth. The ability to deliver such insights might even increase the value of the model. Hence, one could think about additional analysis based on the most critical factors, for example, to improve customer loyalty when using a churn model.

2021.AI’s GRACE AI Platform provides an easy way for companies or governments to create tangible metrics for fairness, transparency, and explainable AI. We link these metrics to your impact and risk assessments, and effectively measure these metrics continuously, while automatically restricting model behavior.

You might also like…