- Sensitivity analysis: This method provides a change in the response when the weightage of the pixels are changed. It represents the impact made by the deletion of a certain feature in the decision making process. It does not indicate relative importance of the features of the input. It fails to capture the importance or relevance of each input feature or the relation between them.
- Deconvolution: This technique provides the heatmap of the matching input pattern for the classified object. It is limited to convolutional neural network.
- Saliency map: It provides the heat map for important regions of the input. It is limited to paying attention of the parts contributing to the class, and does not speak about the other areas of input.
Visual explanation generator
It generates the explanation text through LSTM with relevant features derived from the heatmap. The attributes of the features get mapped on to the words in the explanation. In one implementation, the penultimate layer output is taken as the feature and is trained with the explanation.
Visualization of the model
A classifier model in general, is a black box. The tools Deepvis, RNNvis and LSTMvis help to visualize and interactively plot the activation relevance produced in each layer of the classifier. It also depicts the learnings of each neuron.
2. Explainable AI for text data
Explaining predictions made by models in the textual domain is tricky as textual input is transferred to the models in the form of embeddings. These embeddings are obtained by 4 projecting textual units in a vector space of low dimension. The concepts of relevance and heatmap for images have been explained previously. The same is true for text as well. The difference is that, in the case of text, the unit used for mapping the relevance is a textual unit which, in most cases, is a word.
For obtaining wordwise relevance in a chunk of input text for predictions made by the text classification model, the LRP and sensitivity analysis algorithms, as explained above, are primarily used. However, the usage is in the context of words in the text.
Explainable AI paradigm provides better insights into the AI system and helps to open up the black box of deep learning architecture. The framework can provide visual explanation through a heatmap and descriptive explanation through the text.
The explanation provided by the framework can be used for a variety of applications such as debugging, test case designs and optimal designing of the system.