Hi,
Yes that’s possible. There are some cool libraries out there that can be used to do that:
- ELI5: GitHub - TeamHG-Memex/eli5: A library for debugging/inspecting machine learning classifiers and explaining their predictions
- LIME: GitHub - marcotcr/lime: Lime: Explaining the predictions of any machine learning classifier
I believe it requires some tweaks to make it work on Transformer-based models, but I did this myself some time last year, so it’s definitely possible.
Edit: this Github thread might be helpful.