Hi I am new to hugging face, I have downloaded distilBERT using model = DistilBertModel.from_pretrained("distilbert-base-uncased") and I did not modify the config. But now I want to see the output attentions. Is there any way I can see the attentions without downloading a new model?
Hey @pritam
You can pass output_attentions=True to forward method, and it’ll return attentions. Attnetions are last element of the returned Tuple or if you are on master branch, then you can also pass return_dict=True and call .attentions on the returned object.
output = model(input_ids, output_attentions=True, return_dict=True)
output.attentions # this will give the attentions
1 Like
You can do what @valhalla said. Perhaps the most clean way is changing the config.
config = AutoConfig.from_pretrained(MODEL_ID, output_attentions=True)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID, config=config)
output = model(**inputs) # inputs from default_data_collator
output will consist of logits, pooled output and attention.
2 Likes