Using head_mask in DistilBERT

rbroc · December 2, 2021, 5:27pm

Hi there! I’m trying to pass a head_mask when calling DistilBERT for MLM, but I get a NotImplementedError.
Is it just not possible to use a head_mask with this model - or is there any way to do it?

Topic		Replies	Views
Can we directly use the embeddings from masked language models? 🤗Transformers	0	518	December 15, 2021
Model did not return a loss --- but why? 🤗Transformers	0	656	April 27, 2023
Output_attention = True after downloading a model Beginners	2	3030	August 29, 2020
Question on HuggingFace Model Beginners	0	608	September 6, 2022
Finetuning on MLM task Models	0	561	June 29, 2021

Using head_mask in DistilBERT

Related Topics