I have successfully implemented Longformer’s self attention in BlenderBot Small. However, when I try to load the new model: BlenderbotSmallLongForConditionalGeneration .from_pretrained(‘facebook/blenderbot_small-4096’) , this error occurs:
RuntimeError: Error(s) in loading state_dict for BlenderbotSmallLongForConditionalGeneration:
size mismatch for model.encoder.embed_positions.weight: copying a param with shape torch.Size([4096, 512]) from checkpoint, the shape in current model is torch.Size([512, 512]).
The mismatch between the embedding positions is intentional. I have used this project-
GitHub - patil-suraj/longbart: A long version of BART model based on Longformer model as an example and apparently, during transformers-2.10.0 this embedding positions mismatch was not a problem, because the author of the paper successfully loaded the Long version of Bart(which is similar to Blender). I have tried loading BartForConditionalGeneration.from_pretrained as well, but the same problem occurred again.
My question is: How should I load the new model?
Thanks a lot 