BigBirdPegasus vs BigBird

SajjadAyoubi · September 27, 2021, 5:07pm

I’ve seen the BigBirdPegasus and BigBird pages on transformers doc but don’t understand the difference.
BigBirdPegasus is like pretrained BigBird Encoder+pretrained BigBird Decoder then fine-tuned for summarization?

Topic		Replies	Views
BigBirdPegasus with attention_type="original_full" vs T5 🤗Transformers	0	252	March 11, 2022
Comparison of methods for large token inputs Intermediate	0	320	July 5, 2023
Possible wrong BigBirdTtokenizationFast special token initialization in pretrained model Models	3	415	July 29, 2021
Out of Memory training google/big-bird-roberta-base Models	0	876	December 22, 2021
Bigbird pretraining Beginners	3	882	March 16, 2022

BigBirdPegasus vs BigBird

Related topics