Hi, I have some general questions regarding BERT and distillation. I want to compare the performance of BERT with different model size (transformer block number). Is it necessary to do distillation? If I just train a BERT with 6 Layers without distillation, does the performance look bad? Do I…

BERT model size (transformer block number)

TonyK August 21, 2020, 8:45pm 5

Hi,

OK. I will try to do that. Thank you very much!!

Best,
ZLK