BORT: Optimal Subarchitecture Extraction for BERT

FL33TW00D · December 4, 2020, 6:34pm

Hi guys,
Wondering if anyone has read the new paper from the Alexa team regarding BERT size reduction.

If anyone has any thoughts on it or would like to discuss please comment here.

Thanks

Jung · December 5, 2020, 4:16am

Super interesting, thanks for sharing!! Perhaps @VictorSanh can give us the best comments

Wondering if the same technique can be efficiently used for the giant models like T5-11B and GPT-3

Topic		Replies	Views
CPU based Bert question answering model Models	0	359	November 18, 2021
Fine-tuning BERT with multiple classification heads 🤗Transformers	10	2130	January 19, 2024
Bert Model with Different Architectures Understanding and Make Custom Model (architecture) Using Transformer Library 🤗Transformers	0	348	November 29, 2022
Using Batch Encodings 🤗Transformers	0	298	July 12, 2022
Finetuning BERT as seq2seq for Relation extraction with multiple outputs Beginners	0	243	December 8, 2023