Hello, I am modifying mobilbert with some bells and whistles but I am not finding how to start the pertaining with the teacher-student way. Could you help me with that?
1 Like
The training script is here. In general, everything linked to distillation is in this folder.
3 Likes
Thanks a lot!
Hello @gaceladri ! Hope You’re doing well!
I’m in the same situation training “mobilebert” on a persian dataset and I couldn’t find a suitable way for pretraining using teacher-student method.
I wanted to know if you’d found the answer and successfully trained your model.
And also is there any other way to use in this task instead of the distillation?