It appears there’s no manual for this model, so I’m forced to ask some questions to clear things up. The question applies to both DynaBERT models on Huggingface (I believe they are architecturally the same but pre-trained on different datasets). I am using “DynaBERT_MNLI”
Assuming I re-train it from scratch similarly to a typical BERT model, how can I can configure the width/depth weights to extract the sub-set models?
If I save/load the model via Pytorch’s interface, will the configuration for weights percist?
Are we able to change the weights numerous times in the model object’s lifetime, or is it done only once at initialization?
Are there any specific API calls I need to make after training to generate the sub-set models, or are they generated on the fly during training?
Perhaps there exists an example use of this model, it would be great help it someone could provide it. The model’s documentation is very baren.