Can we use different input shape for transformer model like DistilBERT?
I am going to build a language model with MLM, over DistilBERT model. As we know the input shape of the model is (* , 512), so how we can use (*, 64) as an input? (That is due to not that long sequences)
I use Trainer for training the model for domain adaption and DistilBERT as a model.