Retrain T5 using unsupervised learning with MLM

Hi,
I have built my own corpus. Now, I want to retrain T5 on that corpus. I don’t want to do fine-tune that model on a particular task. I just want to add my corpus data above the T5-base model. Can you all have any suggestions and advise.
Thanks

1 Like