Prepare data to fine-tune T5 model on unsupervised objective

Hi, I couldn’t find a way to fine-tune the T5 model on a dataset in a specific domain (let’s say medical domain) using the unsupervised objective. Does the current version of Huggingface support this? Basically, all I need is to prepare the dataset to train the T5 model on the unsupervised objective, which could itself be very tricky. Any pointer on this is highly appreciated. P.S: I am looking for something in PyTorch and not Tensorflow. @valhalla @clem

1 Like