T5 training from scratch

Beginners
1

Hi all,
I would like to train a T5 model (t5-base version) without loading the pretrained weights, if I write the following:
from transformers import T5Config, T5Model
config = T5Config.from_pretrained(‘t5-base’)
model = T5Model(config)
It will produce the t5-base sized T5 model without loading the checkpoint weights?
Thanks

2

Yes, that’s correct. You will definitely see the “pretrained weights” download progress bar in the other case :slight_smile:

3

Thanks for your reply, if I will see the download progress bar it means that the pretrained weights have been loaded?
The fact that the T5Config was initialized by “from_pretrained” doesn’t imply downloading something?

4

That’s only config (eg. hyperparameters) :wink:

1 Like
5

There will be a little progress bar of < 1mb. The weights are > 1GB.

6

Yeah, I’ve noticed it. Thank you all!