I customized the run_clm.py example to be able to pretrain seq2seq models like T5, hence named run_slm.py. I think it might save time for other people interested in something similar. If you agree, please let me know to create a PR. Thanks! 
1 Like