I would like to pretrain and then fine tune on longT5 on a custom dataset. Ideally, I would train a tokenizer on my data, then merge tokenizers with longT5 tokenizer and then pretrain from the published longT5 checkpoint. has anyone tried this or is aware of any good resources?
Related Topics
Topic | Replies | Views | Activity | |
---|---|---|---|---|
Anyone have idea how we can finetune a model using Trainer API? | 0 | 428 | April 22, 2022 | |
Fine tune LongT5 mdoel | 4 | 846 | December 15, 2022 | |
How to pretrain randomized language model with custom dataset | 0 | 56 | May 15, 2024 | |
How is T5 pretrained? | 3 | 465 | July 12, 2021 | |
Pretraining T5 from scratch using MLM | 0 | 294 | February 15, 2024 |