I would like to pretrain and then fine tune on longT5 on a custom dataset. Ideally, I would train a tokenizer on my data, then merge tokenizers with longT5 tokenizer and then pretrain from the published longT5 checkpoint. has anyone tried this or is aware of any good resources?
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Customizing T5 tokenizer for finetuning | 1 | 618 | May 2, 2024 | |
| Anyone have idea how we can finetune a model using Trainer API? | 0 | 446 | April 22, 2022 | |
| Fine tune LongT5 mdoel | 4 | 917 | December 15, 2022 | |
| Preprocessing for T5 Denoising | 1 | 2723 | May 20, 2021 | |
| Tutorial: Fine-tuning with custom datasets – sentiment, NER, and question answering | 19 | 12844 | February 12, 2024 |