Hey guys, I am trying to fine-tune a MarianMT model that translates from Japanese to English. I have some new tokens (around 50). So, I made the following steps: tokenizer.add_tokens(new_tokens) model.resize_token_embeddings(len(tokenizer)) Then I fine-tuned the model on kde4 dataset. Since Mari…

Adding New Tokens to MarianMT Model

kumarme072 February 2, 2024, 5:34pm 2

Fine tuning is enough go ahead

Topic		Replies	Views
Add_tokens + finetune 🤗Transformers	0	534	February 25, 2022
Can we fine-tune right away after add_tokens? Beginners	0	860	January 31, 2023
LM finetuning on domain specific unlabelled data Beginners	6	4705	April 21, 2021
Enhance a MarianMT pretrained model from HuggingFace with more training data Beginners	4	2741	May 29, 2021
RoBERTa from scratch with different vocab vs. fine-tuning Intermediate	9	2245	August 20, 2020