Fine Tuning Git Model for Malayalam Image Captioning

馃Transformers
1

Hello everyone,

I am currently working on a project to fine-tune the GIT model for image captioning in the Malayalam language. I am following this tutorial Transformers-Tutorials/Fine_tune_GIT_on_an_image_captioning_dataset.ipynb at master 路 NielsRogge/Transformers-Tutorials 路 GitHub which fine-tunes the model on English captions. However, I am unsure about the changes that I need to make for Malayalam.

Here are some of my questions:

  1. The tutorial uses an English tokenizer. What should I use for Malayalam text? Is there a pre-trained tokenizer available that I can use, or do I need to train my own?
  2. During model training, how can I ensure that the model鈥檚 output vocabulary matches the vocabulary of the Malayalam tokenizer?
  3. Are there any Malayalam-specific considerations I should take into account when evaluating the model鈥檚 performance?

I would really appreciate any guidance or resources that can help me with this task. Thank you in advance for your time and help!