How to load weights in the pix2struct delpot model

processor = Pix2StructProcessor.from_pretrained('deplot_model')
model = Pix2StructForConditionalGeneration.from_pretrained('deplot_model')

I am trying to fine-tune the model by extending the vocabulary. To do this, I have added tokens and resized the embedding shape, but I would like to know how to directly access the weights since I will be initializing the newly added weights using different initialization options (random, Xavier, He, etc…). For the other transformer models,


seemed to work but I realized that model.embeddings does not exist in Pix2StructForConditionalGeneration. So is there a way to directly access the weights for the Pix2struct model?


You can use the model.resize_token_embeddings method for this purpose:

from transformers import Pix2StructProcessor , Pix2StructForConditionalGeneration

processor = Pix2StructProcessor.from_pretrained("google/deplot")
model = Pix2StructForConditionalGeneration.from_pretrained("google/deplot")

print("Number of initial token embeddings:", model.get_input_embeddings().num_embeddings)

processor.tokenizer.add_tokens(['newWord', 'newWord2'])

print("Number of final token embeddings:", model.get_input_embeddings().num_embeddings)
1 Like

Thank you @nielsr !