How to load weights in the pix2struct delpot model

Models
1 
processor = Pix2StructProcessor.from_pretrained('deplot_model')
model = Pix2StructForConditionalGeneration.from_pretrained('deplot_model')

Hi,
I am trying to fine-tune the model by extending the vocabulary. To do this, I have added tokens and resized the embedding shape, but I would like to know how to directly access the weights since I will be initializing the newly added weights using different initialization options (random, Xavier, He, etc…). For the other transformer models,

model.embeddings.word_embeddings.weight

seemed to work but I realized that model.embeddings does not exist in Pix2StructForConditionalGeneration. So is there a way to directly access the weights for the Pix2struct model?

2

Hi,

You can use the model.resize_token_embeddings method for this purpose:

from transformers import Pix2StructProcessor , Pix2StructForConditionalGeneration

processor = Pix2StructProcessor.from_pretrained("google/deplot")
model = Pix2StructForConditionalGeneration.from_pretrained("google/deplot")

print("Number of initial token embeddings:", model.get_input_embeddings().num_embeddings)

processor.tokenizer.add_tokens(['newWord', 'newWord2'])
model.resize_token_embeddings(len(processor.tokenizer))

print("Number of final token embeddings:", model.get_input_embeddings().num_embeddings)
1 Like
3

Thank you @nielsr !