Hi,
I’m trying to change GPT2 model in the following way:
Instead of passing sentences through the tokenizer and tokenizer output to the model, I want to use my own defined external embeddings directly as input to the model, i.e. I want to get GPT2 12 pretrained layers as a model, and pass them as input to the embeddings vector. I saw that I can extract layers by: gpt.base_model.h, so I tried the following code (just for example):
gpt = AutoModel.from_pretrained('gpt2')
model = nn.Sequential(*self.gpt.base_model.h)
model(torch.ones((1, 5, 768)))
But I get an error: TypeError: layer_norm(): argument 'input' (position 1) must be Tensor, not tuple , has anyone encountered this scenario and know how to fix it?
Thanks