I’m trying to change GPT2 model in the following way:
Instead of passing sentences through the tokenizer and tokenizer output to the model, I want to use my own defined external embeddings directly as input to the model, i.e. I want to get GPT2 12 pretrained layers as a model, and pass them as input to the embeddings vector. I saw that I can extract layers by:
gpt.base_model.h, so I tried the following code (just for example):
gpt = AutoModel.from_pretrained('gpt2') model = nn.Sequential(*self.gpt.base_model.h) model(torch.ones((1, 5, 768)))
But I get an error:
TypeError: layer_norm(): argument 'input' (position 1) must be Tensor, not tuple , has anyone encountered this scenario and know how to fix it?