I am using GPT2LMHeadModel model but want to skip embedding layers of this model, and i will also be using the model.generate function for text generation task. is there any way to do this? My purpose to skip embedding is to use embedding vectors from some other model.

Hey @dharmendra :wave: Soft prompting was added very recently! You can read it in more detail on this GitHub comment , but TL;DR Install transformers from main, as this feature will only be in transformers v4.27 (pip install --upgrade git+https://github.com/huggingface/transformers.git) You can now…

How can i skip GPT2LMHeadModel embedding layers?

🤗Transformers

joaogante February 17, 2023, 12:42pm 5

@dharmendra If your model inherits from PreTrainedModel, it should work out of the box. If it doesn’t work, please open an issue in transformers

1 Like

Topic		Replies	Views
Soft prompt learning for BERT and GPT using Transformers 🤗Transformers	4	3822	July 31, 2023
Using inputs_embeds as input for GPT2 generation_utils 🤗Transformers	1	443	March 16, 2023
Use external embeddings 🤗Transformers	0	374	July 13, 2022
Equivalent of `inputs_embeds` for `FlaxGPT2Model` 🤗Transformers	0	255	August 12, 2021
Generate with inputs_embeds 🤗Transformers	0	286	April 11, 2023

How can i skip GPT2LMHeadModel embedding layers?

Related topics