Generating model embeddings from Conditional Generation models

gchhablani · September 23, 2023, 12:35am

I’m trying to break apart BLIP2 from LAVIS (https://github.com/salesforce/LAVIS/blob/main/lavis/models/blip2_models/modeling_t5.py) which uses HuggingFace PreTrained Model to generate sentence embeddings from T5 model.
So, questions:

What do I replace this with to get the embeddings: https://github.com/salesforce/LAVIS/blob/e4040b13d6120062829ee9625f016f3cd3dd16e6/lavis/models/blip2_models/blip2_t5.py#L296-L304 ?
What is passed to decoder_input_ids in Conditional Generation models during generation when there is no input_ids but input_embeds?

Topic		Replies	Views
Using huggingface generate() with custom model 🤗Transformers	1	1041	August 26, 2023
Can one get an embeddings from an inference API that computes Sentence Similarity? Beginners	7	3517	November 2, 2023
Using model.generate() in parrellel / faster? Beginners	0	245	October 11, 2023
Run tensorflow transformer T5 model with huggingface generate() function return bad reply 🤗Transformers	2	494	January 4, 2023
What model-pairs are supported by the assistant decoding generation in Huggingface AutoModelForCausalLM? 🤗Transformers	1	85	March 13, 2024