Hi, i have tried to combine the ViT (BeiT weights 16patch-384) as Encoder with a Bert Model as Decoder. (Like Microsofts new arxiv TrOCR paper ) [u] If i use the EncoderDecoderModel it does not support pixel_values for the encoder. feat_extractor = ViTFeatureExtractor.from_pretrained("microsof…

Hi, EncoderDecoderModel is meant to combine any bidirectional text encoder (e.g. BERT) with any autoregressive text decoder (e.g. GPT2). We’re planning to add a VisionEncoderDecoderModel (recently we’ve added SpeechEncoderDecoderModel , which allows you to combine any speech autoencoding model such …

Using EncoderDecoderModel

🤗Transformers

Felix92 October 28, 2021, 2:42pm 5

@nielsr
yes, I’ve just looked at your changes However, I still have the same problems with model.generate () in relation to the validation step and the possibility of exporting the model to ONNX

PS: i have removed the decoder_input_ids after your push today but this notebook is not up to date

Topic		Replies	Views
Unable to use model.generate for Vision encoder decoder model Beginners	3	951	March 6, 2024
EncoderDecoderModel generate text for a ViT as encoder 🤗Transformers	0	620	November 27, 2021
EncoderDecoderModel for Machine Translation 🤗Transformers	0	440	May 21, 2022
EncoderDecoderModel converts classifier layer of decoder Beginners	2	517	October 26, 2021
Combining encoder from one model and a decoder for another for image reconstruction Beginners	0	331	December 15, 2022

Using EncoderDecoderModel

Related topics