Hi, i have tried to combine the ViT (BeiT weights 16patch-384) as Encoder with a Bert Model as Decoder. (Like Microsofts new arxiv TrOCR paper ) [u] If i use the EncoderDecoderModel it does not support pixel_values for the encoder. feat_extractor = ViTFeatureExtractor.from_pretrained("microsof…

Hi, EncoderDecoderModel is meant to combine any bidirectional text encoder (e.g. BERT) with any autoregressive text decoder (e.g. GPT2). We’re planning to add a VisionEncoderDecoderModel (recently we’ve added SpeechEncoderDecoderModel , which allows you to combine any speech autoencoding model such …

Using EncoderDecoderModel

🤗Transformers

nielsr October 28, 2021, 2:23pm 4

The VisionEncoderDecoderModel class is now available (as well as TrOCR): Vision Encoder Decoder Models — transformers 4.12.0.dev0 documentation

Topic		Replies	Views
VisionEncoderDecoder/TrOCR Models	0	703	October 21, 2021
Img2seq model with pretrained weights Beginners	7	1223	November 18, 2021
EncoderDecoderModel generate text for a ViT as encoder 🤗Transformers	0	623	November 27, 2021
How to implement custom vision encoder-decoder? 🤗Transformers	1	695	August 1, 2023
Replacing the decoder of an xxxEncoderDecoderModel 🤗Transformers	2	1701	December 16, 2023

Using EncoderDecoderModel

Related topics