I am trying to train Vision Encoder Decoder with VIT encoder and Hindi GPT2( surajp/gpt2-hindi at main ) decoder, for Hindi Image captioning, which my team are doing as a part of Huggingface course project. Currently my code is this For creating a dataset import torch from torch.utils.data import D…

Hi, You were already on the good way! The only “mistake” I see here is that GPT2 doesn’t have a CLS token. The CLS token is only defined for encoder-only Transformers such as BERT, RoBERTa. So in this case, the decoder start token can be set to the bos (beginning of sequence) token: model.config.d…

Error Training Vision Encoder Decoder for Image Captioning

Intermediate

nielsr November 20, 2021, 9:21am 4

I’ve updated my previous comment. It might make more sense to use the BOS token instead of the padding token as decoder start token.

This was also done for this demo.

2 Likes

Topic		Replies	Views
Finetuning Vision Encoder Decoder Models with huggingface causes ValueError: expected sequence of length 11 at dim 2 (got 12) Beginners	0	493	March 12, 2023
Unable to use model.generate for Vision encoder decoder model Beginners	3	1127	March 6, 2024
Using EncoderDecoderModel 🤗Transformers	4	1075	October 28, 2021
FlaxVisionEncoderDecoderModel decoder_start_token_id Beginners	1	453	January 13, 2022
How can i implement custom model to use Seq2SeqTrainer class 🤗Transformers	0	444	November 8, 2023

Error Training Vision Encoder Decoder for Image Captioning

Related topics