I want to use EncoderDecoderModel.from_pretrained() where time-series-transformer is the encoder and gpt2 as decoder

Intermediate
1

I want to use EncoderDecoderModel.from_pretrained() where huggingface/time-series-transformer-tourism-monthly is the encoder and openai-community/gpt2 as decoder but as mentioned in Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models, TS transformer does not have a tokenizer and hence not sure what to use as encoder input ids

2

Hi,

The Time-Series Transformer itself already has an encoder-decoder architecture, and is not supported by EncoderDecoderModel. This class is only meant to stitch together text encoder-only models with text decoder-only models (and encoder-only models like BERT which have support to be used as decoder).

3

@nielsr But I saw a function called get_encoder() in the transformer encoder-decoder modelling files.

4

Kalsel Pintar - To use EncoderDecoderModel.from_pretrained(), you can load the Transformer model for the encoder using TimeSeriesTransformer.from_pretrained() and the GPT-2 model for the decoder using GPT2LMHeadModel.from_pretrained(). Then, you can combine them using EncoderDecoderModel.from_encoder_decoder_pretrained(). Here’s the code example:

from transformers import TimeSeriesTransformer, GPT2LMHeadModel, EncoderDecoderModel

# Load pre-trained encoder (TimeSeriesTransformer)
encoder_model = TimeSeriesTransformer.from_pretrained("your_time_series_transformer_model_name")

# Load pre-trained decoder (GPT-2)
decoder_model = GPT2LMHeadModel.from_pretrained("gpt2")

# Combine encoder and decoder
model = EncoderDecoderModel.from_encoder_decoder_pretrained(encoder_model, decoder_model)

Make sure to replace "your_time_series_transformer_model_name" with the name of the TimeSeriesTransformer model you want to use.