Can I use "AutoModel For Sequence Classification" class for generative models?

🤗Transformers
1

I want to use AutoModelForSequenceClassification with a llama 7b model
How will the input flow in the model if I load the model with this class ?

2

Yes it works, I used a GPT2-XL to do NLU tasks.
But it also depends on the weights you use whether it has a class called transformers.XXXForSequenceClassification, where XXX is the model class/ name.
Such as here: https://github.com/huggingface/transformers/blob/v4.33.2/src/transformers/models/gpt2/modeling_gpt2.py#L1376

3

Hi, did you try this? Does it give better results as compared to the BERT models? I am trying the same, but the model is overfitting on the training dataset.