How to use models in tranformers?

Beginners
1

I am very new to using transformers. I have just started using them, and I require them for a classification task, Where I have tweets from topics, and I need to find their hate score. I have been suggested to use the GroNLP/hateBERT · Hugging Face model to do so.

But, I was trying it out by the following code:

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("GroNLP/hateBERT")

model = AutoModelForMaskedLM.from_pretrained("GroNLP/hateBERT")

Then I used the following code:

from transformers import pipeline

classifier = pipeline('text-classification', model=model, tokenizer=tokenizer)

But I am getting an error:
The model 'TFBertLMHeadModel' is not supported for text-classification. Supported models are ['TFAlbertForSequenceClassification', 'TFBertForSequenceClassification', 'TFCamembertForSequenceClassification', 'TFConvBertForSequenceClassification', 'TFCTRLForSequenceClassification', 'TFDebertaForSequenceClassification', 'TFDebertaV2ForSequenceClassification', 'TFDistilBertForSequenceClassification', 'TFElectraForSequenceClassification', 'TFFlaubertForSequenceClassification', 'TFFunnelForSequenceClassification', 'TFGPT2ForSequenceClassification', 'TFGPTJForSequenceClassification', 'TFLayoutLMForSequenceClassification', 'TFLayoutLMv3ForSequenceClassification', 'TFLongformerForSequenceClassification', 'TFMobileBertForSequenceClassification', 'TFMPNetForSequenceClassification', 'TFOpenAIGPTForSequenceClassification', 'TFRemBertForSequenceClassification', 'TFRobertaForSequenceClassification', 'TFRoFormerForSequenceClassification', 'TFTapasForSequenceClassification', 'TFTransfoXLForSequenceClassification', 'TFXLMForSequenceClassification', 'TFXLMRobertaForSequenceClassification', 'TFXLNetForSequenceClassification'].

Can someone suggest how to modify this code so I can use the hateBERT model to find the hateful score? And any resources to learn about using any generic models would be very helpful.

2

Hello and welcome @selfisheagle! :hugs:

You’re getting an error because you loaded a model with AutoModelForMaskedLM instead of AutoModelForSequenceClassification, which is what you want for your classification task:

from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

tokenizer = AutoTokenizer.from_pretrained("GroNLP/hateBERT")
model = AutoModelForSequenceClassification.from_pretrained("GroNLP/hateBERT")
classifier = pipeline("sentiment-analysis", model=model, tokenizer=tokenizer)

For more resources, check out the docs about how to use AutoModel to load a model for a task or Chapter 2 of the Hugging Face course.

3

Thanks for your response.
When I ran the code you gave by replacing the sentiment-analysis part with text classification, I thought it would give hate vs no hate classification. But instead, I get the response.

Some weights of the model checkpoint at GroNLP/hateBERT were not used when initializing BertForSequenceClassification: [‘cls.predictions.decoder.weight’, ‘cls.predictions.transform.LayerNorm.bias’, ‘cls.predictions.transform.LayerNorm.weight’, ‘cls.predictions.transform.dense.bias’, ‘cls.predictions.decoder.bias’, ‘cls.predictions.transform.dense.weight’, ‘cls.predictions.bias’]

  • This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
  • This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
    Some weights of BertForSequenceClassification were not initialized from the model checkpoint at GroNLP/hateBERT and are newly initialized: [‘classifier.weight’, ‘classifier.bias’]
    You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
    So, do I have to train still the model for hate vs no hate classification, I thought hateBERT would already have done something along those lines, and I don’t have to pre-train the model.