Fine-tune BERT and Camembert for regression problem

Beginners
11

looks like the model is initialized incorrectly, for regression we need use num_labels=1, and you can do it using two ways

config = BertConfig.from("...", num_labels=1)
model = BertForSequenceClassificatio.from_pretrained("...", config=config)

or

model = BertForSequenceClassificatio.from_pretrained("...", num_labels=1)

creating the model from config and the again using from_pretrained will override the config params. So in your code the model still has num_labels=2

The first is it seems odd that torch.LongTensor would make sense for a regression problem

Yes, the doscstring should be corrected. But you can still pass float tensor for a regression problem.

2 Likes