I want to fine tune a
RobertaForSequenceClassification task on
microsoft/codebert-base model. This
microsoft/codebert-base model hasn’t been trained for Sequence-Classification task.
Can I load this pre-trained model inside a SequenceClassification function and fine tune it on my dataset?
model = RobertaForSequenceClassification.from_pretrained( "microsoft/codebert-base" ) Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at microsoft/codebert-base and are newly initialized: ['classifier.dense.weight', 'classifier.out_proj.weight', 'classifier.out_proj.bias', 'classifier.dense.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
While loading I get this message which is expected as the model isn’t trained on the task and thus would not have the weights.
Can I proceed with fine-tuning this
RobertaForSequenceClassification model or would I need to define my own classifier layer on top of
RobertaModel and train that?