I have a finetuned RobertaForSequenceClassification model that I have already trained and saved to disk. I now want to load it, and instead of using it for classification tasks, extract the embeddings it generates and outputs, or “pooled/pooler output”. From my understanding, I can load the model using X.fromPretrained() with “output_hidden_states=True”. If I load the model using:
model = AutoModel.from_pretrained(model_dir, num_labels=2, output_hidden_states=True)
I get the warning about weights not being initialized: “roberta.pooler.dense.weight”, “roberta.pooler.dense.bias”, and understand they are randomly initialized on loading the model. Also from my understanding, I can still use this model to generate what I believe to be the pooler output by using something like:
pooler_output = model(input_ids, attention_mask=attention_mask)
Since the AutoModel does not load the weights/bias from the saved model, it leads to random results that I don’t want.
I’ve now read two closed issues [1, 2] that gave me some insight on how to generate this pooler output from XForSequenceClassification models. I don’t understand that from the first issue, the poster “concatenates the last four layers” by using the indices -4 to -1 of the output. In my mind this means the last index of the hidden state’s output of an XForSequenceClassification model is what is needed to retrieve pooler output. However, in the second link, and in the RobertaClassificationHead code, they use the hidden representation of the [CLS] token only, something like:
outputs = model(input_ids, attention_mask=attention_mask)
last_hidden_state = outputs[0]
cls_representation = last_hidden_state[:,0,:]
However, when I do this by loading and using a RobertaForSequenceClassification model and instead using outputs[1] (since I do not use labels for the forward function), all rows in the resulting Tensor are equal. I am not sure if this is intended, but if so, then how is this useful for classification if I want to use this as input into another classifier as mentioned by the poster of the second link and the RobertaClassificationHead code? I am not sure if I missed something or there’s something wrong with my trained model, but from what the first poster explains and how they take the last four layers, do I want to take outputs[-1][:,0,:] instead, or is outputs[0] actually the first embedding layer in XForSequenceClassification models rather than the final layer?