Hi,
I am fine-tuning a Model for Sequence Classification i.e., I want to modify both the weights from the pre-trained language model and the weights from the classification head. Once I do that, I want to compare the embeddings from the freshly fine-tuned model and the ones from the original pre-trained model.
How can I retrieve the embeddings from the Model For Sequence Classification as it does not have any last_hidden_states features?
Cheers,
Giuseppe
I found a solution: it was simple but IMHO it was nowhere on the web clearly stated. You can save the model, then reload it and use it with hidden states.
trainer.save_model(working_data_path)
model = AutoModel.from_pretrained(working_data_path).to(device)
preds = model(**inputs, output_hidden_states=True)
The usual last_hidden_state can be retrieved from preds["hidden_states"][-1] .