How to obtain [CLS] embeddings from fine-tuned BERT model (using Transformers Trainer)

Beginners
1

I am sorry if this question has an obvious answer, I am a beginner and struggling to understand the inner workings of Transformers Trainer

I have seen many answers explaining how to obtain the [CLS] embeddings from a BERT model when writing your own training loop using Pytorch. In that case, the embeddings are the last hidden-state/first element of the output tuple, which one can obtained by doing the following:

model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("example sentence")).unsqueeze(0)
outputs = model(input_ids)
last_hidden_states = outputs[0]

I am having some trouble understanding how I can obtain these embeddings when fine-tuning a pretrained model with the Huggingface Transformers Trainer object. When using that, I create a trainer object as follows:

trainer = Trainer(

    model=model,

    args=training_args,

    train_dataset=small_train_dataset,

    eval_dataset=small_eval_dataset,

    compute_metrics=compute_metrics,

)

And finetune the model by calling

trainer.train()

Where can I find the [CLS] embeddings here?

Thank you for your suggestions

2

I think it’s the same? You pass the model object into the trainer and I believe it finetunes that model object as part of the train call. So that same model object should give you what you’re looking for

I think the below code from your post above should work.

input_ids = torch.tensor(tokenizer.encode("example sentence")).unsqueeze(0)
outputs = model(input_ids)
last_hidden_states = outputs[0]```