Get embedding from finetuned BertForSequenceClassification model

🤗Transformers
1

How do I get the embedding tensors from my fine-tuned BertForSequenceClassification model? The issue is ‘BertForSequenceClassification’ object has no attribute ‘embeddings’.

However, I can successfully get the BERT embedding for use in Tensorboard’s Embedding Projector tool with this code:

import transformers
from transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoConfig
from torch.utils.tensorboard import SummaryWriter

import tensorflow as tf
import tensorboard as tb

# https://github.com/pytorch/pytorch/issues/30966#issuecomment-582747929
tf.io.gfile = tb.compat.tensorflow_stub.io.gfile

writer = SummaryWriter(log_dir='tensorboard_embedding_custom2')

model = transformers.BertModel.from_pretrained('bert-base-uncased')

tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')
words = tokenizer.vocab.keys()
word_embedding = model.embeddings.word_embeddings.weight
writer.add_embedding(word_embedding,
                     metadata  = words,
                     tag = f'word embedding')


writer.flush()
writer.close()
2

Hello! :wave:

For BertForSequenceClassification you can do almost exactly what you did with BertModel:

>>> model = BertForSequenceClassification.from_pretrained("bert-base-uncased")
>>> model.bert.embeddings.word_embeddings.weight
Parameter containing:
tensor([[-0.0102, -0.0615, -0.0265,  ..., -0.0199, -0.0372, -0.0098],
        [-0.0117, -0.0600, -0.0323,  ..., -0.0168, -0.0401, -0.0107],
        [-0.0198, -0.0627, -0.0326,  ..., -0.0165, -0.0420, -0.0032],
        ...,
        [-0.0218, -0.0556, -0.0135,  ..., -0.0043, -0.0151, -0.0249],
        [-0.0462, -0.0565, -0.0019,  ...,  0.0157, -0.0139, -0.0095],
        [ 0.0015, -0.0821, -0.0160,  ..., -0.0081, -0.0475,  0.0753]],
       requires_grad=True)
1 Like