How to add RNN layer on top of Huggingface BERT model

🤗Transformers
1

I am working on a binary classification task and would like to try adding RNN layer on top of the last hidden layer of huggingface BERT PyTorch model. How can I extract the layer-1 and contact it with LSTM layer?

tokenizer = BertTokenizer.from_pretrained(model_path)
# Load BertForSequenceClassification, the pretrained BERT model with a single linear classification layer on top.
model = BertForSequenceClassification.from_pretrained(model_path, num_labels=len(lab2ind))
2

We can use BertModel instead of BertForSequenceClassification

https://huggingface.co/transformers/model_doc/bert.html#bertmodel

And feeds hidden states output to LSTM

1 Like
3

Thank you for your reply. I am trying this now. But does add LSTM on top of Bert needs to train BERT from scratch?

4

You could use BertModel.from_pretrained and then LSTM :wink:

1 Like