Training on Domain specific Dataset

Beginners
1

Hi ,
I want to train a sentiment analysis multi-label classifier and in addition to training the final output layers, I’d like to train the hidden BERT layers as well.
I want to understand how much improvement can I get in my metric(F1 score) by feeding it my domain-specific data.
All the documents /references I have seen thus far only point to training the final output layer that generates classification. Is there a way to train various hidden layers of BERT using (let’s say) BERT Base?
Thanks in advance,
Devesh

1 Like
2

My understanding is that if you don’t specifically freeze any of the layers you will always train the whole model.

If you want to train only particular layers, you can add a condition to this code:

model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

for param in model.bert.parameters():
    param.requires_grad = False
3

Hi @neuralpat ,
Thanks so much for taking the time and responding to my queries.I’m going to try it out.
I am also wondering if there is a way to freeze specific layers(eg. Top 3 ) and train ?

4

Hi,

model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

for param in model.bert.parameters():
    param.requires_grad = False

as far I understand, the code above is for ALL params
but you could easily limit to 3 elements :

model.bert.parameters() is a generator

i=0
for i,el in enumerate(model.parameters()):
  if i<3:
    el.requires_grad=False
  else:
    el.requires_grad=True

You can check the result with the result with:

for param in model.parameters():
    print(param)