I am currently using a modified scibert model that is finetuned using masked language modelling via a large corpus and new tokens are also added. I am planning to use this model as a embedding layer for my another model. Currently I get this warning.
Some weights of BertModel were not initialized from the model checkpoint at athiban2001/cord-scibert and are newly initialized: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
I am only going to using this bert model as a embedding layer only. So do I need to worry about this warning for extracting embeddings for a sub tokens?