Hello, I was wondering if it’s possible to finetune a language model on a corpus for masked language modelling or causal language modelling for the purpose of feature extraction (i.e. embeddings).
I know I can finetune on a corpus for a specific task (like MLM or CLM) but to my understanding that only makes changes in the final layer of the model. I’m trying to pull the last hidden state of a model after embedding encoded text inputs. If I finetune and then pull the last hidden state that would not be any different from doing that to the pretrained model. Is it possible to unfreeze the last two layers of a language model and then finetune it?