Hi everyone! First of all, I’m new to huggingface, transformers and NLP in general. My question is in regards to fine-tuning.
I have a dataset of scientific documents in a csv (about 50,000) with titles and abstracts. I tried training it on a gpt2 model and even used methods like freezing the first layer, training it on five epochs then unfreezing all layers and fine-tuning it.
The problem is that once the training is completed, when I use it to generate text, it seems to “forget” everything else and only generate text that are solely related to my dataset. For instance, if I ask it to complete a query like “the weather outside is very sunny and…” it gives me very scientific answers as if it “forgot” everything from before I fine-tuned it with my dataset.
Is there any way around this? Sorry if its a stupid question but I’m very new to this and would love to learn more about how to use it the proper way