I want to further pre-train GPT2 in some few specific texts and the library provides scripts for this.
However, I found the model in the scripts belong to AutoModelForCausalLM, and GPT2 in my case is LMHead. Can this model be used for continuing pre-training as well?
If it can should I use the script or the model?
Thanks a lot!