Hi. I want to train LLaMA from scratch.
I don’t want to finetune it, but I wanna try pretrain it from scratch and see how it goes.
How could I do it?
Thanks
Hi. I want to train LLaMA from scratch.
I don’t want to finetune it, but I wanna try pretrain it from scratch and see how it goes.
How could I do it?
Thanks
Generally, you initialize the model with random weights as shown here and then train the model like any other.
Except you can’t. To understand why, please check Table 1 and Table 15 in the LLaMa paper. However, I’d really like to hear back from you if you actually can train LLaMa from scratch.
Thanks a lot
I just wanted to know how to pre-train LLaMA, and try it for fun.
I did not intended to actually use the custom pre-trained model.