Can I pretrain LLaMA from scratch?

Beginners
1

Hi. I want to train LLaMA from scratch.
I don’t want to finetune it, but I wanna try pretrain it from scratch and see how it goes.
How could I do it?

Thanks

2

Generally, you initialize the model with random weights as shown here and then train the model like any other.

Except you can’t. To understand why, please check Table 1 and Table 15 in the LLaMa paper. However, I’d really like to hear back from you if you actually can train LLaMa from scratch. :metal:

1 Like
3

Thanks a lot
I just wanted to know how to pre-train LLaMA, and try it for fun.
I did not intended to actually use the custom pre-trained model. :grinning:

1 Like
4

Updated link to the LLaMA paper: https://arxiv.org/pdf/2302.13971.pdf