Hi. I want to train LLaMA from scratch. I don’t want to finetune it, but I wanna try pretrain it from scratch and see how it goes. How could I do it? Thanks

Generally, you initialize the model with random weights as shown here and then train the model like any other. Except you can’t. To understand why, please check Table 1 and Table 15 in the LLaMa paper . However, I’d really like to hear back from you if you actually can train LLaMa from scratch. …

Can I pretrain LLaMA from scratch?

Beginners

mdroth August 9, 2023, 5:00pm 7

For those who want to build Llama from scratch, this article might be quite useful.

2 Likes

How does one reinitialize the weights of a Hugging Face LLaMA v2 model the official way as the original model?

Topic		Replies	Views
Failed to train Llama model Models	1	914	July 15, 2023
How to get a LLaMA v2 model with less than 7B parameters? Beginners	0	1591	August 24, 2023
Loading model from_pretrained with dummy parameter 🤗Transformers	0	338	April 16, 2023
Fine Tuning GGML model Beginners	3	3021	September 24, 2023
How does one reinitialize the weights of a Hugging Face LLaMA v2 model the official way as the original model? 🤗Transformers	4	1983	January 20, 2024

Can I pretrain LLaMA from scratch?

Related Topics