In the introduction about llama, it says that llama2 was trained on bfloat16. The model weight uploaded to Hugging Face is on float16. I’m wondering why the current work based on llama2 loads it on float16 other than bfloat16.
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Why are some weights FP32 in Llama 3.1 405B FBGEMM FP8 Quantization? | 7 | 371 | September 27, 2024 | |
| How to Load Llama-3.3-70B-Instruct Model in Float8 Precision? | 1 | 191 | December 11, 2024 | |
| How does one reinitialize the weights of a Hugging Face LLaMA v2 model the official way as the original model? | 4 | 3994 | January 20, 2024 | |
| How to choose std for weight init for llama 2 after reinitialize? | 0 | 453 | January 19, 2024 | |
| Model Size Mismatch | 0 | 284 | May 11, 2024 |