Train llama on domain specific dataset and on instruction format dataset

Models
2

I was thinking about that for Polish. Few thoughts:

  1. I assume you mean fine-tuning with LoRA and not the llama2 itself

  2. If Llama2 has not been trained on significant portion of a foreign language then fine-tuning a chat model with LoRA might not help a lot - I recently read a paper that says exactly this, don’t remember where.

  3. People still doing this, not sure how successful, for example: davidkim205/komt-Llama-2-13b-hf-lora · Hugging Face

  4. If you want to train base model (a not a chat model) to include more Bulgarian, then you have to go through RLHF, which is difficult and costly.

1 Like