Which parameters to use for fine-tuning Llama-2-7b?

Beginners
1

Hello,

I am attempting to fine-tune a Llama-2-7b model for a dialogue summarisation task. I would like to use PEFT LORA fine-tuning, but I am a bit lost in which parameters to use. My dataset contains about 4k examples.

Specifically:

  • Which rank (r) should I use?
  • What is the best lora_alpha? I have read that there exists a common practice to make it twice the rank?
  • Which target_modules should I select?
  • What is the best dropout value?
  • Which learning rate should be used? Is there a standard or is it experimental?

I have already read through the standard documentation for Llama and PEFT, but this does not clarify these parameters, it often only states “these are the parameters you want to adjust yourselves”. Can you provide answers or sources where it is better explained what parameters fit?

Thank you in advance!

1 Like
2

The optimal LoRA hyperparameters always change depending on the fine-tuning goals and model characteristics, so maybe they’re not documented…

Well, for popular models like Llama 2, I think it’s best to just grab the parameters from someone who successfully fine-tuned it.

1 Like