PEFT LoRA GPT-NeoX - LoraConfig

🀗Transformers
1

I have written a training script that makes use of the Accelerate and PEFT libraries to finetune GPT-NeoX and while the training loop runs without error the script does not generate an adapter_config.json file. My intuition is that there is something about my LoraConfig object that is not properly parameterized resulting in a silent failure.

Does anyone have any suggestions on how to best parameterize LoraConfig for GPT-NEoX family of models?

My current LoraConfig:

    peft_config = LoraConfig(
        r=16,
        lora_alpha=32,
        lora_dropout=0.05,
        target_modules = ["query_key_value", "xxx"],
        bias="none",
        task_type=TaskType.CAUSAL_LM,
        fan_in_fan_out=False,
    )

My full training script can be found here.

2

Hello @eusip!
Thanks for the issue!
Indeed you need to slightly tweak the trainer to add a callback to properly save your Peft models, please have a look at what have been suggested in Incorrect Saving Peft Models using HuggingFace Trainer · Issue #96 · huggingface/peft · GitHub and let us know if this works!

1 Like
3

@ybelkada again you have saved the day! Thanks for your help!

1 Like