PEFT LoRA GPT-NeoX - LoraConfig

I have written a training script that makes use of the Accelerate and PEFT libraries to finetune GPT-NeoX and while the training loop runs without error the script does not generate an adapter_config.json file. My intuition is that there is something about my LoraConfig object that is not properly parameterized resulting in a silent failure.

Does anyone have any suggestions on how to best parameterize LoraConfig for GPT-NEoX family of models?

My current LoraConfig:

    peft_config = LoraConfig(
        target_modules = ["query_key_value", "xxx"],

My full training script can be found here.

Hello @eusip!
Thanks for the issue!
Indeed you need to slightly tweak the trainer to add a callback to properly save your Peft models, please have a look at what have been suggested in Incorrect Saving Peft Models using HuggingFace Trainer · Issue #96 · huggingface/peft · GitHub and let us know if this works!

1 Like

@ybelkada again you have saved the day! Thanks for your help!

1 Like