I have written a training script that makes use of the Accelerate and PEFT libraries to finetune GPT-NeoX and while the training loop runs without error the script does not generate an adapter_config.json file. My intuition is that there is something about my LoraConfig object that is not properly parameterized resulting in a silent failure.
Does anyone have any suggestions on how to best parameterize LoraConfig for GPT-NEoX family of models?
My current LoraConfig:
peft_config = LoraConfig( r=16, lora_alpha=32, lora_dropout=0.05, target_modules = ["query_key_value", "xxx"], bias="none", task_type=TaskType.CAUSAL_LM, fan_in_fan_out=False, )
My full training script can be found here.