Ref_model in DPOTrainer

🤗Transformers
1

Hi, I’m using Mistral-7B-Instruct-v0.1 model with peft loaded in the DPOTrainer model.
now when I’m giving ref_model by creating a copy of this PeftLoaded model by create_reference_model() fn then the below piece of code is running infinitely and my Jupyter Lab got freezed. not responding anymore.
but by passing ref_model = None it’s running peacefully.

Now I want to know if I don’t pass the ref_model rather than passing, is there anything will be affected, I know DPOtrainer will create a copy on it’s own for a ref_model, but why passing ref_model is not working or is there any impact.

trainer = DPOTrainer(
    model = combined_model,
    ref_model=None,
    args=args,
    train_dataset=small_dataset,
    eval_dataset=eval_dataset,
    tokenizer=combined_tokenizer,
    beta=dpo_args["beta"],
    loss_type=dpo_args["loss_type"],
    max_length = 512,
    max_prompt_length = 128,
)