I’ve fine-tuned several models with sft using the script here (trl/examples/scripts/sft.py at main · huggingface/trl · GitHub)
However, I’ve wanted to store my data in the “conversation” style and then explicitly set the chat-format in the training pipeline, using e.g. setup_chat_format (from Supervised Fine-tuning Trainer). But I’m having trouble using this. The sft script takes the modelname as a command line argument (see example below)
accelerate launch --config_file examples/accelerate_configs/multi_gpu.yaml --num_processes=1 examples/scripts/sft.py --model_name MYMODEL–dataset_name MYDATA --per_device_train_batch_size 2 --gradient_accumulation_steps 1 --learning_rate 2e-4 --save_steps 20_000 --use_peft --lora_r 16 --lora_alpha 32 --lora_target_modules q_proj k_proj v_proj o_proj --load_in_4bit --output_dir MYOUTPUTDIR
I initially though I could just modify the sft script by adding something like the following (from trl/docs/source/sft_trainer.mdx at main · huggingface/trl · GitHub)
from transformers import AutoModelForCausalLM, AutoTokenizer
‘’‘Load model and tokenizer’‘’
model = AutoModelForCausalLM.from_pretrained(MYMODEL)
tokenizer = AutoTokenizer.from_pretrained(MYMODEL)
‘’‘Set up the chat format with default ‘chatml’ format’‘’
model, tokenizer = setup_chat_format(model, tokenizer)
But this doesn’t work. I get the following error when trying to run the resulting modified sft script.
ValueError(“You passed model_kwargs to the SFTTrainer. But your model is already instantiated.”)
If I understand correctly the problem is my instantiating the model and applying the setup_chat_format to it while still having the model as a parameter of the sft script. Is there some way to get the effect of setup_chat_format while still just passing the model as a kwarg to the sft script? Should I use something like the formatting_func parameter of SFTTrainer to pass a chat formatter?
Thanks in advance for any insight!