I an trying to fine Tune Gemma 7B IT for a dataset. The code is being run in kaggle. both an existing run and current fails with this error
‘TrainingArguments’ object has no attribute ‘model_init_kwargs’
My install block is
!pip install -q -U torch --index-url https://download.pytorch.org/whl/cu117
!pip install -q -U transformers==“4.38.2”
!pip install -q accelerate
!pip install -q -i Simple index bitsandbytes
!pip install -q -U datasets
!pip install -q -U git+https://github.com/huggingface/trl
!pip install -q -U git+https://github.com/huggingface/peft
parameter block is below (The exact same code was running two months ago !)
peft_config = LoraConfig(
lora_alpha=16,
lora_dropout=0,
r=64,
bias=“none”,
task_type=“CAUSAL_LM”,
target_modules=[“q_proj”, “k_proj”, “v_proj”, “o_proj”,
“gate_proj”, “up_proj”, “down_proj”,],
)
training_arguments = TrainingArguments(
output_dir=“logs”,
num_train_epochs=5,
gradient_checkpointing=True,
per_device_train_batch_size=1,
gradient_accumulation_steps=8,
optim=“paged_adamw_32bit”,
save_steps=0,
logging_steps=25,
learning_rate=2e-4,
weight_decay=0.001,
fp16=True,
bf16=False,
max_grad_norm=0.3,
max_steps=-1,
warmup_ratio=0.03,
group_by_length=False,
evaluation_strategy=‘steps’,
eval_steps = 112,
eval_accumulation_steps=1,
lr_scheduler_type=“cosine”,
report_to=“tensorboard”,
)
trainer = SFTTrainer(
model=model,
train_dataset=train_data,
eval_dataset=eval_data,
peft_config=peft_config,
dataset_text_field=“text”,
tokenizer=tokenizer,
max_seq_length=max_seq_length,
args=training_arguments,
packing=False,
)