I’m fine-tuning the “decapoda-research/llama-7b-hf” model. How can I periodically write predictions to a file during training, where I can specify the frequency?
I have searched all over for suggestions, including setting the predict_with_generate=True parameter, or printing stuff from the compute_metrics function, but nothing has really worked.
Here is the trainer:
trainer = transformers.Trainer(
model=model,
train_dataset=train_data,
eval_dataset=val_data,
args=training_arguments,
data_collator=data_collator,
compute_metrics=compute_metrics,
)
And here is my TrainingArguments:
training_arguments = transformers.TrainingArguments(
per_device_train_batch_size=MICRO_BATCH_SIZE,
gradient_accumulation_steps=GRADIENT_ACCUMULATION_STEPS,
warmup_steps=100,
max_steps=TRAIN_STEPS,
learning_rate=LEARNING_RATE,
fp16=True,
logging_steps=10,
optim="adamw_torch",
evaluation_strategy="steps",
save_strategy="steps",
eval_steps=50,
save_steps=50,
output_dir=OUTPUT_DIR,
save_total_limit=4,
load_best_model_at_end=True,
report_to="tensorboard",
)