I am doing hyperparameter tuning as suggested in below link Hyperparameter Search using Trainer API (huggingface.co).
After tuning, when I try to train the model with best parameters I am not getting the same performance.
the best trial - Trial 2 finished with value: 0.8181818181818182 and parameters: {âweight_decayâ: 0.18182315978815689, âlearning_rateâ: 2.8534508921816434e-05, âwarmup_stepsâ: 83}. Best is trial 2 with value: 0.8181818181818182.
Training with best parameters -
model = AutoModelForSequenceClassification.from_pretrained(âdistilbert-base-uncasedâ, num_labels=2)
training_args = TrainingArguments(output_dir=âtest_trainerâ, evaluation_strategy=âepochâ,num_train_epochs=1,per_device_train_batch_size=32,per_device_eval_batch_size=32,weight_decay=0.18182315978815689, learning_rate= 2.8534508921816434e-05, warmup_steps=83)
trainer = CustomTrainer(
#model=model,
args=training_args,
train_dataset=tokenized_train_dataset,
eval_dataset=tokenized_test_dataset,
compute_metrics=compute_metrics,
model_init=model_init
)
trainer.train()
