When trying to use EarlyStopping for Seq2SeqTrainer, e.g. patience was set to 1 and threshold 1.0:
training_args = Seq2SeqTrainingArguments(
output_dir='./',
num_train_epochs=3,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
logging_steps=1,
save_steps=5,
eval_steps=1,
max_steps=10,
evaluation_strategy="steps",
predict_with_generate=True,
report_to=None,
metric_for_best_model="chr_f_score",
load_best_model_at_end=True
)
early_stop = EarlyStoppingCallback(2, 1.0)
trainer = Seq2SeqTrainer(
model=model,
args=training_args,
train_dataset=valid_data.with_format("torch"),
eval_dataset=test_data.with_format("torch"),
compute_metrics=compute_metrics,
callbacks=[early_stop]
)
trainer.train()
The model continues training until max_steps
instead of stopping after the stopping criteria is met.
I’m not sure if this is a bug or maybe some argument is missing in when I use Seq2SeqTrainer, a working code to replicate the issue can be found on Huggingface EarlyStopping Callbacks | Kaggle
After the max_steps
, if we do some probing, somehow the early_stopping_patience_counter
has been reached but the training didn’t stop
>>> early_stop.early_stopping_patience_counter
2