[Maybe Bug] When using EarlyStopping Callbacks with Seq2SeqTraininer, training didn't stop

🤗Transformers DeepSpeed
#1

When trying to use EarlyStopping for Seq2SeqTrainer, e.g. patience was set to 1 and threshold 1.0:

training_args = Seq2SeqTrainingArguments(
    output_dir='./',
    num_train_epochs=3,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    logging_steps=1,
    save_steps=5,
    eval_steps=1,
    max_steps=10,
    evaluation_strategy="steps",
    predict_with_generate=True,
    report_to=None,
    metric_for_best_model="chr_f_score",
    load_best_model_at_end=True
)

early_stop = EarlyStoppingCallback(2, 1.0)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=valid_data.with_format("torch"),
    eval_dataset=test_data.with_format("torch"),
    compute_metrics=compute_metrics,
    callbacks=[early_stop]
)

trainer.train()

The model continues training until max_steps instead of stopping after the stopping criteria is met.

I’m not sure if this is a bug or maybe some argument is missing in when I use Seq2SeqTrainer, a working code to replicate the issue can be found on Huggingface EarlyStopping Callbacks | Kaggle

After the max_steps, if we do some probing, somehow the early_stopping_patience_counter has been reached but the training didn’t stop

>>> early_stop.early_stopping_patience_counter
2
Problem with EarlyStoppingCallback
#2

Also asked on python - Why did the Seq2SeqTrainer not stop when the EarlyStoppingCallback criteria is met? - Stack Overflow

#3

Found the issue, when the save_state is not in pace with the eval_steps the patience and threshold for the early stopping works in a different way, where the patience will kick in after the first save_state is met.

This might be a feature / bug.

Setting the eval_steps to be the same as the save_state would return the early stopping behavior as we intuitively expect.

Details on python - Why did the Seq2SeqTrainer not stop when the EarlyStoppingCallback criteria is met? - Stack Overflow