I am training GPT-J with DeepSpeed. The training loss kept decreasing, which is good, but the validation loss starts climbing as early as 2 epochs. However, the oddest thing is that the training loss drop and validation loss jump seem synchronized, and always happen at the boundary of each epoch. It…

Why my training loss drops at epoch boundaries?

AndreaSottana September 16, 2022, 11:15am 2

Hi @dunalduck0 are you using TensorFlow or PyTorch out of curiosity?
I know this post is old, but did you manage to find an answer in the end? I’m facing a similar issue

Topic		Replies	Views
Training loss increases suddenly at the beginning of each epoch 🤗Transformers	1	1621	March 29, 2022
Trainer's step loss always drops sharply after each epoch regardless of model / data 🤗Transformers	3	2082	March 28, 2023
The training loss(logging steps) will drop suddenly after each epoch? Help me plz! Orz 🤗Transformers	1	1159	August 26, 2022
Loss behaviour for bert fine-tuning on QNLI Models	3	4367	October 15, 2021
[trainer] 'train_loss' different from 'loss' 🤗Transformers	4	4589	March 31, 2023

Why my training loss drops at epoch boundaries?

Related topics