Loss becoming nearly zero in first 5K steps when training LM from scratch

🤗Transformers
1

I am training the ALBERT LM model from scratch.

I have already trained it for Hindi and Bangla and it was working fine but when I am training on Gujarati and Telugu, the loss is becoming zero in 5K steps.

What could be the reason for the sudden drop in the loss? Can anyone suggest what could be the cause or how to debug such issue?
Any suggestions?

1 Like
2

Usually, this means you are training on your validation data, so I’d triple-check your training set and validation set don’t contain the same texts, or that there is no leak from one to the other.

3

I am not using any validation data. I have masked the raw corpus after sentencpiece tokenization and put it to training.

4

Then it’s just the model learning your training set. If you really want to know how it would fare on new data, you need to use a validation set and compute the loss on it.

5

Have used eval set for hindi, And it was taking lot of time to for evaluation so I went without eval data.

6

But without eval data you don’t know how your model actually perform. You may be overfitting the training dataset

7

But without eval data you don’t know how your model actually performs. You may be overfittumg

8

agreed. But what can be cause of sudden drop in loss?

9

hi, have you resolved your issue. I have a similar problem

10

@008karan @dinesh I have a similar problem (here the post). Have you resolved your issue?

@sgugger I’m using validation set too, but train loss becames zero in 4 epochs.I tried to continue the training with others 4 epochs with lower learning rate, but the train loss starts at zero, so I have no improvement.

11

Hi, I encounter a similar problem. :dizzy_face:
Have you guys fixed it? Any help?