Additional random tqdm progress bars while Training

Beginners
1

Hi,

Suddently, I started getting additional progress bars while training. Here is the snapshot. I realized that I am getting train_batch_size (8 in this case) bars between every training step update progress bar. It only started from today. All these additional print statements are drastically slowing down the training and the notebook is getting hanged. Any idea how to fix this?

I have disabled tqdm in Trainer but the main tqdm is getting disabled but not these 8 additional bars between each step.

Thanks.

image
image1275×680 160 KB

2

Is this on TPU?

3

I met the same problem. Not in TPU, I tried in GPU or CPU separately, both were the same.

4

I downgraded ‘datasets’ from 1.12.0 to 1.11.0, problem was solved.

3 Likes
5

Oh thanks for helping us find the reason!

6

There should be a patch release of Datasets fixing this issue soon :slight_smile:

1 Like
7

Thanks for the responses. @yechenzhi downgrading datasets worked. :smiley: Thank you :smiley: