When using seq2seq finetune.py I am getting this warning:
wandb: WARNING Step must only increase in log calls.
and because of this issue, it is not reporting all the stats in my wandb console.
I found this which seems to be relevant: https://github.com/wandb/client/issues/1530
Here is a more complete log:
Epoch 1: 50%|█████ | 1/2 [00:00<00:00, 2.62it/s, lowandb: WARNING Step must only increase in log calls. Step 1 < 3; dropping {'val_avg_loss': 3.4621164798736572, 'val_avg_rouge1': 0.0, 'val_avg_rouge2': 0.0, 'val_avg_rougeL': 0.0, 'val_avg_rougeLsum': 0.0, 'val_avg_gen_time': 0.008627939224243163, 'val_avg_gen_len': 3.0, 'step_count': 3, 'epoch': 1}.
Epoch 2: 50%|█████ | 1/2 [00:00<00:00, 2.65it/s, lowandb: WARNING Step must only increase in log calls. Step 2 < 4; dropping {'val_avg_loss': 2.1513829231262207, 'val_avg_rouge1': 19.3801, 'val_avg_rouge2': 8.7686, 'val_avg_rougeL': 18.0454, 'val_avg_rougeLsum': 19.2525, 'val_avg_gen_time': 0.11329512596130371, 'val_avg_gen_len': 66.0, 'step_count': 4, 'epoch': 2}.
Epoch 3: 50%|█████ | 1/2 [00:00<00:00, 2.61it/s, lowandb: WARNING Step must only increase in log calls. Step 3 < 5; dropping {'val_avg_loss': 1.903280258178711, 'val_avg_rouge1': 27.2808, 'val_avg_rouge2': 11.1652, 'val_avg_rougeL': 23.9808, 'val_avg_rougeLsum': 27.4063, 'val_avg_gen_time': 0.10656294822692872, 'val_avg_gen_len': 61.0, 'step_count': 5, 'epoch': 3}.
Epoch 4: 50%|█████ | 1/2 [00:00<00:00, 2.71it/s, lowandb: WARNING Step must only increase in log calls. Step 4 < 6; dropping {'val_avg_loss': 1.3527400493621826, 'val_avg_rouge1': 13.7018, 'val_avg_rouge2': 7.5888, 'val_avg_rougeL': 11.3654, 'val_avg_rougeLsum': 11.3237, 'val_avg_gen_time': 0.20975174903869628, 'val_avg_gen_len': 121.0, 'step_count': 6, 'epoch': 4}.
Any thoughts on this?