Hi @mariosasko
I am actually having an idea why the loss would behave differently in streaming and non-streaming mode, it would be great if you could confirm please.
When I am training with streaming (i.e. iterable dataset), the logger only sees one epoch which is the chosen number of training steps.
Then I am afraid there is no reshuffling of the dataset during training … am I right ?
Question here is what is the best way to fix this please ?
Is there a place where I should configure the length of the dataset, which is known in advance in my case ?
Or should I make a callback every length dataset / batch size to manually shuffle the dataset ?