Does Trainer.train repeat streaming dataset when max_steps is not reached?

tzuhsial · May 26, 2023, 6:39pm

I am using a streaming dataset streaming=True, and running preprocessing on the fly. I know the number of instances in the dataset. I am using Trainer and would like to iterate the streaming dataset 10 times by setting max_steps

Does the Trainer.train automatically repeats the streaming dataset if max_steps is not reached?

Topic		Replies	Views
TrainingArguments class - max_steps formula when using streaming dataset 🤗Transformers	1	2305	September 14, 2023
Explicitly set number of training steps using Trainer 🤗Transformers	5	5690	September 16, 2020
There seems to be not a single sample in your epoch_iterator, stopping training at step 0! This is expected if you're using an IterableDataset and set num_steps (5000000) higher than the number of available samples Beginners	2	1248	April 19, 2023
Trainer fails to resume training from a checkpoint, claiming there's not enough samples in the dataset Intermediate	1	1128	May 29, 2023
Trainer epoch does not go through all training data? Beginners	4	3086	January 22, 2021

Does Trainer.train repeat streaming dataset when max_steps is not reached?

Related Topics