I have an IterableDataset. It is finite and has a __len__() method. However, I cannot use it for more than 1 epoch in Trainerdue to the dataset has been iterated through and no more samples left. With num_train_epochs > 1 in TrainingArguments, after one epoch, I get the error ValueError: Batch does…

Thanks @John6666 . with_format() won’t work in my case because it is a method for the class datasets.IterableDataset while my class DataStreamer is inherited from the class torch.utils.data.IterableDataset. The solution that eventually worked was in the first page you cited (which I visited before…

Using an IterableDataset for 1+ epochs in Trainer

Beginners

John6666 January 1, 2025, 11:58pm 2

Maybe this?

Topic		Replies	Views
Using IterableDataset with Trainer - `IterableDataset' has no len() 🤗Transformers	7	13969	December 17, 2024
Using IterableDataset, Trainer never calls `set_epoch` to increment epoch 🤗Datasets	1	651	October 2, 2023
There seems to be not a single sample in your epoch_iterator, stopping training at step 0! This is expected if you're using an IterableDataset and set num_steps (5000000) higher than the number of available samples Beginners	2	1602	April 19, 2023
Is IterableDataset automatically reshuffled after each epoch in Trainer? Beginners	0	94	October 31, 2024
Training a Tokenizer on a Streamed Dataset Beginners	5	1308	May 30, 2023

Using an IterableDataset for 1+ epochs in Trainer

Related topics