I have a huggingface dataset and I want to make a dataloader from it, which is 1) infinite 2) shuffles the data. I tried with this version, but this does not work with accelerator, and making the infinite loop with generator loop, accerelator does not modify the types automatically. How should I write this?
dataloader = DataLoader( dataset, batch_size=batch_size, collate_fn=data_collator, num_workers=data_args.dataloader_num_workers, pin_memory=True) while True: for batch in dataloader: yield batch
thanks @lhoestq for any suggestion on this.