When resuming a run, the dataloader restarts from the beginning. Is there a simple way to make it resume from the index at which we finished the previous run?
No, that’s not baked in in Accelerate, could be a nice feature request (but doing it efficiently will require a bit of time ). For now you should manually skip the first batches if it’s really important to you (if you’re using iterable datasets, we won’t be able to do anything more efficient anyway).
1 Like