After quite a bit of digging, I realized that when I call
dataloader, model = accelerator.prepare( dataloader, model )
even_batches hyperparameter defaults to True, and this is expected behavior in that case. That is, my last batch cannot be split evenly over the number of devices, so a few samples are taken from the start of the dataset. However:
When I change my accelerator with
accelerator = Accelerator(even_batches=False) this does not do anything! I can confirm that
even_batches is set to False, and I get no warning about this.
Digging a bit more, the
dataloader returned by prepare is actually a
DataLoaderDispatcher, which does not seem to care about the
even_batches argument. I can see here that it simply rounds up the batch size, and here that it then samples from the first batch.
So, how can I get the accelerated dataset to stop doing this? Can the docs be updated to reflect the fact that setting
even_batches to False does nothing (I suspect with certain buried dataset types, etc. it does something, not sure though)?