Accelerator .prepare() replaces custom DataLoader Sampler

🤗Accelerate
1

I want to use a DataLoader that uses a custom sampler you can find at vision/references/classification/sampler.py at main · pytorch/vision · GitHub

When doing :

print(dataset, dataset.sampler)
dataset = accelerator.prepare(dataset)
print(dataset, dataset.sampler)

I get the following print:

<torch.utils.data.dataloader.DataLoader object at 0x7f987fbc4e50> <utils.imagenet_dataloader.RASampler object at 0x7f987fbc4e20>
<accelerate.data_loader.DataLoaderShard object at 0x7f98518b6da0> <torch.utils.data.sampler.SequentialSampler object at 0x7f98518b6b60>

This means that my RASampler got turned into a SequentialSampler.
Is this a normal behaviour? Since it seems I can’t manually restore my sampler afterwhile, this is quite a problem.
Could you tell me how to solve this problem?

2 Likes
2

Hi @HTess
Did you find a solution to this?

3

Yes. The original sampler is still there I think as well? Can you check sampler.sampler? Or sampler.batch_sampler

4

Hi @muellerzr

I wanted to use a custom sampler with my dataloader. Will the sampler behaviour remain the same after passing through accelerate.prepare() ? Or will it be changed to a SequentialSampler() ?

5

As mentioned, the custom sampler will be used. This new sampler is simply just distributing all of the batch across your GPUs. So it goes old sampler → new sampler to dispatch.

To test this, you can try including a print statement in your custom sampler and iterate after prepare

2 Likes
6

Hi! Is there a way to get the state of the custom sampler after .prepare() method was used?
I would like to save the state of the custom sampler, but I am not sure if it’s possible?

1 Like