Find_unused_parameters parameter to Huggingface SM Estimator not doing anything?

[jahmanson]

I’m having the same issue (except with XLNet and the DataCollatorForPermutationLanguageModeling). Still haven’t resolved it. Any progress on your end?

XLNet pre-training fails with multiple GPUs on Sagemaker

I’m initializing the run_plm example on Sagemaker from my local machine but getting the following error message: RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument find_unused_parameters=True to torch.nn.parallel.DistributedDataParallel Seems like I’m setting that flag using the “ddp…