Distributed Training run_summarization.py

philschmid · July 29, 2021, 6:45am

Thanks for opening the thread. Happy to help you.
Could still share the full cloudwatch logs? sometimes the errors are a bit hidden.

I saw you changed the instance ml.p3dn.24xlarge to ml.p3.16xlarge and kept the batch_size this could be the issue. Could reduce the batch_size to 2 or change the instances type?

Topic		Replies	Views
Distributed Training on Sagemaker Amazon SageMaker	13	2728	August 5, 2021
Sagemaker gpt-j train file error Amazon SageMaker	27	2912	February 22, 2024
Multi Instance Training Error Amazon SageMaker	5	1585	October 29, 2021
OutOfMemoryError: CUDA out of memory while trying to replicate this notebook on sagemaker: https://github.com/huggingface/notebooks/blob/main/sagemaker/24_train_bloom_peft_lora/sagemaker-notebook.ipynb Amazon SageMaker	4	1687	June 16, 2023
Distributed training with Sagemaker 🤗Transformers	0	305	June 26, 2023

Distributed Training run_summarization.py

Related topics