OutOfMemoryError: CUDA out of memory while trying to replicate this notebook on sagemaker: https://github.com/huggingface/notebooks/blob/main/sagemaker/24_train_bloom_peft_lora/sagemaker-notebook.ipynb

albiross · June 16, 2023, 8:49am

Hi guys, i am facing the same issue. I have set 'per_device_train_batch_size': 1 and restard the notebook instance but the problem still appear. Any suggestions ?
Thanks

Topic		Replies	Views
Distributed Training on Sagemaker Amazon SageMaker	13	2739	August 5, 2021
Sagemaker gpt-j train file error Amazon SageMaker	27	2913	February 22, 2024
"No space left on device" when using HuggingFace + SageMaker Amazon SageMaker	39	25739	October 10, 2023
Cuda memory error on unchanged workshop 1 notebooks Amazon SageMaker	1	791	December 1, 2021
Distributed Training run_summarization.py Amazon SageMaker	3	935	July 30, 2021

OutOfMemoryError: CUDA out of memory while trying to replicate this notebook on sagemaker: https://github.com/huggingface/notebooks/blob/main/sagemaker/24_train_bloom_peft_lora/sagemaker-notebook.ipynb

Related topics