Hi Huggingface community, I encountered the “no space left on device” error when a sagemaker training job tries to download a large model from huggingface hub. To reproduce this issue, simply replace the model id with “EleutherAI/gpt-neox-20b” in this notebook . Both g5.8xlarge and p4d.24xlarge insta…

@philschmid Thanks for the suggestion. I found that I need to also set os.environ['HF_HOME'] = '/tmp' to make it work.

"no space left on device" when downloading a large model for the Sagemaker training job

Keertiraj July 18, 2024, 7:20am 5

I followed the above steps. I am still facing the error. I have opened a new topic for this issue: Llama3-8B-Instruct finetuning - No space left on device error

@philschmid thank you for your help

Topic		Replies	Views
"No space left on device" when using HuggingFace + SageMaker Amazon SageMaker	39	23568	October 10, 2023
SageMaker OS Error No Space Left On Device while trying to train Falcon40B Amazon SageMaker	3	1168	August 24, 2023
No space left on device when trying to run batch inference - HF not using EBS storage? Amazon SageMaker	2	972	February 3, 2022
"OS Errorr: No space left on device" when trying to load a trained model from S3 Amazon SageMaker	1	1090	December 28, 2023
Sagemaker endpoint - no space left on device with large models Amazon SageMaker	3	4373	February 21, 2023