“No space left on device” when using Robust Speech Challenge + AzureML

Beginners
1

Hi there

I have been trying to train a model as per the [Robust Speech Event](https://github.com/neelan-elucidate-ai/transformers/tree/main/examples/research_projects/robust-speech-event)

Training begins but seems to fill up /dev/sdb1 location which has a total of 119 GB. This happens even when i force the cache to be on a different drive as suggested here : both manually in the functions and via env var.

This happens regardless of the dataset used (at fist I thought it was an issue with "load_dataset("mozilla-foundation/common_voice_7_0", "en")

I suspect the very large model and data need a very big tmp location but it should be clearing the tmp not filling it up.

Any ideas @patrickvonplaten @osanseviero @anton-l ?

2

seems it was the output dir not the cache_dir causing the issue. seem to be running now after changing output dir

1 Like