Monitoring GPU usage & utilization (nvidia-smi)

oran-sh · August 20, 2023, 2:58pm

Hello,
I’m running training on SM with HuggingFace estimator.
How can I run ‘nvidia-smi’ every ~1minute to monitor GPU usage and utilization?
Thanks!

Desjajja · August 31, 2023, 7:24am

watch -n60 'nvidia-smi'

oran-sh · August 31, 2023, 10:51am

Where do I insert this command?

Topic		Replies	Views
API Scaling algoritms and configuration Inference Endpoints on the Hub	2	1121	September 30, 2022
Why does all my gpu memory get used with a small model? Beginners	5	1667	March 13, 2022
Accelerate on single GPU doesnt seem to work Beginners	2	2825	May 16, 2023
Out of Memory error with multi-gpu training but no error with just one gpu? Amazon SageMaker	0	306	December 12, 2023
Accelerate sees only one GPU on multi-GPU Sagemaker instance 🤗Accelerate	1	1058	May 2, 2023