I am training my model with deepspeed
using hugging face Trainer API. I was wondering how Trainer/Accelerate
handles logging of loss
/ compute_metrices
. My observation is that deepspeed
creates separate processes per gpu for training and each process has its own Trainer. With this observation I have the following hypothesis:
The
acceleare
will gather all the losses andavg
them in one of the processes before logging them in tensor board.
So my questions are:
- Is this hypothesis correct, or do I need additional handling for reporting if I am using
deepspeed
? - Is this hypothesis valid during evaluation phase as well?
- Is this hypothesis valid for
compute_metrics
function as well? If not, how do I write process-safecompute_metrics
function to report global statistics?
Thanks.