Hello! I don’t have a solution for you, but Im running the exact same setup since I don’t have access to A100’s yet. Could you please read this and see if I am on the right track, because if I run into the error you are having I can assist you.
My post: DeepSpeed integration for HuggingFace Seq2SeqTrainingArguments