@yuerlong fp16=True in training argument was giving me an error so I removed that argument.
FYI - Issue on Kosmos-2 model training on new dataset