@aclifton314 many thanks for your reply. Sorry, my result from multiple GPU by using Trainer API is very strange in comparison with using 1 GPU, would you please share a sample of code that u used multiple GPU and results was reasonable? many thanks.