I suspect I have some configuration problem that causes no gain from a dual GPU setup.
I have a 4080 and a 3090 which have a similar number of CUDA cores, so I’ve expected some gain training Bert Large (toy example).
I’ve tested with a relatively small training size of 10000 sampled from the Yelp review dataset.
What I’ve found 4080 alone is the fastest 54.449 samples/sec (batch size manually set to 18 which was the max fit to vram).
Second fastest 4080 and 3080 together 51.828 samples/sec (batch size manually set to 18)
Slowers 3090 alone 43.912 samples/sec (batch size manually set to 32).
I use multiple loader workers (set 20 without any deeper consideration), pin to memory.
Can this be caused by some sort of overhead and the relatively small training size? (Note: the tests I’ve done haven’t used random seed to sample the training data not sure it that can be responsible for a consistent lower performance with dual GPU)
Any specific setting in training args, I have to consider?
Additional note. With dual GPUI training I not see the 3090 can pull the same wattage than 3090 alone (270W vs 340W). Power supply 1600W EVGA.