Learning rate for the `Trainer` in a multi gpu setup

Not necessarily, because it’s a huristic that people recommend to do so, but it’s also recommended to test yourself at your discretion.

What’s really happening is the number of steps increases that we’re stepping the learning rate, so if you want the same LR from situation A to B you should try multiplying the learning rate.

However again: test yourself first. Sometimes it’s not necessary