Fixing the random seed in the Trainer does not produce the same results across runs

[sgugger]

For full reproducibility you need to instantiate your model inside the Trainer by using the model_init argument (or setting a seed before instantiating your model). You have random weights in your model head and those are different in your two runs in the code you show.