Make predictions with the Dropout on

Beginners
1

The default behavior of Trainer(...) when evaluating model is disabling Dropout. Concretely, y_pred for M runs will be exactly the same

for i in range(M):
    logits, labels, metrics = trainer.predict(tokenized_datasets["eval"])
    y_pred = np.argmax(logits, axis=2)
    ...

Now I am trying to apply Monte Carlo Dropout trick introduced this this answer. This requires to turn the Dropout on while making predictions on the validation set.

I am wondering how I achieve this goal. Any input is appreciated

2

I don’t think this is possible with the Trainer class as it is, but you can derive this class and then change the relevant methods.

In your case, I think you need to change the evaluation_loop method and delete the model.eval() line.

I think it would be better to keep the model.eval() line and set only the Dropout layer to train mode like it is shown in this post.

There might be some more changes that you need to make.
You can find the Trainer source code here.

Hope this helps :slight_smile: .