@yurii , thanks for the reply.
I think I confuse others by using the term “inference.” Here I am doing is to “forward” the model without using decoder_input_ids and labels, cause I’d like to compute some unsupervised loss on unlabeled data. Plus, I don’t want to break the auto-grad graph, so I think model.generate() is not a good choice for my case?
Could you show me where the code snippet or document about automatically doing the shift_to_right thing? I could’t find it myself. Thanks a lot.