That’s actually a mistake in the documentation, it should be “by shifting the labels” instead of “by shifting the input_ids”. Can you open a PR to fix this?
Sure, I will 
Seems like the implementation is correct
Yes, now everything makes sense, thank you!