Hi, I’m trying to improve my understanding of the transformer and surely details matter.
I was wondering how HF handled the shifted outputs to satisfy (along with the mask) the auto-regressive property.
Assume the inputs are (x_0,x_1,...x_T) and the outputs are (y_0,y_1,...,y_T). As we want to train an auto-regressive-like model, we wish that pred(y_k)= f(x_0,...,x_T, y_0,...y_{k-1}), a simple way to do this is to shift the elements to the right by one and mask elements that are now to the right of each token.
I can think of 2 ways to shift the outputs to the right by one :
- (<SHIFT>, y_0, ...,y_{T-1}, y_T)
- (<SHIFT>, y_0, ..., y_{T-1})
In my view, the second approach makes more sense. As y_T is the last output, it won’t be used to generate a token that would come after, i.e, it would never appear on the RHS of the pred function we introduced above.
What approach was followed by hugging face ?
Thanks