How to modify each decoding step in ProphetNet Transformer

Hi everyone, I am working on a project which needs modification on each decoding step of the ProphetNet model. At each decoding step, I want to concatenate an embedding representation with decoding output representation before it passes to the softmax layer. I am not sure which script/code needs modification for this purpose. I am new to the transformer libaray. Can you please suggest or provide some references? It will be helpful. Thank you!

Hi @valhalla and @s4sarath, any suggestions for this. It need not be a prophet-net model, it can t5 or BART model in general. Thank you!

Hi Kaushal,

I am also not too familiar with transformers library. It’s isn’t not achievable. But you might need little bit of effort.

I suggest you can try it inside you model layer itself. Decoding can be untouched. Just make a condition in your model, that at inference pass the extra embedding for concatenation. It’s fairly straightforward in PyTorch. But not so in TF2 , if you want to serialize.

I suggest try these in transformers PyTorch implementation. From my personal experience, their TF implementation is no where close to perfect ( which is no one’s fault ), TF requires effort.

1 Like

Thank you @s4sarath for the directions. Let me try your suggestions.